Commit graph

371 commits

Author SHA1 Message Date
Julian Knutsen
47eb06e66a docs(bazel): make bazel test the primary build and test path (ga-96smfk.23)
Bazel is beads' build and test system: bazel.yml's lanes gate every PR,
and nogo (lint + vet), gofmt and the repository guards exist only as
Bazel targets. The docs, the Makefile and the hooks now say so.

Makefile: `make test` runs bazel.yml's test lane (`bazel test //...
--config=ci`), `make check` adds the testing.Short policy and the
`make ci-pr-lint` nogo gate, and `make check-docs` runs the docsync and
doc-freshness Bazel targets before the pinned-release CLI flag check.
BAZEL_FLAGS passes --config=fork-cache or --config=remote-exec through.
The plain-Go recipes stay as explicit twins: test-go, check-go,
check-docs-go. No workflow called the primary names, and
TestWorkflowsNameTheirTestEngine keeps it that way.

Hooks: pre-push runs scripts/pre-push-suite.sh for branch pushes whose
commits change Go or Bazel inputs: the test lane with
--config=remote-exec when some rc names an executor, else
--config=fork-cache. Without bazel, or with BD_PREPUSH_SUITE=go, it runs
make test-go under a banner saying that is not what CI gates on.

Docs: AGENTS.md, CONTRIBUTING.md, engdocs/TESTING.md (a per-lane command
table and the fork-cache/remote-exec setup), README, RELEASING, the PR
template and the .bazelrc header. Bazelisk is a required prerequisite;
go test is an inner-loop convenience CI does not enforce.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 07:12:47 +00:00
Julian Knutsen
8d7d7fef16
build(rbe): adopt gascity's toolchain-only worker-env pin; check the copy nightly (#7346)
gascity now builds the worker-env manifest from the worker's toolchain,
not the image's patch revisions. The manifest records the arch, the OS
release, Go and dolt, and the upstream releases of the libraries and
tools actions reach (major.minor; git and yq at their major). The Debian
epochs and Ubuntu revisions are gone, so the Blacksmith image's routine
security updates no longer move the hash. That is what stranded every
remote action on 2026-10-07: coreutils 9.4-3ubuntu6.1 -> 6.2 and libc6
2.39-0ubuntu8.6 -> 8.8 forced re-pins (gascity #7266, beads #7338).

- tools/rbe/worker-env.txt: byte-identical copy of gascity's new
  manifest. platforms/BUILD.bazel pins its sha256,
  sha256:6be1381cf08e7fd66684cfce6f26aca57de2561e722c505d1f2f578e9fdeafa7.
  The manifest now also measures python3, curl, openssl, unzip and yq,
  which //scripts:scripts_test runs from the executor.
- scripts/rbe_worker_env_test.go: the manifest must be a rendering of
  gascity's tools/rbe/worker-env (sorted; one arch, dolt, go, os and yq
  line; packages at most major.minor, installed). A copy of an older
  gascity manifest with dpkg revisions, a hand edit or a truncated copy
  fails here.
- tools/rbe/worker-env-sync, run by nightly.yml: compares the manifest
  and the pin with gascity's main byte for byte. If they differ, it
  prints the diff and the pin to commit, and fails.

Merge this together with the gascity change. Pool workers run gascity
main's provisioning, so they serve the new hash only once that change
lands, and beads' actions on any other pin queue. This supersedes #7338
(the old-format re-pin). The new key misses the cache once.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 23:18:24 -07:00
Julian Knutsen
a4509deb2d
ci(bazel): nogo replaces go vet and golangci-lint (ga-96smfk.18) (#7341)
* ci(bazel): nogo replaces go vet and golangci-lint (ga-96smfk.18)

Lint and vet now run as nogo inside the Bazel build, so they gate wherever
the Bazel lanes execute (rbe-west), and the golangci-lint and go vet jobs
leave pr.yml and main.yml.

- //tools/nogo (own go.mod, joined to beads' for go_deps only through
  gazelle.go.work): go test's vet checks (cmd/go's defaultVetFlags, the set
  scripts/ci/go-test-vet.sh ran) plus the eight linters .golangci.yml
  enables, each a wrapper over the library version golangci-lint v2.10.1
  uses (depguard, errcheck, forbidigo, gosec, misspell, sloglint, unconvert,
  unparam).
- .golangci.yml stays the single lint policy: Bazel embeds it and
  tools/nogo/internal/golangci applies it as golangci-lint does (enabled
  set, settings, run.tests: false as library-unit-only analysis, lax
  generated-file detection, path/path-except/text exclusion rules), with
  strict decoding so an unimplemented key fails the build.
- gascity's rules_go //nolint-range patch and x/tools ctrlflow patch,
  byte-identical.
- Windows/darwin: cross-configured, not a golangci leg. The pure lane's
  release cross-compile (//tools/bazel:release_cross, every library and
  binary for all eight release platforms, cgo off) runs nogo on every
  compile, a superset of golangci-lint's windows and darwin legs.
  scripts/pr-lint (make ci-pr-lint, bd preflight) runs `bazel build
  --config=nogo //...` plus release_cross for windows_amd64,darwin_arm64.
  release_cross skips tools/nogo, a separate Go module `go build ./...`
  never compiled.
- pr.yml: scripts-go-checks (vet) and pr-lint-wrapper (x3) are gone, with
  their ci-gate entries; main.yml: pr-lint-wrapper (x6) and go-vet-cache.
  scripts/ci/go-test-vet.sh and install-golangci-lint.sh are deleted.
- pre-commit hook and .pre-commit-config.yaml run `make lint-changed
  LINT_CHANGED_SCOPE=staged` (nogo over the staged packages); Makefile adds
  lint, vet and lint-changed. pr-lint.sh no longer runs gofmt
  (//scripts/repochecks:fmt_test owns it; ga-96smfk.41).
- internal/testutil/integration: five gosec findings in integration-tagged
  library files golangci-lint never built, now linted by the integration
  lanes, carry reasoned //nolint directives.

Parity: with the exclusion rules removed, golangci-lint and nogo report the
same 3,916 findings natively and 3,859 each for windows and darwin (file,
line, linter, text); with .golangci.yml as committed both report none.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): equivalence.py leaves nested Go modules' tests out of go test's set

tools/nogo is its own Go module, so `go test ./...` in the root never runs
its tests; Bazel's unit lane does. Its packages are neither expected nor
extra.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 05:17:22 +00:00
Julian Knutsen
ce661175ff
ci(bazel): retire the docker A/B variants and pr.yml's container jobs (#7340)
The dolt-server lane (--config=doltserver) has been the only pre-merge run
of the Dolt-backed domain, uow, tracker, doctor/fix and protocol suites
since BAZEL_COVERS_FORKS went "true": bazel-coverage reports pr_lanes=true
for every pull_request and merge_group, so pr.yml's test-domain-uow and
contract-corpus never run. An A/B run of every requires-docker target
against its dolt-server twin on the same commit shows identical test sets
and statuses (2683 testcases) except TestDoltServerFingerprint/container,
the container half of the fingerprint, which needs docker by design.

Delete the six requires-docker sh_tests, test:docker in .bazelrc, the
requires-docker tag from the taxonomy and lane filters, bazel.yml's
dolt-lane dispatch input and image pull, and pr.yml's test-domain-uow and
contract-corpus jobs with their ci-gate wiring. The testcontainers backend
in internal/testutil stays: it is plain go test's default. pull-dolt-image.sh
stays for pr-risk.yml's legacy tiers. No required check changes: the
rulesets require only CI Gate / Required and PR Risk Gate / Required.

Refs ga-96smfk.21

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 04:13:24 +00:00
Julian Knutsen
3f1c6038f0
ci: warn-only shard-balance budget for bazel-server-storage (#7339)
rbe-ci-cost-latency-study.md recommendation 4: this tier's last shard
has trailed the rest by 4.3-4.5 min in 2 of 8 runs. tools/bazel/shard_budget.py
reads the job's BEP file and prints ::warning when a sharded target's
slowest shard exceeds max(150s, 2x median); --fail-on-imbalance (not
wired in yet) switches that to ::error and a nonzero exit once the
budget has run clean for a while.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-07 04:05:33 +00:00
Julian Knutsen
5e73670866
ci(bazel): cross-compile every release target with Bazel, retire the go build job (#7332)
pr.yml's check-release-target-cross-compilation ran `CGO_ENABLED=0
GOOS/GOARCH go build -tags gms_pure_go ./...` for the eight release targets
on two GitHub/Blacksmith runners. It is now a step of bazel.yml's pure-Go
lane (BAZEL_PURE, already required by ci-gate):
scripts/ci/bazel-release-cross-compile.sh builds //tools/bazel:release_cross
once, remotely, with --//tools/bazel:release_platforms set from
scripts/ci/release-targets.txt (still the single source of truth, pinned to
.goreleaser.yml). release_cross's split transition configures every
go_library/go_binary for each rules_go platform with cgo off, so all eight
platforms build in parallel in one invocation.

- tools/bazel/release_cross.bzl: the split-transition rule and the
  string-list flag.
- tools/bazel/go_srcs.py (make bazel-sync, checked by bazel-sync-check and
  TestBazelGoSrcsBlocksCurrent) keeps the release_cross target list; the
  script checks with bazel query that every Bazel package holding Go
  targets is reached before it builds.
- //tools/bazel:pure_bd_has_no_cgo_only_deps: Bazel compiles gozstd's pure
  stubs where `go build` rejects them, so the pure bd of every platform
  must not link gozstd.
- //internal/storage/embeddeddolt/cmd:cmd (cgo-only main, skipped by
  `go build ./...` under CGO_ENABLED=0) is tagged cgo-only and incompatible
  with pure builds.
- A failed build is re-run per platform (cache hits) to name each failing
  GOOS/GOARCH.

Same cgo-off coverage boundary as the retired job. Refs ga-96smfk.20.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 03:18:37 +00:00
Julian Knutsen
bb9b9e5020
ci(bazel): run the upgrade smoke tests under Bazel; retire cross-version-smoke.yml (#7331)
cross-version-smoke.yml resolved the newest 5 GitHub releases (30 on a tag
push) with `gh release list`, then built the candidate and downloaded each
release on a GitHub runner for scripts/upgrade-smoke-test.sh.

//tests/upgrade_smoke has one target per release in the newest 30 the
release catalog pins a linux/amd64 asset for (@bd_releases//:versions.bzl):
upgrade_smoke_<release>_test runs the script against the non-race
bd_for_tests with that release pinned as PREV_BIN (sha256-verified when
Bazel fetches it; the script checks it reports the release and downloads
nothing). A release added to the catalog joins the window. Tagged
pr-core-only, so bazel-test (not the integration lane) runs them remotely
on every PR, merge group and
push, cached until the candidate, the script or the release changes.

The tested set is now reviewed rather than resolved at run time: stable
releases from the catalog, where the workflow took GitHub releases
including prereleases. cross-version-smoke.yml and the F7c checks of its
chunk fold, path filter, runners and binary cache are removed (the
path-filter and binary-cache checks had no other workflow left).

Refs: ga-96smfk.19

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 02:49:39 +00:00
Julian Knutsen
7e0adfd353
ci(bazel): run the historical-upgrade corpus under Bazel; retire migration-test.yml (#7330)
* chore(migration-test): extend the release catalog through v1.3.1

The catalog stopped at v1.1.2 (the v1.2 upgrade census). The migration
corpus already upgrades from v1.2.2, and the Bazel targets that replace
migration-test.yml and cross-version-smoke.yml take their pinned release
binaries from the catalog (@bd_releases), so it has to cover the releases
since. Regenerated with `catalog generate` after raising maximumVersion to
v1.3.1: five new stable versions (v1.2.0, v1.2.1, v1.2.2, v1.3.0, v1.3.1;
three with a linux/amd64 asset) and five new proxy prereleases; no existing
entry, drift record or repository-only exclusion changed. The count
constants and the identity digest are updated to match, and the README now
says how to add a release.

Refs: ga-96smfk.19

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): run the historical-upgrade corpus under Bazel; retire migration-test.yml

migration-test.yml ran scripts/migration-test/run.sh --version <release> for
14 reviewed releases in 3 shards on GitHub runners: it downloaded each
release archive, built v0.9.1 from source with Go 1.25 through the module
proxy, installed libicu74 and the dolt 2.1.8 runtime, all over the network.

//tests/migration now has one target per corpus release
(historical_upgrade_<release>_test) plus legacy_bridge_test, all hermetic:
- Published releases come from @bd_releases (the release catalog's
  sha256-verified assets) through a per-target directory
  (HISTORICAL_RELEASE_DIR); lib/binary.sh uses it instead of downloading and
  still checks each binary's self-reported version.
- v0.9.1 is //tests/migration:bd_source_tag, the same `go mod download` +
  `go build -trimpath` with CGO and Go 1.25.0, run offline as a build
  action: the module files come from tests/migration/source_tag_modules.json
  through a file:// GOPROXY tree (tools/bazel/go_module_proxy.bzl), the
  release's own zip is the catalog's source_zip, and go checks every
  dependency against the release's go.sum (SOURCE_TAG_SQLITE_BIN).
- The external Dolt runtime is @dolt_test_runtime_linux_amd64 (dolt.bzl's
  dolt_release gains a version attribute; 2.1.8 is pinned to versions.sh's
  digest), passed as DOLT_BIN to the server-Dolt and wisp-plane releases.
- The workers already provide jq, python3, sqlite3 and libicu74.

The targets are untagged, so bazel-test runs them remotely on every PR,
merge group and push; result caching replaces the path filter. Every
release also passed inside a loopback-only network namespace.
scripts/migration_corpus_bazel_test.go keeps the BUILD lists equal to
versions.sh's corpus, the catalog digests equal to STRICT_RELEASE_SHA256,
the source-tag lock equal to the catalog's source_zip, and the Dolt runtime
pin equal to versions.sh's. migration-test.yml and the policy tests that
pinned its shards, caches, apt step and runner are removed.

Refs: ga-96smfk.19

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): run the historical-upgrade targets in the PR-core lane only

The integration lane runs every untagged target too, and its
check_testcases.py requires Go testcases, which a shell test's test.xml does
not list. A new pr-core-only tag keeps //tests/migration out of
--config=integration (bazelIntegrationExcludedTags adds it), so the corpus
runs once, in bazel-test, against the release-shaped bd_for_tests (no
integration tag). build_source_tag.sh carries the build-tags opt-out: it
builds a historical release with that release's own go.mod, which predates
gms_pure_go.

Refs: ga-96smfk.19

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 02:16:11 +00:00
Julian Knutsen
9a91f0629e
ci(bazel): run the differential regression suite under Bazel; retire regression.yml (#7324)
regression.yml ran `go test -tags=regression ./tests/regression/...` on a
GitHub runner: it built the candidate with `go build`, downloaded the v0.49.6
baseline release over the network, and pulled the dolt-sql-server image.
None of that works on rbe-west workers (no network for fork lanes, no docker).

- tools/bazel/bd_releases.bzl: a module extension that reads
  scripts/migration-test/release-catalog.json and declares one repository per
  catalog release with a linux_amd64 asset digest (bd_release_<version>,
  verified against the catalog's sha256), behind the @bd_releases hub
  (@bd_releases//:<version>, plus versions.bzl). Adding a release is a
  catalog change only; nothing downloads until a test declares one.
- //tests/regression:regression_test: the suite's files build under
  `regression || integration`, so gazelle generates the target. It runs the
  injected non-race bd_for_tests (BEADS_TEST_BD_BINARY) against
  @bd_releases//:v0.49.6 (BD_REGRESSION_BASELINE_BIN), both over a hermetic
  dolt sql-server (BEADS_TEST_DOLT_SERVER=local, required), in 8 shards.
  TestMain checks the baseline reports BASELINE_VERSION, so the label and the
  file cannot drift. Tagged dolt-server-cmd: bazel.yml's bazel-cmd-dolt lane,
  which pr.yml's gate requires, runs it remotely; test result caching stands
  in for the old path detector.
- A plain `go test -tags=integration ./...` (nightly) compiles the files but
  skips the suite; only -tags=regression (make test-regression) or Bazel
  runs it.
- regression.yml, main.yml's warm-up compile for it and the policy tests that
  pinned its jobs are removed.

Refs: ga-96smfk.19

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 01:41:12 +00:00
Julian Knutsen
b1fe68e63a
ci(bazel): run pr.yml's repository guards as Bazel tests (#7329)
fast-checks, PR Policy, and the Linux leg of check-doc-freshness-platforms
ran repository guards on GitHub runners. They now run as Bazel tests in
bazel.yml's `test` lane (remote, cached), each over only the files it reads:

- tools/bazel/repo_subset.bzl picks a narrow, path-selected slice of
  //:repo_files at analysis time. scripts/repochecks/run_check.sh copies that
  slice into a throwaway git checkout and runs the unchanged guard there.
- //scripts/repochecks: build_tags, workflow_tags, go_install_guidance,
  winget_portable_alias, migration_hygiene (checks A and B),
  fmt (registered SDK gofmt), doc_freshness, workapi_frontend_boundary,
  types_gen_drift (`make api-check` drift, oapi-codegen from go.mod).
- //scripts:doc_freshness_required_test (integration lane, -required-suite)
  and //scripts/gitattributespolicy:gitattributespolicy_required_host_test
  cover the Linux doc-freshness leg.
- check-versions stays covered by //internal/versioncheck:versioncheck_test.
  check-testing-short stays covered by //scripts:scripts_test.

pr-policy-wrapper is deleted. fast-checks keeps only the two checks that
need the PR base: migration hygiene's delta half (MIGRATION_HYGIENE_SCOPE=
delta) and the .beads diff. check-doc-flags stays, because it builds bd from
the pinned release tag. The example Go modules leave .bazelignore so gofmt
and the grep guards see them. check_testcases.py now accepts --not-go
//pkg:all.

Refs: ga-96smfk.20

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 00:59:18 +00:00
Julian Knutsen
72534f8e57
ci(bazel): check the Dolt server fingerprint on the dolt-server lane; retire pr.yml's fingerprint job (#7328)
pr.yml's every-PR "Test (Dolt server fingerprint)" job installed dolt, pulled
the dolt-sql-server docker image and ran TestDoltServerFingerprint against
both backends. The rbe-west workers have no docker, and the container backend
is now used only by the container-backed legacy jobs, which run only where
bazel-coverage does not cover pr_lanes.

- //internal/testutil:testutil_dolt_test runs testutil_test with the local
  backend selected and required (BEADS_TEST_DOLT_SERVER=local,
  BEADS_TEST_REQUIRE_DOLT_CONTAINER=1) on the dolt-server lane, which
  pr.yml's gate requires: the fingerprint's local half against the pinned
  dolt, plus the TestLocalDoltServer_* tests every other configuration
  skips under BEADS_TEST_SKIP=dolt.
- The container half moves back to test-domain-uow's first test step (where
  it was before D2 step 3), so it runs exactly where containers are used.
  //internal/testutil:testutil_docker_test is its docker-lane A/B control.
  TestPinnedDoltCLIMatchesContainerImage still pins the image tag to the
  dolt release.
- test-dolt-server-fingerprint, its ci-gate need/result and its policy
  tests are removed; TestDoltServerFingerprintRunsWhereEachBackendRuns
  pins the new placement.

Refs: ga-96smfk.19

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 00:59:16 +00:00
Julian Knutsen
c292964571
ci(bazel): run the managed-local proxied lifecycle lane under Bazel; retire proxied-local-smoke.yml (#7326)
proxied-local-smoke.yml built bd and the cmd/bd test binary on a GitHub
runner, installed the pinned dolt, and ran `-test.run ^TestManagedLocalProxied`
under `sudo unshare --net` with only loopback up, to prove the managed-local
lifecycle (bd -> loopback proxy -> dolt sql-server) needs no outbound network.

//cmd/bd:bd_managed_local_test runs the same tests (BEADS_TEST_PROXIED_LOCAL=1,
BEADS_TEST_SKIP=dolt, the non-race bd_for_tests as the bd under test, the
pinned dolt from the test wrapper's PATH) through tools/bazel/go_test_offline.sh,
which enters an unprivileged user + network namespace with only loopback up
and fails if outbound network is reachable: the job's netns, without sudo,
on the rbe-west workers. Tagged dolt-server-proxied, so bazel.yml's
bazel-proxied lane (remote-only, required by pr.yml's gate) runs it; test
result caching replaces the workflow's path filter. The job's "tests are
compiled in" assertion becomes scripts/managed_local_lane_test.go, which
evaluates cmd/bd's build constraints for a linux cgo build and requires each
of the 17 pinned tests.

The target joins the pinned retired-lane targets in
TestBazelRetiredLanesCannotBeNarrowed. proxied-local-smoke.yml and the policy
tests that pinned its job and its dolt version are removed (the Bazel dolt
pin is already enforced against install-dolt.sh).

Refs: ga-96smfk.19

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 00:59:14 +00:00
Julian Knutsen
cb0dcbecf2
ci(bazel): hermetic LLVM C toolchain and pinned rbe_worker exec platform (#7325)
beads' MODULE.bazel registered no C toolchain, so -race and cgo builds used
the autodetected local_config_cc of the machine that ran analysis. From a
client with lld installed (cherry) every remote cgo/-race link failed with
"collect2: fatal error: cannot find 'ld'" (the workers have no lld), and the
host gcc path, lld availability and gcc-version inputs keyed every Go action
to the client machine.

Port gastownhall/gascity's design:
- toolchains_llvm 1.11.0 with a pinned LLVM 22.1.8 slice
  (third_party/cc_toolchain/llvm_dist.bzl) and a sysroot of sha256-pinned
  Ubuntu 24.04 debs (deb_sysroot.bzl), without ICU (ICU policy);
  stock toolchains_llvm LLVM on other hosts.
- .bazelrc: BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 so nothing falls back to the
  host compiler.
- //platforms:rbe_worker, selected unconditionally, whose worker-env exec
  property is the sha256 of tools/rbe/worker-env.txt (a copy of the shared
  rbe-west OSS pool's manifest, owned by gascity), so the worker host is in
  every action key.
- scripts/bazel_hermetic_cc_test.go and scripts/rbe_worker_env_test.go guard
  both.

Refs: ga-96smfk.20

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-07 00:59:11 +00:00
Bee
bc34f1d77c
ci: upgrade workflows also run on schema-migration changes (#7150)
#7147 gave migration-test, conformance and cross-version-smoke a shared
paths filter: non-test Go files plus build inputs. The schema migrations
are .up.sql files embedded into the binary (go:embed in
internal/storage/schema), so they match no entry. A PR that adds a
migration with only a _test.go beside it skipped all three workflows,
where before #7147 it ran all three. d26230d0d and 63a471a1e have that
shape.

Add internal/storage/schema/migrations/** to the shared base in each
paths block and in advisoryPathFilterBase, which pins the base across
the three files.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-07 00:40:04 +00:00
Julian Knutsen
8761d2c40e
ci(bazel): run the conformance e2e tier under Bazel; retire conformance.yml and the docs-mintlify docsync job (#7321)
conformance.yml ran two tiers on a GitHub runner. Tier 1 (the embedded-Dolt
TestConformance) already runs as the embedded lane's
embeddeddolt_conformance_{core,audit}_test. Tier 2, the real-binary CLI
corpus (`go test -tags 'gms_pure_go e2e' ./test/conformance`), had only a
go_library under Bazel. Its two test files now build under `e2e ||
integration`, so gazelle generates //test/conformance:conformance_test
(integration-only, against the injected non-race bd_for_tests) and
--config=integration runs it remotely in bazel-integration, which pr.yml's
gate requires. PR Core's go test builds neither tag, so the --config=ci
equivalence check is unchanged; scripts/conformance.sh keeps working with
-tags e2e as the local go test entrypoint. With both tiers covered,
conformance.yml and main.yml's warm-up compiles for it are removed.

docs-mintlify.yml drops its docsync job (`go test ./test/docsync`, run by
bazel-test as //test/docsync:docsync_test on every PR) and keeps only
Mintlify's broken-link check, which needs the network (npx mint) and the
base branch's history; its path filter narrows to what that check reads.

Refs: ga-96smfk.17

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 22:42:25 +00:00
Bee
82872a81a5
fix(ci): rbe pre-warm survives a failed gh run list under bash -e, and clamps an over-long worker count (#7162) (#7163)
Actions runs the dispatch step as bash -e, so a failing gh run list ended the
step at the assignment and the could-not-read warning never printed. The
assignment now tolerates the failure. A RBE_PREWARM_WORKERS too long for the
shell's integer compare slipped past the max-4 clamp and the loop dispatched
without bound; two or more digits is now decided in the case (all zeros stays
the kill switch). The behaviour test runs the script with bash -e, as Actions
does, under a 30s deadline, and pins the clamp.

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 22:35:02 +00:00
Julian Knutsen
0500fc6f40
ci(bazel): run the scripts and source-walking tests remotely; retire their go test legs (#7320)
//scripts:scripts_test and //scripts/check-versions:check-versions_test
drop their host-tools/no-remote-exec/no-remote-cache tags and run on the
rbe-west workers like every other target (the worker image ships bash, git,
python3, jq, yq, curl, openssl, unzip and coreutils; a missing tool fails the
test under Bazel rather than skipping it). Go comes in hermetically as the
registered SDK; check-versions.sh's `go build` is stood in for by the
Bazel-built checker.

The tests that walked the source checkout now get it as data:
tools/bazel/go_srcs.py keeps a managed `repo_files` filegroup in every
package and aggregates them into //:repo_files (exactly the tracked files
Bazel can see; make bazel-sync-check fails on a package without one).
scripts_test and docsync_test take it instead of hand-listed filegroups,
and every TEST_SRCDIR/IsBazel skip and go-test-only block in them is gone.
The release formula under .bazelignore'd .beads/ comes in through a
@beads_formulas local repository; npm-package/ and .claude/ are visible to
Bazel again (only their node_modules/worktrees stay ignored), so the npm
manifest release check and the AGENTS.md skill links run under Bazel too.
TestZZStdioNotLeaked falls back to TestMain's pre-run streams instead of
skipping in a shard without its baseline.

tools/bazel/equivalence_allowlist.txt shrinks from 20 skip entries to the
two cmd/bd tests of plain go test's own bd build fallback, which Bazel never
takes; pr-preflight-platforms now runs them on Linux, macOS and Windows. The
scripts-go-checks scripts-test and allowlisted legs, their scripts
(scripts-go-test.sh, allowlisted-go-tests.sh, run_allowlisted_go_tests.py)
and main.yml's --compile-only warm for them are removed. The vet leg stays
(ga-96smfk.18).

Refs: ga-96smfk.16

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 22:20:16 +00:00
Julian Knutsen
9f60d71dae
ci(main): push to main runs the Linux test tiers under Bazel only (#7316)
bazel.yml already runs on every push to main and executes the PR Core,
pure-Go, dolt-server, embedded, proxied, server-storage, integration and
cmd/bd Dolt-server tiers on rbe-west in about five minutes. main.yml re-ran
the same tiers as legacy go test jobs (62 jobs, about 380 runner-minutes a
push), and its runs were all failing or cancelled. Remove the duplicated
Linux jobs: the ubuntu -race -short leg with its Codecov upload, PR Core,
the pure-Go compile check, domain+uow, the 6+8 integration shards, the 15
proxied shards, and the embedded build plus its storage, conformance and
cmd shards.

The macOS -race -short suite (rbe-west has no macOS workers) moves to
Blacksmith macOS and restores the macOS saver's caches. The cache savers
that fed the fork path from the removed jobs move to venue matrices: the
macOS saver gains a GitHub-hosted leg, and the Linux saver gains a
GitHub-hosted race leg (the embedded build used to seed it).

Refs ga-96smfk.14

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 20:52:01 +00:00
Julian Knutsen
e72cd8b315
ci: merge-queue readiness for the required gates; cache test results (fresh nightly) (#7264)
* ci: make the required gates merge-queue ready

The default-branch ruleset requires "CI Gate / Required" and "PR Risk
Gate / Required". Prepare both for a GitHub merge queue (ALLGREEN,
squash), so enabling the ruleset's merge_queue rule is the only step left.

- pr.yml, pr-risk.yml: merge_group pinned to types [checks_requested].
- D2: bazel-coverage covers every merge_group run (same repository, CI
  secrets, actor github-merge-queue[bot], bazel.yml mode remote), so the
  retired legacy tiers stay retired in the queue and ci-gate requires
  their Bazel lanes to have run remotely and passed; a missing secret or
  the farm switch off is red, as on a same-repo PR.
- Diff bases on merge_group: the package gates diff
  merge_group.base_sha..head_sha (not HEAD^, which misses all but the
  last commit of a rebase-method entry), the .beads/issues.jsonl guard
  now runs on merge_group against base_sha (ci-gate no longer excuses
  its skip), and check-doc-flags passes BD_DOCS_DIFF_BASE so CLI-docs
  drift already on main cannot eject every queued PR.
- Test result reuse: on a merge_group run and on attempt 2+ of a PR run,
  the six always-execute Bazel lanes append --config=reuse-test-results
  (--cache_test_results=auto), so tests whose action key is unchanged
  reuse the farm's recorded PASS and only what the merge changed runs.
  Failures are never reused, eviction retries stay 0, and each PR's first
  attempt, push to main and nightly still execute every test.
- scripts/ci_merge_queue_test.go simulates merge_group through the shared
  evalGHExpr evaluator (now with true/false/null literals, hyphenated
  identifiers and case-insensitive ==) and the workflows' own scripts:
  triggers and gate names, concurrency groups, every pull_request field
  read, the coverage and rbe decisions, result reuse, the package-gate,
  .beads and doc-drift diff bases.
- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md: new "Merge Queue" section (how
  to queue, settings, what runs, result reuse rationale, ejection and
  flakes, rollback).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci: cache Bazel test results except nightly; queue trust and sizing

Review follow-ups to the merge-queue readiness change.

- Test results (S1): --nocache_test_results also stops Bazel writing
  results, so a PR's first run never fed the cache and the merge-queue
  reuse config could only hit on later reuse runs. Drop it from
  test:embedded, test:doltserver-proxied, test:doltserver-integration and
  test:sole-run (which keeps --experimental_remote_cache_eviction_retries=0),
  and drop test:reuse-test-results and BAZEL_REUSE_RESULTS. Results are now
  cached (read and write) on PRs, merge groups and pushes to main.
  nightly.yml passes the new fresh-test-results input, so every `bazel
  test` appends --config=fresh (--nocache_test_results) and the suite
  re-executes nightly. The docker lane keeps --nocache_test_results.
  Measured locally with a disk cache: a default run writes and the next
  reads ("(cached) PASSED"); a fresh run neither reads nor writes.
- Security headers (S2): bazel.yml now states that a merge_group run
  executes the queued commit's workflows, so a queued fork or Dependabot
  PR's CI edits run with every CI secret, the pre-warm app key included
  (merging-level trust), and records the maintainer rule: review fork
  changes to .github/, .bazelrc, tools/bazel, MODULE.bazel, BUILD files
  and CI scripts before queueing. Same note in the topology doc.
- Queue sizing (S3): recommend max_entries_to_build 3 and
  check_response_timeout_minutes 60 (max_entries_to_merge 5, ALLGREEN,
  SQUASH unchanged).
- Nits: the .beads guard fails closed on a missing diff base, and its
  test runs under GitHub's default `bash -e`; evalGHExpr follows GitHub's
  loose equality (true == 'true' is false, false == '' is true, missing
  context keys are null) with a typed-context variant; the
  detect-package-gates comment notes the SQUASH/HEAD^ equivalence and
  path-filter scope; the doc covers re-run reuse, flake re-sampling via
  dispatch, stale queue runs and how to confirm reuse on the farm.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 13:54:15 -07:00
Julian Knutsen
4493eb8d23
ci(pr): run same-repo macOS preflight/doc-freshness legs on Blacksmith macOS (#7231)
pr-preflight-platforms and check-doc-freshness-platforms now resolve their
same-repo-macos marker like the Linux/Windows legs: blacksmith-6vcpu-macos-26
for same-repo PRs and merge_group, GitHub-hosted macos-latest for forks,
Dependabot and every other event. The label is pinned to macos-26 because
GitHub's macos-latest is macos-26-arm64 today, so both paths run the same
OS/arch; 6 vCPU is already 2x macos-latest's 3 vCPU.

Blacksmith cannot see GitHub-saved caches, so main.yml gains
blacksmith-macos-go-build-cache: a push-to-main-only saver on the same label
(same job guard as the other Blacksmith savers) that builds ./cmd/bd and runs
the shared warm-non-race-cache.sh, saving the module cache and a non-race
GOCACHE keyed by go.sum + UTC day. The PR legs stay restore-only. main.yml's
GitHub-hosted macOS full-suite leg is unchanged and keeps seeding the fork
path.

Policy tests: the macOS marker arm is evaluated for same-repo/merge_group/
fork/deleted-head/dependabot/push/schedule and must match the flat macOS
const; the saver joins the PR-guard and per-day-key sweeps and the cache
ownership inventory; TestBlacksmithMacOSSaverMatchesPRLegs pins the
label/path/key/warm-up contract between saver and PR legs.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 13:19:49 -07:00
Julian Knutsen
0773e6334e
docs(agents): contributor-only agent instructions and documented-issue intake (#7223)
Makes the repo's agent and contributor surface describe developing beads, not using bd, and requires every PR to link a documented issue.

- Contributor-only root AGENTS.md with a routing table; cmd/bd/AGENTS.md and engdocs/AGENTS.md seam files; CLAUDE.md symlinks so Claude Code loads nested files.
- Bug and feature issue forms, a PR template that requires Closes #, CONTRIBUTING sections for the triage and pipeline labels, and the opt-in fork-cache during dev setup.
- ROADMAP.md for 1.4: versioned beads (#6132), Graph Preview 2 (#7170), contributor onboarding (#7228).
- docsync guards: root AGENTS.md at or under 20 KB; every AGENTS.md symlinked, routed, and with resolving links, paths, and guard-test names.

Agent-Signature: claude-code-claude-opus-5-5-high on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 08:43:53 +00:00
Julian Knutsen
ec72c8c563
ci(bazel): drop //cmd/bd:bd_test from the integration lane where the cmd/bd Dolt-server tier is required (#7230)
bazel-integration (--config=integration) and bazel-cmd-dolt
(--config=doltserver-cmd, //cmd/bd:bd_dolt_server_test) both run the
integration-build cmd/bd bd_test; the second without BEADS_TEST_SKIP=dolt,
so it is a superset. Once pr.yml requires bazel-cmd-dolt, the integration
lane's bd_test run is a duplicate.

bazel.yml gains a boolean workflow_call input, cmd-dolt-required (default
false). When true, bazel-integration tests //... -//cmd/bd:bd_test. pr.yml
passes it, and a policy test pins it to pr.yml's BAZEL_CMD_DOLT_REQUIRED
(a `with:` cannot read env), so the drop only happens where ci-gate
requires BAZEL_CMD_DOLT. bazel-farm.yml, nightly.yml, push and dispatch
leave it unset and keep bd_test. While the flag is "false" (current main)
nothing changes; with #7226's flip the input flips with it.

Coverage: both lanes share the same `if:` (modes remote, fork-ro, fork-rw,
cache; Dependabot included) and bazel-gate.sh accepts their skips in the
same modes (skip, local). TestBazelIntegrationExcludesBdTestOnlyWhereCmdDoltCovers
pins that, the exact exclusion, the flag/input equality, the gate's
conditional requirement, and that no other caller passes the input.

Rollback: commit BAZEL_CMD_DOLT_REQUIRED "false" and cmd-dolt-required:
false together in pr.yml.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:19:47 -07:00
Julian Knutsen
15d74c08ff
ci: require the cmd/bd Dolt-server tier (BAZEL_CMD_DOLT_REQUIRED=true) (#7226)
Advisory since #7185. On the farm it passed on every PR run based on
current main; its only failures (3) were branches predating #7219 and
reproduced a real test bug that #7219 fixed. Flip the committed flag so
CI Gate / Required enforces BAZEL_CMD_DOLT. Rollback: set it back to
"false".

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:04:39 -07:00
Julian Knutsen
630eae9b0c
ci(pr): share restored GOCACHE with the native init credential step (#7227)
pr-preflight-platforms' "Exercise native init gateway credential shell"
step had no GOCACHE env, so it fell back to the runner's empty default
cache and cold-compiled all of ./cmd/bd's test binary on every PR, right
after the hook-timeout step had compiled the identical binary (same tags,
same platform-default CGO) into the restored non-race cache. Job logs:
213s of a ~350s macos-latest leg, ~117s on Windows; on Linux the rerun
with a shared cache is a relink only (~8s).

- pr.yml: point that step, and the three Windows-only Go steps, at the
  restored non-race GOCACHE.
- main.yml: the macOS test leg (the only macOS non-race seeder) now runs
  warm-non-race-cache.sh, so the PR macOS legs restore the test packages
  and test-only deps they compile, not just the non-test ./cmd/bd graph.
- Policy tests: every run step in pr-preflight-platforms and
  check-doc-freshness-platforms must use the restored GOCACHE, and the
  macOS seeder must warm before it saves.

No test selection, matrix, or gate wiring changes.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:01:23 -07:00
Julian Knutsen
516e0b4bf0
ci: autofix pushes with the gastownhall-autofix App token (no more approve-and-run) (#7215)
* ci(autofix): push bot fixes with a scoped GitHub App token

github-actions[bot] pushes left PR checks action_required instead of
retriggering, and the DOCS_AUTOFIX_TOKEN fallback secret was never
configured. bazel-autofix.yml and docs-autofix.yml now mint a short-lived,
repo-scoped (contents:write) installation token for the gastownhall-autofix
App via the `autofix` environment on same-repo PRs, falling back to the
default workflow token (and its manual re-run note) when the App token
isn't available; fork PRs are unaffected and still only get the apply-recipe
comment.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(ci): job environment parses as a name or a mapping; gofmt

ciWorkflowJob.Environment was a string, so every test that loads all
workflows failed on deploy-pages-redirect.yml's mapping form
(environment: {name, url}).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-04 21:27:06 -07:00
Rongjun GENG
3c65c99c29
ci: trigger pull_request CI on hotfix/** branches, not just release/** (#7148) (#7198)
pr.yml and pr-risk.yml (and the three advisory workflows that already
list release/** beside it -- conformance.yml, cross-version-smoke.yml,
regression.yml) trigger pull_request only on main and release/**. A PR
based on hotfix/1.3.1 (or any sibling hotfix/** line) matched neither
pattern, so it ran almost no CI: #7146, a backport to hotfix/1.3.1,
showed 17 checks -- 15 cross-version historical smokes and triage/label
jobs, 2 skipped -- no Lint, no unit shards, no embedded tier, no CI
Gate / Required, no PR Risk Gate / Required.

pr.yml's own comment states exactly why release/** is there:
release-prep and release-fix PRs are based on the release branch, not
main, and without it they run almost no CI before the tag build. A
hotfix line is in the identical position. Added hotfix/** alongside
release/** in all five files.

New TestPullRequestWorkflowsTriggerOnHotfixBranches pins hotfix/**
presence in each file's pull_request.branches list -- red before this
change (observed [main release/**] in all five), green after. The
existing TestPRRiskBazelCoverageJob equality check (pr.yml and
pr-risk.yml's pull_request triggers must match) still passes, since
both were updated identically. go vet/gofmt clean.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
2026-10-04 20:30:49 -07:00
Julian Knutsen
5e951a2ff3
perf(issueops): batch create/import without per-row round trips; on-demand graph checks (#7205)
* perf(storage): cut batch-create round trips (import, bulk create, apply-batch)

A multi-issue create (import, bulk create, CreateIssuesWithFullOptions) ran
~20 SQL statements per issue, and its dependency pass ran a recursive cycle
CTE per edge over the union of both dependency tables, which Dolt cannot
index — every recursion step rescanned the edge tables, so an import whose
edges form chains went superlinear (400 issues: 73 s of 82 s in that CTE).

issueops batch create, same transaction and same stored outcome:
- createBatchCache: one IN-list read of every explicit id's presence in both
  planes (replacing two COUNT probes per issue), one read of the batch's
  stored labels (each issue's new labels then land in one multi-row INSERT),
  created/label_added events buffered and written in one pass under the ids
  InsertDerivedEvent would mint (same digest, same lowest free ordinal), and
  runs of brand-new rows written with multi-row INSERTs (same ODKU clause)
  then finished in order.
- Dependency pass: wisp routing and target presence from batch reads; the
  cycle and hierarchy checks walk an in-memory copy of the edge tables,
  loaded once and kept in step with every edge the pass writes (a pair it
  already holds is re-read, since a clientFoundRows connection reports a
  matched duplicate as written).
- createBlockedRecomputeIDs seeds all created edges and expands once.

Shared paths:
- Blocked-state recompute splits ids once per call by whether they own a
  dependency row; edgeless ids (every freshly created issue) skip the two
  union UPDATEs for one plain unmark, run after the chunk's union statements
  on the first pass so the unions see what they saw before.
- appendChildrenInTx reads children per IN-list chunk, not per parent.
- ExecuteCreate hands its BatchContext to the batch body instead of having
  it re-read the same config (-6 statements per apply-batch create item).

Not changed: one version-history row per created issue (the version
contract, version_history.go), journal order and content, and the per-row
bodies, which stay the path for singular creates.

Equivalence: internal/storage/createbatchequiv runs an import scenario
(upserts, duplicate ids, wisps, generated ids, cycles, hierarchy conflicts,
dangling/external targets, stale is_blocked, comments, journal and
versioned history on) through the fast and per-row bodies on the embedded
and server engines and compares every written table and the skip reports.

Re-pinned TestLargeBatchApplyStatementCounts*: embedded 7009/14014/846 ->
6396/12790/793, server 7008/14014/846 -> 6396/12791/793 (breakdown in
pinnedEmbeddedStatementCounts).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(storage): batch-create review fixes — lazy edge graph, opt-in recompute probe, insert budget

Review of the batch-create round-trip change (S1-S3 and nits):

- The dependency pass's in-memory graph no longer loads both edge tables.
  It reads a node's outgoing edges through the issue_id index the first
  time a walk reaches it: the batch's own endpoints up front, then one
  IN-list read per BFS level. Memory and reads scale with the subgraph
  the batch's walks touch, not the edge tables, so a two-edge create on a
  large database no longer materialises every edge. A written edge whose
  source was never loaded loads that source (row included).
- The blocked-state no-edge probe runs only when a create asks for it or
  when a chunk holds at least 32 ids. Single-issue recomputes on close,
  reopen, update and dependency edits issue exactly the statements they
  did on main (the reopen sqlmock expectation is back to main's).
- Multi-row issue INSERTs are bounded by an estimated 16 MiB of bound
  text as well as 100 rows. When a multi-row statement fails, its rows are
  replayed one at a time, and the batch continues if every row lands
  instead of returning the original error.
- Nits: document the binary-collation assumption behind Go string
  equality; DisableCreateFastPathsForTest documents that it is
  process-global and non-parallel, and panics on an overlapping caller;
  drop the unreachable skip branch; explain why wisp targets need no
  presence read.

Re-pinned TestLargeBatchApplyStatementCounts* classic 40: 793 -> 786
(the +7 probes on dep-add recomputes are gone; net vs main is -60, all of
it ExecuteCreate's config reuse). 356/712 unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(storage): search the batch dependency graph from both ends

The lazy per-node load (previous commit) still walked forward from the
check's start, so on a large stored graph a single create's cycle check
read every node reachable from its target — about 800 statements per
create on a 2000-issue fixture. The cycle and hierarchy questions a batch
asks almost always end at a row the batch just created, whose only
incoming edges are the batch's own.

depGraph now answers each reachability question with a bidirectional
search: forward from the start along outgoing edges (issue_id index),
backward from the goal along incoming edges (the three target-column
indexes, read as one UNION ALL per table), always expanding the smaller
frontier and loading a level's nodes in that direction once. The batch's
endpoints are loaded in both directions up front, so a create's checks
usually finish with no further read (a chain import, too: each new
edge's goal has no incoming edge yet). Rows are keyed by (table, source,
target), so a row read from both ends counts once. An incoming read
ignores a row whose matching column is not its first set target column,
as the CTE's COALESCE does. A written pair that neither loaded end can
vouch for is re-read instead of assumed new.

On a 50k-issue / 100k-edge database, a create with two blocks edges
now runs 27 statements, against the full-scan or per-node walks before
(numbers in the review-fix summary).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(storage): hold batch create to golden digests from the pre-change code

The batch-create equivalence test compared the fast paths with the same
code's per-row bodies, so changes the fast-path switch does not gate —
the recompute seed merge, the IN-list children read, ExecuteCreate's
context reuse, the prepare/finish split — were checked only by older
contract tests. Its batch was also too small to cross a chunk boundary
and never set RejectStaleUpserts.

createbatchequiv now runs four scenarios, each through the fast and
per-row bodies, and checks both against a per-table row count and
SHA-256 recorded from ebe3b6bcb (before the fast paths existed;
CREATEBATCHEQUIV_GOLDEN_DIR re-records them):
- small: the original scenario;
- import458 and import458-reject-stale: the review's 458-issue batch
  (every chunk boundary, wisps and cross-plane edges, cycles, hierarchy
  conflicts, duplicate ids across deferred runs, generated and
  hierarchical ids, stale upserts rejected and accepted), plus new edges
  whose blocked-state fallout reaches stored rows outside the batch (a
  spawner's waiter, an upserted parent's existing child);
- apply: an apply-batch through ExecuteCreate, dep adds and an update.

The recorded digests match on the embedded and server engines (the
server test now sets the fixture's prefix, so generated ids agree).
Dropping the spawner-waiter seeding is invisible to the fast-vs-per-row
comparison and caught by the golden check.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(storage): large-graph create benchmark

batchbench measures creates against a database already holding a large
dependency graph (batchfixtures.LargeGraphInserts: N issues, ~2N
acyclic edges with wide reachable sets and a deep parent-child forest,
committed as a Dolt version): ExecuteCreate under an existing parent with
an existing blocker, a store create with two blocks edges, and an
apply-batch of creates each with a parent-child and a blocks dep_add.
It logs wall time, statements and Go allocation per shape on the
embedded and server engines. Opt in with BEADS_BENCH_LARGE_GRAPH=<issues>
(and BEADS_BENCH_LARGE_GRAPH_CREATES); it asserts nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(storage): read incoming dependency edges one target column at a time

The bidirectional graph search read a node's incoming edges with one
UNION ALL over the three target columns per table. Dolt plans that as a
scan: 0.7 s per read on a 100k-edge table, where each leg alone is a 1 ms
lookup on its own index (EXPLAIN shows IndexedTableAccess per leg). Issue
the three legs as separate reads into one row set; a row matched through
two columns still counts once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(storage): run the 458-issue equivalence scenarios outside the race lane

The 458-issue golden scenarios took the embedded tier's shard over its
20-minute limit under -race (the in-process engine's own instrumentation;
about 35 s each without it). They move to
TestCreateBatchFastPathsMatchPerRowLarge_Embedded, skipped under -race and
added to nightly.yml's non-race embedded step, following the precedent of
the 712-item statement-count and 1000-item wall-clock cases. The light
scenarios (small, apply) stay in TestCreateBatchFastPathsMatchPerRow_Embedded
on every lane. The server backend (built without -race in its full
suite) runs both: TestCreateBatchFastPathsMatchPerRow_Dolt and, as its own
top-level test so the shard manifest's hash places it on another shard
(about four minutes there), TestCreateBatchFastPathsMatchPerRowLarge_Dolt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 13:56:12 -07:00
Julian Knutsen
fa3d50eb19
ci(bazel): zstd for the anonymous fork cache when it advertises it (rbe-west E4) (#7202)
* ci(bazel): zstd cache transfers for the anonymous fork cache only, behind RBE_CACHE_ZSTD (E4)

E4 of the rbe-west egress reduction (design A.5), reworked for the
read-only zstd server shape. rbe-west's trusted MAIN and OSS schedulers
(:443 to :50052) and rbe-fork (:8444) advertise no compressor, and Bazel
refuses a remote that does not advertise zstd. So only the anonymous fork
cache (rbe-cache :8443, nativelink-anon, which keeps zstd) may ask for it.

- **`write-bazelrc.sh`:** no `--remote_cache_compression` on the trusted
  remote-exec or rbe-fork paths. In mode cache, with `RBE_CACHE_ZSTD=1`, it
  adds `build:fork-cache --remote_cache_compression`. Empty or 0 adds
  nothing; another value warns and stays off, rather than failing every
  cache lane.
- **`bazel.yml`:** every setup-bazel caller passes
  `RBE_CACHE_ZSTD: ${{ needs.rbe.outputs.mode == 'cache' &&
  vars.RBE_CACHE_ZSTD || '' }}`, so only cache runs see the variable.
  Merging changes nothing, and rollback is the variable, with no revert.
- **`.bazelrc`:** fork-cache no longer sets the flag; a comment points at
  the gate.

Tests:
- `TestBazelCacheModeReachesTheRC` evaluates `vars.` right-hand sides and
  simulates `RBE_CACHE_ZSTD=1`. Mode cache's rc carries exactly the
  fork-cache zstd line; the remote, fork-ro, fork-rw and local modes carry
  no compression at all.
- `TestSetupBazelRCWriter` runs `RBE_CACHE_ZSTD` = 1, 0, empty and a bad
  value in mode cache, and remote-exec with 1 (never compressed).
  `TestSetupBazelRCWriterForkMode` pins rbe-fork's exact lines with
  `RBE_CACHE_ZSTD=1`.
- The bazel.yml setup-env pin includes `RBE_CACHE_ZSTD`.
- `checkBazelrcForkCache` rejects the flag in any `.bazelrc` config, with
  fork-cache and remote-exec fixtures.
- New `TestBazelRemoteCacheCompressionOnlyForkCache` scans `.bazelrc`, the
  workflows and the local action files. The flag must appear once: the
  writer's fork-cache echo.

**Merge order:** after rbe-west's anonymous process advertises zstd. Even
then the flag stays off until `RBE_CACHE_ZSTD=1` is set. If rbe-cache stops
advertising zstd, fork-cache's fallback flags make lanes build locally.

**Action keys:** unchanged; the flag only changes ByteStream resource
names (`compressed-blobs/zstd/`).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): zstd for the fork cache by a GetCapabilities probe, not a repository variable (E4)

Review finding 3: GitHub gives fork pull_request runs no repository
variables (bazel.yml: forks "see no secrets and no vars"), so
vars.RBE_CACHE_ZSTD was always empty exactly where the switch matters. Fork
run 37220594372 logs RBE_VAR_ON: false. The variable could never turn zstd
on for fork-cache clients, and once on it could not follow rbe-cache
rolling zstd back.

- New setup-bazel/cache-zstd-probe.sh, a byte copy of gascity's
  tools/rbe/cache-zstd-probe.sh: one unary REAPI GetCapabilities call
  (instance oss) to rbe-cache :8443 over curl's HTTP/2, decoded with
  python3's standard library. It exits 0 only when
  cache_capabilities.supported_compressors lists ZSTD, the field Bazel
  checks. Any other result exits nonzero: not listed, an HTTP or gRPC
  error, a refused connection, a timeout (connect 3s, total 5s), an
  unparsable answer, or no curl or python3. It writes nothing to stdout
  and one verdict line to stderr. No new action and no new dependency.
- write-bazelrc.sh: in mode cache, `if bash .../cache-zstd-probe.sh` adds
  build:fork-cache --remote_cache_compression. The probe never fails the
  script. RBE_CACHE_ZSTD and its bad-value warning are gone.
- bazel.yml: the ten RBE_CACHE_ZSTD env lines are gone (back to the
  pre-E4 setup env).

Tests:
- New TestCacheZstdProbe runs the real script against a TLS HTTP/2
  stand-in for rbe-cache, which checks the exact request. Passes: zstd
  advertised (packed, and unpacked among others). Fails: rbe-cache's real
  2026-10-04 answer (no compressors), deflate only, zstd for batch updates
  only, zstd outside cache_capabilities, a gRPC error trailer, no
  grpc-status, trailers-only UNIMPLEMENTED, HTTP 502, a compressed or
  truncated message, a truncated field, not gRPC, a timeout, a refused
  connection, an untrusted certificate. It also pins the probe's endpoint
  and instance to .bazelrc's fork-cache.
- TestSetupBazelRCWriter: the fork cache gets exactly the zstd line after
  --config=fork-cache against a zstd stand-in, and none against the live
  answer or a refused port. Remote-exec and rbe-fork, with a zstd
  stand-in, never probe and never carry compression.
- TestBazelCacheModeReachesTheRC again rejects any vars. in setup-bazel's
  env, and no mode's rc carries compression (the probe is refused there).
- The writer harnesses default RBE_CACHE_PROBE_URL to a refused loopback
  port, so no test reaches rbe-cache.

Live check, 2026-10-04: rbe-cache answers without supported_compressors,
so the writer's mode cache rc has no flag until the E1 cutover.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:26:54 -07:00
Julian Knutsen
ebe3b6bcb0
test,ci(cmd/bd): run the Dolt-gated cmd/bd suite (advisory Bazel lane), fix in-process runner leaks and rotted tests; init --quiet stays quiet (#7185)
* test(cmd/bd): reset cobra flags between in-process runs; keep them out of parallel tests

runBDInProcess and runBDInProcessAllowError execute the package-level
rootCmd tree again and again in one process, and cobra never resets a
flag between Execute calls: a value one invocation parsed (create's
--title, --labels, list's --status, ...) and its Changed bit stayed on the
FlagSet, so the next invocation saw it, and a slice flag that had been
set appended to the stale value instead of replacing it. With a Dolt test
server (the only setup in which these tests run) that failed about 20
TestCLI_* tests depending on suite order.

TestMain now snapshots every flag of the tree (snapshotCommandFlags,
after all init()s registered them, before any test runs) and both runners
put the tree back before and after each run (resetCommandFlags): the
pointee of each flag.Value is restored (for pflag's scalar types that is
the bound variable; for slice types it also clears the internal "changed"
bit, so the next Set replaces), slice contents via SliceValue.Replace,
and Changed=false. TestResetCommandFlagsRestoresEveryFlag sets every flag
of the real tree and checks the reset. The ad-hoc --title reset in
TestCLI_CreateRejectsEmptyTitle_ProxiedServerMode is no longer needed.

Seven TestCLI_Comment* tests called t.Parallel() around the runners.
inProcessMutex serializes the runners only against each other, so their
chdir, stdio swap, env and config re-initialization raced every other
parallel test (data races on config globals; TestCobraParallelPolicyGuard
failing to read its own source files after a concurrent chdir). They run
serially now, and the guard treats the two runners like rootCmd.Execute.

Also two stale TestCLI_Create* cases: DryRunWithRigPrefix exercised a
create --prefix flag that no longer exists, and the flag-like-title cases
passed "--help" or "--foo-bar" where cobra parses them as flags; they now
pass the title after "--", the only way it reaches create's own guard.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): fix the ADO round-trip and bulk-delete tests that rotted unseen

Both need the integration tag and a Dolt test server, which no CI lane
gave cmd/bd (BEADS_TEST_SKIP=dolt everywhere), so they failed unnoticed:

- TestADORoundTrip* stored ado.pat in the database. ado.pat is a secret
  read only from config.yaml or AZURE_DEVOPS_PAT (config.IsYamlOnlyKey),
  so the tracker found no PAT. The fixture sets AZURE_DEVOPS_PAT.
- TestBulkDeleteNoResurrection expected GetIssue to return (nil, nil) for
  a deleted issue; the Dolt store reports storage.ErrNotFound.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): fix parity, completion and notion tests that only fail with a Dolt server

These are Dolt-gated (integration tag + test server), so no CI lane ran
them and they rotted:

- TestParityCreate*/TestParityReopenErrorExits1Silently: bd create
  auto-routes via routing.DetectUserRole("."), which warns "beads.role
  not configured (GH#2950)" on stderr when the working directory's git
  config has no beads.role. The suite pins the configured case, so the
  parity env now sets beads.role=maintainer in a test-owned global git
  config ($XDG_CONFIG_HOME/git/config; the role lookup scrubs every
  GIT_CONFIG_* variable, so those cannot be used).
- TestIssueIDCompletion: issueIDCompletion resolves a database path
  before looking at the open store and returns nothing when there is
  none; the test now sets dbPath to its store's path.
- TestIssueIDCompletion_UsesWorktreeFallbackWhenStoreNil and
  TestGetNotionConfigReadsDBPathWhenStoreUnset: the code under test
  reopens the store from metadata.json, which names a database but not
  the per-test branch of the shared database that newTestStore uses, so
  the reopened store saw none of the test's rows. They now use their own
  database (newTestStoreIsolatedDB).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): fix stale doctor/e2e/autocommit/readonly Dolt-gated tests

These need the integration tag and a Dolt test server, which no CI lane
gave cmd/bd, so they rotted unseen:

- TestE2E_DoctorFixMetadataRoundtrip, TestE2E_MigrateDoltMetadata,
  TestDoctorCheckHealthReportsVersionMismatchOnRepoLocalPort: bd init now
  defaults to embedded mode, where `bd sql` and `bd dolt start` are
  refused. Init with --server on a repo-local server of their own
  (repoLocalServerEnv: the suite's shared-server port vars cleared, server
  stopped at cleanup) and start it explicitly, since BEADS_TEST_MODE=1 never
  auto-starts one. MigrateDoltMetadata's DELETE also needed `key` quoted.
- TestDoctor{Conventions,Pollution}MaxRows_EnvOnly_Exits2: bdRunRaw drops
  every BEADS_* variable, so the subprocess hit dolt.New's test-database
  firewall (exit 1) before the row cap; pass BEADS_TEST_SERVER=1.
- TestDoltAutoCommit_On_WritesAdvanceHead: the embedded engine keeps its
  data under .beads/embeddeddolt, not .beads/dolt; findDoltRepoDir looks
  in both.
- TestValidateCheck_AllClean: bd doctor validate runs five checks since
  the cross-table duplicates check was added.
- TestConfigValidateReadOnlyIsHermetic: readonly commands take a shared
  workspace gate, whose lock file (.beads/dolt.gate.lock) workspacegate
  creates on first use by design; compare the .beads tree without gate
  lock files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): undo each in-process execution's env and storage-mode leaks

Every in-process run of the bd command tree (runBDInProcess and the many
tests that call rootCmd.Execute directly) leaves process state behind:
PersistentPreRun exports the selected workspace as BEADS_DIR, loads its
.beads/.env, and records the workspace's mode in serverMode /
proxiedServerMode. With a Dolt test server, later tests in the same shard
then resolved a deleted fixture ("no active beads workspace found",
"migration sidecar is missing"), ran subprocess bd against the wrong
workspace ("recorded as proxied-server in .beads/metadata.json"), or took
the proxied path in RunE-level tests ("proxied-server UOW provider not
initialized", a nil-UOW panic in reopenProxiedResolve that killed a shard).

TestMain now registers a cobra initializer/finalizer pair
(installExecuteIsolation) that snapshots the environment and the two mode
globals when an execution starts and restores them when it ends; the
runners' restoreEnvSnapshot shares the helper. TestDoltConfigEnvironmentOverrides
used os.Setenv + defer os.Unsetenv, which deleted the suite's
BEADS_DOLT_SERVER_PORT (the test server's port) for every later test; it
uses t.Setenv now.

Full integration-tagged cmd/bd suite against a hermetic dolt (16 shards):
18 order-dependent failures and the shard panic gone; the 27 remaining
failures all fail alone too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(init): keep --quiet quiet through the Claude/Codex/Cursor auto-setup

`bd init --quiet` ("Suppress output") still printed the project
integration installers' progress to stdout ("Installing Claude hooks for
this project...", "Claude Code integration installed", ...): the
installers write to os.Stdout unconditionally. Point stdout at the null
device around them while quiet; their failures keep going to stderr as
before. Found by TestInitCommand's quiet case, which asserts no output and
had not run in CI (Dolt-gated).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(init): bring the Dolt-gated init tests up to the current init contract

These tests need a Dolt test server, so no CI lane ran them and they
drifted from the product:

- Embedded is init's default mode, so the server-mode assertions
  (.beads/dolt, opening the database on the test server, dolt_mode
  "server" in metadata.json) now pass --server (TestInitCommand,
  TestInitAlreadyInitialized, TestInitRedirect, TestInitBEADS_DIR,
  TestInitDoltMetadata, TestInitDoltMetadataNoGit, TestInitDatabaseFlag's
  metadata subtests). The in-process ones reset rootCmd's flags around
  themselves so --server cannot leak into later in-process inits.
- Each server init gets its own --database: the suite shares one test
  server, and a "test" database another test created belongs to another
  project (init refuses: PROJECT IDENTITY MISMATCH).
- The marker tests seed a current-era version witness: the legacy-upgrade
  guard refuses a non-empty .beads/dolt without one.
- The redirect test provisions the canonical target with config.yaml: bd
  ignores a redirect to a directory with no project files (#4692).
- Shared-server inits get a free port of their own
  (sharedServerTestEnv): the inherited BEADS_DOLT_SERVER_PORT is the
  suite's test server, which a shared-server start refuses as another
  project's, and the default shared port is 3308, a developer's live
  server.
- --shared-server's config.yaml is now nested YAML (dolt: shared-server:
  true); parse it instead of matching the old flat line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): run the Dolt-gated tests that skipped on a bare dolt.New

setupTestDB (config, kv and Linear round-trip tests), setupGatedTestDB
(mol ready --gated) and the rename-prefix and bd sql tests opened stores
with dolt.New(&dolt.Config{Path: ...}): no server port, no
CreateIfMissing, so against a running test server they got "database ...
not found" and skipped as "Dolt server not available". They now use the
suite's newTestStore helpers (skip only when no test server runs), and
what that exposed is fixed test-side:

- TestCustomStatusConfig: custom_statuses is read back ORDER BY name;
  look statuses up by name / expect name order.
- TestLinearRoundTripCoreFields, TestLinearPullMilestonesCreatesEpicHierarchy:
  linear.api_key is a yaml-only secret (LINEAR_API_KEY), push now needs an
  explicit linear.state_map, and batch push uses TeamLabels and
  issueBatchCreate, which the mock server now answers.
- TestSqlCommand: bd sql refuses in the default embedded mode; set
  serverMode for the server-backed store, and stop discarding RunE errors.
- TestE2E_AutoStartedRepoLocalServerPersistsAcrossCommands skipped on
  this process's embedded mode although every bd it runs is a subprocess
  on a --server workspace; gate on the dolt binary instead and stop the
  auto-started server at cleanup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): no fixed "unused" port, no swallowed JSON, no leaked front-door server

Three more things the Dolt-backed cmd/bd run exposed:

- TestTestServerConnection, TestSharedServerCfgNilHonorsSharedServer and
  the two TestLegacyGuard* e2e tests pointed bd at 127.0.0.1:59999 as a
  port nothing listens on. Any process on the host may hold it; a
  listener that accepts and resets turns the expected "connection
  refused" / "Dolt server unreachable" into another error. They take a
  just-freed port from closedLoopbackPort (testutil.FindFreePort) now.
- TestDoltSetConfigJSONOutput and TestDoltSetConfigWithUpdateConfig
  created .beads 0755, so bd's permissions warning landed in the JSON
  under test and the tests skipped ("output not pure JSON") every run.
  The fixture uses 0700, and non-JSON output fails instead of skipping.
- TestMigrateDoltModeFrontDoor's closing show/list auto-start a
  repo-local server after the reverse migration; nothing stopped it, so
  the suite's leak check failed (when the opt-in
  BEADS_TEST_MIGRATION_FRONTDOOR suite runs). It stops it at cleanup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): advisory cmd/bd Dolt-server tier (bazel-cmd-dolt)

No CI lane ran cmd/bd's Dolt-gated tests (TestCLI_*, the init and
store-backed suites, the doctor/e2e tests): main.yml's integration shards,
nightly's Full Test Suite, and Bazel's --config=prcore/ci and
--config=integration all pass BEADS_TEST_SKIP=dolt, so cmd/bd's TestMain
never starts a test server and those tests skip; the proxied and embedded
tiers run only their manifests' tests. They rotted unseen.

//cmd/bd:bd_dolt_server_test (tag dolt-server-cmd) runs the whole
integration-tagged cmd/bd suite, the integration build's bd_test, against
a hermetic dolt sql-server: BEADS_TEST_DOLT_SERVER=local starts it from
the pinned dolt in the test wrapper's runfiles, and
BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 fails the target instead of skipping
into a cached pass. No test selection: the Go binary shards itself over
every top-level test (32 shards), so each runs exactly once, and
check_testcases.py fails a shard that ran none. --config=doltserver-cmd
has exactly --config=integration's build flags, so the two lanes share
bd_test's compile; prcore/ci and integration exclude the tag.

bazel.yml's new bazel-cmd-dolt job runs it exactly where bazel-integration
runs (same if: remotely on the farm for same-repo PRs, fork-ro/fork-rw,
and locally with the read-only cache in mode cache for forks). It is
advisory: job-level continue-on-error keeps a failure out of the call's
aggregate (BAZEL), and pr.yml's ci-gate reads its result (BAZEL_CMD_DOLT)
but requires it only while the committed top-level flag
BAZEL_CMD_DOLT_REQUIRED is "true" (now "false"; flipping it is the one
change needed). bazel-gate.sh accepts its skip in modes skip and local.

Policy tests: a third lane category in TestBazelLaneIsGatedAlongsideLegacy
(bazelFlagGatedLanes: gate env wired, never statically required, added by
the run script exactly when the pinned flag is "true"), the gate
simulation over every mode with the flag off (advisory: nothing it
reports reddens the gate) and on (exactly a gated lane), the cache-mode
and execution-mode checks, the tag taxonomy and prcore/integration
exclusions, and TestBazelCmdDoltJob (job steps, timeouts, .bazelrc
config equal to integration's build with no dolt skip or test selection,
the target's dolt-server rules, and bd_test's env repeated).

Measured locally (all fixes in this series, 32 shards, 16 at a time on a
shared host): 3267 top-level tests, all shards pass, max shard 313s (one
5001-issue export test, 214s, sets it), mean 100s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(init): hand the project installers a writer instead of swapping os.Stdout

The --quiet fix earlier in this series pointed the process-global
os.Stdout at the null device around the Claude/Codex/Cursor installers.
That works, but it mutates a global from inside a cobra Run: anything else
writing to stdout meanwhile (a goroutine, a future concurrent installer)
would vanish, and a concurrent read of os.Stdout is a data race.

The installers already write through env structs with stdout/stderr
fields. setup.ProjectInstallOutput carries the two writers, and
InstallClaudeProjectTo / InstallCodexProjectTo / InstallCursorProjectTo
use them (the old entry points keep the process streams). init passes
QuietProjectInstallOutput() under --quiet: progress discarded, error lines
still on stderr. The proxied-server init path (domain adapter
InstallClaudeProject now takes quiet) does the same, so its
"Installing Claude hooks..." is quiet too.

Consistent error policy under --quiet: errors still reach stderr. init's
own "Warning: failed to setup <installer>" lines were suppressed under
--quiet while the installers' "Error:" lines were not, and some installer
failures (Cursor rules) print nothing themselves; the warnings now print
whether or not quiet, like the installers' errors.

Coverage in required lanes: TestProjectInstallersReportToTheGivenOutput
(cmd/bd/setup) and TestEmbeddedInitA/basic, which now asserts that
`bd init --quiet` with agents enabled installs the Claude hooks and prints
none of the installers' progress (verified to fail with the quiet output
disabled).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(export): exercise the incremental-export fallback with a small threshold

TestTryIncrementalExport_ThresholdExceededFallsBack seeded
incrementalExportThreshold+1 = 5001 issues to cross the threshold. Batch
create is one transaction but still several statements per issue, so under
-race against the shared test server it took ~214 s, the longest test in
cmd/bd's Dolt-backed suite by far. The behaviour under test is only
"more changed issues than the threshold -> full export fallback".

incrementalExportThreshold becomes a var (same value), and the test, which
is not parallel, sets it to 20 and seeds 21 issues: 214 s -> 0.6 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): tighten the flag-reset self-test, guards and canaries

Review follow-ups:
- TestResetCommandFlagsRestoresEveryFlag no longer skips a flag it cannot
  set: a flag type without a sample value, a Set error, or a non-pointer
  Value (which resetCommandFlags could not restore) is an error, so a new
  flag type cannot go untested. resetCommandFlags documents that it also
  resets flag-bound package globals.
- TestCobraParallelPolicyGuard treats .ExecuteC/.ExecuteContext/
  .ExecuteContextC like .Execute (the per-execution isolation writes the
  mode globals; a parallel execution would race).
- TestConfigValidateReadOnlyIsHermetic ignores only the gate a readonly
  command takes (.beads/dolt.gate.lock and its .info sidecar), not any
  "*.gate.lock*" file, so a new unexpected gate still trips the canary.
- TestCLI_CreateRejectsFlagLikeTitles: a bare `bd create --foo-bar` must
  fail and create no issue (bd-2c0's intent), next to the "--" cases.
- TestCustomStatusConfig cites why statuses come back in name order: the
  normalized custom_statuses table (#2961, 8537d943b) has no ordinal and is
  read ORDER BY name, like custom_types; the test predates it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): run the cmd/bd Dolt-server tier in 16 shards

Each shard pays ~30-40 s of fixed cost (TestMain plus its own dolt
sql-server), so 32 shards spent roughly 40% of the lane's executor time on
setup and added 32 farm actions to every PR touching cmd/bd. With the
5001-issue export test cut to under a second, 16 shards measure (locally,
--nocache_test_results): max 186 s, mean 137 s, 2,187 executor-seconds
(vs ~2,800-3,200 at 32), all 3,267 top-level tests, every shard passing.
TestBazelCmdDoltJob pins the count; bazel.yml's timing comment is
restated (cache mode ~20-30 minutes).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 03:06:54 -07:00
Julian Knutsen
683993660f
fix(proxy): fail fast with a clear MySQL error when an external upstream is unreachable (#7181)
* fix(proxy): fail fast with a MySQL error when the external upstream is unreachable

A proxied-server command against a stopped external Dolt upstream stalled
~20-30s before failing with a bare "invalid connection". The local db proxy
stays up, so on a refused backend dial (or a front such as stunnel/socat
that accepts and closes because its own upstream is gone) it simply closed
the client connection. The MySQL driver reports that as invalid
connection / unexpected EOF, which the uow bootstrap ping (#6003) classifies
as a transient drop and retries with exponential backoff for its whole 30s
budget. Retrying cannot help: the proxy already proved the upstream at its
own startup, and the outage is the upstream's.

The proxy now answers such a connection itself with a MySQL ERR packet
(2003, HY000) in place of the server greeting, naming the upstream and the
cause, exactly as a real server refuses a connection before the handshake.
The driver surfaces it as a *mysql.MySQLError, which the bootstrap retry
already treats as permanent, so the command fails in ~0.1s with an
actionable message. Only shapes that prove the upstream is not serving
qualify: ECONNREFUSED/ENOENT/EHOSTUNREACH/ENETUNREACH on dial, or a backend
that reaches EOF having sent zero bytes (a MySQL server always speaks
first). A dial timeout or a drop after the greeting keeps the plain close
and the client's transient retry.

External backends only: a managed (local sidecar) backend is owned by the
proxy and its dial can be refused transiently while bd drives it (the
full proxied lane caught TestProxiedServerInitExplicitBeadsDirIsAuthoritative
hitting exactly that mid-init when this applied to every backend), so
managed backends keep the bare close and the client's bootstrap retry.
TestExternalProxyOutageAndRestoreReDialsTCPAndUnix asserted the old
silent close during an outage; it now asserts the ERR packet.

With the latency fixed, TestProxiedOutageReconnectAcceptanceMatrix is
renamed into the proxied discovery pattern as
TestProxiedServerOutageReconnectAcceptanceMatrix and placed in the Bazel
30-shard block by gen_proxied_shard_manifest.py's incremental --write
(shard 8). The frozen legacy 15-shard block is untouched; the test
hash-distributes there like every other test added since it was frozen.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(proxy,uow): retry the proxy's upstream-outage report for ~1s; hedge the zero-byte case

Review follow-up to the fail-fast change.

A zero-byte close from the upstream is not proof it is down: Dolt closes
queued connections without a byte when max_connections/back_log are full,
when a queued connection times out, and while it shuts down for a
restart; an L4 front does the same while its target is replaced. Its
report now says "closed the connection before the MySQL greeting (down,
restarting, or at its connection limit)" instead of "unreachable", and
the comment no longer claims proof.

Refusals and endpoint churn also lost #6003's retry entirely, so an
external Dolt restart or a Service with no ready endpoints failed a
command at once. The client now retries the proxy's report
(proxy.IsUpstreamOutageError: 2003 with the "beads db proxy: " prefix)
for upstreamOutageRetryWindow (1s) from the first time it sees it, with
the backoff clamped so the last attempt lands at the window's end; other
transient errors keep the 30s budget. A real outage still fails in about
a second, inside the acceptance test's 3s bound.

Also: the dial report drops Go call-site wrapping ("dial tcp H:P:
connection refused"), truncation keeps UTF-8 intact, the comment states
how the packet differs from a real server's pre-handshake ERR (keeps the
'#'+SQLSTATE marker for vitess; 2003 is a client code chosen on purpose),
a Stats counter for reported outages, and the isTransientPingError
paragraph is reflowed and corrected.

Tests: pingWithRetry retries the report inside the window and gives up
at its end without using the full budget; openDB against a front that
plays the proxy (its real packet) and comes back 300ms after the first
connection succeeds in one call, and fails in about the window when it
never does; NewProxyServer enables the report for external backends
only; the dial message carries no Go type names; rune-safe truncation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(proxied): fail, not skip, socat-bridged subtests in the Bazel lane

The external-unix front-door matrix subtest and both outage/reconnect
matrix subtests bridge an external endpoint with socat and skipped when
it was missing, so they could stop running without anyone noticing (the
parent test still passes). requireSocat now fails them when
BEADS_TEST_REQUIRE_SOCAT=1, which //cmd/bd:bd_proxied_test sets (its
remote workers have socat; a recent main run's external-unix subtest
passed there). The legacy GitHub 15-shard jobs have no socat and leave
it unset, so the subtests keep skipping there. The check now runs before
the upstream Dolt server is started, so a skip no longer pays for one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(proxied): measured weight for the outage matrix; pin it to legacy shard 1

Give TestProxiedServerOutageReconnectAcceptanceMatrix a duration entry
(80s: local runs measured 55-79s alone, 60-70s in its shard, and review
68-109s under load) and re-place it with the generator's incremental
--write: it moves from 30/8 to 30/11, one line, nothing else moved. The
heaviest estimated shard is now 435s against a ~357s median; a repack
would even that out but moves most assignments, so it is left to a
separate CI-only change that refreshes all durations together.

Legacy 15-shard block: the test would hash onto shard 14, already ~13.5
of the 15 min timeout on main. It is pinned to shard 1 (~6.5 min in
recent main runs) under the block's reviewed-pin rule; it skips there
today (no socat), and the pin keeps shard 14 safe if that changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 02:37:58 -07:00
Julian Knutsen
b8a9545cbe
ci(setup-bazel): 4 GB client heap on every runner (R4) (#7174)
The Bazel client's JVM defaults to 25% of RAM, about 2 GB on the 8 GB
2 vCPU Blacksmith runners every remote lane runs on. //... analysis fits
today with little headroom, and the plan moves more lanes and targets
onto these runners. write-bazelrc.sh now writes
`startup --host_jvm_args=-Xmx4g`: a startup option, so it changes no
action key and only restarts the server. Verified with Bazel 9.2.0:
`bazel --bazelrc=<generated rc> info max-heap-size` reports 4294MB.

gascity's W1 byte-copies this composite, so the change lands here first.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:04:39 -07:00
Julian Knutsen
c41079528d
ci: pin the Actions allow-list for the selected-and-SHA-pinned policy (R3) (#7175)
beads allows every action today (allowed_actions: all, no SHA pinning),
so a fork PR's workflow can call any action, including Blacksmith's
sticky-disk action that commits state later trusted runs restore.
.github/allowed-actions.json is the exact body of the selected-actions
setting: GitHub-owned actions plus gascity's list plus the third-party
actions beads uses. That includes DeterminateSystems/nix-installer-action,
which determinate-nix-action calls, and peter-evans/create-pull-request.

Every uses: under .github is already pinned to a full commit SHA. The new
ci_actions_policy_test.go keeps it that way: each reference is local or
owner/repo[/path]@<40-hex>, and allowed by the file, so the repository
setting can switch to allowed_actions=selected with
sha_pinning_required=true without breaking a workflow. Negative controls:
an @v6 tag ref and an unlisted owner/repo each fail the test.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:03:40 -07:00
Julian Knutsen
b1a0690249
ci: pin four hash-placed proxied tests off overloaded legacy shards 5 and 3 (#7172)
Main's "Test (proxied-server cmd 5/15)" has hit the 15m -test.timeout in
6 of the 9 Main runs since #7129 where the shard ran to completion.
That PR renamed 12 proxied tests into TestProxiedServer* discovery.
They are not in the frozen 15-shard legacy block, so they hash-place,
and two landed on shard 5:
TestProxiedServerStorageClass (nine parallel inits, the heaviest of the
12) and TestProxiedServerServeRefusesAStaleCheckpointWithGone. Shard 5
went from 430-620s of job time to 800-950s. In those runs the timeout
fired while TestProxiedServerMolReadyGated's parallel subtests were still
queued behind the rest of the shard. That test is the victim, not a
pathological one. Shard 3 (also hash-loaded) runs 810-870s, within about
a minute of the same limit.

Measured locally (4 pinned CPUs, GOMAXPROCS=4), shard 5 took 897s with
its current test set and 524s with the pre-#7129 set.

Pin those two tests, plus shard 3's two heaviest hash-placed tests
(TestProxiedServerListComments and
TestProxiedServerServeRefusesAStreamOnADisabledJournal), onto the legacy
shards with the lowest measured job time: 14, 7, 4 and 1. Every test
already listed in the block stays where it is, so shard 5 is back to its
pre-#7129 test set. pr-risk.yml's fork jobs read the same block, so they
get the same relief. The script's 15m timeout, the job and step
timeouts, and the Bazel 30-shard block are unchanged. The block header
and the generator's legacy header now say a reviewed pin is allowed.

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 00:15:08 -07:00
Julian Knutsen
4b2852ce21
fix: macOS main job — /private/var symlink in test repo pin; bash 3.2-safe CI scripts (#7165)
* fix(git): canonicalize PinNoRepositoryUnderForTesting root so the pin matches on macOS

On macOS t.TempDir()/os.TempDir() live under /var/folders/..., but /var is
a symlink to /private/var and getcwd(2) reports /private/var/folders/....
underPinnedRootForTesting compared the raw pin against os.Getwd() with a
string prefix test, so the pin never matched there and
TestPinNoRepositoryUnderForTestingSurvivesResetCaches failed on every Main
macOS run (GetRepoRoot() returned the /private/var repo instead of "").

Store the pin both as given (absolute) and symlink-resolved, and match the
working directory in both raw and resolved form. Canonicalization can only
widen the fence, never narrow it (e.g. Bazel runfiles symlink trees keep
matching exactly as before).

TestPinNoRepositoryUnderForTestingMatchesThroughSymlinks reproduces the
macOS shape on any platform with a symlinked repo root; it fails without
this change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(ci): make run-go-test-binary.sh and install-golangci-lint.sh parse under bash 3.2

The Main macOS job runs the scripts package tests with macOS's /bin/bash
3.2, where two constructs abort under `set -u`:

- run-go-test-binary.sh expanded the usually-empty PASSTHROUGH array bare;
  bash < 4.4 treats that as an unbound variable ("PASSTHROUGH[*]: unbound
  variable"), failing TestTestScriptPrebuiltTestBinaryContract and
  TestRunGoTestBinaryDefaultsTimeoutWhenOmitted. Use the ${arr[@]+...}
  guard.
- install-golangci-lint.sh used `declare -A`; bash 3.2 parses it as an
  indexed array and `[amd64]=` dies as an arithmetic lookup ("amd64:
  unbound variable"), failing
  TestInstallGolangciLintFailsClosedOnChecksumMismatch. Replace it with a
  case function and update TestGolangciLintInstallScriptPinned's pin check.

Reproduced and verified against a bash 3.2.57 build: all four tests fail
on origin/main and pass with this change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:08:51 -07:00
Julian Knutsen
59dcd83bbd
ci(bazel): use client-id for the rbe-prewarm app token; drop local plan paths (#7164)
actions/create-github-app-token (v3.2.0, pinned) deprecates its app-id
input in favor of client-id. rbe-prewarm's mint step now passes the
bazel-allocator App's public Client ID (Iv23ligrqVEhlamZnoPU, from
GET /apps/bazel-allocator) as a reviewed literal: it is not a credential,
and a literal keeps it policy-tested with the rest of the step instead of
in an unversioned repository variable. The private key stays a secret.

RBE_POOL_APP_ID is now unused, so its workflow_call declaration in
bazel.yml and its pass-through in pr.yml and nightly.yml are removed,
and a policy test rejects any workflow mentioning it again. The
repository secret can be deleted.

engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md's "rbe-west Pre-warm" section now
documents the client-id, the single remaining secret, and
RBE_PREWARM_WORKERS (default 1, clamped to 4, 0 = kill switch, repo
variable currently 2). It also no longer cites a maintainer's local
planning-notes path; a new tracked-file policy test fails if one
reappears.

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 23:03:41 -07:00
Julian Knutsen
5f98170500
ci: require the Linux-cross-built Windows test jobs (WINDOWS_PREBUILT_REQUIRED=true) (#7161)
The prebuilt twins of the Windows liveness and worktree-remove jobs passed
alongside the native ones on every PR since F4 (#7135): #7139, #7145,
#7147, #7155, #7156, #7159, #7160 — 1-1.5 min vs 3-8 min native. Flip the
committed flag so ci-gate enforces the prebuilt pair; the native pair keeps
running as advisory and is the rollback (set the flag back to "false").

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 22:44:58 -07:00
Julian Knutsen
88d9b645a6
ci(bazel): pre-warm rbe-west's OSS worker pool when a same-repo PR's Bazel lanes start (#7159)
* ci(bazel): best-effort pre-warm rbe-west's OSS worker pool on PR open

bazel.yml's Bazel lanes each separately wait ~120s at "N actions, 0
running" while gastownhall/gascity's rbe-west NativeLink worker pool
scales up from zero. Add an advisory rbe-prewarm job (never a `needs`
of any lane, never in bazel-gate.sh's vocabulary, job-level
continue-on-error so it can never fail bazel.yml's own run) that
mints a short-lived installation token for a "bazel-allocator" GitHub
App (installed on gastownhall/gascity only, Actions read/write +
Metadata read) and dispatches gastownhall/gascity's
rbe-worker-pool.yml ahead of time, only when this run's lanes will
actually target rbe-west's "oss" instance (modes remote and fork-rw;
fork-ro targets a separate pool this doesn't serve).

Adds RBE_POOL_APP_ID / RBE_POOL_APP_PRIVATE_KEY as new optional
workflow_call secrets, threaded through every caller (pr.yml,
bazel-farm.yml, nightly.yml); either secret missing (the app not yet
provisioned) or repo var RBE_PREWARM_WORKERS=0 disables pre-warming
cleanly. Policy tests (scripts/bazel_rbe_prewarm_test.go, plus updates
to the existing bazel.yml structural tests) pin the mode gate, the
credential scope, the dispatch target, and that the job can never gate
or fail a run; each invariant was hand mutation-tested. Docs added to
engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md ("rbe-west Pre-warm").

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): close rbe-prewarm token-theft path, fix stale secret-count docs (B1)

Security review of bdef342d5 found a BLOCKER (B1): the rbe-prewarm job
checked out inputs.checkout-sha (the fork head, reachable via
bazel-farm.yml's pull_request_target path) and then ran
.github/scripts/rbe-prewarm.sh with the minted gastownhall/gascity
dispatch token present. Script-file edits don't require workflow-approval
the way workflow-file edits do, so a VIP fork or same-repo branch pusher
could edit that script to steal the shared bazel-allocator key.

Fix: delete the checkout step and the .sh file; inline the dispatch logic
into the step's own `run:` block, whose bytes always come from the
trusted (approval-gated) workflow definition even under
pull_request_target. Mint the token in the immediately preceding step;
nothing after the mint step touches a repo-relative path. Stop forwarding
RBE_POOL_APP_ID/RBE_POOL_APP_PRIVATE_KEY from bazel-farm.yml (that fork
path now gets a cold start instead of pre-warming) and narrow the job's
`if:` to mode == 'remote' only (fork-rw never carried secrets anyway, so
it was dead weight - S1).

Also addresses the review's should-fix items:
- S2: document the widened trust boundary (the bazel-allocator key is
  shared with gascity's own rbe-west scaler; leak blast radius; rotation
  breaks the scaler, so the kill switch is unset-secret /
  RBE_PREWARM_WORKERS=0, not rotation) as an accepted risk in bazel.yml's
  header and engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md.
- S3: fix stale "exactly four secrets" claims (pr.yml/nightly.yml pass 6;
  bazel-farm.yml now correctly passes 4 again) and the false "never a
  long-lived credential" claim.
- S4: flag the continue-on-error-preserves-caller-success assumption as
  reasoned-through but not live-validated in this sandbox.
- N1: drop the unused RBE_PREWARM_IDLE_MINUTES knob entirely.
- N4: strengthen the dispatch-target test to check exact gh command
  strings and ban hardcoded repo/workflow literals outside $POOL_REPO/
  $POOL_WORKFLOW.

Test changes (scripts/bazel_rbe_prewarm_test.go rewritten; ci_workflow_test.go,
bazel_farm_workflow_test.go, bazel_cache_mode_test.go updated for the new
job shape): new TestRBEPrewarmNoCheckoutStep,
TestRBEPrewarmNothingAfterMintTouchesARepoPath,
TestRBEPrewarmTokenOnlyInDispatchStep, and
TestRBEPrewarmAppSecretsOnlyExpectedCallers directly target the B1
mutation class; TestRBEPrewarmDispatchScriptBehavior keeps the fake-gh
behavioral coverage for the now-inlined dispatch script. Verified by hand
that both of the reviewer's mutations (widen the `if:` back to include
fork-rw; add a step after mint running a checked-out script) are now
caught.

Verification: go test ./scripts/..., go vet, gofmt, actionlint on the
four touched workflow files, make ci-pr-policy, make bazel-sync-check,
and `bazel test //scripts:scripts_test` all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 22:31:17 -07:00
Julian Knutsen
1137aeb70d
ci(bazel): embedded-Dolt tier 20/5 → 50/15 duration-balanced Bazel shards (F1) (#7156)
* ci(bazel): embedded-Dolt tier 20/5 -> 40/14 duration-balanced shards (F1)

Give the Bazel-only bazel-embedded job its own duration-balanced manifest
blocks for //cmd/bd:bd_embedded_test (20 -> 40 shards) and
//internal/storage/embeddeddolt:embeddeddolt_embedded_test (5 -> 14
shards), decoupled from PR Risk's/main.yml's legacy frozen 20/5-shard
fork/push jobs, mirroring F2's bd_proxied_test precedent.

- Measured every TestEmbedded* / embeddeddolt test's wall time in
  isolation; split 21 cmd-tier and 1 storage-tier single top-level test
  over 180s into disjoint sub-tests (3 irreducible exceptions left:
  TestEmbeddedUpdateConcurrent 475s, TestEmbeddedCloseConcurrent 308s,
  TestLargeBatchApplyStatementCounts712_Embedded 261s).
- New scripts/ci/gen_embedded_{cmd,storage}_shard_manifest.py +
  _embedded_shard_manifest_lib.py generate/check duration-balanced
  manifest blocks (LPT packing, incremental --write/--repack, exact-once
  --check), chosen at N=40/M=14 where the heaviest shard reaches the
  irreducible-test floor.
- Fixed 18 stale legacy 20-shard cmd manifest entries (and 1 storage
  entry) left over from earlier test splits, which would have hard-failed
  every shard of PR Risk's real test-embedded-cmd jobs on their next run.
- New scripts/embedded_shard_count_test.go policy tests tie the
  Bazel-only block's shard count to bazel.yml's check_shard_coverage.py
  argument and catch generator staleness; generalized
  TestBazelRetiredLanesCheckListedTestsRan's live-shard-count dispatch
  from bd_proxied_test-only to a map covering both embedded targets;
  removed TestBazelEmbeddedJobMirrorsEmbeddedTier's now-invalid
  BUILD.bazel-shard-count-equals-PR-Risk-matrix assertion.
- Verified: go test ./scripts/..., bazel test //scripts:scripts_test,
  actionlint, make ci-pr-policy, make bazel-sync-check all pass; full
  local bazel test of both targets at 40/14 shards passes 2/2; mutation
  testing confirms each new/changed policy test fails exactly when it
  should.

See ~/beads-bazel-plan/f1/impl-report.md for full detail.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(embedded-dolt): skip 712-item statement-count assertion under -race (F1)

TestLargeBatchApplyStatementCounts712_Embedded genuinely costs ~260-300s
under -race (confirmed by solo re-measurement: 298.60s) -- 14014 real SQL
statement round-trips through the race-instrumented in-process Dolt engine,
not an artifact and not splittable (one indivisible ApplyBatchInTx
transaction). It was the storage tier's single-test floor for CI shard
packing.

Apply the exact precedent TestLargeBatchApplyWallClock_Embedded already set
for its 1000-item shape: skip this assertion under raceEnabled (the cost is
race-instrumentation overhead on the Dolt engine's own internal
goroutine/lock machinery, not this test's own logic) and extend nightly.yml's
"Embedded Dolt batch-apply suite (non-race)" step's -run regex so the full
pinned 14014-statement baseline still runs, non-race, every night. Verified
directly: under -race the assertion now skips in 0.00s; without -race it
still runs and asserts Total=14015 (within the existing +/-2 tolerance) in
36.76s.

Updates scripts/ci_workflow_test.go's
TestNightlyFullTestRunsEmbeddedDoltBatchApplySuite to match the extended
regex.

Storage tier's new single-test ceiling under the per-PR -race lane drops
from ~260.7s to 152.38s (TestBatchApplyContract).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* ci(bazel): correct cmd-tier long-pole durations, re-pack embedded manifolds (F1)

Investigated coordinator item 2 (single-test long poles): TestEmbeddedUpdateConcurrent
(recorded 475.39s) and TestEmbeddedCloseConcurrent (recorded 308.06s), plus
the two other >180s cmd-tier entries from the same capture
(TestEmbeddedInitA 196.94s, TestEmbeddedEpicCloseEligibleReason 192.47s).

Root cause: NOT host contention, NOT backoff/retry sleeps (bdRunWithFlockRetry
contributes negligibly), NOT race-instrumented-subprocess overhead alone.
These numbers were captured via `bazel --nohome_rc test --config=embedded`
with the then-current 20-shard legacy manifest (~10-11 tests/shard) and
`-test.parallel=4`. Go's t.Parallel() starts each parallel test's elapsed
clock when its function is invoked, before the Parallel() call blocks it
waiting for a free one-of-4 slot -- so a test queued behind several other
heavy shard-mates has that queueing wait baked into its own recorded
`--- PASS: TestX (Ys)` time. This is shard-composition-dependent scheduling
delay, not a cost intrinsic to the test.

Confirmed two ways in an isolated clone (not this worktree, per the
no-plain-go-test-from-a-beads-checkout rule):
- Solo (no rivals at all): 151.83s / 106.48s / 63.58s / 66.08s.
- All four together with the real -race -test.parallel=4 flags and exactly
  4 tests requested (so nobody queues -- parallel slots == test count):
  141.34s / 113.80s / 69.79s / 74.63s.
Both agree closely and both are far below the capture-time numbers.

F1's LPT packer already isolates the heaviest tests into shards with few or
no other heavy rivals (TestEmbeddedUpdateConcurrent lands alone in its own
40-shard-block shard), so under the new manifest these tests see little to
no queueing delay in real CI. None of the four needed splitting, parallel
subtests, or raceEnabled-scaled N: once correctly measured, none exceeds the
180s single-test guideline (new corrected weights: 155.0 / 115.0 / 72.0 /
77.0s). Corrected scripts/ci/embedded_cmd_test_durations.json accordingly
(with a doc-comment explaining the methodology and both confirmations) and
re-packed the 40-shard Bazel-only block (--repack --write); --check passes
(exact-once coverage of all 241 discovered tests). The frozen legacy 20-shard
block is untouched and independently re-verified exact-once by simulating
embedded-test-shard.sh across all 20 shards (241/241 tests, no
duplicates/misses/errors).

New cmd-tier single-test ceiling: 179.47s (TestEmbeddedCloseAlreadyClosed,
likely itself somewhat inflated by the same mechanism but already under the
180s line as recorded, so no action needed).

Also updates scripts/ci/embedded_storage_test_durations.json: the prior
commit's raceEnabled-skip for TestLargeBatchApplyStatementCounts712_Embedded
means it now costs ~0s in the per-PR -race lane bazel-embedded actually
packs for (not its real ~260-300s non-race cost, which nightly.yml still
exercises) -- weight set to 1.0 with a doc comment tying it to the skip in
large_batch_apply_measure_test.go. Re-packed the 14-shard Bazel-only block
accordingly; --check passes (exact-once coverage of all 387 discovered
tests, excluding TestMain/TestConformance per that file's existing
convention). The frozen legacy 5-shard block is untouched and independently
re-verified exact-once via embedded-storage-test-shard.sh simulation
(388/388 tests, no duplicates/misses/errors).

Verification: go test ./scripts/... (includes the nightly.yml policy test),
make ci-pr-policy, make bazel-sync-check, actionlint all pass. Real local
`bazel --nohome_rc test --config=embedded //cmd/bd:bd_embedded_test
//internal/storage/embeddeddolt:embeddeddolt_embedded_test`
(--local_test_jobs=8, both targets together, on a shared/noisy sandbox host)
passes both targets; per-shard wall time from each shard's test.xml: cmd/bd
max 218.9s (shard 23/40, min 20.6s), storage max 251.1s (shard 5/14, min
0.98s) -- down from the prior oversubscribed sweep's 783.41s / 351.94s and
well under the single-test-floor-based ~475s projection this slice started
from. Projected tier lane time (max of the two targets, real dedicated CI
runners should do at least this well): ~251s, under the 5-minute target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(embedded): un-vacuous TestEmbeddedDepB edges, split epic-close test, trim concurrency stress N (F1 review)

Coordinator review follow-ups on the F1 embedded-Dolt resharding slice:

- TestEmbeddedDepB (one of TestEmbeddedDep's two CI-speed split halves) was
  asserting its tree_* subtests against an edgeless graph, passing
  vacuously; restores the same parent-child/blocks edges TestEmbeddedDepA
  builds so those subtests exercise real multi-node traversal (review S2).
- TestEmbeddedEpicCloseEligibleReason split into
  TestEmbeddedEpicCloseEligibleReasonFlag/...Output for CI shard balance,
  matching the project's established single-concern split pattern.
- TestEmbeddedUpdateConcurrent/TestEmbeddedCloseConcurrent's
  issuesPerWorker reduced 5->2: under real concurrent-shard-load
  measurement these were genuine (not queueing-artifact) serialized-write
  costs; 2 preserves every invariant the test checks (duplicate-ID
  detection across all workers, one non-decreasing-list-count check per
  worker) at a fraction of the serialized work.
- large712ShapeName: the 712-shape's race skip and its zero-match guard now
  share one named constant instead of two independent literal shape-name
  strings, so a future rename of the shape can't silently skip the pinned
  statement-count assertion under race (review N2).
- Comment-only fixes (review N5) in files referencing test names since
  renamed/split by this slice's earlier work: close_proxied_integration_test.go,
  graph_apply_test.go, storage_class_proxied_integration_test.go,
  tests/regression/discovery_test.go, plus doc-comment path cleanups (N1:
  drop unverified ~/beads-bazel-plan paths/bead ids in favor of the
  committed duration JSON files and engdocs/TESTING.md).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* ci(bazel): re-derive embedded-Dolt durations under concurrent load; cap cmd shards at Bazel's 50-shard limit (F1 round 2)

Coordinator review required re-deriving every per-test duration weight from
a single, consistent, CI-realistic measurement (all shards running
concurrently, -test.parallel=4, race-enabled) instead of the prior pass's
mix of crowded-shard captures and solo/4-together spot-corrections, and
required measuring per-shard wall time via BEP testAttemptDuration rather
than a shard's outer test.xml <testsuites time> (which can include a
different, already-finished shard-mate's setup/teardown).

- scripts/ci/embedded_cmd_test_durations.json (242 tests) and
  scripts/ci/embedded_storage_test_durations.json (387 tests) rewritten in
  full from one consistent sweep. Every entry is now a fresh measurement;
  none is a carried-over capture-time value or an unverified estimate. Both
  files' _comment blocks retract the earlier "queueing delay baked into a
  test's own reported elapsed time" theory (testing.go's (*T).Parallel does
  not add wait time to a blocked test's own clock) and document the real
  methodology.
- Only one cmd-tier test remains over 200s (TestEmbeddedListFiltering,
  239.70s), already under the ~240s ceiling; no further top-level splits
  were needed this round.
- New finding: Bazel hard-caps shard_count at 50 per sh_test-like rule
  ("Having more than 50 shards is indicative of poor test organization";
  a native analysis-time limit, not configurable). The previous 88-shard
  plan for //cmd/bd:bd_embedded_test (chosen as the LPT-packed floor before
  this limit was discovered) is reverted to 50, Bazel's real ceiling.
  Repacking the fresh weights at 50 shards gives a heaviest shard of
  409.6s -- above the single-test floor of 239.7s, because 50 bins is not
  enough headroom over the suite's ~20145 measured test-seconds to flatten
  every bin to that floor. This makes the <=5 min lane target unreachable
  for the cmd tier via resharding alone within Bazel's native per-rule
  limit; see ~/beads-bazel-plan/f1/impl-report.md section 3c for the full
  writeup and the two remaining levers (suite-wide speedups or a
  multi-Bazel-rule split), neither attempted in this slice.
- internal/storage/embeddeddolt gets 15 Bazel-only shards (unaffected by
  the 50-shard cap): LPT-packed heaviest shard 225.3s, within target.
- cmd/bd/BUILD.bazel, internal/storage/embeddeddolt/BUILD.bazel,
  .github/workflows/bazel.yml, engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md,
  scripts/embedded_shard_count_test.go, scripts/pr_risk_bazel_coverage_test.go,
  scripts/ci/gen_embedded_cmd_shard_manifest.py updated to the corrected
  50/15 shard counts (and, for the generator, N3: --write without an
  explicit total_shards now errors instead of silently regenerating the
  frozen legacy 20-shard block).
- .github/scripts/embedded-{cmd,storage}-test-shards.txt: re-packed
  Bazel-only blocks at 50/15; frozen legacy 20/5 blocks untouched except for
  N5 fixes to stale entries left by this slice's earlier test splits
  (TestShardScriptsListOnlyRealTests catches this class of bug; both
  legacy blocks reconfirmed exact-once).
- .github/workflows/nightly.yml / scripts/ci_workflow_test.go (N4): the
  embedded-dolt batch-apply nightly step now runs with `if: ${{
  !cancelled() }}` and its own explicit `-timeout 20m`, independent of the
  job-level timeout, with both halves now asserted by
  TestNightlyFullTestRunsEmbeddedDoltBatchApplySuite.

Verified: gofmt -l clean, go vet ./scripts/... clean, go test ./scripts/...
green, both generators' --check pass at 50/15 (and legacy 20/5 remain
exact-once via the frozen-block invariant).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(ci): rename embedded manifest staleness tests out of the ^TestEmbedded skip

The allowlisted go-test leg runs with -skip ^TestEmbedded, so tests named
TestEmbedded*ShardManifestGeneratorNotStale never ran there and the leg
failed with "matched no test that ran".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 21:30:03 -07:00
Julian Knutsen
84ce244857
ci: same-repo Linux/Windows PR jobs on Blacksmith with main-seeded warm build caches (F7b) (#7155)
* ci: move same-repo PR Linux/Windows/macOS legs to Blacksmith, shard preflight (F7b)

PR-path jobs for same-repo PRs/merge_group now run on Blacksmith instead
of GitHub-hosted runners, following F4/F7a's ternary-expression and
restore-only-cache patterns; forks/Dependabot are unaffected:

- scripts-go-checks, pr-lint-wrapper: same-repo Blacksmith 4vcpu Linux.
- check-doc-freshness-platforms, pr-preflight-platforms: restructured
  from a top-level matrix.os list to an include:-only "matrix marker"
  shape (runner: same-repo-linux/-windows/-macos) so each host can
  resolve an independent same-repo Blacksmith selection via one chained
  ternary runs-on expression.
- Six standalone Windows jobs (test-windows-liveness,
  worktree-remove-windows, test-windows-small,
  test-windows-liveness-prebuilt, worktree-remove-windows-prebuilt,
  windows-make-shell) move to same-repo Blacksmith Windows
  (blacksmith-{2,4}vcpu-windows-2025, public beta). CGO/mingw-w64 and
  Chocolatey preinstall are unconfirmed on that image; both risks are
  flagged with one-line rollback instructions at the call sites.
- pr-preflight-platforms' Windows leg is split into two shards (common
  steps on shard 1, the three Windows-only heavy steps on shard 2) to
  shorten its critical path; shard 2 downloads Go modules directly
  instead of sharing shard 1's runner/cache.

main.yml gains cache savers for the newly same-repo-Blacksmith PR
consumers, since Blacksmith and GitHub-hosted runners cannot see each
other's actions/cache entries even for identical keys:

- blacksmith-go-build-cache warms the Blacksmith-selection race/non-race
  GOCACHEs scripts-go-checks and pr-preflight-platforms/check-doc-
  freshness-platforms' ubuntu legs need, push-to-main only. Its non-race
  leg shares scripts/ci/warm-non-race-cache.sh with build-artifacts, and
  compiles tools/bazel/run_allowlisted_go_tests.py's allowlisted
  packages via its new --compile-only flag, so the two paths and the
  warm-up can never drift from what they actually run.
- pr-lint-wrapper, go-vet-cache, test-windows gain a venue: [blacksmith,
  github] matrix leg to seed both runner selections pr.yml's
  same-repo-Blacksmith-or-fork-hosted consumers can land on.

Updates the policy-test suite (ci_workflow_test.go,
ci_blacksmith_runner_test.go, pr_lanes_bazel_coverage_test.go) to cover
the new runner selections, matrix-marker shape, shard gating, and cache
topology, including a dedicated inventory/writer check for
blacksmith-go-build-cache.

Deferred follow-up (not done here): sharding check-doc-freshness-
platforms and the standalone Windows jobs, and moving/sharding macOS
legs beyond what check-doc-freshness-platforms'/pr-preflight-platforms'
matrix-marker form already covers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): F7b review response -- msys2/Blacksmith-saver guards, macOS stays hosted (B1/B2/S3)

Addresses ~/beads-bazel-plan/f7b/review.md's two cache-safety blockers and
one follow-up gap found after F7b moved same-repo PR Linux/Windows/macOS
legs onto Blacksmith and added push-to-main Blacksmith cache seeders:

- B1: windows-make-shell's msys2/setup-msys2 step caches its installed
  packages by default, which implicitly SAVED a GitHub Actions cache entry
  from a same-repo PR the moment it ran on a Blacksmith runner (confirmed
  via a real cache-list entry from a refs/pull/.../merge ref). Disabled with
  a fail-closed `cache: ${{ runner.environment == 'github-hosted' }}`, and
  generalized the sweep (TestBlacksmithReachablePRJobsDisableDefaultCachingActions)
  to cover any default-caching setup-* action on a Blacksmith-reachable job,
  not just this one.

- B2: blacksmith-go-build-cache, pr-lint-wrapper, go-vet-cache and
  test-windows (main.yml) had no (or an incomplete) same-repo push-to-main
  guard, so a same-repo fork with push access to its own main could run --
  and save a cache entry from -- a paid Blacksmith job. Added the exact
  push-to-main/same-repo `if:` F7c's own blacksmith-setup-go-cache already
  uses, pinned by TestBlacksmithSaverJobsGuardedAgainstPullRequest.

- S3: the macOS leg of pr-preflight-platforms/check-doc-freshness-platforms
  resolved to Blacksmith for same-repo PRs; Blacksmith bills macOS at
  roughly 20x its Linux/Windows rate and no push-to-main Blacksmith-macOS
  saver exists, so that leg would pay the premium for a guaranteed cold
  cache. macOS now stays unconditionally on GitHub-hosted macos-latest.

Also pins S7 (blacksmith-go-build-cache/go-vet-cache/test-windows's cache
keys are bounded by go.sum+UTC-day, not github.sha) and S6/S8 (their
20-minute timeouts), both already in place but previously unpinned, and
replaces run_allowlisted_go_tests.py's fragile GO_TEST_FLAGS[-2:] slice with
an explicit BUILD_FLAGS constant (N1).

Re-ran the reviewer's 15 mutations (~/beads-bazel-plan/f7b/review/mutate.py):
all now killed or structurally moot (the Windows preflight shard split this
review asked to revert-unless-helpful was reverted, in an earlier commit on
this branch, removing the vulnerability surface 6 of them targeted).

scripts/BUILD.bazel: add the new ci_f7b_review_test.go to scripts_test's
srcs -- bazel test caught it missing (file existed on disk but was not
compiled into the Bazel-built test binary).

Verification: go vet/test/build, gofmt, bazel --nohome_rc test
//scripts:scripts_test, actionlint -shellcheck, shellcheck, make
ci-pr-policy, make bazel-sync-check all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(ci): drop local review-doc path from comment

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 17:58:31 -07:00
Julian Knutsen
6be9c861a8
ci: advisory workflows on Blacksmith with restore-only caches, folded matrices, path filters (F7c) (#7147)
* ci(advisory): move Conformance/Regression/Migration/Smoke/Docs/Proxied-Local to Blacksmith, fold matrices, path-filter, drop redundant nix-build PR trigger (F7c)

Reduces per-PR GitHub-hosted job load for advisory (non-required) workflows
per the "PR CI -> 5 min" plan, slice F7c. None of these workflows gate the
required CI Gate / PR Risk Gate checks.

- conformance.yml: same-repo PR/merge_group runs move to Blacksmith
  (4vcpu); PR trigger gets the shared "upgrade-relevant code" path filter
  (any non-test Go change or build input); also now runs on push to main.
- regression.yml: fold detect-regression-need into regression's first step
  (gated via `if:`), eliminating a whole job; move to Blacksmith.
- migration-test.yml: fold the 14-version x cache-identity matrix into 3
  shards (src/pre-dolt/dolt-runtime), looping every version in a shard via
  a new "Verify explicit historical upgrades" step so a bad version does
  not hide the rest; gains the shared path filter; move to Blacksmith.
- cross-version-smoke.yml: fold the 1-job-per-version matrix into chunks of
  5 (delegated to upgrade-smoke-test.sh's native SMOKE_VERSIONS support);
  gains the shared path filter; move to Blacksmith.
- docs-mintlify.yml, proxied-local-smoke.yml: move to Blacksmith
  (2vcpu/4vcpu); add missing timeout-minutes where absent.
- nix-build.yml: drop the pull_request trigger — pr-risk.yml's required
  test-nix job (nix run + nix flake check) is a strict superset of
  `nix build .#default`, pinned by a new regression test.
- main.yml: add blacksmith-setup-go-cache, a push-to-main-only job that
  warms setup-go's implicit cache with cache: true, since Blacksmith
  runners cannot see GitHub-saved actions/cache entries that the six
  workflows above now rely on via setup-go's own caching. Carves a scoped,
  documented exception into assertPinnedGoCacheActions for this one job;
  every other setup-go step in main.yml/pr.yml/pr-risk.yml still must pin
  cache: false.

Forks and Dependabot PRs are unaffected (still ubuntu-latest, no secrets),
via the same conditional runner expression used elsewhere in the repo.

New policy tests in scripts/ci_f7c_advisory_test.go pin: path-filter
identity across the three filtered workflows, full version-set coverage
of the migration-harness fold (no scenario dropped), the per-shard/per-
chunk retry-all-versions-even-on-failure loop, fork/dependabot runner
selection semantics, no Blacksmith job reads secrets, and that the new
cache-seed job exists and that its six consumers don't disable the cache
it warms. All new tests and their underlying behaviors were mutation-
tested (each fails when the behavior it pins is reverted).

Caveat for rollout: the Blacksmith cache-seed job assumes Blacksmith
labels have branch-protected cache isolation from fork/untrusted PRs at
the org level (spec-f7.md Group B precondition) — this needs confirming
with an org admin before merge, since the seed job's cache becomes shared
infrastructure the six advisory workflows implicitly trust.

Replayed the last 60 merged PRs' changed-file lists against the new path
filter: 20/60 would now skip Conformance/Migration/Smoke (matches the
spec's estimate), and none of the 60 touched non-test Go code while being
classified as skip.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,advisory): F7c review B1/B2 - pin migration fallback, restore-only Blacksmith caches

B1: migration-test.yml's non-Blacksmith historical-upgrades fallback was
`ubuntu-latest`, which actions/runner-images#14748 schedules to move from
24.04 to 26.04 starting 2026-10-19. 26.04 drops libicu74, which the v0.55.4
fixture's ICU ABI depends on (and which "Install test dependencies" installs
explicitly). Pin the fallback to the literal `ubuntu-24.04` label instead, and
add the matching sameRepoBlacksmith4vcpuNoble constant so the pin itself is
tested. Confirmed via `git show <pre-F7c>` that none of the other five
workflows F7c touched ever had a pinned-OS fallback, so this regression was
scoped to migration-test.yml only.

B2: Blacksmith cannot see GitHub-hosted cache entries, and a same-repo PR
writing a cache entry another PR or main later reads back unverified is a
poisoning vector. Make every advisory consumer (conformance, regression,
migration-test, cross-version-smoke, docs-mintlify, proxied-local-smoke)
disable setup-go's own implicit cache on self-hosted runners and restore-only
from a self-defined `blacksmith-sg-v1-...` key namespace (not setup-go's own
implicit key, so there is no dependency on Blacksmith's $ImageOS matching
GitHub-hosted resolution). Only main.yml's new blacksmith-setup-go-cache job
(push-to-main only, always trusted) saves: it always restores, unconditionally
re-runs every warm-up command (no longer gated on a cache hit, so a stale or
partial restore self-heals), and always saves under the same key. Delete the
now-unneeded goCacheImplicitCacheExceptions carve-out in ci_workflow_test.go.

Also convert cross-version-smoke.yml's smoke-binaries-* and regression.yml's
regression-baseline-* caches from a monolithic actions/cache (auto-saves on
any miss) into an explicit restore (always, modulo the job's own gate) / save
(gated off pull_request) pair, closing the same poisoning vector for
per-release binaries. migration-test.yml's historical-dolt-* cache is left
unchanged: it is already checksum-verified on every read by
scripts/migration-test/lib/binary.sh.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(ci,advisory): F7c review SF - behavioral tests, path filters, F7a naming coordination

Replace substring-matching tests with behavioral ones that execute the real
embedded bash/jq against synthetic input, closing the reviewer's 12 surviving
mutations:

- TestMigrationHarnessLoopExecutesBehaviorally runs the real "Verify explicit
  historical upgrades" step against a stub run.sh: every version in a shard
  still runs after an earlier one fails (closes `exit 0`/`break` mutations),
  failures are ::error:: annotated and written to $GITHUB_STEP_SUMMARY, a bad
  or empty SHARD_VERSIONS fails closed without ever invoking run.sh, and a
  hung version is bounded by `timeout --kill-after` without hiding the version
  queued after it. The step's env now exports SHARD so its run: body has no
  literal ${{ }} GH-expression syntax left for a test harness to choke on.
- TestMigrationHarnessCacheHashMatchesDerivation pins each shard's cache-hash
  against sha256(join(",", <shard's cache-identity strings>))[:16], recovered
  from the pre-fold 14-entry table (git show 362eecc6bf), closing a stale-
  cache-hash mutation.
- TestMigrationHarnessDoltRuntimeStepGatedByShard pins the exact gate
  expression for "Install Dolt test runtime".
- TestCrossVersionSmokeChunksEveryResolvedVersion now executes the real jq
  chunking and join programs (via the new ghExtractJQProgram helper) against
  synthetic version lists of several sizes, and asserts the smoke step's run:
  command is an exact match - closing jq-slice-off-by-one and
  dropped-SMOKE_VERSIONS mutations a substring check could not catch.
- TestRegressionStepsAfterDetectAreGated pins that every step after
  regression.yml's detect step stays gated on
  steps.detect.outputs.run_regression == 'true', closing an
  ungated-regression-step mutation.
- TestBlacksmithSetupGoSeedExistsForAdvisoryConsumers,
  TestAdvisoryBlacksmithConsumersAreCacheRestoreOnly (replacing
  TestAdvisoryBlacksmithConsumersKeepImplicitSetupGoCache),
  TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers, and
  TestAdvisoryBinaryCachesAreRestoreAlwaysSavePRGated cover the B2
  restore-only/seeder-saves cache topology end to end, including a
  seeder/consumer key-match check and scoping the "no save in a consumer job"
  check to the blacksmith-sg-v1- namespace specifically so migration-test.yml's
  legitimate, checksum-verified historical-dolt-* cache is not flagged.

Path filters: add scripts/migrate-legacy-to-current.sh to migration-test.yml
(the harness invokes it directly but it lives outside scripts/migration-test/)
and internal/storage/embeddeddolt/conformance_test.go plus backend/conformance/**
to conformance.yml, with a new TestConformancePushPathsMatchPullRequestPaths
pinning push.paths and pull_request.paths stay identical.

Coordination with F7a (ci/f7a-blacksmith-fold, lands before F7c): F7a
independently defines its own sameRepoBlacksmith4vcpu/2vcpu constants and a
TestSameRepoBlacksmithExpressionSemantics, and is moving a real GH-Actions
expression evaluator into a shared scripts test helper file. Per the
coordinator, rename this branch's same-named test to
TestF7cAdvisorySameRepoBlacksmithExpressionSemantics (deliberately left as the
same simple literal mirror it always was, not a competing evaluator) so it
does not collide with F7a's helper on rebase; at rebase time this test should
be reduced or replaced in favor of F7a's shared evaluator.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(ci): update stale nix-build PR-trigger and renamed check-name references (F7c review S6)

F7c (bb81136203) dropped nix-build.yml's pull_request trigger as redundant
with PR Risk's test-nix job, and folded cross-version-smoke.yml's one-job-
per-version matrix into one job per 5-version chunk. Update the docs that
still described the old topology:

- CONTRIBUTING.md: nix build .#default no longer runs on PRs; point at PR
  Risk's required test-nix job instead.
- update-vendor-hash.yml: same correction in its header comment and the
  GITHUB_TOKEN-retrigger caveat.
- engdocs/CI_TEST_SURFACE_AUDIT.md: migration-test.yml now also runs on PRs
  (path-filtered, advisory) across 3 shards instead of 14 per-version legs;
  nix-build.yml's row now reflects push-to-main-only.
- engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md: replace `Upgrade smoke
  (<version> -> candidate)` with the current `Upgrade smoke (chunk N)` job
  name in both the "do not require directly" list and the Cross-Version
  Smoke section; rewrite the Nix Build section to describe nix-build.yml as
  push-to-main-only (no longer a PR check to warn about requiring directly)
  and remove the now-unreachable `nix build .#default` bullet from the
  "do not require" list.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* ci(advisory): rebase onto F7a shared Blacksmith helper, drop duplicate const/evaluator

F7a (PR #7139, ci/f7a-blacksmith-fold) landed its own sameRepoBlacksmith4vcpu
const and TestSameRepoBlacksmithExpressionSemantics, plus a real GitHub
Actions expression evaluator (evalGHExpr/mustEvalGHRunsOn) in a new shared
scripts/ci_blacksmith_runner_test.go. This branch rebased onto
origin/ci/f7a-blacksmith-fold and reconciles the two:

- Removed this branch's duplicate sameRepoBlacksmith4vcpu definition in
  ci_workflow_test.go (rebase conflict); it's now served solely by F7a's
  shared const. sameRepoBlacksmith4vcpuNoble (F7c's B1 fix, fallback pinned
  to ubuntu-24.04 instead of ubuntu-latest) is kept as its own const, as
  instructed, since it is F7c-specific and not shared.
- Replaced TestF7cAdvisorySameRepoBlacksmithExpressionSemantics's
  hand-written Go mirror of the same-repo Blacksmith boolean logic with real
  evaluation of each F7c advisory job's actual, as-parsed runs-on expression
  text via F7a's evalGHExpr/mustEvalGHRunsOn, across the same event-shape
  truth table F7a's own test uses (same-repo/fork/dependabot/deleted-fork-
  head/merge_group/push/pull_request_target/schedule/workflow_dispatch).
  This covers all 8 F7c advisory Blacksmith jobs, including
  migration-test.yml's historical-upgrades job falling back to the literal
  ubuntu-24.04 label rather than ubuntu-latest. Confirmed via a cp-backed
  mutation (flipping the Noble fallback back to ubuntu-latest) that this new
  test, unlike the deleted hand-written mirror, actually catches a
  regression in the real expression text.
- Merged two more rebase conflicts in ci_workflow_test.go: the
  assertNoUnmanagedGoCacheSteps main.yml allowlist (F7a's
  windows-test-binaries-cache plus F7c's blacksmith-setup-go-cache) and a
  subsequent non-conflicting-but-incomplete 3-way merge of
  TestSameRepoBlacksmithRunners' main.yml entry (added
  windows-test-binaries-cache alongside blacksmith-setup-go-cache so the "no
  other Blacksmith label leaks" sweep covers both).

Did not merge TestBlacksmithAdvisoryJobsReadNoSecrets into F7a's
TestBlacksmithJobsReadNoSecrets (ci_workflow_test.go) even though the shared
helper file's doc comment suggests it; out of scope for this round per the
coordinator's explicit instructions, left as a follow-up.

Re-ran full verification after rebase + reconciliation: go build/vet/test
./scripts/..., bazel --nohome_rc test //scripts:scripts_test, actionlint
-shellcheck=shellcheck across all touched workflows, make ci-pr-policy, make
bazel-sync-check - all green. gofmt -l clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,advisory): close F7c re-review blocker X1 and S1-S5/N1 gaps

Re-review (~/beads-bazel-plan/f7c/rereview.md) found one blocker and five
should-fix gaps against ci/f7c-advisory-workflows:

X1 (blocker): migration-test.yml's historical-dolt-* cache was still a bare
actions/cache that auto-saves on any key miss on Blacksmith for same-repo
PRs. tar -P extraction plus lib/binary.sh's cp -f following a planted
symlink can redirect a write before sha256 verification ever runs, so the
"checksum-verified" exemption was unsound. Converted to the same
restore-always/save-off-PR split as smoke-binaries-*/regression-baseline-*,
dropped the exemption comments, and added it to advisoryBinaryCaches.

S1: TestRegressionDetectStepBehavioral executes regression.yml's detect
step's actual bash against a scratch git repo, closing mutation M13 (detect
step hard-wired to emit run_regression=false).

S2: TestBlacksmithReachableAdvisoryJobsNeverSaveACache sweeps every job in
every F7c advisory workflow (plus main.yml) whose runs-on resolves to a
Blacksmith label under a same-repo pull_request/merge_group context, and
fails on any bare actions/cache, any un-gated actions/cache/save, or any
setup-go/setup-node/setup-python step that would still cache there. Also
added a job-level `if: github.event_name == 'push' && github.ref ==
'refs/heads/main' && github.repository == 'gastownhall/beads'` guard to
main.yml's seeder job (TestBlacksmithSeederGuardedAgainstPullRequest) so a
future pull_request/merge_group trigger on main.yml can't turn the seeder
into a same-repo-PR-reachable cache writer.

S3: advisoryBinaryCaches' save-gate check is now exact-equality, not
strings.Contains, so a `... || true` gate can no longer slip past it.

S4: conformance.yml's path filters now include
internal/storage/embeddeddolt/test_fixture_test.go, which holds TestMain
for the whole package (moved out of conformance_test.go) and was previously
excluded by the blanket !**_test.go negation.

S5: main.yml's blacksmith-setup-go-cache seeder keyed its save on
github.sha, so every push to main - regardless of whether go.sum changed -
wrote a new multi-GB ~/go/pkg/mod + ~/.cache/go-build entry, risking LRU
eviction of F7a's own Blacksmith-seeded caches. Added a "Compute cache
date" step and keyed on the UTC calendar day instead, capping writes to at
most once per day; the same step/key was added to all 6 consumers so their
restore key stays byte-identical to the seeder's save key
(TestBlacksmithSetupGoCacheKeysMatchAcrossSeederAndConsumers).

N1: flipped `cache: ${{ runner.environment != 'self-hosted' }}` to
`cache: ${{ runner.environment == 'github-hosted' }}` on setup-go's own
implicit cache across all 7 call sites, so an empty/unknown
runner.environment value fails closed (caching off) instead of open.

Verification: go build/vet/test ./scripts/..., bazel --nohome_rc test
//scripts:scripts_test, actionlint -shellcheck=shellcheck
.github/workflows/*.yml, make ci-pr-policy, make bazel-sync-check all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,advisory): close merge_group gap in binary-cache save gates

Running TestBlacksmithReachableAdvisoryJobsNeverSaveACache against every
Blacksmith trust context it is reachable under (instead of stopping at the
first match found, which made the check's coverage depend on Go's
randomized map iteration order) surfaced a real gap the S2 sweep was meant
to close: smoke-binaries-*, regression-baseline-* and historical-dolt-*'s
save gates were all `if: github.event_name != 'pull_request'`, a deny-list
that also admits a hypothetical future merge_group event. All three jobs'
runs-on expressions already resolve to a Blacksmith label under
merge_group, so a merge_group run would have been able to write those
caches too - the same same-repo-writer risk X1 closed for pull_request,
just not currently reachable since none of these three workflows has an
`on: merge_group:` trigger today.

Changed all three save gates to an exact allow-list of the triggers they
actually intend to write from (push, workflow_dispatch) rather than a
deny-list of the one trigger they don't. Updated
TestAdvisoryBinaryCachesAreRestoreAlwaysSavePRGated's wantSaveIf strings to
match, and made TestBlacksmithReachableAdvisoryJobsNeverSaveACache check
every blacksmithTrustContexts entry per job (sorted for determinism)
instead of breaking on the first Blacksmith-reachable context, so this
class of context-dependent gate gap can't hide behind map iteration order
again.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(ci,advisory): directly assert main.yml has no PR-reachable trigger

TestBlacksmithSeederGuardedAgainstPullRequest's job-level guard defuses the
vulnerability structurally, but a mutation that only edits main.yml's on:
block (adding pull_request:) without touching the job-level if: was not
caught by any existing test (confirmed: mutate.py's N6 probe survived).

Add TestMainWorkflowHasNoSameRepoPRReachableTrigger, the direct half of
F7c review fix S2's "pin the seeder's trust boundary" ask: assert main.yml's
on: has none of pull_request/pull_request_target/merge_group/workflow_run,
and still has push. Confirmed this kills N6 (manually applied and reverted
via a cp backup).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* build(bazel): auto-sync BUILD files

Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/37159233834
(`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy).
Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see
scripts/bazel-autofix-push.sh.

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-03 16:03:39 -07:00
Julian Knutsen
d2844318a1
fix(tests): prlintmake runs go.mod's toolchain from the Bazel Go SDK (#7138)
Under Bazel, prlintmake_test resolved go.mod's toolchain (go1.26.7)
through whatever go the executor has on PATH. When that is another
release, gofmt-bin.sh's GOTOOLCHAIN=go1.26.7 fallback downloads the
toolchain from proxy.golang.org on every run, and fails where actions
have no network: rbe-west's fork pool runs every action in its own
network namespace with loopback only (NETNS=1). Three tests failed
there (TestGofmtBinMatchesGoModToolchain, TestGofmtBinIgnoresPathGofmt,
TestFmtCheckReportsGofmtVersion) and passed with NETNS=0.

The registered SDK is go.mod's toolchain (bazel_policy_test pins them),
so declare it as data and have testGo put it first on PATH. gofmt-bin.sh
then takes its "host go already is the pinned one" path and fetches
nothing. Plain `go test` is unchanged.

Co-authored-by: gascity-agent <agent@gascity.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 13:33:54 -07:00
Julian Knutsen
fac9a1e32c
ci: same-repo Linux PR jobs on Blacksmith; fold tiny checks and cross-compile legs (F7a) (#7139)
* ci: F7a same-repo Blacksmith moves and job folds (pr.yml, pr-risk.yml)

Move cache-free same-repo pr.yml/pr-risk.yml jobs onto the org's Blacksmith
runner pool (forks/Dependabot stay ubuntu-latest), and fold several
independent single-purpose jobs into fewer multi-step jobs, to cut same-repo
PR queue time without changing either required gate's name or its token
contract:

- fast-checks: folds check-build-tags, check-version-consistency,
  check-migration-hygiene, check-no-beads-changes and fmt-check into one
  Blacksmith job with five independent steps; ci-gate reads each step's
  outcome via job outputs into the same CI_GATE_REQUIRED token the standalone
  job used, plus a FAST_CHECKS backstop token for a lost job.
- check-release-target-cross-compilation: 8 per-target matrix legs collapse
  to 2 (unix/desktop), driven by the new scripts/ci/release-targets.txt
  manifest (kept in sync with .goreleaser.yml) and
  scripts/ci/check-release-cross-compile.sh.
- advisory-reports: folds build-examples and complexity-report (both
  already advisory) into one job, keeping their pre-fold
  step-level continue-on-error/timeout design.
- windows-make-shell: 3-way host matrix folds into one job with an
  aggregator step reproducing the old native/MSYS2/Cygwin pass/fail rules;
  test-windows-doltversion and test-windows-dbproxy-server fold into
  test-windows-small (neither was gated).
- Runner moves: fast-checks, advisory-reports,
  check-release-target-cross-compilation, check-doc-flags,
  pr-policy-wrapper, test-dolt-server-fingerprint (pr.yml) and test-nix
  (pr-risk.yml) move to the same-repo Blacksmith expression at their
  measured vCPU size.

Policy tests: extend TestSameRepoBlacksmithRunners for the moved/added jobs,
add TestSameRepoBlacksmithExpressionSemantics, TestBlacksmithJobsReadNoSecrets
and TestPRCIGateFastChecksTokens, retarget the complexity/build-examples and
dolt-server-fingerprint tests, rewrite
TestReleaseTargetCrossCompilationMatrixMatchesGoreleaser against the new
manifest, and teach the ci-gate simulation helper to resolve
needs.fast-checks.outputs.*. Document the moves/folds and the Blacksmith-off
revert set in engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): windows-make-shell/test-windows-small fail natively, raise advisory-reports headroom (review SF-1, SF-2, N-2, N-3)

- windows-make-shell: drop continue-on-error from install/exercise steps
  (kept only on the Cygwin setup step), delete the hand-written "Fail if a
  required Windows Make shell leg failed" aggregator, and delete the unused
  native/msys2/cygwin job outputs. A failing leg now reds the job via native
  GitHub Actions propagation instead of relying on an aggregator step that a
  mutation showed could silently undershoot (SF-1, N-3).
- test-windows-small: same fix - drop continue-on-error from doltversion/
  dbproxy-server steps and delete their aggregator step.
- Add timeout-minutes: 10 to the MSYS2 and Cygwin "Set up ... GNU Make"
  steps, so a hung installer download can no longer take out the whole job
  (SF-2).
- advisory-reports: raise timeout-minutes from 45 to 60 for more headroom
  above the sum of its step timeouts (N-2).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(ci): real GH-expression evaluator shared with F7c, behavioral xcompile test, secret/run-command coverage (review SF-3, SF-5, SF-6, N-1, N-4, N-6)

- New scripts/ci_blacksmith_runner_test.go: a real GitHub Actions expression
  evaluator (tokenizer + recursive-descent parser for ==, !=, &&, ||, !,
  parens, string literals, dotted lookups, with GitHub's short-circuit-
  returns-operand && / || semantics) that evaluates the REAL pinned
  sameRepoBlacksmith{2,4,8}vcpu expression strings from pr.yml/pr-risk.yml,
  replacing the old TestSameRepoBlacksmithExpressionSemantics' tautological
  hand-written Go mirror (SF-3). This file is the shared helper F7c's own
  TestSameRepoBlacksmithExpressionSemantics should rebase onto - see its
  doc comment for the "Public surface" (the three consts, evalGHExpr, and
  mustEvalGHRunsOn) (SF-5).
- ci_workflow_test.go: drop the old consts/test now living in the shared
  file; extend TestBlacksmithJobsReadNoSecrets to also scan job.If and
  step.If for secret references, not just env/with/run (N-4; container/
  services credentials and matrix-marker resolution stay deferred to F7b);
  pin fast-checks' build-tags step invoking all 3 scripts and the
  migration-hygiene BASE_SHA env, and add
  TestPRWindowsSmallDbproxyServerIsCGODisabled (N-6); add
  TestPRWindowsMakeShellLegsFailTheJob and
  TestPRWindowsMakeShellSetupStepsHaveTimeouts pinning the SF-1/SF-2 fixes.
- New scripts/check_release_cross_compile_test.go: behavioral test for
  check-release-cross-compile.sh using a fake `go` on PATH - asserts a
  failing target propagates as non-zero, every target in the group is still
  attempted, every invocation passes CGO_ENABLED=0 and the gms_pure_go tag,
  and an unknown group fails loudly with zero invocations (SF-6).
- check-release-cross-compile.sh: document the deliberate sequential-build
  (vs. the spec's 2-wide suggestion) choice and its revisit trigger (N-1).
- scripts/BUILD.bazel: add the two new test files to scripts_test's explicit
  srcs list (not a glob); verified against `make bazel-sync-check`.

All new/changed tests mutation-tested: each production-code mutation was
caught by the relevant test, then reverted via `cp` from a /tmp backup and
diff-verified byte-identical (never git stash / git checkout --).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(ci): fix stale check names and aggregator description in CI_REQUIRED_CHECK_TOPOLOGY.md (review N-7)

- Replace the stale "Check build-tag policy", "Check version consistency",
  "Check for .beads changes" and "Check formatting" entries under "Do not
  require these existing check names" with the actual post-fold name,
  "Fast checks (build tags, versions, migrations, beads diff, fmt)", and add
  the cross-compilation and windows-make-shell fold's actual check names.
- Rewrite the windows-make-shell fold description: it no longer has a
  hand-written aggregator step (deleted per SF-1 in the prior commit), and
  "the separate cygwin job" is now described correctly as a pre-fold matrix
  leg, not a job.
- Update the advisory-reports timeout mention from 45 to 60 minutes (N-2).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* docs(ci): drop local design-doc path from topology doc

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 13:13:33 -07:00
Julian Knutsen
f0f8879ac4
ci(bazel): fork PRs execute on rbe-west's fork pool (fork-ro/fork-rw), cache fallback (#7136)
* ci(bazel): fork and Dependabot PRs execute remotely on rbe-fork when rbe-west's mint is open (F9)

Fork and Dependabot pull_request runs get no secrets, no OIDC token and no
repository variables, so rbe-west decides for them. bazel.yml's rbe job asks
rbe-fork-mint's /v1/status (rbe-mint.ops.gascity.com:8444) once per run:

- open, tier ro: mode fork-ro (instance oss-fork, the fork pool, nothing
  cached for anyone);
- open, tier rw (allowlisted author's own fork, allowlisted trigger): mode
  fork-rw (instance oss, the OSS pool);
- closed, refusing, unreachable or anything else: mode cache, exactly as
  today (local execution plus the anonymous read-only cache).

In the fork modes every lane runs, remotely, from a GitHub-hosted runner:
setup-bazel makes an EC P-256 key and a CSR, uploads the CSR as an artifact
of this run (the mint's proof), gets a ~2 h certificate
(fork-credential.sh, a byte copy of gascity's tools/rbe/fork-credential.sh
from F6) and writes an rc for grpcs://rbe-fork.ops.gascity.com:8444 with
the mint's instance. A lane whose certificate names another tier than the
rbe job fails ("re-run all jobs"). Fork lanes add --config=sole-run like
mode remote.

BAZEL_COVERS_FORKS ships "false" in pr.yml and pr-risk.yml: forks and
Dependabot keep their legacy tiers, and the Bazel lanes run beside them
(remotely while rbe-fork is open). F13 commits "true"; then a covered fork
run in mode cache is red, like a covered same-repo PR without the secret.

Safe while rbe-fork is dark: with the gate closed the mint's port resets
connections and the status probe falls back to mode cache in seconds; with
the mint up but disabled, canaried or rw off it answers open false or 403,
also mode cache.

Not in this change (later slices): bazel-farm.yml and its inputs stay (F12),
bazel.yml keeps its push trigger and pr.yml/pr-risk.yml gain none (F14).
keep their go build fallback on GitHub-hosted runners.

Tests: the rbe decision against a stubbed mint (ro, rw, closed, rw closed,
canary, busy, garbage, unknown tier, unreachable); the coverage and gate
matrices learn the fork modes, BAZEL_COVERS_FORKS and mint answers that
change between runs; TestBazelCacheModeReachesTheRC drives setup-bazel's
fork steps against a stand-in mint and checks each lane's rc;
fork-credential.sh's client contract and write-bazelrc.sh's fork mode and
exclusivity; the endpoint pin is allowed as exactly its one line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): fork-credential.sh retries the mint's 429 and never sleeps after the last try

A 429 (per-minute rate, live-certificate quota) may pass on its own like a
502, so it is retried with the same 10/20/30 s backoff (one minute, the
rate window); the loop no longer sleeps after the fourth failed attempt.
The cert request and the rbe job's /v1/status probe give up connecting
after 5 s, so a closed gate (dropped SYNs) costs seconds, not minutes.
Byte-identical to gascity's tools/rbe/fork-credential.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: gascity-agent <agent@gascity.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 12:10:14 -07:00
Julian Knutsen
c1fc671fcf
ci: cross-build Windows test binaries on Linux (advisory side-by-side), drop PR macOS full suite (F4) (#7135)
* ci(F4): advisory Linux-cross-built Windows test binaries, macOS cache restore

Side-by-side rollout slice of the PR CI -> 5 min plan (F4):

- Add pr.yml `windows-test-binaries`: a Linux job that cross-compiles
  bd.exe and two go-test-c binaries (cgo and nocgo) for windows/amd64
  via mingw (CC/CXX=x86_64-w64-mingw32-*, matching release.yml's
  goreleaser setup), driven by a committed manifest
  (scripts/ci/windows-test-binaries.txt) and build script
  (scripts/ci/build-windows-test-binaries.sh). Artifacts are uploaded
  and consumed by two new advisory "-prebuilt" twins
  (test-windows-liveness-prebuilt, worktree-remove-windows-prebuilt)
  that run the identical test selection/flags/env (including
  BEADS_TEST_REPO_ROOT) against the prebuilt binaries via
  scripts/ci/run-go-test-binary.sh and scripts/test.sh's new
  BEADS_TEST_PREBUILT_TEST_BINARY short-circuit.
- The original native Windows jobs are untouched and remain required;
  the new jobs are advisory only (not in CI_GATE_REQUIRED) for this
  rollout phase. Flipping to the cross-built path later is a one-line
  CI_GATE_REQUIRED change.
- Add main.yml `windows-test-binaries-cache`: seeds the Go build cache
  for the cross-compile toolchain/packages on a schedule, mirroring
  the existing module/build cache warmers; reorder build-artifacts'
  warm-cache step to precede the module-cache save for a cleaner
  build -> warm -> save grouping.
- Restore macOS's native build-cache save step in main.yml (lost
  during an earlier job consolidation).
- Remove the non-required "Test (macos-latest)" full-suite job from
  PRs; the native macOS gate jobs still run, so required coverage is
  unaffected. This does drop a non-required pre-merge macOS race
  signal; see docs/notes for the accepted tradeoff.
- Add scripts/windows_test_binaries_manifest_test.go: policy tests
  pinning the manifest's three advisory-phase entries, pinning the
  mingw toolchain to release.yml's, checking consumer jobs only
  reference manifest-listed binaries, checking upload/download
  artifact names and `needs` match, checking the prebuilt jobs select
  exactly the same tests (-run literals) as their native counterparts,
  and checking none of the new jobs reference secrets (fork safety).
  All mutation-tested.
- Add two scripts/ci_workflow_apt_get_timeout_test.go budget entries
  for the new "Install mingw cross-compiler" steps.
- Fix two latent Bazel-only scripts_test failures surfaced while
  wiring the new test file into scripts/BUILD.bazel: materialize
  cmd/bd's sources via the existing //cmd/bd:go_srcs filegroup so
  sourceRepoRoot(t)-based package-directory checks work under Bazel's
  runfiles root, and strip Bazel's own TEST_TIMEOUT env var before
  exercising scripts/test.sh's hardcoded default in
  test_script_test.go (name collision with test.sh's own override
  knob).

Verified: go test ./scripts/...; bazel --nohome_rc test
//scripts:scripts_test; actionlint; shellcheck; make ci-pr-policy;
make bazel-sync-check; a from-scratch /tmp clone cross-build using
the exact job commands (cold 130.2s, warm/cache-hit 30.3s, PE32+
x86-64 validity via file/objdump, byte-identical SHA-256 across
cold/warm/traced runs); wine -test.list confirms
TestPrimeBinaryPortfolio/TestIsServerProbablyRunning* present in the
cgo binary and the full TestWorktreeRemove* family (including
Windows-specific variants) present in the nocgo binary.

Scoping note: only 2 of the spec's 4 consumer jobs (liveness,
worktree-remove) get prebuilt twins, and only 3 of 8 spec-listed
manifest binaries are built this slice; preflight, doc-freshness and
credential-fixture consumers are deferred as documented follow-up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): address F4 review findings (SF-1..SF-6 + nits)

Addresses the F4 review (~/beads-bazel-plan/f4/review.md): no blockers, six
substantive findings plus several nits.

- SF-1: main.yml's windows-test-binaries-cache job now seeds the Windows
  cross-compile cache from both runner selections pr.yml's consumer job can
  land on (Blacksmith for same-repo PRs/merge_group, ubuntu-latest for
  forks/Dependabot) via a 2-leg matrix with identical keys/paths, instead of
  a single ubuntu-latest leg Blacksmith runners can never see.
- SF-2: removed the two consumer-less "Warm non-race Go build cache
  (test-only packages)" steps from main.yml (build-artifacts, test macOS
  leg); they had no real consumer and cost ~270s on main's macOS leg for
  nothing.
- SF-3: scripts/ci/run-go-test-binary.sh now injects the same
  -test.timeout=10m default (when -timeout is omitted) and unconditional
  -test.paniconexit0 flag `go test` itself injects; added
  timeout-minutes to test-windows-liveness-prebuilt; added a direct
  regression test (TestRunGoTestBinaryDefaultsTimeoutWhenOmitted) for the
  omitted-timeout branch, which no path through scripts/test.sh can reach
  (test.sh always supplies its own -timeout default first).
- SF-4: scripts/test.sh's BEADS_TEST_PREBUILT_TEST_BINARY branch now runs
  run-go-test-binary.sh as a child process (capturing and re-exiting its
  status) instead of `exec`-ing it, so the EXIT trap (beads_test_env_cleanup)
  still runs and the per-run BEADS_TEST_ENV_ROOT temp dir (and, with
  BEADS_TEST_SHARED_SERVER=1, the shared dolt sql-server) is cleaned up
  instead of leaked/orphaned.
- SF-5: closed the five surviving mutations (cgo/nocgo swap between the
  Windows prebuilt twins, dropped -count=1, an unguarded twin in
  ci-gate.needs, a goreleaser compiler change, a PKGDIR change) with
  dedicated policy tests, and replaced the three-edits-per-job required/
  advisory toggle with a single committed WINDOWS_PREBUILT_REQUIRED flag
  (pr.yml top-level env) in the style of the existing BAZEL_RETIRES_LEGACY_*
  flags: both Windows pairs are now unconditionally in ci-gate.needs and
  always evaluated, and only which pair lands in CI_GATE_REQUIRED vs.
  CI_GATE_SKIPPED_OK is decided by the flag, in one place.
- SF-6: dropped the //cmd/bd:go_srcs data dep from scripts_test; the
  prebuilt-binary contract test now exercises ./scripts/ci (already a data
  dep of this target) instead of ./cmd/bd, since it never builds or runs the
  real package.
- Nits: removed dangling ~/beads-bazel-plan references and a nonexistent
  test name from committed comments; log the resolved mingw-w64 version
  when cross-compiling; removed a no-op shellcheck disable comment;
  documented why run-go-test-binary.sh's BEADS_TEST_REPO_ROOT re-export
  doesn't race beads_test_env_enter's unset; rewrote the pr.yml tombstone
  comment for the removed macOS PR job to state the actual, narrower
  tradeoff instead of an overclaiming "main.yml covers this" statement.

Verified: bazel --nohome_rc test //scripts:scripts_test passes; actionlint
clean on pr.yml/main.yml; shellcheck clean on all edited scripts (only
pre-existing, unrelated warnings remain elsewhere); make bazel-sync-check
and make ci-pr-policy both pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(scripts): isolate prebuilt-binary contract test from ambient hermetic env

TestTestScriptPrebuiltTestBinaryContract's "cleans up BEADS_TEST_ENV_ROOT
after the binary exits" subtest (added for F4 review finding SF-4) passed
locally but failed in CI's scripts-go-checks/scripts-test leg.

scripts/ci/scripts-go-test.sh (the CI driver for this package's `go test
./scripts/...` run) calls beads_test_env_enter in its own shell before
invoking go test, so BEADS_TEST_ENV_ACTIVE=1 and an outer
BEADS_TEST_ENV_ROOT are already present in this test binary's own
os.Environ(). runTestScriptPrebuiltBinary built the scripts/test.sh
subprocess's env from os.Environ() (minus TEST_TIMEOUT), so that ambient
state leaked through. Inside the subprocess, scripts/test.sh's own
beads_test_env_enter call (test.sh:17) saw the already-active guard and
no-opped: no new root, no new EXIT trap armed in that process - so the
fake binary observed the outer, still-alive root, and the subtest's
"directory is gone after test.sh exits" assertion failed, even though the
SF-4 fix (run the prebuilt binary as a child instead of exec'ing it) is
correct.

Every real production caller of scripts/test.sh's prebuilt path (pr.yml's
Windows liveness/worktree-remove prebuilt steps) runs it as the first
command of a fresh GitHub Actions `run:` step, never nested under an
already-entered hermetic env, so this nesting is purely an artifact of
this test binary's own ambient environment under scripts-go-test.sh, not
a real CI behavior gap.

Strip BEADS_TEST_ENV_ACTIVE/_ROOT/_DISABLE/_KEEP (alongside the existing
TEST_TIMEOUT filter) from the child env in runTestScriptPrebuiltBinary so
the subtest deterministically exercises scripts/test.sh's real top-level,
non-nested invocation regardless of what ambient hermetic-env state this
test binary itself happens to be running under.

Verified: reproduced the CI failure locally by exporting
BEADS_TEST_ENV_ACTIVE=1/BEADS_TEST_ENV_ROOT before `go test ./scripts/...`;
confirmed the fix resolves it; re-ran bazel test //scripts:scripts_test
and scripts/ci/scripts-go-test.sh directly, both pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 11:49:56 -07:00
Julian Knutsen
bc19f6e478
ci: parallel PR Lint matrix with pinned golangci-lint, fold duplicate Lint, split scripts-go-checks (F5) (#7127)
* ci(lint,scripts): fold duplicate Lint job; matrix PR Lint and scripts-go-checks (F5)

Lint (golangci-lint-action, only-new-issues) was functionally identical to
pr-lint-wrapper's native leg (H7): same pinned binary, config, flags, and
diff scope. Fold it into the wrapper in both pr.yml and main.yml, and drop
LINT from pr.yml's ci-gate.

pr-lint-wrapper becomes a 3-leg matrix (native/windows/darwin, fail-fast:
false) under the same PR_LINT_WRAPPER gate id, installing golangci-lint
v2.10.1 from a pinned, sha256-verified release binary
(scripts/ci/install-golangci-lint.sh) instead of go install/the Action, with
a per-leg golangci-lint cache (pr.yml restores only; main.yml's matrix is
the sole saver). scripts-go-checks becomes a 3-leg matrix
(scripts-test/vet/allowlisted, fail-fast: false) under the same
SCRIPTS_GO_CHECKS gate id, giving go vet its own build cache instead of
sharing the race build cache. main.yml gains a go-vet-cache job to warm that
cache on push; build-artifacts remains the Go module cache's sole saver.

Splitting each job's previously-sequential passes into parallel matrix legs,
combined with removing the duplicate Lint job outright, takes wall-clock
time off the PR gate's critical path.

Updates scripts/ci_workflow_test.go and scripts/pr_lanes_bazel_coverage_test.go
for the new matrices, cache topology, and the removed LINT id; adds
TestGolangciLintInstallScriptPinned. Updates engdocs/LINTING.md and the live
prose in engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md (its disclaimed "historical
snapshot" gate-wiring block is left as-is).

Verified: go test ./scripts/..., bazel --nohome_rc test //scripts:scripts_test
(after adding engdocs/LINTING.md and .githooks/pre-commit as scripts_test
data), actionlint, shellcheck, make ci-pr-policy, make bazel-sync-check.
Confirmed lint-leg parity against current main with the pinned binary (0
issues PR-scoped; the same two pre-existing unparam findings full-tree,
unchanged) and scripts-go-checks leg parity standalone. Mutation-tested the
new/updated policy assertions with six deliberate breaks (dropped matrix
leg, reintroduced LINT, corrupted the pinned sha256, removed a cache-save
step, dropped the vet leg, reintroduced the lint job); each was caught with
an on-target failure.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(ci): review-fix pass on F5's lint/scripts-go-checks matrix (SF-1..SF-5, N-1..N-8)

Addresses all SHOULD-FIX and NIT findings from the review of 7e858e2714
(no BLOCKERs):

- SF-1: lint/vet cache saves now run on !cancelled() instead of bare
  cache-miss, so a red lint/vet run (main is currently lint-red on two
  pre-existing, unrelated unparam findings) still warms the cache for
  PR legs.
- SF-2: ci-measurements.yml now exports golangci-lint's install dir onto
  PATH in the same step, since $GITHUB_PATH only takes effect in later
  steps.
- SF-3: adds a behavioral test (fake curl/tar) proving the install
  script's sha256 verification fails closed.
- SF-4: drops the commit sha from the golangci-lint cache key (saves
  only on config/dependency change, not every push) and trims the
  windows/darwin legs to cache only the small golangci-lint result
  cache, not their large GOCACHE.
- SF-5: removes the vet cache's dead non-race-build-cache fallback
  restore key (different path list, can never hit) in both pr.yml and
  main.yml.
- N-1..N-8: doc/comment accuracy fixes, restores the "moved code reads
  as new" caveat, adds a module-cache restore step and timeout-minutes
  to pr-lint-wrapper/go-vet-cache, pins fetch-depth/GOCACHE via new
  policy-test assertions, and flags a superseded doc snapshot.

New commit on top of 7e858e2714; existing commit trailers are left
untouched per explicit instruction. Not pushed, no PR opened.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* build(bazel): auto-sync BUILD files

Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/37105481015
(`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy).
Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see
scripts/bazel-autofix-push.sh.

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-03 09:36:20 -07:00
Julian Knutsen
6e9a15507e
ci(bazel): proxied-server tier 15 → 30 duration-balanced shards; un-hide 12 proxied tests (F2) (#7129)
* ci(bazel): raise proxied-server tier from 15 to 30 shards (F2)

bazel.yml's bazel-proxied lane (//cmd/bd:bd_proxied_test, run only with
remote execution) was sharing PR Risk's legacy 15-shard, frozen
bd-init-cost-proxy manifest block, so its longest shard was the long
pole of the Bazel-only lanes. Give it its own 30-shard block in
.github/scripts/proxied-cmd-test-shards.txt, bin-packed
longest-processing-time-first by measured wall-time
(scripts/ci/proxied_test_durations.json, captured from two bazel-proxied
job failures' testlogs) instead of the bd-init count proxy, via
scripts/ci/gen_proxied_shard_manifest.py's new --weights=duration mode.
Coverage is proven equal to the full discovered test set by two
independent methods (sorted-name diff and an actual 30-shard invocation
of proxied-test-shard.sh in list-only mode), both exact matches with no
duplicates.

cmd/bd:bd_proxied_test's shard_count moves to 30; PR Risk's and
main.yml's own test-proxied-cmd jobs keep their 15-shard matrix and
frozen manifest block untouched, per D2's retirement design (the Bazel
lane only needs to run the same tests, not the same split). Updated the
two coupled policy tests (scripts/ci_workflow_test.go's
TestBazelDoltServerTiersMirrorPRRisk, scripts/pr_risk_bazel_coverage_test.go's
TestBazelRetiredLanesCheckListedTestsRan) to allow the Bazel target's
shard_count to differ from its legacy job's own matrix size, and
engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md / engdocs/TESTING.md to document
the split.

Verified: go test ./scripts/...; bazel --nohome_rc test
//scripts:scripts_test; actionlint on bazel.yml; make bazel-sync-check;
make ci-pr-policy; 3 representative shards (1, 15, 30 of 30) run
directly against Bazel-built race bd_test/bd_for_tests binaries and the
pinned dolt 2.2.0 release, all green, 55-81s wall time each (the
duration data's absolute seconds are from a contention-era sample and
run far faster here; the LPT balance itself, not the absolute SLA, is
the property that matters).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): close fork-PR gap in proxied 30-shard manifest validation (review B1/S1)

TestShardScriptsListOnlyRealTests only ran proxied-test-shard.sh at PR
Risk's legacy 15-shard matrix, never at the Bazel-only bazel-proxied
lane's 30-shard manifest block. bazel-proxied only runs in mode=remote
(skipped on fork PRs), so a fork PR could rename, duplicate, or corrupt
a TestProxiedServer*/TestServerMode* manifest entry in the 30-block and
merge green via go test ./scripts/... (which runs on every PR), only
breaking the required lane afterward on main and every same-repo PR.

Fix: the test now computes every shard total it must validate per
script (this job's own PR Risk matrix size, every distinct total found
in that script's own committed manifest, and - for proxied-test-shard.sh
- cmd/bd:bd_proxied_test's live shard_count from BUILD.bazel) and runs
the full per-shard validation loop at each one, relying on the script's
own duplicate/stale-name guards. It also sums each total's "manifest: N"
counts across all shards and fails if they sum to zero, catching a
silently-deleted block that would otherwise fall back to 100% hashing
undetected.

Also fold in S1: replace the independently hard-coded `30` overrides in
ci_workflow_test.go and pr_risk_bazel_coverage_test.go with
bazelProxiedShardCount(t), a new shared helper that reads
cmd/bd:bd_proxied_test's shard_count live from cmd/bd/BUILD.bazel. This
turns the check_shard_coverage.py argument comparison in
pr_risk_bazel_coverage_test.go into a genuine cross-file pin (BUILD.bazel
vs bazel.yml) instead of two literals that could drift together
unnoticed. ci_workflow_test.go's tiers loop no longer asserts
BUILD.bazel's shard_count against a value read from the same file
(tautological); it instead asserts pr-risk.yml's and main.yml's legacy
test-proxied-cmd matrices equal each other, restoring the pin S1 found
removed on this branch relative to base.

Verified against the reviewer's mutations (duplicate entry, bogus
entry, whole-block deletion, stale rename, matrix divergence, BUILD
shard_count drift, bazel.yml arg drift): each now fails go test
./scripts/..., where it previously passed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): add generator --write/--check and freeze legacy manifest block (review S3/N1/N4/N5)

gen_proxied_shard_manifest.py:
- Add --write (rewrite only the target total_shards block in place,
  leaving every other block's header/body text untouched) and --check
  (regenerate in memory and diff against the committed block, exiting
  1 with the exact --write command needed if stale). Both operate via
  new split_blocks()/write_block()/check_block() helpers that round-trip
  the real manifest file losslessly.
- N1: fix the duration-weights header's false claim about the
  fallback ratio.
- N5: warn on stderr when proxied_test_durations.json has duration
  captures for tests that no longer exist in cmd/bd/*_test.go (renamed,
  split, or deleted since the last capture).

proxied-cmd-test-shards.txt:
- N4: mark the legacy 15-shard block's header as frozen for
  pr-risk.yml's/main.yml's legacy fork/push jobs: it is deliberately
  not kept in sync with newly-added tests (those hash-fallback
  instead), so regenerating it is not routine maintenance. Only the
  header text changed; all 149 explicit assignment lines and the
  "known blind spots" callouts are untouched (verified: a fresh
  --write would pack all 164 currently-discovered tests explicitly,
  silently changing which shard runs which test for this block, which
  is exactly what "frozen" means to avoid).
- Regenerate the Bazel-only 30-shard block's header via
  `gen_proxied_shard_manifest.py 30 --weights=duration --write` to
  describe --write/--check instead of the old manual-copy-paste
  instructions. proxied_test_durations.json has not changed yet (S2
  is still in progress), so this run only touched header text; the 164
  assignment lines are unchanged.

pr_risk_bazel_coverage_test.go:
- Add TestProxiedShardManifestGeneratorNotStale: runs
  gen_proxied_shard_manifest.py 30 --weights=duration --check under
  plain `go test` (skipped under `bazel test`, whose runfiles hold
  neither the generator's sources nor cmd/bd) so a stale 30-block
  (e.g. committed by hand, or left behind after adding tests without
  regenerating) fails scripts-go-checks on every PR, including forks.

tools/bazel/equivalence_allowlist.txt: allowlist the new test's guarded
Bazel skip.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): rename 14 undiscovered proxied-gated tests; N2/N3/N6 doc fixes

cmd/bd/*_test.go: 14 tests call requireProxiedServerEnv/
requireSharedProxiedServer but their names did not match the shard
scripts' discovery regex ^Test(ProxiedServer|ServerMode), so no CI lane
ever set BEADS_TEST_PROXIED_SERVER=1 for them and they silently skipped
everywhere (pre-existing gap, not caused by F2; list was in
~/beads-bazel-plan/f2/review/undiscovered_gated.txt).

Fixed by renaming rather than widening discovery: 162 of the 164
already-discovered tests call one of the two gate functions in the first
few lines of their body, so name and gate already correlate almost
perfectly, but the 14 tests' names are each styled differently with no
single regex term that would cover all of them without a hand-maintained
list. Renaming lets the existing discovery mechanism, hash-fallback
behavior, and both shard-coverage policy tests pick them up with zero
other changes. Checked each of the 14 names for cross-references first;
each appeared only in its own file plus at most one doc-comment in a
sibling file, both updated alongside the rename. No collisions with
existing test names.

Discovery went from 164 to 178 tests. Regenerated the Bazel-only 30-shard
block (gen_proxied_shard_manifest.py 30 --weights=duration --write) to
pack the 14 new tests; this is the first --write this round to actually
change assignments, not just header text. The frozen legacy 15-shard
block (N4) was deliberately left alone: the 14 renamed tests now
hash-fallback into it, exactly like every other test added since it was
last generated.

cmd/bd/BUILD.bazel (N6): restore the bd_proxied_test comment sentence
bazel.yml's mirror comment kept but BUILD.bazel's had dropped: PR Risk
builds its bd-cmd-test with -race, as here; main.yml's test-proxied-cmd
twin uses build-artifacts' non-race bd-cmd-test, so this lane is the
stricter of the two.

.github/workflows/bazel.yml, .bazelrc (N2/N3): remove stale timing
language ("30 shards of ~5-10 minutes" / ".bazelrc's leftover "15 shards
of ~10 minutes") never reconciled with this task's own local
representative-shard measurements (55-81s/shard); replaced with the
concurrency argument (30 simultaneous dolt sql-servers don't fit a
GitHub-hosted runner regardless of any single shard's wall time). Also
removes the "within 5s of each other" balance claim (S2): the underlying
duration weights are contaminated by -test.parallel overlap in the
top-level <testsuite time> figures they were derived from; a corrected
re-measurement is in progress (separate commit once it lands).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): re-derive proxied duration weights without overlap double-counting (review S2)

scripts/ci/proxied_test_durations.json's 178 entries are now each a
clean, single-job (no concurrent heavy bazel/dolt job on the box),
-test.parallel=1 measurement: a fresh manifest-aligned 30-shard sweep's
own <testsuite time> per top-level test where errors="0" (including
failing-but-complete suites, whose elapsed time is still trustworthy),
plus, for the handful too slow to finish inside Bazel's default
per-target timeout, a `bazel run` leaf-subtest-only sum that never adds
a parallel group's own PASS/FAIL line on top of its already-overlapping
children's lines. An earlier pass at this data summed every PASS/FAIL
line indiscriminately, double-counting parallel subtest groups.

A first attempt at re-measuring the slow stragglers overlapped in
wall-clock time with a second, unrelated heavy sweep and came out 2-3x
inflated versus the same tests measured cleanly; that contaminated data
was discarded in favor of isolated re-runs. TestProxiedServerDep2
initially looked hung during its isolated re-run (15+ minutes with no
PASS/FAIL output and no CPU growth in the top-level test binary) but
was not: each of its subtests does real `bd init --proxied-server`
work in a short-lived child process, and Go buffers a parallel
subtest group's output until the group's last member finishes. Letting
it run to completion (2357.72s) rather than guessing avoided reporting
bad data.

seconds_per_init_fallback is recomputed as the weighted average
seconds-per-init across all 178 measured tests (sum of durations / sum
of inits), for use only when a brand-new test has no entry yet.

Re-packing the 30-shard manifest with this corrected data changes the
shard-balance picture: TestProxiedServerDep2 and TestProxiedServerClose
are each large enough, alone, to dominate their own shard (~2358s and
~1105s locally) while every other shard clusters around 741-876s. The
"within 5s" claim this replaces was never earned by the underlying
data; .github/workflows/bazel.yml now documents the real, uneven
spread and flags that the two outlier shards' local numbers already
exceed the job's 25m step timeout on their own merits (unverified
against RBE's actual per-shard throughput, since this job has only
ever run remote).

Verified: go vet ./scripts/..., go test ./scripts/... (includes the
manifest/coverage policy tests), actionlint, make bazel-sync-check,
make ci-pr-policy all pass. Exact-once coverage re-confirmed: 178
manifest assignments for 178 discovered tests, zero duplicates, zero
gaps.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(cmd/bd): apply re-review candidate fixes; revert 2 unready renames (R-B1)

Re-review found the prior round's 14 renames would turn the required
Bazel + legacy 15-shard lanes red once discovered:

- TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix/external-unix and
  TestProxiedServerOutageReconnectAcceptanceMatrix/external-unix-socket
  built a dolt.sock path under t.TempDir(), which nests Bazel's hermetic
  tmp-root plus the (64-byte-truncated) test name deep enough to exceed
  sockaddr_un's 108-byte sun_path limit. Both now use a new shortSocketPath
  helper (a short os.MkdirTemp("", "bds") directory) instead.

- runProxiedDeadline raced a deadline-watcher goroutine against
  exec.Cmd.Start()/Run()'s write to cmd.Process, caught by -race. Replaced
  with cmd.Cancel, which exec.CommandContext only invokes after Start, so
  the read is ordered after Start's write.

With those two fixes applied, TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix
passes outright. TestProxiedServerOutageReconnectAcceptanceMatrix's race is
fixed, but both its subtests still fail a real "exceeded 3s bound" assertion
against a stopped external upstream -- a genuine latency issue, not a test
bug -- so it is reverted back to its original name
(TestProxiedOutageReconnectAcceptanceMatrix) with a comment explaining why,
pending a bead against the underlying latency.

TestServerModeSharedServeGlobalReportsTheServedDatabase also reverted back
to TestSharedServerModeServeGlobalReportsTheServedDatabase: `bd serve
--global` refuses to auto-apply 4 pending schema migrations (v65->v69) to a
shared/proxied server database and exits 1 with a consent-required usage
message instead of starting, per the #5920 migration-consent guard. Also a
genuine product/test-fixture question, not fixed here; reverted with a
comment, pending a bead.

The other 12 of the original 14 renames are kept; all 12 verified passing
in both lanes (see commits that follow for the manifest update and full
verification).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): re-derive proxied weights from parallel=4 CI-identical sweep (R-S1)

The prior round's duration data (committed in 6be9605d9f) was captured at
-test.parallel=1 (serial). That fixed a real double-counting bug, but
serial timings don't model this lane, which always runs -test.parallel=4
(--config=doltserver-proxied's own setting). Re-review measured near-zero
correlation (r=0.08) between the serial-derived predicted shard load and
actual parallel=4 wall time -- e.g. TestProxiedServerRecomputeBlocked was
weighted 812s serially but measured 33s actual at parallel=4.

Replaced with a full fresh capture: `bazel --nohome_rc test
--config=doltserver-proxied //cmd/bd:bd_proxied_test` (local execution
only, the exact config bazel.yml's proxied-server tier uses), 30/30 shards
passing, 0 failures, 0 races. Every one of the 176 discovered tests'
top-level <testsuite time> was captured directly from this run's
bazel-testlogs test.xml files -- no leaf-summation workaround was needed
this time, since none of the 176 hit Bazel's per-target default timeout
running alongside the rest of its shard.

Re-ran `gen_proxied_shard_manifest.py 30 --weights=duration --repack
--write` with these corrected weights: a deliberate full reweight, not an
incremental add, since every weight changed. The new packing clusters all
30 shards within a few seconds of ~355-360s each (176 tests summing to
10712s / 30), a dramatic improvement over the serial data's wrongly
implied two-outlier-shard spread (TestProxiedServerDep2 alone at ~2358s,
TestProxiedServerClose alone at ~1105s -- both artifacts of serial
timing, not real per-shard cost). Updated bazel.yml's paragraph to report
this honestly instead of the prior, false "exceeds 25-minute timeout"
claim, which was itself downstream of the serial/parallel mismatch.

Verified: the 12 tests kept renamed in the previous commit, plus
TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix's socket-path fix,
all pass under the new 30-shard Bazel manifest (this capture run) and
under the legacy 15-shard hash-fallback script (ran the 7 shards their
hash lands in: 2, 3, 4, 5, 8, 10, 11 -- all pass).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,bazel): incremental --write, --repack, name-coverage --check (R-S2)

The prior --check compared the committed 30-shard block byte-for-byte
against a fresh from-scratch LPT pack, and --write always did a full
repack. That meant any weight drift (even from proxied_test_durations.json
alone, with no test added/renamed/removed) failed --check, and two
unrelated PRs each adding one proxied test would both force a full
repack and conflict on unrelated shard lines -- re-review demonstrated
one new test moving 73 assignment lines under the old full-repack
behavior.

Redesigned:

- --check now verifies exact-once NAME coverage of the currently
  discovered TestProxiedServer*/TestServerMode* set against the
  committed block -- missing, stale, or duplicated names fail with an
  actionable message -- but does NOT compare shard assignments, so a
  weights-only change no longer forces --check to fail.
- --write is now incremental by default: every already-assigned test
  keeps its current shard, stale (no longer discovered) names are
  dropped, and only newly-discovered names are LPT-placed onto whichever
  shard is currently lightest. A full from-scratch LPT pack now requires
  the explicit --repack flag (used in the preceding commit for the
  deliberate full reweight; ordinary new-test additions should use the
  incremental default).

Mutation-tested directly (not just reviewed): verified --check correctly
flags a stale name as "stale/unknown" and a newly added test as "missing"
with the exact --write command to fix it; verified incremental --write
touches only the lines for the added/removed names; verified --repack
--write produces a materially different (full rebalance) diff from the
incremental default.

Updated TestProxiedShardManifestGeneratorNotStale's comment in
pr_risk_bazel_coverage_test.go to describe the new name-coverage-only
--check semantics (it previously described the old byte-identical-packing
behavior; the test's own assertion -- exit code only -- was already
correct and unaffected).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 09:36:05 -07:00
Julian Knutsen
ef988df84b
fix(ci,tests): un-hide TestEmbedded* tests, fence cmd/bd tests from ancestor .beads, cut #7051 race long poles, fix main lint (#7128)
* fix(lint): drop unparam findings in primeGitCmd and expandStep

primeGitCmd's error return was always nil (GH#4927 made every path
infallible but left the signature); drop it and update both call sites.

expandStep's depth parameter was always called with 0 from every caller;
recursion depth is tracked internally by expandStepWith via its own
depth+1 calls on children, so the parameter was dead. Hardcode the 0
start inside expandStep and drop the parameter.

Fixes golangci-lint unparam findings:
  cmd/bd/prime.go:306:67: primeGitCmd - result 1 (error) is always nil
  internal/formula/expand.go:197:49: expandStep - depth always receives 0

make ci-pr-lint is clean (native + windows/amd64 + darwin/arm64 non-CGO).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(dolt): give each TestBatchApplyContract subtest its own context

TestBatchApplyContract's subtests all shared one 90s context built once
in newDoltBatchApplyFixture. BoundsTheItemCount legitimately needs most
of that budget against the server-Dolt tier (ApplyBatchInTx's create
path calls ExecuteCreate once per item, ~5-8 round trips each, for all
1000 items at the MaxApplyBatchItems bound raised by #7051). Once
earlier subtests had spent part of the shared budget, BoundsTheItemCount
would blow the deadline and every subtest that ran after it in the same
parent would fail too with "context deadline exceeded", masking the
real failure behind unrelated red subtests (nightly run 37089445362,
shard 2/16: BoundsTheItemCount at 78.9s, deadline exceeded).

Give each t.Run its own fresh testContext(t) instead of sharing one,
mirroring openStoreWithOwnBudget's established rationale for the same
bug class (be-gvnsq). The store and its copy-on-write branch stay
shared across subtests, and subtests still run in sequence; only the
per-subtest deadline is decoupled. No production code changed: the
underlying per-item round-trip cost is genuine, documented, deliberate
behavior (see applyCreate's doc comment), and batching it safely would
be a larger, riskier change than this CI flake warrants.

Verified via bazel --nohome_rc test --config=doltserver-integration
//internal/storage/dolt:dolt_server_full_test --test_filter=TestBatchApplyContract
-t- --test_output=all: shard 2/16 (same shard as the original failure)
now passes, TestBatchApplyContract in 77.68s with BoundsTheItemCount at
66.08s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(embeddeddolt): skip redundant 1000-item wall-clock case under -race

TestLargeBatchApplyWallClock_Embedded's 1000-item shape was taking
597-651s under --config=embedded (-race), inflated by race-instrumenting
the in-process Dolt engine's own internal goroutine/lock machinery on
every statement issueops.ApplyBatchInTx issues. The Dolt server (TCP)
tier doesn't pay this cost: its sql-server runs in a separate,
non-raced subprocess (dolt_race_off), but the embedded backend runs
Dolt entirely inside this test binary, so -race instruments all of it.

This pushed the embedded-storage CI lane's shard 5 over its 19-minute
per-shard test timeout (panic: test timed out after 19m0s).

The 1000-item case carries no pass/fail assertion on duration or count
(the test is explicitly a logging-only measurement, not a regression
gate), so skipping it under race removes no coverage. Real 1000-item
(issueops.MaxApplyBatchItems) correctness coverage is unaffected: it
still runs and is asserted by TestBatchApplyContract/BoundsTheItemCount
in this same package, under the same race lane.

Adds the established raceEnabled build-tag pair (see
internal/workapi and cmd/bd for the sibling idiom) to this package so
the skip can be conditioned on -race.

Verified via Bazel (--config=embedded, --nohome_rc):
  TestLargeBatchApplyWallClock_Embedded: 651.23s -> 86.79s
  (1000 now SKIPped; 356 still runs and passes)
  Shard 5 of embeddeddolt_embedded_test: FAIL -> PASS

Not fixed in this commit: TestBatchApplyContract/BoundsTheItemCount's
embedded instantiation still takes ~903-945s under -race and leaves
shard 4 of embeddeddolt_embedded_test over its 19-minute budget. Unlike
the wall-clock test, that subtest carries real correctness assertions
via the shared backend/conformance.RunBatchApplyBoundsTheItemCount, and
shrinking it would require a cross-backend conformance-test design
decision (e.g. a per-fixture item-count override) that trades away real
MaxApplyBatchItems boundary coverage for one backend - judged out of
scope for an isolated test-only fix here. Flagging for a deliberate
follow-up rather than weakening the assertion to make it fast.

TestLargeBatchApplyStatementCounts_Embedded and the dolt-tier
TestLargeBatchApplyStatementCounts_Dolt were checked and left
untouched: neither includes the 1000-item shape, neither showed any
slowness or flakiness, and both pin exact statement-count regression
baselines that should not be disturbed without cause.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(cmd/bd): fence test-binary beads/git discovery against ancestor .beads

cmd/bd's test binary ambiently discovers its beads workspace the same way
production bd does: via internal/beads.FindBeadsDir's worktree-fallback and
ancestor-walk logic. When this package's checkout is itself a linked git
worktree (e.g. a sibling checkout like beads-bazel-y next to beads), the
first cached git-context call made anywhere in the test binary - before any
individual test has had a chance to chdir into its own isolated fixture -
permanently answers isWorktree=true/a real --git-common-dir for the rest of
the process. beads.FindBeadsDir then treats that as license to read (and
tests that don't isolate properly could write) the main checkout's shared
.beads database.

Confirmed via direct reproduction in a disposable scratch main+worktree pair
under /data/tmp (never /data/projects/beads): running a correctly-built `bd
--version` from the worktree checkout's cmd/bd directory resolves and warns
about the fake main checkout's .beads permissions, demonstrating the
fallback is reachable from plain ambient discovery, not just from a
contrived unit test.

internal/git.PinNoRepositoryForTesting seeds the process-wide git context
cache with a "not a git repository" sentinel as the first statement of
testMainInner, making the worktree-fallback vector inert by default. Tests
that need real git behavior for a fixture they control already call
git.ResetCaches/beads.ResetCaches after chdir'ing there (the existing
cmd/bd/git_test_helpers.go idiom), which overrides the pin for their own
scope - this is unchanged.

Separately, testMainInner now appends (never overwrites) a
BEADS_CEILING_DIRECTORIES entry for the package's own repo root, bounding
any literal ancestor-directory walk that starts at or under this checkout -
closing a second vector (an ancestor walk escaping past a nested worktree
root into a directory-with-.beads further up) that the pin alone doesn't
address. The existing pollution guard is extended to also watch the
worktree-fallback .beads directory (resolved via a direct, cache-independent
git shell-out) as a defense-in-depth detection backstop.

Verified via two full before/after `go test -tags "cgo integration" -v .`
runs of cmd/bd in a disposable linked-worktree scratch clone: identical
pass/fail sets before and after (no regressions), and zero filesystem
writes observed (inotify-monitored) in the scratch main checkout's .beads
in either run - the existing subprocess test helpers already discipline
themselves to set an explicit cmd.Dir/BEADS_DIR, so no currently-live test
was caught actively polluting the fallback directory in this repro. This
change is defense-in-depth against the mechanism itself (and against
regressions reintroducing it), rather than a fix for an empirically
isolated failing test in the current suite.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): make embedded-dolt and go:embed tests discoverable by their CI shards

embedded-test-shard.sh discovers cmd/bd's embedded-dolt tier by grepping
`^func TestEmbedded` out of cmd/bd/*_embedded_test.go — a filename-glob
match, not a package-wide one. 11 TestEmbedded* functions lived in files
that didn't match that glob (auto_import_upgrade_test.go,
dep_list_cross_db_test.go, and four files mixing TestEmbedded* with
unrelated TestProxiedServer*/TestServerMode*/TestWithFetchOneExtra_*
siblings: max_rows_test.go, store_factory_test.go,
purge_wisps_plane_integration_test.go, types_create_agreement_test.go).
They compiled and would run under a plain `go test`, but were invisible
to every CI lane that drives this tier off that discovery (the 20-shard
embedded cmd matrix), so they never ran in CI at all.

Fixed by relocating them into *_embedded_test.go files: a straight
`git mv` for the two files that were purely TestEmbedded* content, and a
function-level split for the four mixed files (moving just the
TestEmbedded* function itself; shared helpers/types stay put since
Go doesn't care which file in a package defines a symbol). The newly
discoverable tests aren't in embedded-cmd-test-shards.txt — consistent
with the ~55 pre-existing discovered-but-unlisted tests already in this
suite, they fall back to the manifest's documented hash(name)%shards
assignment. Verified via `BEADS_TEST_SHARD_LIST_ONLY=1` across all 20
shards: all 212 tests (201 + 11) are now listed exactly once, no shard
comes up empty. Also verified for real: built the cmd/bd test binary
and ran all 11 against a prebuilt bd binary (BEADS_TEST_BD_BINARY) from
the repo root, matching how CI invokes the prebuilt CMD_BINARY — all 11
pass.

Separately, 5 functions had a `TestEmbedded` prefix for an unrelated
reason — go:embed'd content (agents_test.go's AGENTS.md template tests,
schema_test.go's migration-source duplicate-version check) — and so were
being silently caught and skipped by the repo-wide `-skip '^TestEmbedded'`
pattern PR Core/main CI lanes use to exclude the Dolt-embedded tier from
./... runs. Renamed to a TestGoEmbed* prefix so they run where every
other unit test in their packages does; no other references to the old
names exist in the repo. Confirmed with `-skip '^TestEmbedded'` locally:
all 5 now run (previously would have been silently skipped).

Repo-wide grep for stray `^func TestEmbedded` outside cmd/bd,
internal/storage/embeddeddolt and internal/storage/dolt now returns only
internal/tracker's TestEmbeddedDoltExternalRefChangedAfterUsesHistoryFastPath,
which is genuinely about the Dolt-embedded backend and correctly named.

cmd/bd/BUILD.bazel regenerated via `bazel run //:gazelle` to reflect the
renames/new files; it also picked up internal/ceiling as a go_test dep,
a sync gap left over from test_repo_beads_guard_test.go's import in the
prior commit on this branch.

Verified: go build/vet (default, -tags cgo) clean; gofmt clean; go test
./scripts/...; bazel --nohome_rc test //scripts:scripts_test; bazel
--nohome_rc build //cmd/bd:bd_embedded_test; bazel --nohome_rc run
//:gazelle (cmd/bd/BUILD.bazel diff only); python3 tools/bazel/go_srcs.py
--check (make bazel-sync-check); make ci-pr-policy; make ci-pr-lint
(gofmt + golangci-lint native/windows/darwin, 0 issues). cmd/bd test
binary runs were built from cmd/bd (not the module root) and executed
with BEADS_TEST_BD_BINARY set to a binary built the same way, to avoid
buildEmbeddedBD's `go build -o X .` resolving "." against the wrong
package when invoked outside `go test`'s own working directory — not a
bug in this change, reproduced identically against an untouched,
already-correctly-named test (TestEmbeddedVersion). /data/projects/beads/.beads
checked before and after: no change beyond the pre-existing 166 untracked
.beads/formulas/*.toml files already present at session start.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(embeddeddolt): shrink BoundsTheItemCount's real apply under -race

TestBatchApplyContract/BoundsTheItemCount's embedded instantiation
measured 903-945s under -race (shard 4's single biggest long pole),
because the in-process cgo Dolt engine's own goroutine/lock machinery
runs inside the race-instrumented binary for a full 1000-item apply.

Every backend's ApplyBatch validates via the same storage.PlanApplyBatch
before opening any transaction, so the refusal half of this subtest is
already O(1) regardless of count, and the exact 1000/1001 boundary is
already pinned, for real and sub-millisecond, by
TestPlanApplyBatchAcceptsTheBoundary in internal/storage/batch_apply_test.go.
That leaves only "does this backend's own write body durably land N
accepted rows" as the expensive, backend-specific thing this subtest's
"at bound" half needs N real items for - and N does not need to be 1000
to prove that.

Split RunBatchApplyBoundsTheItemCount into a thin wrapper (unchanged,
still used by internal/storage/dolt and internal/storage/uow at the
full MaxApplyBatchItems) plus a new RunBatchApplyBoundsTheItemCountAtScale
taking an explicit atBoundItems. Embedded's subtest now applies 150 items
under -race (65.33s locally, down from 749.22s unmodified on the same
host) and the full 1000 otherwise, via the existing raceEnabled idiom.

internal/storage/dolt's non-race BoundsTheItemCount still covers the
shared inner write body at the full 1000-item count every PR. No CI lane
covers embedded's own post-commit wrapper at 1000 items anymore after
this change (the only non-race embeddeddolt build is compile-only and
never executed) - a known, reported, deliberate gap, documented in both
the updated large_batch_apply_measure_test.go comment and the task's
impl-report.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(git): make test-repository pin survive ResetCaches

PinNoRepositoryForTesting wrote its "no repository" sentinel into the
same gitCtx variable ResetCaches() unconditionally clears, so any test
fixture that calls ResetCaches() after the pin is set (runInDir's
cleanup, resetRepoCachesForTest, 25+ call sites) silently un-pins the
whole test binary for every later test. A reviewer's probe test proved
the bypass: GetRepoRoot() returned the real checkout path after a single
runInDir round-trip.

Replace it with PinNoRepositoryUnderForTesting(root), a separate,
directory-scoped pin checked live against os.Getwd() on every
getGitContext() call, before the gitCtxOnce/gitCtx one-shot cache is
touched at all. The pin survives ResetCaches() for any working directory
at-or-under the pinned root, while fixtures that chdir OUTSIDE that root
(the normal t.TempDir() pattern) still get real git detection.

Add TestPinNoRepositoryUnderForTestingSurvivesResetCaches, reproducing
the reviewer's exact repro and proving the fix: pins a real repo root,
confirms unpinned after chdir+ResetCaches outside the root, and confirms
STILL pinned after chdir+ResetCaches back inside the root (mimicking
runInDir's cleanup).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(cmd/bd): widen guard fallback coverage, document -C/doctor gap

Two review follow-ups on the test-repo .beads guard:

1. Honest scope documentation: FindBeadsDirFrom/ResolveBeadsDirForRepo's
   -C/doctor paths shell out to git directly with the ceiling env var
   stripped, so PinNoRepositoryUnderForTesting (see previous commit) does
   not fence them. Document this residual gap explicitly rather than
   implying full coverage; covering it would need those call sites to
   consult the pin too, which is out of scope here.

2. The guard's fallback-.beads watch list only watched metadata.json and
   skipped watching entirely when the main .beads directory was absent —
   exactly the layout where the worktree fallback is actually in use.
   Restructure to check the main and fallback directories independently,
   add config.yaml/routes.jsonl to the flat watch list, and add a
   recursive snapshot of dolt/ (snapshotDoltTree) so a real pollution
   write anywhere under the fallback's Dolt-backed state is caught.
   diffSnapshots is generalized to also report newly-appeared keys, since
   a recursive tree snapshot can legitimately gain entries between
   before/after in ways the old fixed-name flat list never could.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): pin MaxRows embedded tests off their shared race shard

TestEmbeddedMaxRowsNonListPaths and TestEmbeddedMaxRowsList both hashed
to embedded cmd shard 16, which measured 746s wall under -race
(605s + 214s) against a 19-20 minute budget on a 192-core host with
effectively unbounded test.parallel; CI's 4-way parallelism would be
worse. Neither test had ever run under race in CI before this branch
surfaced them to their own shards.

Both tests spawn many independent bd subprocesses per subtest
(bdInit + exec.Command), so their cost is dominated by subtest/subprocess
count rather than a single large-N parameter; buildEmbeddedBD also never
passes -race to the spawned bd binary, so there is no raceEnabled-style
"shrink N" lever analogous to BoundsTheItemCount's that would apply here
without just deleting coverage. Pin them to shards 11 and 13 instead,
the two lightest shards by current manifest+hash-fallback test count
(verified via BEADS_TEST_SHARD_LIST_ONLY=1, which lists shard
composition without executing any tests).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(embeddeddolt): correct coverage comments, range-check, document build tags

- large_batch_apply_measure_test.go's doc comment above
  TestLargeBatchApplyWallClock_Embedded had three factual errors: the
  150-item sibling subtest does not always run at 150 "race or not" (it's
  1000 without -race); the 356-item shape is not "non-race only" (it runs
  under race too, as the same comment's next line said); and the
  server-Dolt full suite does not run on "every PR" (full_embedded in
  .github/scripts/ci-embedded-tier.sh is unconditional on merge_group/push
  but conditional on PRs, per pr-risk.yml's gating). Rewrote the comment to
  be self-contained (no out-of-repo citation) and accurate, and reworded
  the related t.Skip message. Fixed the same "every PR" overstatement in
  batch_apply_contract_test.go's BoundsTheItemCount race-branch comment.
- Added a range-check to
  conformance.RunBatchApplyBoundsTheItemCountAtScale: atBoundItems must be
  1..publicops.MaxApplyBatchItems, since that's the production bound this
  contract pins.
- Documented why race_detector_on_test.go/race_detector_off_test.go use
  `cgo && race`/`cgo && !race` instead of the plain `race`/`!race` pair
  used by internal/workapi and cmd/bd: every consumer of raceEnabled here
  is itself cgo-gated, so the matching cgo tag keeps this file out of
  non-cgo builds too, where it would otherwise compile with no consumer
  and trip an unused-lint finding.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(dolt): give BoundsTheItemCount 3x timeout budget

The server-Dolt tier's BoundsTheItemCount subtest measured 66s against
the package's shared 90s testTimeout — a thin margin on a loaded CI
runner. Give it a dedicated 3x (270s) context.WithTimeout instead of
raising the shared testTimeout constant, since that constant also scales
every other subtest's 5*testTimeout cleanup bound package-wide; a local
override keeps the blast radius to the one expensive subtest.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(ci): run embedded-Dolt batch-apply suite non-race in nightly

The nightly full-test job's main suite (go test -race ... ./...) never
sets BEADS_TEST_EMBEDDED_DOLT, so every skipUnlessEmbeddedDolt-gated test
in internal/storage/embeddeddolt — including TestBatchApplyContract and
the 1000-item case of TestLargeBatchApplyWallClock_Embedded — is skipped
on every CI lane, not just this one. Add a dedicated non-race step so a
real 1000-item apply through the embedded backend's own
version-commit-published-after-the-tx wrapper gets exercised somewhere:
non-race because the embedded engine's own goroutine/lock machinery runs
inside this test binary and the race detector dramatically inflates its
wall-clock (see race_detector_on_test.go), which would defeat the wall-
clock test's purpose and reintroduce the kind of shard-timeout pressure
the MaxRows shard-pin commit just worked around elsewhere.

Verified locally from a fresh /tmp clone (outside any .beads ancestor,
per this task's safety rules):
BEADS_TEST_EMBEDDED_DOLT=1 go test -tags gms_pure_go -run '^(TestBatchApplyContract|TestLargeBatchApplyWallClock_Embedded)$' ./internal/storage/embeddeddolt
PASS, 178.3s total (BoundsTheItemCount 81.06s, WallClock/1000 63.14s),
comfortably inside the job's 45-minute timeout.

Add TestNightlyFullTestRunsEmbeddedDoltBatchApplySuite to
scripts/ci_workflow_test.go, pinning the step's exact run command, its
BEADS_TEST_EMBEDDED_DOLT=1 env, that it runs after the main suite step,
and that it stays non-race.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 04:18:14 -07:00
Julian Knutsen
aaae7e6aff
ci: package gates start after rbe, MCP pytest -n 8, gate jobs on Blacksmith (F3) (#7126)
* ci(bazel): move package-mcp/package-npm gates into bazel.yml (F3)

Package gates now build bd via Bazel (no --config=ci) when the rbe job
enables it and fall back to go build otherwise, so forks/Dependabot/rbe-off
runs are unaffected. pr.yml's CI Gate keeps PACKAGE_MCP/PACKAGE_NPM required
and never-skippable, now sourced from needs.bazel.outputs.*; the retired
detect-package-gates/package-mcp/package-npm pr.yml jobs and
DETECT_PACKAGE_GATES are deleted. MCP pytest pins -n 8 explicitly with
pytest-xdist locked via uv lock. rbe, bazel-coverage, and both required gate
jobs move to Blacksmith for same-repo PRs (gate names unchanged).

Also closes a gap in detect-package-gates.sh (missing bazel.yml in the
force-both-gates path list, since the job definitions moved there) and wires
integrations/beads-mcp's manifest/lock into scripts_test's Bazel data so the
new pytest-xdist pin check runs under `bazel test` too.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): review fix-up for F3 package gates (SF-1, SF-2, nits)

Review of 65f378d241 (~/beads-bazel-plan/f3/review.md, no blockers) found:

- SF-1: the package gates' `bazel build //cmd/bd:bd_for_tests` resolved to
  a different output configuration (k8-fastbuild) than `bazel test
  --config=ci`'s bd_for_tests (k8-fastbuild-ST-..., via test:ci ->
  test:prcore's --@rules_go//go/config:race), so it got no action-cache
  reuse from the test job. Add the matching race flag to both jobs' build
  step; verified with cquery that both invocations now cquery to the
  identical bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path,
  and with a real build that it's an action-cache hit once bazel test
  --config=ci has run. Update the pinned Run string in
  pr_lanes_bazel_coverage_test.go and the impl report's cache-reuse claim.
- SF-2: drop stale comments/doc text saying the package gates download
  bazel-ci-build-artifacts; they build their own bd now (bazel.yml,
  pr.yml, engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md).

Nits: pin both required gates' display names in TestSameRepoBlacksmithRunners
(nothing else reads job.Name, so a rename would otherwise go uncaught); pin
the package jobs' Python/uv/Node setup against main.yml's copies; add
timeout-minutes to both gate jobs and detect-ci-tier (same-repo PRs now run
on paid Blacksmith minutes, not a free runner's 360-minute default); shut
down Bazel and remove the RBE client key right after each package job
packages its Bazel-built bd (nothing downstream needs either); add
scripts/ci/package-bazel-bd.sh to detect-package-gates.sh's both-gates
trigger arm and pin it in TestDetectPackageGatesCoversBazelWorkflow.

Every fix mutation-tested (mutate -> test fails with a specific message ->
restore from a /tmp backup, diff-confirmed identical -> test passes again).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 00:03:43 -07:00
Julian Knutsen
e2b78f7d7a
ci(bazel): run integration on fork PRs via the read-only cache; retry Go module fetches (#7120)
* ci(bazel): retry Go module fetches and scale download timeouts in every mode

A fork run failed Bazel / test on one proxy.golang.org TLS handshake
timeout. gazelle's go_repository fetches modules with the go command
(fetch_repo), which never retries, and Bazel's downloader retry flags do
not reach it. setup-bazel's generated rc now sets
--repo_env=GOPROXY=https://proxy.golang.org|https://proxy.golang.org|direct:
with "|" the go command moves to the next entry on any error, so the proxy
is tried twice and then the module's origin. fetch_repo checks every
module against its go.sum hash whatever the source. Verified with Bazel
9.2.0: an unreachable first entry still fetches all 438 go_deps repos,
and without a fallback the fetch fails.

--http_timeout_scaling=2.0 doubles the timeouts of Bazel's own
downloader (the Go SDK, http_archive), which already retries.

Both flags are key neutral (repository fetching only) and apply in every
mode, because repository fetching always runs on the runner.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): run the integration lane for fork and Dependabot PRs in cache mode

bazel-integration was remote-only, so fork and Dependabot PRs (mode
cache) had no PR-time integration run at all: main.yml's integration jobs
run only after merge. With rbe-west's anonymous read-only cache, every
action trusted CI already ran is a hit, so the GitHub-hosted runner builds
and tests only what the PR changed.

Simulated in an ubuntu:24.04 container with 4 CPUs and only the generated
rc (no user rc). This setup's action keys match CI's: the ci lane
reproduced the fork run's 6550 processes with 4847 cache hits.
- Main HEAD: 143 s (4656 hits, 12 local, 113 of 115 tests cached).
- A cmd/bd change: 170-290 s.
- Cache closed (cold, all local): about 30 minutes.

- bazel.yml: the lane's if is now mode remote or cache, never local
  (rbe=off). The step timeout goes from 45 to 75 minutes (job 85), so a
  closed cache (the farm's kill switch) still leaves fork PRs green.
- bazel-gate.sh: mode cache no longer allows BAZEL_INTEGRATION to skip.
  Mode local still does.
- New TestBazelCacheModeReachesTheRC runs, end to end, the real decision
  step for 22 starts:
  - fork, Dependabot and same-repo PRs, push, rbe=cache/off dispatches
    and bazel-farm, each with the variable on and unset;
  - same-repo runs with and without the secret;
  - a fork PR that "has" the secrets, and one passing fork-farm.
  For each start it evaluates every lane's if and setup-bazel env, runs
  write-bazelrc.sh, and checks bazel-gate.sh's skips. A cache-mode rc
  carries --config=fork-cache and the fetch hardening, and no executor,
  TLS file, instance, upload, header, BES or masked secret.
- ci_workflow_test.go: the integration if, the run modes, and a
  closed-cache floor for the step timeout.
- Comments in bazel.yml, pr.yml and bazel-farm.yml, and the topology
  engdoc. The engdoc also covers:
  - fork-cache's slow-blob behaviour: 15 s read deadlines trip the
    breaker, then "Lost inputs" and one build restart, still green;
  - why the endpoint's FindMissingBlobs denial never affects cache-only
    clients.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-02 17:15:56 -07:00
Julian Knutsen
0588af25d1
ci: retire pr.yml's legacy PR Core, build, pure-Go, domain+uow and contract jobs where the gated Bazel lanes run them (D2 step 3) (#7097)
* ci: retire pr.yml's legacy PR Core, build, pure-Go, domain+uow and contract jobs where the gated Bazel lanes run them (D2 step 3)

On same-repo, non-Dependabot pull_requests while the committed
BAZEL_RETIRES_LEGACY_PR_LANES flag is "true" (bazel-coverage's new
pr_lanes output; the job stays identical in pr.yml and pr-risk.yml),
pr.yml's build-artifacts, pr-core-wrapper, check-cmd-bd-puregeo-tests,
test-domain-uow and contract-corpus stand down. ci-gate accepts those
skips only then, and BAZEL_PR_LANES_RETIRED requires the same run's
Bazel test, pure-Go/js-wasm and dolt-server lanes to have run remotely
and passed. Revert: commit the flag "false" in both workflows.

What the retired jobs did besides their tests keeps running on every PR:
- the package gates take bazel-ci-build-artifacts (bazel-test's bd, same
  layout) where build-artifacts stands down;
- the Dolt server fingerprint moves to its own required job;
- the Go tests Bazel does not run or skips (equivalence allowlist) run
  under go test in pr-preflight-platforms' Linux leg, each required to
  pass (tools/bazel/run_allowlisted_go_tests.py).

In mode remote the three lanes' bazel test invocations add
--config=sole-run (no cached test results, no whole-invocation retry on
remote cache eviction), as steps 1 and 2 do for their lanes.

Policy tests: decision matrix and named cases extended to the third
flag, gate simulation for the pr.yml tier, artifact consumers, pinned
lane steps and rc lines, a repo-wide narrowing scan of BUILD rules and
.bzl files, and the allowlisted-test runner.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(ci): record the retired pr.yml PR Core, build, pure-Go, domain+uow and contract jobs in the check topology

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(ci): only pr.yml's workflow env sets a BAZEL_RETIRES_* flag, and only the decision step reads one

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(pr): keep go test ./scripts/..., go test's vet and the Bazel-skipped tests on every PR (D2 step 3 review F1, F1b, F3, L2)

Review of D2 step 3: retiring PR Core removed the only required pre-merge
`go test` of the scripts packages, whose policy tests (the D2 guards
among them) check part or all of their rules under go test only, and
go test's implicit vet for every package (rules_go's go_test runs none).

New required job scripts-go-checks (SCRIPTS_GO_CHECKS), on every PR, in
PR Core's environment (dolt, git and dolt identity, test-env):
- scripts/ci/scripts-go-test.sh: `go test ./scripts/...` with PR Core's
  flags;
- scripts/ci/go-test-vet.sh: `go vet` with cmd/go's defaultVetFlags over
  ./..., policy-tested equal to the running toolchain's list;
- scripts/ci/allowlisted-go-tests.sh, moved here from
  pr-preflight-platforms.

TestBazelOnlySkipsAreAllowlisted also requires every test that runs part
of its checks under go test only (a Bazel-guarded return/continue, or a
go-test-only block that reports failures) to live under ./scripts, the
only packages go test still runs on every PR; and step 3's
TestRunAllowlistedGoTestsScript gets its allowlist entry.

Package gates on a covered PR whose Bazel call did not run remotely (the
farm switch off) now fail in their own step naming
BAZEL_PR_LANES_RETIRED instead of on a missing artifact (L2).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(ci): record scripts-go-checks, the skip-allowlist check and the re-run-safe uploads for D2 step 3

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-02 06:08:52 -07:00