Commit graph

12 commits

Author SHA1 Message Date
Julian Knutsen
873162dafa
ci(bazel): dolt-int gates server-storage + cmd-dolt in mode remote (item 5 A2) (#7535)
* ci(bazel): dolt-int gates the server-storage and cmd-dolt tiers in mode remote (item 5 A2)

In mode remote, folds bazel-server-storage (--config=doltserver-integration)
and bazel-cmd-dolt (--config=doltserver-cmd) into one bazel-dolt-int job
(--config=dolt-int), following the dolt-race pattern from #7514. The
members have different gating categories (server-storage plain-gated,
cmd-dolt flag-gated/advisory), so the union carries continue-on-error: true
and a bespoke --label-scoped Shard balance step. All other modes keep the
two lanes running unchanged. Action keys are byte-identical between the
member configs and the union (17 + 46 = 63 TestRunner actions), so cached
results carry over either direction and a revert is a safe rollback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): dolt-int per-tier results, integration gating, key-parity wording (review)

Coordinator review of #7535 found three must-fixes: integration:"off" was
ignored in remote mode (the union's members already honor it); job-level
continue-on-error made a server-storage-only failure read success on
push/nightly/dispatch, where main stays red; and the PR's key-parity claim
overstated byte-identical actionKeys when only the remote action digest
(not the local actionKey) is unaffected by moving --test_env/--test_arg
onto the targets' own env/args.

Fixes:
- bazel-dolt-int's if now adds `&& inputs.integration != 'off'`, matching
  the members (#7491's review); bazelLaneRunModes and
  TestBazelDoltIntArmsExactlyOnePerMode cover the "off" case.
- Job-level continue-on-error removed; only the `bazel test` step carries
  it, and a new always() "Classify tier results" step
  (tools/bazel/classify_tier_results.py, with its own unit tests and a Go
  wrapper) computes server_storage_result/cmd_dolt_result from the BEP and
  decides job.status per-tier. Confirmed with bazel-gate.sh/ci-gate.sh on
  synthetic empty-output inputs that a dead job still fails the gate.
- Corrected the local-actionKey-vs-remote-digest wording in .bazelrc,
  bazel.yml's job comment and CI_REQUIRED_CHECK_TOPOLOGY.md.
- Should-fix: test step 23->25 / job 58 minutes with reasoning; --label
  coverage added to shard_budget_test.py; stale lane comment fixed in
  tests/regression/BUILD.bazel:46; TestBazelDoltIntSoleRunIsInTheUnion's
  docstring corrected (members bake sole-run's line in directly, never via
  --config=sole-run).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): fail-closed classify, per-tier check_testcases, pinned argv (review)

Final review round for PR #7535 (maintainer-city-c3, bee-ghosttrack,
julianknutsen):

- classify_tier_results.py now fails every tier shut (exit 1) on a
  non-PASSED label outside every --tier list, so a newly tagged target
  added without updating the step's --tier args cannot silently read as
  success (must-fix 1). Its --tier/--check label sets are policy-tied to
  the BUILD files' own tag sets by a new
  TestBazelDoltIntTierLabelsMatchTheTags.
- A missing or non-integer --rc (a timed-out or cancelled test step)
  takes the same fail-everything-shut branch instead of crashing argparse
  with no outputs written (bee-ghosttrack's should-fix 2).
- check_testcases.py gained a repeatable --label flag, and bazel.yml's
  shared "Every target and shard ran tests" step is split into check-ss
  and check-cd, each continue-on-error and folded into its own tier via
  classify's new --check NAME=OUTCOME, so a zero-testcase cmd-dolt shard
  no longer reddens the job on push/nightly (must-fix 2).
- classify's run: argv is now pinned by exact equality
  (bazelDoltIntClassifyRun), and every flag except --tier/--check is
  --StoreOnce; a repeated --tier/--check name or an unrecognized flag is
  now a hard parser error (must-fix 3).
- "Upload test logs" now also fires on steps.test.outcome == 'failure',
  not just failure() (must-fix 4).
- shard_budget.py --label now fails closed (::error, exit 1) on a label
  with no executed shard in the BEP at all, instead of silently reporting
  "no sharded test target executed two or more shards" (should-fix 3,
  carried across both review rounds).
- Updated engdocs/TESTING.md, CI_REQUIRED_CHECK_TOPOLOGY.md, .bazelrc and
  scripts/bazel_dolt_int_test.go/ci_workflow_test.go comments for the
  nits: stale "Advisory (continue-on-error: true)" wording, the
  bazelUnionHasFlagGatedMember comment's stale claim about bazel-dolt-int,
  "byte-identical" without specifying remote action digest vs. local
  actionKey, and "is unset" vs. ci-gate.sh's actual "was skipped" branch
  for an empty union output.

Verified: python3 -I on all four tools/bazel/*_test.py (classify 29,
check_testcases 7, shard_budget 19) green; GOTOOLCHAIN=go1.26.7 go test
./scripts/... green; bazel --nohome_rc test //scripts:scripts_test
--config=ci green.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): shard_budget --label fail-closed only on a truly unknown label

The previous commit's fail-closed --label check compared against
read_shard_durations, which only records executed (non-cached) shards, so
a target whose every shard was a cache hit this run had zero entries and
was wrongly treated the same as a --label typo. This was not caught by
the new tests (none exercised an all-cached target) and broke this PR's
own dolt-int CI run: bazel-dolt-int's Shard balance step went red on
//internal/storage/dolt:dolt_server_full_test with "no executed shard for
it", even though that target was correctly, fully cache-hit.

Fix: a new read_tested_labels() reads every label with any testResult
event (cached or executed), and --label's fail-closed check now tests
against that set instead of read_shard_durations. A fully-cached target
still legitimately reports "nothing to judge" through evaluate(); only a
label absent from the BEP altogether is now an error. Added the missing
test row (every shard cached) that would have caught this before push.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(bazel): pin bazel-dolt-int's Upload test logs if-condition

A full 42-mutation re-run for PR #7535's second review round found that
nothing pinned this job's own "Upload test logs" if-condition (must-fix 4):
the other lanes' upload-logs steps are each pinned, but bazel-dolt-int's
(unlike dolt-race, a brand-new job) had no equivalent assertion, so a future
accidental revert to the old failure()-only condition would go uncaught.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-13 09:17:54 +00:00
Julian Knutsen
85464f567a
ci(bazel): dolt-race gates the dolt-server and proxied tiers in mode remote (item 5 A1b) (#7514)
* ci(bazel): dolt-race gates the dolt-server and proxied tiers in mode remote (item 5 A1b)

In mode remote, bazel-dolt-race (`bazel test //... --config=dolt-race`, the
union of --config=doltserver and --config=doltserver-proxied) runs both tiers
in place of bazel-doltserver and bazel-proxied, and is no longer advisory:
its job.status is both the bazel-doltserver and the bazel-proxied
workflow_call output (union result first, member's otherwise). In the fork
modes, cache and local, the two member jobs run exactly as before: there one
client does both tiers' work without in-flight deduplication, so a union
would cost the sum of the tiers rather than the max. Exactly one arm runs in
any mode, so gate ids, pr.yml, ci-gate.sh and bazel-gate.sh are unchanged.

- bazel.yml: dolt-race if: mode == 'remote' (every event that reaches it);
  no continue-on-error; result output and always() recorder; test step
  timeout 25 -> 20 (job 55 >= step sum 46 + 5, < 60). Members' if: their old
  rule && mode != 'remote'.
- scripts/bazel_dolt_race_test.go (was bazel_dolt_race_shadow_test.go):
  union-of-members config, sole-run lines in the union, gating shape, step
  and job timeouts, exactly one arm per mode x event with the call outputs
  evaluated as GitHub does, and the union's 62 test actions within
  --config=remote-exec's --jobs=64.
- Gate simulations evaluate the workflow_call output expressions, and a
  gated lane's reporter is the lane or its union lane.
- aquery: all 62 TestRunner actions under --config=dolt-race are
  byte-identical (key, argv, env, outputs) to --config=doltserver (with and
  without sole-run) and --config=doltserver-proxied on origin/main.
- Docs: CI_REQUIRED_CHECK_TOPOLOGY.md "dolt-race Union Lane", TESTING.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): dolt-race test step 23 min so Bazel logs a 1200 s TIMEOUT first

The proxied tier caps each test at 1200 s; a 20-minute step let GitHub
kill a hung shard before Bazel wrote its TIMEOUT test.log and goroutine
dump. Step 23, job 56 (step sum + 5, under the merge queue's 60).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): say the dolt-race job timeout bounds the job, not the run

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 04:35:17 +00:00
Julian Knutsen
c1ccb728ec
ci(bazel): mode-cache readers use the remote repo contents cache via rbe-cache (#7476)
* ci(bazel): mode cache reads the remote repo contents cache through rbe-cache

Mode cache (fork and Dependabot PRs while rbe-fork is closed, rbe=cache
dispatches) now reads Bazel's remote repo contents cache: the trees rrc-seed
writes on main pushes, served by rbe-cache's anonymous GetActionResult and
ByteStream.Read (Caddy :8443 -> nativelink-anon -> AC_OSS, read only).
setup-bazel's write-bazelrc.sh adds the trusted readers' two lines to its
fork-cache rc (passed with --bazelrc, so the startup line applies):

  startup --experimental_remote_repo_contents_cache
  common --loading_phase_threads=64

The committed .bazelrc stays untouched (design R2): the flag is a startup
option and cannot live in build:fork-cache, and developer and macOS
machines never read.

Gated on the new .github/actions/setup-bazel/cache-rrc-probe.sh (a copy of
gascity's tools/rbe/cache-rrc-probe.sh): one GetCapabilities to rbe-cache
(connect 3s, total 5s). Only an answer (gRPC 0, cache_capabilities present)
writes the lines; anything else writes none and logs a ::notice:: with the
reason, and the script never fails on it. Kill switch: fork_rrc_read=on in
that script; set it to off in a one-line PR to stop mode cache reading.

Measured (scratch origin/main, no credentials, 8 CPUs, cold output base,
`build --nobuild //...`): repository rules 665-672 -> 38-41 (the rest are
bd_releases/dolt_release and local or toolchain repos, not reproducible),
analysis 26.5-26.7s -> 12.1-14.8s, wall about 36s -> 25-32s.

Tests: TestSetupBazelRCWriter runs the writer against a stand-in rbe-cache
(answers, answers with zstd, PERMISSION_DENIED, refused): exactly the
reader's lines after --config=fork-cache and any zstd line only when it
answers, a notice otherwise; remote-exec and rbe-fork never get them nor
probe. TestBazelCacheModeReachesTheRC checks no simulated mode carries them
with the probe refused. TestCacheRRCProbe runs the real probe (errors,
malformed, timeout, refused, untrusted cert, kill switch off without a
request).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): mode cache rrc review fixes: reader opt-in, containment, AC probe

Scope. write-bazelrc.sh enabled the remote repo contents cache for every
mode cache job, bazel-release-cross included, whose configs
rrc-lane-commands.txt does not seed. setup-bazel now takes BAZEL_RRC_READ
("true" or empty, anything else fails): only with it does mode cache probe
rbe-cache and add the reader's lines. The 8 trusted reader jobs pass it,
and nothing else. TestBazelRRCOptInMirrorsTheTrustedReaders pins that set
to bazelRRCReadLanes; TestBazelRRCLaneCommandsCoverEveryReader now checks
every job that enables the flag either way (reader step or opt-in) is a
reading lane whose bazel configs are all seeded. TestSetupBazelRCWriter
covers a job without the opt-in (no probe, no lines) and bad values.

Runner. The 5 readers that run in mode cache (test, pure, integration,
doltserver, cmd-dolt) fall back to ubuntu-24.04 instead of ubuntu-latest:
one runs-on line each; mode remote stays Blacksmith, and the remote-only
readers and every other job are unchanged.

Visibility. write-bazelrc.sh writes one step-summary line in mode cache:
whether the job reads, and why (the probe's reason, with rbe-cache's
GetCapabilities TCP connect time, DNS excluded).

Probe. cache-rrc-probe.sh is re-copied from gascity: it also asks
GetActionResult for a fixed key no action has and a ByteStream Read of
that digest as a blob, and passes only if both answer NOT_FOUND (gRPC 5),
proving Caddy, nativelink-anon, the action cache and the CAS answer; it
reports the TCP connect time minus the DNS lookup. serveCapabilities
answers both; TestCacheRRCProbe adds their failure answers.

Containment and limits. The rrc-verify issue body and
CI_REQUIRED_CHECK_TOPOLOGY.md no longer say to set RBE_REPO_CONTENTS_CACHE
off (mode cache never sees it, and off also stops rrc-verify): commit
fork_rrc_read=off in both probe scripts (beads has no fresh-merge, so an
open fork PR keeps its old switch until it is pushed again); for immediate
server-side containment, rbe-cache-gate close on core; and, because the
mode remote readers read the same oss action cache entries, set the
variable to seed on a mismatch (no lane reads, rrc-seed and
rrc-verify keep running), off only to stop seeding too, and leave it as
it is when the run only could not check. The doc also says a
closed or down cache only falls back at probe time (after loading, files
are fetched lazily and losing rbe-cache fails the command, bazel#30218),
and notes the 512 MiB anonymous CAS limit under gascity's R4.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): bazel-dolt-race opts in to the rrc read like the other trusted readers

#7464 added bazel-dolt-race as a trusted reader after this branch was cut, so
TestBazelRRCOptInMirrorsTheTrustedReaders now covers it: pass BAZEL_RRC_READ
(it runs only in mode remote, so this changes nothing today). It never runs in
mode cache, so it keeps the lanes' ubuntu-latest fallback: the cache-reader
runner rule (ubuntu-24.04) excludes it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 17:42:12 +00:00
Julian Knutsen
eb204363c9
ci(bazel): add advisory dolt-race shadow lane (item 5 phase 1) (#7464)
* ci(bazel): add advisory dolt-race shadow lane (item 5 phase 1)

Add --config=dolt-race, the union of --config=doltserver and
--config=doltserver-proxied with one tag filter for both tiers, and an
advisory bazel-dolt-race job that runs it beside bazel-doltserver and
bazel-proxied. The shadow measures one client per group of lanes against
the two lanes it would replace before any cutover (item 5 slice A1a).

The job runs in mode remote on pull_request and merge_group events only.
It has job-level continue-on-error, no job or workflow_call output, and no
gate names it, so it cannot change CI Gate or PR Risk Gate. Its
ci-analytics step is copied verbatim with LANE=dolt-race, and rrc seeds
the new config.

Policy tests pin the config as the exact union of its members' lines (tag
filter aside), the job's run modes and events, its advisory shape, the
analytics lane map and the rrc command list. aquery shows every member
target's TestRunner action (actionKey, argv, env, inputs, outputs) is
unchanged under the union config.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): give the dolt-race shadow timeout headroom; fix gate-wait wording

Review follow-up on the advisory bazel-dolt-race job (maintainer-city-c3,
findings 1, 2 and 5):

- Raise the job's timeout-minutes from 30 to 35. continue-on-error covers
  a failed or timed-out step, but not a job GitHub cancels at its own
  timeout, which could make the bazel call non-success and turn CI Gate
  red. TestBazelDoltRaceShadowIsAdvisory now reads both timeouts from the
  workflow and requires job >= test step + 10 (and a step timeout at all).
- Correct the docs and comments: the shadow cannot fail the gate, but
  CI Gate still waits for it, because its needs on the bazel call
  completes only when every job in it has finished.
- Correct the checks comment: check_testcases covers every target in the
  union, including the dolt-server tier its own lane never checked.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): bound every dolt-race shadow step; job timeout covers their sum

Review follow-up (S1): a hang in any bazel-dolt-race step without a
timeout would run into the job's own timeout, and a job GitHub cancels
there is not covered by continue-on-error. Every step now has a
timeout-minutes, and the job's timeout covers their sum:

- checkout 5, rrc read 1, setup-bazel 10, flaky query 5, checks 3 each,
  test-log upload 3. The test step (25) and the analytics steps (3, 2)
  keep bazel-proxied's values; bazel-proxied sets no timeout on the
  others, so these bound them.
- The step timeouts sum to 60 minutes, so the job timeout goes from 35 to
  65.
- TestBazelDoltRaceShadowIsAdvisory now requires a timeout on every step
  and job >= sum(step timeouts) + 5, read from the workflow, alongside the
  existing job >= test step + 10.

Also cite the farm maintainer's confirmation of NativeLink 1.7.1's
in-flight Execute dedupe in the job comment.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): keep dolt-race shadow timeout under the merge queue's

Tighten the shadow's non-test step budgets so the job timeout (step sum
+ 5) is 55 minutes, and pin that it stays below the 60-minute merge
queue check timeout: CI Gate waits for the shadow, so a hung shadow
must not drop a group from the queue.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 10:13:49 +00:00
Julian Knutsen
f53010a23b
ci(bazel): remote repo contents cache: main-push seed, lane readers, nightly verify (#7453)
Port gascity's remote repo contents cache wiring (gascity #7467, #7478) to
bazel.yml, gated on the repository variable RBE_REPO_CONTENTS_CACHE
(unset/off: nothing; seed: rrc-seed on push to main; canary: + bazel-test
reads; on: + every test lane reads). Mode remote only.

- rbe job: an rrc step turns the variable into the rrc output.
- Readers: the eight test lanes append startup
  --experimental_remote_repo_contents_cache and common
  --loading_phase_threads=64 to .bazelrc.local before Set up Bazel.
- rrc-seed (push to main, id-token: write): a 30-minute
  rbe-rrc-writer-beads certificate from rbe-west's mint
  (.github/scripts/rrc-writer-credential.sh), then every lane's command
  (.github/scripts/rrc-lane-commands.txt) with --nobuild, --remote_cache
  only and --remote_upload_local_results (.github/scripts/rrc-run-lanes.sh).
- rrc-verify (schedule/dispatch, issues: write): cold fetch of every lane
  and tools/bazel/rrc_verify.py against the cache; a mismatch opens an
  rrc-verify issue.
- pr.yml, nightly.yml and bazel-farm.yml grant id-token: write and
  issues: write on their bazel.yml call (startup_failure otherwise);
  TestBazelWorkflowCallersGrantEveryRequestedPermission guards it.

rrc-writer-credential.sh, rrc_verify.py and rrc_verify_test.py are byte
copies of gascity's, pinned by sha256.

Refs ga-vnycm2.29.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 02:50:26 +00:00
Julian Knutsen
fa3d50eb19
ci(bazel): zstd for the anonymous fork cache when it advertises it (rbe-west E4) (#7202)
* ci(bazel): zstd cache transfers for the anonymous fork cache only, behind RBE_CACHE_ZSTD (E4)

E4 of the rbe-west egress reduction (design A.5), reworked for the
read-only zstd server shape. rbe-west's trusted MAIN and OSS schedulers
(:443 to :50052) and rbe-fork (:8444) advertise no compressor, and Bazel
refuses a remote that does not advertise zstd. So only the anonymous fork
cache (rbe-cache :8443, nativelink-anon, which keeps zstd) may ask for it.

- **`write-bazelrc.sh`:** no `--remote_cache_compression` on the trusted
  remote-exec or rbe-fork paths. In mode cache, with `RBE_CACHE_ZSTD=1`, it
  adds `build:fork-cache --remote_cache_compression`. Empty or 0 adds
  nothing; another value warns and stays off, rather than failing every
  cache lane.
- **`bazel.yml`:** every setup-bazel caller passes
  `RBE_CACHE_ZSTD: ${{ needs.rbe.outputs.mode == 'cache' &&
  vars.RBE_CACHE_ZSTD || '' }}`, so only cache runs see the variable.
  Merging changes nothing, and rollback is the variable, with no revert.
- **`.bazelrc`:** fork-cache no longer sets the flag; a comment points at
  the gate.

Tests:
- `TestBazelCacheModeReachesTheRC` evaluates `vars.` right-hand sides and
  simulates `RBE_CACHE_ZSTD=1`. Mode cache's rc carries exactly the
  fork-cache zstd line; the remote, fork-ro, fork-rw and local modes carry
  no compression at all.
- `TestSetupBazelRCWriter` runs `RBE_CACHE_ZSTD` = 1, 0, empty and a bad
  value in mode cache, and remote-exec with 1 (never compressed).
  `TestSetupBazelRCWriterForkMode` pins rbe-fork's exact lines with
  `RBE_CACHE_ZSTD=1`.
- The bazel.yml setup-env pin includes `RBE_CACHE_ZSTD`.
- `checkBazelrcForkCache` rejects the flag in any `.bazelrc` config, with
  fork-cache and remote-exec fixtures.
- New `TestBazelRemoteCacheCompressionOnlyForkCache` scans `.bazelrc`, the
  workflows and the local action files. The flag must appear once: the
  writer's fork-cache echo.

**Merge order:** after rbe-west's anonymous process advertises zstd. Even
then the flag stays off until `RBE_CACHE_ZSTD=1` is set. If rbe-cache stops
advertising zstd, fork-cache's fallback flags make lanes build locally.

**Action keys:** unchanged; the flag only changes ByteStream resource
names (`compressed-blobs/zstd/`).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): zstd for the fork cache by a GetCapabilities probe, not a repository variable (E4)

Review finding 3: GitHub gives fork pull_request runs no repository
variables (bazel.yml: forks "see no secrets and no vars"), so
vars.RBE_CACHE_ZSTD was always empty exactly where the switch matters. Fork
run 37220594372 logs RBE_VAR_ON: false. The variable could never turn zstd
on for fork-cache clients, and once on it could not follow rbe-cache
rolling zstd back.

- New setup-bazel/cache-zstd-probe.sh, a byte copy of gascity's
  tools/rbe/cache-zstd-probe.sh: one unary REAPI GetCapabilities call
  (instance oss) to rbe-cache :8443 over curl's HTTP/2, decoded with
  python3's standard library. It exits 0 only when
  cache_capabilities.supported_compressors lists ZSTD, the field Bazel
  checks. Any other result exits nonzero: not listed, an HTTP or gRPC
  error, a refused connection, a timeout (connect 3s, total 5s), an
  unparsable answer, or no curl or python3. It writes nothing to stdout
  and one verdict line to stderr. No new action and no new dependency.
- write-bazelrc.sh: in mode cache, `if bash .../cache-zstd-probe.sh` adds
  build:fork-cache --remote_cache_compression. The probe never fails the
  script. RBE_CACHE_ZSTD and its bad-value warning are gone.
- bazel.yml: the ten RBE_CACHE_ZSTD env lines are gone (back to the
  pre-E4 setup env).

Tests:
- New TestCacheZstdProbe runs the real script against a TLS HTTP/2
  stand-in for rbe-cache, which checks the exact request. Passes: zstd
  advertised (packed, and unpacked among others). Fails: rbe-cache's real
  2026-10-04 answer (no compressors), deflate only, zstd for batch updates
  only, zstd outside cache_capabilities, a gRPC error trailer, no
  grpc-status, trailers-only UNIMPLEMENTED, HTTP 502, a compressed or
  truncated message, a truncated field, not gRPC, a timeout, a refused
  connection, an untrusted certificate. It also pins the probe's endpoint
  and instance to .bazelrc's fork-cache.
- TestSetupBazelRCWriter: the fork cache gets exactly the zstd line after
  --config=fork-cache against a zstd stand-in, and none against the live
  answer or a refused port. Remote-exec and rbe-fork, with a zstd
  stand-in, never probe and never carry compression.
- TestBazelCacheModeReachesTheRC again rejects any vars. in setup-bazel's
  env, and no mode's rc carries compression (the probe is refused there).
- The writer harnesses default RBE_CACHE_PROBE_URL to a refused loopback
  port, so no test reaches rbe-cache.

Live check, 2026-10-04: rbe-cache answers without supported_compressors,
so the writer's mode cache rc has no flag until the E1 cutover.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:26:54 -07:00
Julian Knutsen
ebe3b6bcb0
test,ci(cmd/bd): run the Dolt-gated cmd/bd suite (advisory Bazel lane), fix in-process runner leaks and rotted tests; init --quiet stays quiet (#7185)
* test(cmd/bd): reset cobra flags between in-process runs; keep them out of parallel tests

runBDInProcess and runBDInProcessAllowError execute the package-level
rootCmd tree again and again in one process, and cobra never resets a
flag between Execute calls: a value one invocation parsed (create's
--title, --labels, list's --status, ...) and its Changed bit stayed on the
FlagSet, so the next invocation saw it, and a slice flag that had been
set appended to the stale value instead of replacing it. With a Dolt test
server (the only setup in which these tests run) that failed about 20
TestCLI_* tests depending on suite order.

TestMain now snapshots every flag of the tree (snapshotCommandFlags,
after all init()s registered them, before any test runs) and both runners
put the tree back before and after each run (resetCommandFlags): the
pointee of each flag.Value is restored (for pflag's scalar types that is
the bound variable; for slice types it also clears the internal "changed"
bit, so the next Set replaces), slice contents via SliceValue.Replace,
and Changed=false. TestResetCommandFlagsRestoresEveryFlag sets every flag
of the real tree and checks the reset. The ad-hoc --title reset in
TestCLI_CreateRejectsEmptyTitle_ProxiedServerMode is no longer needed.

Seven TestCLI_Comment* tests called t.Parallel() around the runners.
inProcessMutex serializes the runners only against each other, so their
chdir, stdio swap, env and config re-initialization raced every other
parallel test (data races on config globals; TestCobraParallelPolicyGuard
failing to read its own source files after a concurrent chdir). They run
serially now, and the guard treats the two runners like rootCmd.Execute.

Also two stale TestCLI_Create* cases: DryRunWithRigPrefix exercised a
create --prefix flag that no longer exists, and the flag-like-title cases
passed "--help" or "--foo-bar" where cobra parses them as flags; they now
pass the title after "--", the only way it reaches create's own guard.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): fix the ADO round-trip and bulk-delete tests that rotted unseen

Both need the integration tag and a Dolt test server, which no CI lane
gave cmd/bd (BEADS_TEST_SKIP=dolt everywhere), so they failed unnoticed:

- TestADORoundTrip* stored ado.pat in the database. ado.pat is a secret
  read only from config.yaml or AZURE_DEVOPS_PAT (config.IsYamlOnlyKey),
  so the tracker found no PAT. The fixture sets AZURE_DEVOPS_PAT.
- TestBulkDeleteNoResurrection expected GetIssue to return (nil, nil) for
  a deleted issue; the Dolt store reports storage.ErrNotFound.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): fix parity, completion and notion tests that only fail with a Dolt server

These are Dolt-gated (integration tag + test server), so no CI lane ran
them and they rotted:

- TestParityCreate*/TestParityReopenErrorExits1Silently: bd create
  auto-routes via routing.DetectUserRole("."), which warns "beads.role
  not configured (GH#2950)" on stderr when the working directory's git
  config has no beads.role. The suite pins the configured case, so the
  parity env now sets beads.role=maintainer in a test-owned global git
  config ($XDG_CONFIG_HOME/git/config; the role lookup scrubs every
  GIT_CONFIG_* variable, so those cannot be used).
- TestIssueIDCompletion: issueIDCompletion resolves a database path
  before looking at the open store and returns nothing when there is
  none; the test now sets dbPath to its store's path.
- TestIssueIDCompletion_UsesWorktreeFallbackWhenStoreNil and
  TestGetNotionConfigReadsDBPathWhenStoreUnset: the code under test
  reopens the store from metadata.json, which names a database but not
  the per-test branch of the shared database that newTestStore uses, so
  the reopened store saw none of the test's rows. They now use their own
  database (newTestStoreIsolatedDB).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): fix stale doctor/e2e/autocommit/readonly Dolt-gated tests

These need the integration tag and a Dolt test server, which no CI lane
gave cmd/bd, so they rotted unseen:

- TestE2E_DoctorFixMetadataRoundtrip, TestE2E_MigrateDoltMetadata,
  TestDoctorCheckHealthReportsVersionMismatchOnRepoLocalPort: bd init now
  defaults to embedded mode, where `bd sql` and `bd dolt start` are
  refused. Init with --server on a repo-local server of their own
  (repoLocalServerEnv: the suite's shared-server port vars cleared, server
  stopped at cleanup) and start it explicitly, since BEADS_TEST_MODE=1 never
  auto-starts one. MigrateDoltMetadata's DELETE also needed `key` quoted.
- TestDoctor{Conventions,Pollution}MaxRows_EnvOnly_Exits2: bdRunRaw drops
  every BEADS_* variable, so the subprocess hit dolt.New's test-database
  firewall (exit 1) before the row cap; pass BEADS_TEST_SERVER=1.
- TestDoltAutoCommit_On_WritesAdvanceHead: the embedded engine keeps its
  data under .beads/embeddeddolt, not .beads/dolt; findDoltRepoDir looks
  in both.
- TestValidateCheck_AllClean: bd doctor validate runs five checks since
  the cross-table duplicates check was added.
- TestConfigValidateReadOnlyIsHermetic: readonly commands take a shared
  workspace gate, whose lock file (.beads/dolt.gate.lock) workspacegate
  creates on first use by design; compare the .beads tree without gate
  lock files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): undo each in-process execution's env and storage-mode leaks

Every in-process run of the bd command tree (runBDInProcess and the many
tests that call rootCmd.Execute directly) leaves process state behind:
PersistentPreRun exports the selected workspace as BEADS_DIR, loads its
.beads/.env, and records the workspace's mode in serverMode /
proxiedServerMode. With a Dolt test server, later tests in the same shard
then resolved a deleted fixture ("no active beads workspace found",
"migration sidecar is missing"), ran subprocess bd against the wrong
workspace ("recorded as proxied-server in .beads/metadata.json"), or took
the proxied path in RunE-level tests ("proxied-server UOW provider not
initialized", a nil-UOW panic in reopenProxiedResolve that killed a shard).

TestMain now registers a cobra initializer/finalizer pair
(installExecuteIsolation) that snapshots the environment and the two mode
globals when an execution starts and restores them when it ends; the
runners' restoreEnvSnapshot shares the helper. TestDoltConfigEnvironmentOverrides
used os.Setenv + defer os.Unsetenv, which deleted the suite's
BEADS_DOLT_SERVER_PORT (the test server's port) for every later test; it
uses t.Setenv now.

Full integration-tagged cmd/bd suite against a hermetic dolt (16 shards):
18 order-dependent failures and the shard panic gone; the 27 remaining
failures all fail alone too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(init): keep --quiet quiet through the Claude/Codex/Cursor auto-setup

`bd init --quiet` ("Suppress output") still printed the project
integration installers' progress to stdout ("Installing Claude hooks for
this project...", "Claude Code integration installed", ...): the
installers write to os.Stdout unconditionally. Point stdout at the null
device around them while quiet; their failures keep going to stderr as
before. Found by TestInitCommand's quiet case, which asserts no output and
had not run in CI (Dolt-gated).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(init): bring the Dolt-gated init tests up to the current init contract

These tests need a Dolt test server, so no CI lane ran them and they
drifted from the product:

- Embedded is init's default mode, so the server-mode assertions
  (.beads/dolt, opening the database on the test server, dolt_mode
  "server" in metadata.json) now pass --server (TestInitCommand,
  TestInitAlreadyInitialized, TestInitRedirect, TestInitBEADS_DIR,
  TestInitDoltMetadata, TestInitDoltMetadataNoGit, TestInitDatabaseFlag's
  metadata subtests). The in-process ones reset rootCmd's flags around
  themselves so --server cannot leak into later in-process inits.
- Each server init gets its own --database: the suite shares one test
  server, and a "test" database another test created belongs to another
  project (init refuses: PROJECT IDENTITY MISMATCH).
- The marker tests seed a current-era version witness: the legacy-upgrade
  guard refuses a non-empty .beads/dolt without one.
- The redirect test provisions the canonical target with config.yaml: bd
  ignores a redirect to a directory with no project files (#4692).
- Shared-server inits get a free port of their own
  (sharedServerTestEnv): the inherited BEADS_DOLT_SERVER_PORT is the
  suite's test server, which a shared-server start refuses as another
  project's, and the default shared port is 3308, a developer's live
  server.
- --shared-server's config.yaml is now nested YAML (dolt: shared-server:
  true); parse it instead of matching the old flat line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): run the Dolt-gated tests that skipped on a bare dolt.New

setupTestDB (config, kv and Linear round-trip tests), setupGatedTestDB
(mol ready --gated) and the rename-prefix and bd sql tests opened stores
with dolt.New(&dolt.Config{Path: ...}): no server port, no
CreateIfMissing, so against a running test server they got "database ...
not found" and skipped as "Dolt server not available". They now use the
suite's newTestStore helpers (skip only when no test server runs), and
what that exposed is fixed test-side:

- TestCustomStatusConfig: custom_statuses is read back ORDER BY name;
  look statuses up by name / expect name order.
- TestLinearRoundTripCoreFields, TestLinearPullMilestonesCreatesEpicHierarchy:
  linear.api_key is a yaml-only secret (LINEAR_API_KEY), push now needs an
  explicit linear.state_map, and batch push uses TeamLabels and
  issueBatchCreate, which the mock server now answers.
- TestSqlCommand: bd sql refuses in the default embedded mode; set
  serverMode for the server-backed store, and stop discarding RunE errors.
- TestE2E_AutoStartedRepoLocalServerPersistsAcrossCommands skipped on
  this process's embedded mode although every bd it runs is a subprocess
  on a --server workspace; gate on the dolt binary instead and stop the
  auto-started server at cleanup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): no fixed "unused" port, no swallowed JSON, no leaked front-door server

Three more things the Dolt-backed cmd/bd run exposed:

- TestTestServerConnection, TestSharedServerCfgNilHonorsSharedServer and
  the two TestLegacyGuard* e2e tests pointed bd at 127.0.0.1:59999 as a
  port nothing listens on. Any process on the host may hold it; a
  listener that accepts and resets turns the expected "connection
  refused" / "Dolt server unreachable" into another error. They take a
  just-freed port from closedLoopbackPort (testutil.FindFreePort) now.
- TestDoltSetConfigJSONOutput and TestDoltSetConfigWithUpdateConfig
  created .beads 0755, so bd's permissions warning landed in the JSON
  under test and the tests skipped ("output not pure JSON") every run.
  The fixture uses 0700, and non-JSON output fails instead of skipping.
- TestMigrateDoltModeFrontDoor's closing show/list auto-start a
  repo-local server after the reverse migration; nothing stopped it, so
  the suite's leak check failed (when the opt-in
  BEADS_TEST_MIGRATION_FRONTDOOR suite runs). It stops it at cleanup.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): advisory cmd/bd Dolt-server tier (bazel-cmd-dolt)

No CI lane ran cmd/bd's Dolt-gated tests (TestCLI_*, the init and
store-backed suites, the doctor/e2e tests): main.yml's integration shards,
nightly's Full Test Suite, and Bazel's --config=prcore/ci and
--config=integration all pass BEADS_TEST_SKIP=dolt, so cmd/bd's TestMain
never starts a test server and those tests skip; the proxied and embedded
tiers run only their manifests' tests. They rotted unseen.

//cmd/bd:bd_dolt_server_test (tag dolt-server-cmd) runs the whole
integration-tagged cmd/bd suite, the integration build's bd_test, against
a hermetic dolt sql-server: BEADS_TEST_DOLT_SERVER=local starts it from
the pinned dolt in the test wrapper's runfiles, and
BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 fails the target instead of skipping
into a cached pass. No test selection: the Go binary shards itself over
every top-level test (32 shards), so each runs exactly once, and
check_testcases.py fails a shard that ran none. --config=doltserver-cmd
has exactly --config=integration's build flags, so the two lanes share
bd_test's compile; prcore/ci and integration exclude the tag.

bazel.yml's new bazel-cmd-dolt job runs it exactly where bazel-integration
runs (same if: remotely on the farm for same-repo PRs, fork-ro/fork-rw,
and locally with the read-only cache in mode cache for forks). It is
advisory: job-level continue-on-error keeps a failure out of the call's
aggregate (BAZEL), and pr.yml's ci-gate reads its result (BAZEL_CMD_DOLT)
but requires it only while the committed top-level flag
BAZEL_CMD_DOLT_REQUIRED is "true" (now "false"; flipping it is the one
change needed). bazel-gate.sh accepts its skip in modes skip and local.

Policy tests: a third lane category in TestBazelLaneIsGatedAlongsideLegacy
(bazelFlagGatedLanes: gate env wired, never statically required, added by
the run script exactly when the pinned flag is "true"), the gate
simulation over every mode with the flag off (advisory: nothing it
reports reddens the gate) and on (exactly a gated lane), the cache-mode
and execution-mode checks, the tag taxonomy and prcore/integration
exclusions, and TestBazelCmdDoltJob (job steps, timeouts, .bazelrc
config equal to integration's build with no dolt skip or test selection,
the target's dolt-server rules, and bd_test's env repeated).

Measured locally (all fixes in this series, 32 shards, 16 at a time on a
shared host): 3267 top-level tests, all shards pass, max shard 313s (one
5001-issue export test, 214s, sets it), mean 100s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(init): hand the project installers a writer instead of swapping os.Stdout

The --quiet fix earlier in this series pointed the process-global
os.Stdout at the null device around the Claude/Codex/Cursor installers.
That works, but it mutates a global from inside a cobra Run: anything else
writing to stdout meanwhile (a goroutine, a future concurrent installer)
would vanish, and a concurrent read of os.Stdout is a data race.

The installers already write through env structs with stdout/stderr
fields. setup.ProjectInstallOutput carries the two writers, and
InstallClaudeProjectTo / InstallCodexProjectTo / InstallCursorProjectTo
use them (the old entry points keep the process streams). init passes
QuietProjectInstallOutput() under --quiet: progress discarded, error lines
still on stderr. The proxied-server init path (domain adapter
InstallClaudeProject now takes quiet) does the same, so its
"Installing Claude hooks..." is quiet too.

Consistent error policy under --quiet: errors still reach stderr. init's
own "Warning: failed to setup <installer>" lines were suppressed under
--quiet while the installers' "Error:" lines were not, and some installer
failures (Cursor rules) print nothing themselves; the warnings now print
whether or not quiet, like the installers' errors.

Coverage in required lanes: TestProjectInstallersReportToTheGivenOutput
(cmd/bd/setup) and TestEmbeddedInitA/basic, which now asserts that
`bd init --quiet` with agents enabled installs the Claude hooks and prints
none of the installers' progress (verified to fail with the quiet output
disabled).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(export): exercise the incremental-export fallback with a small threshold

TestTryIncrementalExport_ThresholdExceededFallsBack seeded
incrementalExportThreshold+1 = 5001 issues to cross the threshold. Batch
create is one transaction but still several statements per issue, so under
-race against the shared test server it took ~214 s, the longest test in
cmd/bd's Dolt-backed suite by far. The behaviour under test is only
"more changed issues than the threshold -> full export fallback".

incrementalExportThreshold becomes a var (same value), and the test, which
is not parallel, sets it to 20 and seeds 21 issues: 214 s -> 0.6 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(cmd/bd): tighten the flag-reset self-test, guards and canaries

Review follow-ups:
- TestResetCommandFlagsRestoresEveryFlag no longer skips a flag it cannot
  set: a flag type without a sample value, a Set error, or a non-pointer
  Value (which resetCommandFlags could not restore) is an error, so a new
  flag type cannot go untested. resetCommandFlags documents that it also
  resets flag-bound package globals.
- TestCobraParallelPolicyGuard treats .ExecuteC/.ExecuteContext/
  .ExecuteContextC like .Execute (the per-execution isolation writes the
  mode globals; a parallel execution would race).
- TestConfigValidateReadOnlyIsHermetic ignores only the gate a readonly
  command takes (.beads/dolt.gate.lock and its .info sidecar), not any
  "*.gate.lock*" file, so a new unexpected gate still trips the canary.
- TestCLI_CreateRejectsFlagLikeTitles: a bare `bd create --foo-bar` must
  fail and create no issue (bd-2c0's intent), next to the "--" cases.
- TestCustomStatusConfig cites why statuses come back in name order: the
  normalized custom_statuses table (#2961, 8537d943b) has no ordinal and is
  read ORDER BY name, like custom_types; the test predates it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): run the cmd/bd Dolt-server tier in 16 shards

Each shard pays ~30-40 s of fixed cost (TestMain plus its own dolt
sql-server), so 32 shards spent roughly 40% of the lane's executor time on
setup and added 32 farm actions to every PR touching cmd/bd. With the
5001-issue export test cut to under a second, 16 shards measure (locally,
--nocache_test_results): max 186 s, mean 137 s, 2,187 executor-seconds
(vs ~2,800-3,200 at 32), all 3,267 top-level tests, every shard passing.
TestBazelCmdDoltJob pins the count; bazel.yml's timing comment is
restated (cache mode ~20-30 minutes).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 03:06:54 -07:00
Julian Knutsen
b8a9545cbe
ci(setup-bazel): 4 GB client heap on every runner (R4) (#7174)
The Bazel client's JVM defaults to 25% of RAM, about 2 GB on the 8 GB
2 vCPU Blacksmith runners every remote lane runs on. //... analysis fits
today with little headroom, and the plan moves more lanes and targets
onto these runners. write-bazelrc.sh now writes
`startup --host_jvm_args=-Xmx4g`: a startup option, so it changes no
action key and only restarts the server. Verified with Bazel 9.2.0:
`bazel --bazelrc=<generated rc> info max-heap-size` reports 4294MB.

gascity's W1 byte-copies this composite, so the change lands here first.

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 01:04:39 -07:00
Julian Knutsen
88d9b645a6
ci(bazel): pre-warm rbe-west's OSS worker pool when a same-repo PR's Bazel lanes start (#7159)
* ci(bazel): best-effort pre-warm rbe-west's OSS worker pool on PR open

bazel.yml's Bazel lanes each separately wait ~120s at "N actions, 0
running" while gastownhall/gascity's rbe-west NativeLink worker pool
scales up from zero. Add an advisory rbe-prewarm job (never a `needs`
of any lane, never in bazel-gate.sh's vocabulary, job-level
continue-on-error so it can never fail bazel.yml's own run) that
mints a short-lived installation token for a "bazel-allocator" GitHub
App (installed on gastownhall/gascity only, Actions read/write +
Metadata read) and dispatches gastownhall/gascity's
rbe-worker-pool.yml ahead of time, only when this run's lanes will
actually target rbe-west's "oss" instance (modes remote and fork-rw;
fork-ro targets a separate pool this doesn't serve).

Adds RBE_POOL_APP_ID / RBE_POOL_APP_PRIVATE_KEY as new optional
workflow_call secrets, threaded through every caller (pr.yml,
bazel-farm.yml, nightly.yml); either secret missing (the app not yet
provisioned) or repo var RBE_PREWARM_WORKERS=0 disables pre-warming
cleanly. Policy tests (scripts/bazel_rbe_prewarm_test.go, plus updates
to the existing bazel.yml structural tests) pin the mode gate, the
credential scope, the dispatch target, and that the job can never gate
or fail a run; each invariant was hand mutation-tested. Docs added to
engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md ("rbe-west Pre-warm").

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci): close rbe-prewarm token-theft path, fix stale secret-count docs (B1)

Security review of bdef342d5 found a BLOCKER (B1): the rbe-prewarm job
checked out inputs.checkout-sha (the fork head, reachable via
bazel-farm.yml's pull_request_target path) and then ran
.github/scripts/rbe-prewarm.sh with the minted gastownhall/gascity
dispatch token present. Script-file edits don't require workflow-approval
the way workflow-file edits do, so a VIP fork or same-repo branch pusher
could edit that script to steal the shared bazel-allocator key.

Fix: delete the checkout step and the .sh file; inline the dispatch logic
into the step's own `run:` block, whose bytes always come from the
trusted (approval-gated) workflow definition even under
pull_request_target. Mint the token in the immediately preceding step;
nothing after the mint step touches a repo-relative path. Stop forwarding
RBE_POOL_APP_ID/RBE_POOL_APP_PRIVATE_KEY from bazel-farm.yml (that fork
path now gets a cold start instead of pre-warming) and narrow the job's
`if:` to mode == 'remote' only (fork-rw never carried secrets anyway, so
it was dead weight - S1).

Also addresses the review's should-fix items:
- S2: document the widened trust boundary (the bazel-allocator key is
  shared with gascity's own rbe-west scaler; leak blast radius; rotation
  breaks the scaler, so the kill switch is unset-secret /
  RBE_PREWARM_WORKERS=0, not rotation) as an accepted risk in bazel.yml's
  header and engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md.
- S3: fix stale "exactly four secrets" claims (pr.yml/nightly.yml pass 6;
  bazel-farm.yml now correctly passes 4 again) and the false "never a
  long-lived credential" claim.
- S4: flag the continue-on-error-preserves-caller-success assumption as
  reasoned-through but not live-validated in this sandbox.
- N1: drop the unused RBE_PREWARM_IDLE_MINUTES knob entirely.
- N4: strengthen the dispatch-target test to check exact gh command
  strings and ban hardcoded repo/workflow literals outside $POOL_REPO/
  $POOL_WORKFLOW.

Test changes (scripts/bazel_rbe_prewarm_test.go rewritten; ci_workflow_test.go,
bazel_farm_workflow_test.go, bazel_cache_mode_test.go updated for the new
job shape): new TestRBEPrewarmNoCheckoutStep,
TestRBEPrewarmNothingAfterMintTouchesARepoPath,
TestRBEPrewarmTokenOnlyInDispatchStep, and
TestRBEPrewarmAppSecretsOnlyExpectedCallers directly target the B1
mutation class; TestRBEPrewarmDispatchScriptBehavior keeps the fake-gh
behavioral coverage for the now-inlined dispatch script. Verified by hand
that both of the reviewer's mutations (widen the `if:` back to include
fork-rw; add a step after mint running a checked-out script) are now
caught.

Verification: go test ./scripts/..., go vet, gofmt, actionlint on the
four touched workflow files, make ci-pr-policy, make bazel-sync-check,
and `bazel test //scripts:scripts_test` all pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 22:31:17 -07:00
Julian Knutsen
f0f8879ac4
ci(bazel): fork PRs execute on rbe-west's fork pool (fork-ro/fork-rw), cache fallback (#7136)
* ci(bazel): fork and Dependabot PRs execute remotely on rbe-fork when rbe-west's mint is open (F9)

Fork and Dependabot pull_request runs get no secrets, no OIDC token and no
repository variables, so rbe-west decides for them. bazel.yml's rbe job asks
rbe-fork-mint's /v1/status (rbe-mint.ops.gascity.com:8444) once per run:

- open, tier ro: mode fork-ro (instance oss-fork, the fork pool, nothing
  cached for anyone);
- open, tier rw (allowlisted author's own fork, allowlisted trigger): mode
  fork-rw (instance oss, the OSS pool);
- closed, refusing, unreachable or anything else: mode cache, exactly as
  today (local execution plus the anonymous read-only cache).

In the fork modes every lane runs, remotely, from a GitHub-hosted runner:
setup-bazel makes an EC P-256 key and a CSR, uploads the CSR as an artifact
of this run (the mint's proof), gets a ~2 h certificate
(fork-credential.sh, a byte copy of gascity's tools/rbe/fork-credential.sh
from F6) and writes an rc for grpcs://rbe-fork.ops.gascity.com:8444 with
the mint's instance. A lane whose certificate names another tier than the
rbe job fails ("re-run all jobs"). Fork lanes add --config=sole-run like
mode remote.

BAZEL_COVERS_FORKS ships "false" in pr.yml and pr-risk.yml: forks and
Dependabot keep their legacy tiers, and the Bazel lanes run beside them
(remotely while rbe-fork is open). F13 commits "true"; then a covered fork
run in mode cache is red, like a covered same-repo PR without the secret.

Safe while rbe-fork is dark: with the gate closed the mint's port resets
connections and the status probe falls back to mode cache in seconds; with
the mint up but disabled, canaried or rw off it answers open false or 403,
also mode cache.

Not in this change (later slices): bazel-farm.yml and its inputs stay (F12),
bazel.yml keeps its push trigger and pr.yml/pr-risk.yml gain none (F14).
keep their go build fallback on GitHub-hosted runners.

Tests: the rbe decision against a stubbed mint (ro, rw, closed, rw closed,
canary, busy, garbage, unknown tier, unreachable); the coverage and gate
matrices learn the fork modes, BAZEL_COVERS_FORKS and mint answers that
change between runs; TestBazelCacheModeReachesTheRC drives setup-bazel's
fork steps against a stand-in mint and checks each lane's rc;
fork-credential.sh's client contract and write-bazelrc.sh's fork mode and
exclusivity; the endpoint pin is allowed as exactly its one line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): fork-credential.sh retries the mint's 429 and never sleeps after the last try

A 429 (per-minute rate, live-certificate quota) may pass on its own like a
502, so it is retried with the same 10/20/30 s backoff (one minute, the
rate window); the loop no longer sleeps after the fourth failed attempt.
The cert request and the rbe job's /v1/status probe give up connecting
after 5 s, so a closed gate (dropped SYNs) costs seconds, not minutes.
Byte-identical to gascity's tools/rbe/fork-credential.sh.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: gascity-agent <agent@gascity.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-03 12:10:14 -07:00
Julian Knutsen
aaae7e6aff
ci: package gates start after rbe, MCP pytest -n 8, gate jobs on Blacksmith (F3) (#7126)
* ci(bazel): move package-mcp/package-npm gates into bazel.yml (F3)

Package gates now build bd via Bazel (no --config=ci) when the rbe job
enables it and fall back to go build otherwise, so forks/Dependabot/rbe-off
runs are unaffected. pr.yml's CI Gate keeps PACKAGE_MCP/PACKAGE_NPM required
and never-skippable, now sourced from needs.bazel.outputs.*; the retired
detect-package-gates/package-mcp/package-npm pr.yml jobs and
DETECT_PACKAGE_GATES are deleted. MCP pytest pins -n 8 explicitly with
pytest-xdist locked via uv lock. rbe, bazel-coverage, and both required gate
jobs move to Blacksmith for same-repo PRs (gate names unchanged).

Also closes a gap in detect-package-gates.sh (missing bazel.yml in the
force-both-gates path list, since the job definitions moved there) and wires
integrations/beads-mcp's manifest/lock into scripts_test's Bazel data so the
new pytest-xdist pin check runs under `bazel test` too.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): review fix-up for F3 package gates (SF-1, SF-2, nits)

Review of 65f378d241 (~/beads-bazel-plan/f3/review.md, no blockers) found:

- SF-1: the package gates' `bazel build //cmd/bd:bd_for_tests` resolved to
  a different output configuration (k8-fastbuild) than `bazel test
  --config=ci`'s bd_for_tests (k8-fastbuild-ST-..., via test:ci ->
  test:prcore's --@rules_go//go/config:race), so it got no action-cache
  reuse from the test job. Add the matching race flag to both jobs' build
  step; verified with cquery that both invocations now cquery to the
  identical bazel-out/k8-fastbuild-ST-.../bin/cmd/bd/bd_for_tests/bd path,
  and with a real build that it's an action-cache hit once bazel test
  --config=ci has run. Update the pinned Run string in
  pr_lanes_bazel_coverage_test.go and the impl report's cache-reuse claim.
- SF-2: drop stale comments/doc text saying the package gates download
  bazel-ci-build-artifacts; they build their own bd now (bazel.yml,
  pr.yml, engdocs/CI_REQUIRED_CHECK_TOPOLOGY.md).

Nits: pin both required gates' display names in TestSameRepoBlacksmithRunners
(nothing else reads job.Name, so a rename would otherwise go uncaught); pin
the package jobs' Python/uv/Node setup against main.yml's copies; add
timeout-minutes to both gate jobs and detect-ci-tier (same-repo PRs now run
on paid Blacksmith minutes, not a free runner's 360-minute default); shut
down Bazel and remove the RBE client key right after each package job
packages its Bazel-built bd (nothing downstream needs either); add
scripts/ci/package-bazel-bd.sh to detect-package-gates.sh's both-gates
trigger arm and pin it in TestDetectPackageGatesCoversBazelWorkflow.

Every fix mutation-tested (mutate -> test fails with a specific message ->
restore from a /tmp backup, diff-confirmed identical -> test passes again).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-03 00:03:43 -07:00
Julian Knutsen
e2b78f7d7a
ci(bazel): run integration on fork PRs via the read-only cache; retry Go module fetches (#7120)
* ci(bazel): retry Go module fetches and scale download timeouts in every mode

A fork run failed Bazel / test on one proxy.golang.org TLS handshake
timeout. gazelle's go_repository fetches modules with the go command
(fetch_repo), which never retries, and Bazel's downloader retry flags do
not reach it. setup-bazel's generated rc now sets
--repo_env=GOPROXY=https://proxy.golang.org|https://proxy.golang.org|direct:
with "|" the go command moves to the next entry on any error, so the proxy
is tried twice and then the module's origin. fetch_repo checks every
module against its go.sum hash whatever the source. Verified with Bazel
9.2.0: an unreachable first entry still fetches all 438 go_deps repos,
and without a fallback the fetch fails.

--http_timeout_scaling=2.0 doubles the timeouts of Bazel's own
downloader (the Go SDK, http_archive), which already retries.

Both flags are key neutral (repository fetching only) and apply in every
mode, because repository fetching always runs on the runner.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): run the integration lane for fork and Dependabot PRs in cache mode

bazel-integration was remote-only, so fork and Dependabot PRs (mode
cache) had no PR-time integration run at all: main.yml's integration jobs
run only after merge. With rbe-west's anonymous read-only cache, every
action trusted CI already ran is a hit, so the GitHub-hosted runner builds
and tests only what the PR changed.

Simulated in an ubuntu:24.04 container with 4 CPUs and only the generated
rc (no user rc). This setup's action keys match CI's: the ci lane
reproduced the fork run's 6550 processes with 4847 cache hits.
- Main HEAD: 143 s (4656 hits, 12 local, 113 of 115 tests cached).
- A cmd/bd change: 170-290 s.
- Cache closed (cold, all local): about 30 minutes.

- bazel.yml: the lane's if is now mode remote or cache, never local
  (rbe=off). The step timeout goes from 45 to 75 minutes (job 85), so a
  closed cache (the farm's kill switch) still leaves fork PRs green.
- bazel-gate.sh: mode cache no longer allows BAZEL_INTEGRATION to skip.
  Mode local still does.
- New TestBazelCacheModeReachesTheRC runs, end to end, the real decision
  step for 22 starts:
  - fork, Dependabot and same-repo PRs, push, rbe=cache/off dispatches
    and bazel-farm, each with the variable on and unset;
  - same-repo runs with and without the secret;
  - a fork PR that "has" the secrets, and one passing fork-farm.
  For each start it evaluates every lane's if and setup-bazel env, runs
  write-bazelrc.sh, and checks bazel-gate.sh's skips. A cache-mode rc
  carries --config=fork-cache and the fetch hardening, and no executor,
  TLS file, instance, upload, header, BES or masked secret.
- ci_workflow_test.go: the integration if, the run modes, and a
  closed-cache floor for the step timeout.
- Comments in bazel.yml, pr.yml and bazel-farm.yml, and the topology
  engdoc. The engdoc also covers:
  - fork-cache's slow-blob behaviour: 15 s read deadlines trip the
    breaker, then "Lost inputs" and one build restart, still green;
  - why the endpoint's FindMissingBlobs denial never affects cache-only
    clients.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-02 17:15:56 -07:00