beads/engdocs
Julian Knutsen 46e670abbf
ci: retire PR Risk's legacy embedded tier where the gated Bazel lane runs it (D2 step 1) (#7071)
* ci(pr-risk): stand down legacy embedded tier where the gated Bazel lane runs it

D2 step 1. A new bazel-embedded-coverage job decides, from the same
expressions as bazel.yml's rbe job, whether pr.yml's Bazel call takes
execution mode remote on a same-repo pull_request (RBE_WEST_WORKERS on,
executor secret present, not a fork). Only then do test-embedded-storage,
test-embedded-conformance and test-embedded-cmd (27 jobs) skip, and PR
Risk's gate accepts those skips only then; a failed or missing decision is
red. Forks, Dependabot, secret-less runs, the farm switch unset and
merge_group keep the legacy tier. build-embedded keeps running: its
artifact feeds the proxied and server Dolt jobs.

The decision job reads no repository code and the secret only as an
emptiness test. scripts/pr_risk_bazel_coverage_test.go runs both actual
decision scripts over every event/var/secret/fork combination and requires
covered <=> (pull_request && Bazel mode remote), pins the shared env
expressions, checks pr.yml still requires BAZEL_EMBEDDED and shares PR
Risk's triggers, and simulates PR Risk's gate over every tier/decision.

Revert: unset RBE_WEST_WORKERS.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(ci): record the retired legacy embedded tier in the check topology

Who keeps the legacy tier, how the decision and gate work, the drift
guard, the residual cross-run race, the conformance.yml duplicate, and
how to revert (unset RBE_WEST_WORKERS).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(pr-risk): retire the embedded tier by committed flag; pr.yml gate fails closed

Review F1. PR Risk's coverage decision read the mutable RBE_WEST_WORKERS
variable in a different run from pr.yml's rbe job, so unsetting it and
re-running one workflow could leave both required checks green with
neither the legacy tier nor the Bazel lane having run the tier.

- Both workflows now commit BAZEL_RETIRES_LEGACY_EMBEDDED (policy-tested
  equal) and run the identical bazel-embedded-coverage job:
  covered = flag && pull_request && !fork && executor secret present.
  RBE_WEST_WORKERS is no longer read by PR Risk at all.
- pr.yml's ci-gate requires the job's result (BAZEL_EMBEDDED_COVERAGE)
  and BAZEL_EMBEDDED_RETIRED, which is red when covered but the Bazel
  embedded lane did not run remotely and pass (mode skip included).
- Revert is a commit of the flag to "false" in both workflows; in-flight
  PRs pick it up on their next push. The kill-switch runbook now flips
  the flag first.

Tests run both decision steps, bazel.yml's rbe step and pr.yml's real
gate step over every event/var/secret/fork/flag combination, with the
variable differing between the two runs, and require no double-green
without the tier.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): never cache embedded-tier test results

Review F2. Since D2 step 1 the Bazel embedded lane is the tier's only
pre-merge run on same-repo PRs, so a cached PASS (stale, or a poisoned
entry in the shared farm action cache) or a cached run that never
re-exercises an intermittent -race finding would stand in for the legacy
jobs' -test.count=1 run. test:embedded now sets --nocache_test_results,
and the policy test requires it and forbids any other rc line that turns
test result caching back on.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(ci): forbid flaky-test retries in the gated Bazel lanes

Review F3. Adding --flaky_test_attempts to a lane's config or
flaky = True to a lane target passed every policy test, and would hide
a flaky failure the legacy jobs (which ran each test once) would have
reported; since D2 step 1 nothing else runs the embedded tier on
same-repo PRs. TestBazelGatedLanesNeverRetryFlakyTests rejects
flaky_test_attempts and runs_per_test_detects_flakes in .bazelrc,
.github (workflows, setup-bazel's generated rc) and tools/bazel, and
flaky = True in every BUILD file and .bzl macro.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(ci): pin the Bazel embedded lane's command line and config

Review F4. An extra --test_arg=-test.short/-test.skip on bazel.yml's
`bazel test //... --config=embedded` line, or a test:embedded
--test_filter in .bazelrc, narrowed the tier while every policy test
passed (check_testcases only rejects a shard with zero tests). Since D2
step 1 this lane is the tier's only pre-merge run on same-repo PRs, so:

- bazel-embedded's test step must be exactly the pinned script;
- the --config=embedded lines must be exactly the pinned list, in order;
- no unconfigured common/build/test line, and nothing setup-bazel's
  generated rc writes, may select or narrow tests (test_filter,
  test_arg, -test.short/run/skip, tag/size/lang/timeout filters).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(bazel): add the D2 policy tests to scripts_test

scripts/pr_risk_bazel_coverage_test.go was missing from
//scripts:scripts_test's srcs, so `bazel test //scripts:scripts_test`
(and the BUILD-sync check) did not see it. The repository-walking
flaky-retry test skips under Bazel, whose runfiles hold no other
package's BUILD files.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): check every listed embedded test ran in its Bazel shard

Review F5. The shard scripts discover tests from source with grep, but
each Bazel test binary holds only its target's srcs, so a discovered test
the binary lacks matches nothing in the shard's -test.run selector and
passes silently (check_testcases.py only rejects a shard with no tests).

tools/bazel/check_shard_coverage.py runs each shard script in list-only
mode (BEADS_TEST_SHARD_LIST_ONLY=1) and requires every Bazel shard's
test.xml to hold exactly the listed top-level tests: missing, extra,
wrong-shard, shard-count mismatch, missing test.xml and a failing script
are errors. bazel-embedded runs it after the tier for
//cmd/bd:bd_embedded_test (20) and embeddeddolt_embedded_test (5); the
policy test pins the suites to pr-risk.yml's jobs and shard counts, and
a fixture test exercises the script.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(ci): correct the D2 enforcement, re-run and merge-queue claims

Review F6.
- The beads-only ruleset enforces both gates on main only; release/**
  PRs run them but merging does not wait for them, before and after D2.
- There is no merge queue and strict is off: merge_group runs are not a
  pre-merge safety net for the retired tier.
- Spell out "Re-run all/failed jobs" semantics under the committed flag,
  the remaining secret-deletion window, and that the revert needs a new
  push to in-flight PRs.
- Record the lane hardening the retirement relies on (no result caching,
  no retries, pinned selection, per-shard coverage check).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(ci): drop TestMain from the embedded shard-coverage check

Re-review G1 (blocking). embedded-storage-test-shard.sh discovers tests
with `grep '^func Test'`, which also lists TestMain(m *testing.M)
(internal/storage/embeddeddolt/test_fixture_test.go). It is never a
<testcase>, so check_shard_coverage.py reported it missing on every real
remote run, turning bazel-embedded and CI Gate red on every same-repo PR.

The checker now drops NOT_TESTS = {TestMain}. The shard scripts are left
unchanged so the legacy jobs' -test.run selection stays byte-for-byte the
same (dropping TestMain there would be harmless to shard assignment,
which hashes per name, but would touch the legacy CI scripts this step
does not otherwise change).

TestEmbeddedShardScriptsListOnlyRealTests runs both real shard scripts
in list-only mode for every shard and requires each listed name to be a
`func Name(t *testing.T)` in the package, or in NOT_TESTS (which may hold
only TestMain, and only if it is not a test). The fixture shard script
now lists TestMain too, so the filter itself is exercised.

Verified on a real remote run (`bazel test //... --config=embedded
--config=remote-exec`, 4/4 targets, 27 test.xml, 582 top-level tests,
348 s uncached): check_testcases.py and check_shard_coverage.py both
pass (199 cmd + 381 storage tests); the previous checker failed on
TestMain.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(pr-risk): decide embedded coverage by actor, not by the executor secret

Re-review G2. covered still read secrets.RBE_WEST_EXECUTOR, which an admin
can delete between PR Risk's run and a re-run of pr.yml: PR Risk stayed
green with the legacy tier skipped, while the re-run computed covered =
false, took mode local and passed with no embedded lane.

Both workflows' bazel-embedded-coverage now decide
covered = flag && pull_request && !fork && github.actor != dependabot[bot]
and read no secret or variable at all (PR Risk now reads no secret).
Dependabot (no Actions secrets) keeps the legacy tier; github.actor,
unlike triggering_actor, survives a re-run by someone else. A same-repo
PR without the secret is covered, so pr.yml's gate goes red (mode local)
instead of passing.

The two-run test matrix now varies the secret as well as RBE_WEST_WORKERS
between PR Risk's and pr.yml's runs, adds the actor to the facts, and
names the missing-secret case (red). Docs updated: the residual is gone,
the cost of failing closed is recorded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(ci): close the remaining ways to narrow the Bazel embedded lane

Re-review G3. Skip-inducing edits outside the pinned --config=embedded
lines and command line still passed every policy test (reviewer
survivors m1, m2, m5, m6, m9, m10, m11), and the runtime checks counted
skipped testcases as run.

TestBazelEmbeddedLaneCannotBeNarrowed now rejects:
- any .bazelrc line of a config the lane uses (unconfigured, embedded,
  remote-exec from setup-bazel's rc, and any config they reference) that
  filters, narrows, re-runs or re-routes tests (test_filter, test_arg,
  -test.*, *_filters, test_env=BEADS_TEST*/GO_TEST*/TESTBRIDGE*,
  --config=, cache_test_results, run_under other than test_env, flaky,
  runs_per_test, sharding strategy);
- any import other than the two gitignored try-imports, and any
  committed rc file besides .bazelrc (try-import would load it in CI);
- -test.short/run/skip/list, BEADS_TEST_SKIP, BEADS_TEST_EMBEDDED_DOLT,
  TESTBRIDGE_TEST_ONLY and similar in tools/bazel/*.sh (test_env.sh,
  go_test_manifest_shard.sh, ...) and anywhere in the setup-bazel action,
  and any --config= there but remote-exec;
- any change to the set of embedded-tagged targets or their args/env.

At run time check_shard_coverage.py fails a shard whose top-level tests
were all skipped, and its new --whole option applies the same check to
the unsharded conformance core/audit targets (added to bazel-embedded's
step). The real remote run still passes (no shard is all-skipped today;
one cmd shard has 1 of 9 tests skipped).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): ask Bazel for flaky test targets before the embedded tier

Re-review G4. The F3 grep wanted a literal `flaky = True`, so
`flaky = _RETRY` (or a macro default) slipped through (reviewer m3).

- bazel-embedded runs `bazel query 'attr(flaky, 1, tests(//...))'` after
  setup-bazel and before the tier, and fails on any result (loading only;
  about 0.2 s warm). Verified locally: with `_RETRY = True` /
  `flaky = _RETRY` on embeddeddolt_embedded_test the query lists it.
- The policy grep now rejects any `flaky =` whose value is not a literal
  False or 0, and TestBazelEmbeddedQueriesFlakyTargets pins the step.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(ci): cover the zero-testcase shard in check_shard_coverage

Re-review G5. Skipping the "listed but did not run" report when a shard's
test.xml has no top-level tests survived the fixture test (reviewer m7);
check_testcases.py catches that shard at run time, but the coverage
check should not depend on it. Two fixture cases: a shard with no
testcases, and one with only a subtest.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(ci): record the G1-G5 embedded-lane hardening in the topology doc

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 02:03:46 -07:00
..
adr fix(init): refuse to silently recreate a missing server-side database as empty (#5791) 2026-09-30 20:07:49 -07:00
decisions docs(pipeline): pin the CLI docs to the released bd; repair main's broken CLI reference (#4876) 2026-07-17 18:03:28 -07:00
design fix(externaldeps): guard explicit claims and batch closes (#6738) 2026-10-01 01:35:54 -07:00
staged-for-removal docs: pin the Dolt version instead of steering users to releases/latest (ga-m0dlk) (#5899) 2026-08-20 22:41:24 -07:00
ADDING_AN_ISSUEOPS_ROLE.md feat(journalops): the mutation journal becomes the third facade namespace (#5552) 2026-08-10 10:58:05 -07:00
AGENT_SIGNING.md docs: move internal/maintainer docs from docs/ to engdocs/ 2026-07-12 10:37:07 -07:00
ATTRIBUTION.md docs: move internal/maintainer docs from docs/ to engdocs/ 2026-07-12 10:37:07 -07:00
BDP_BEAD_GRAPH_PLAN.md BDP bead-graph plan and W-arch design docs (P-1 rulings complete; docs-only) (#6154) 2026-09-28 22:12:19 -07:00
BDP_GRAPH_ARCHITECTURE.md BDP bead-graph plan and W-arch design docs (P-1 rulings complete; docs-only) (#6154) 2026-09-28 22:12:19 -07:00
BDP_GRAPH_CLI_AND_STORAGE_SPEC.md BDP bead-graph plan and W-arch design docs (P-1 rulings complete; docs-only) (#6154) 2026-09-28 22:12:19 -07:00
CI_CLEANUP_PLAN.md fix(preflight): share checkout lint authority (#5780) 2026-09-30 22:41:55 -07:00
CI_REQUIRED_CHECK_TOPOLOGY.md ci: retire PR Risk's legacy embedded tier where the gated Bazel lane runs it (D2 step 1) (#7071) 2026-10-01 02:03:46 -07:00
CI_TEST_SURFACE_AUDIT.md ci: give PR Risk's gate a distinct check name (#6939) 2026-09-28 22:31:43 -07:00
CLAUDE.md docs(test): make TESTING.md canonical (bd-mrs20) (#5226) 2026-08-01 02:30:21 -07:00
COLLISION_MATH.md docs(ci): add Mintlify preview, docsync guard, and broken-links workflow 2026-07-12 10:37:07 -07:00
complexity-baseline.txt ci: add advisory cyclomatic complexity reporting (#6081) 2026-08-30 19:05:17 -07:00
COMPLEXITY_TRACKING.md ci: add advisory cyclomatic complexity reporting (#6081) 2026-08-30 19:05:17 -07:00
CONTRIBUTOR_NAMESPACE_ISOLATION.md docs(ci): add Mintlify preview, docsync guard, and broken-links workflow 2026-07-12 10:37:07 -07:00
DOC_INVENTORY.md fix(preflight): share checkout lint authority (#5780) 2026-09-30 22:41:55 -07:00
ERROR_HANDLING.md docs: port user docs into the Mintlify site tree under docs/ 2026-07-12 10:37:07 -07:00
EXCLUSIVE_LOCK.md docs: move internal/maintainer docs from docs/ to engdocs/ 2026-07-12 10:37:07 -07:00
EXTENDING.md perf(storage): SearchIssueSummaries narrow-projection foundation (be-nu4.3) (#3458) 2026-10-01 01:19:24 -07:00
ICU-POLICY.md fix(ci): require direct workflow build tags (#5679) 2026-10-01 01:24:03 -07:00
INTEGRATION_CHARTER.md docs(ci): add Mintlify preview, docsync guard, and broken-links workflow 2026-07-12 10:37:07 -07:00
INTERNALS.md docs(ci): add Mintlify preview, docsync guard, and broken-links workflow 2026-07-12 10:37:07 -07:00
LINTING.md fix(preflight): share checkout lint authority (#5780) 2026-09-30 22:41:55 -07:00
messaging.md fix(doctor): recommend 'bd prune' instead of nonexistent 'bd cleanup' (#5829) 2026-08-20 05:30:47 -07:00
PERFORMANCE_TESTING.md docs(engdocs): point storage test and bench references at the Dolt packages 2026-07-18 02:43:57 +00:00
PROJECT_CHARTER.md docs(ci): add Mintlify preview, docsync guard, and broken-links workflow 2026-07-12 10:37:07 -07:00
README_TESTING.md docs(test): make TESTING.md canonical (bd-mrs20) (#5226) 2026-08-01 02:30:21 -07:00
RELEASE-STABILITY-GATE.md docs: move internal/maintainer docs from docs/ to engdocs/ 2026-07-12 10:37:07 -07:00
RELEASING.md docs: move internal/maintainer docs from docs/ to engdocs/ 2026-07-12 10:37:07 -07:00
REPO_CONTEXT.md docs(ci): add Mintlify preview, docsync guard, and broken-links workflow 2026-07-12 10:37:07 -07:00
RULES_AUDIT_DESIGN.md docs: move internal/maintainer docs from docs/ to engdocs/ 2026-07-12 10:37:07 -07:00
SECURITY-DEPENDENCY-EXCEPTIONS.md Bump Go to 1.26.5 (#4942) 2026-07-23 18:54:32 -07:00
SERVE_RUNBOOK.md docs: the ext-api-store documentation tranche — post-#5516 auth sweep, wave doc coherence, role checklist refresh (#5545) 2026-08-10 08:05:07 -07:00
TESTING.md docs(testutil): document migration-0032 hang and Ryuk-disabled container-leak caveats (#6001) 2026-09-30 20:17:56 -07:00
TESTING_PHILOSOPHY.md docs(test): make TESTING.md canonical (bd-mrs20) (#5226) 2026-08-01 02:30:21 -07:00
UI_PHILOSOPHY.md docs: move internal/maintainer docs from docs/ to engdocs/ 2026-07-12 10:37:07 -07:00