From 4f9174e6baf8947016fca819dcdaa45a1f980807 Mon Sep 17 00:00:00 2001 From: "gascity-eddie[bot]" <338826946+gascity-eddie[bot]@users.noreply.github.com> Date: Mon, 19 Oct 2026 14:04:30 +0000 Subject: [PATCH] docs(agents): bazel test is authoritative; agents never run go test (#7695) * docs(agents): bazel test is authoritative; agents never run go test Agents run tests only with `bazel test`, also while iterating on one behavior: AGENTS.md shows the focused form (`bazel test //cmd/bd:bd_test --config=ci --test_filter='^TestName$'`), the target naming, that one --test_filter covers every target, that a cached result for unchanged inputs is the same evidence, and that a new test file needs `make bazel-sync` before Bazel runs it. `go test` (`./scripts/test.sh`, `make test-go`) stays a convenience for humans working offline. TESTING.md's focused red/green row and its plain-go-test section say the same. Why: on the maintainer-city factory host, seats' own uncached `go test` reruns of the same packages are the largest tool cost in adopt-pr, while `bazel test` there executes on rbe-west and returns an unchanged target's cached result in seconds. Co-Authored-By: Claude Opus 5.5 (1M context) * docs(agents): check that a focused test ran; exempt golden regeneration Review fixes for #7695 (iterations 1 and 2). The focused form exits 0 with "1 test passes" when the test never ran: a filter that matches nothing, an integration-tagged file --config=ci does not compile, the TestEmbedded* name skip, a -short skip, and a parent test whose every subtest skipped (Go still prints "--- PASS" for the parent). It now adds --test_output=all, and a run counts only when its log shows "--- PASS: " and no "--- SKIP" for the test or a /... subtest; a subtest that skips by design is left out with a subtest filter. engdocs/TESTING.md gains "Check That the Test Ran"; AGENTS.md links there instead of restating it. The section names the tests --config=ci does not run and the way into each: - Dolt-backed: the three --test_env flags, or the test's Dolt lane. - -short-gated: --test_arg=-test.short=false. - integration-tagged: --config=integration; the Dolt-backed ones of cmd/bd, tests/regression and internal/storage/dolt run in their Dolt-server lane target, whose env the suite needs. No lane runs those of other packages with Dolt, so a failure there is compared with the base commit first. - TestEmbedded*: --test_arg=-test.skip= lifts the name skip; a test that then still skips for BEADS_TEST_EMBEDDED_DOLT=1 runs in its package's embedded-tagged target under --config=embedded (the cmd/bd and internal/storage/embeddeddolt targets are named). - files BUILD.bazel does not list: make bazel-sync. A lane's sh_test can hand the test binary its own -test.run, which overrides --test_filter, so a trailing --test_arg=-test.run with sharding disabled selects the test there. In a sharded target the other shards print "testing: warning: no tests to run". The section also gives the target-naming rule. Golden regeneration (make corpus-regen, make githooks-regen, BD_UPDATE_GOLDEN=1 ... go test, -update-ledger-doc) writes the source tree, which a Bazel test cannot do: it runs from its runfiles tree, and on rbe-west update mode fails with "permission denied". AGENTS.md and the new "Regenerating Golden Files" section exempt these go test runs; where a host refuses go test even for them, a maintainer regenerates. The factory seat shim (gastownhall/workflows#102, scripts/seat-bin/go) still refuses update mode and needs the same carve-out there. Also: the plain-go-test opener scopes "humans working offline" to the runner and its go test targets, so the Dolt-backend, fixture and testing.Short() rules still bind agents; make ci-pr-core is marked plain go test; the beads-docs skill's docsync gate is the Bazel target; RELEASING.md and engdocs/RELEASE-STABILITY-GATE.md name the regression suite's Bazel target (//tests/regression:regression_test under --config=doltserver-cmd) in place of make test-regression, which is plain go test. Measured on rbe-west (exit 0 unless noted). Iteration 1, at c4d5409895: - a mistyped filter, and //internal/compact TestNew (integration-tagged) under --config=ci: "testing: warning: no tests to run" - TestNew under --config=integration: "--- SKIP" (BEADS_TEST_SKIP=dolt); with the Dolt flags added it runs (and fails, exit 3, on the local backend: its store finds no "beads" database; outside this PR) - TestConcurrentHeartbeatReclaimClose with the Dolt flags: "--- SKIP" (short mode); plus --test_arg=-test.short=false: "--- PASS" - //cmd/bd:bd_embedded_test, sharding disabled: --test_filter alone ran TestEmbeddedUpdateConcurrent (manifest shard 1's test); the trailing -test.run ran "--- PASS: TestEmbeddedShowBasicsAndJSON" - //scripts:go_test_sources_test TestEmbeddedCmdShardManifestBareWriteIsIdempotent: --config=ci "no tests to run"; plus --test_arg=-test.skip=: "--- PASS" Iteration 2, at b1d25543aa (the same Go inputs): - //internal/storage/dolt:dolt_test TestMoveIssuePersistenceInTxTransitions under --config=ci: "--- PASS" for the parent and "--- SKIP" for all 6 subtests (BEADS_TEST_SKIP=dolt), "1 test passes"; with the Dolt flags and --test_filter='^TestMoveIssuePersistenceInTxTransitions$/^persistent-to-ephemeral$' "--- PASS" for the parent and that subtest only, no "--- SKIP" - //internal/storage/embeddeddolt:embeddeddolt_test TestEmbeddedRemoteMigrateGate_BlocksReopen under --config=ci plus --test_arg=-test.skip=: "--- SKIP" (BEADS_TEST_EMBEDDED_DOLT unset); //internal/storage/embeddeddolt:embeddeddolt_embedded_test under --config=embedded, sharding disabled, trailing -test.run: "--- PASS" (the shard script listed 433 tests; only the named one ran) - --config=ci plus --test_arg=-test.skip=: "--- PASS" for //tools/nogo TestEmbeddedConfigIsTheRepositoryConfig and //internal/tracker TestEmbeddedDoltExternalRefChangedAfterUsesHistoryFastPath - sharding disabled, trailing -test.run: "--- PASS" for //internal/storage/dolt:dolt_server_full_test TestWispDeleteCascade_TransactionDeleteIssue under --config=doltserver-integration, and for //cmd/bd:bd_dolt_server_test TestCLI_ReopenClearsDeferUntil under --config=doltserver-cmd - make check-docs: exit 0 (docsync: 10 test cases ran, none skipped) Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 (1M context) --- .claude/skills/beads-docs/SKILL.md | 3 +- .../beads-docs/references/simplification.md | 5 +- .../beads-docs/references/verification.md | 2 +- AGENTS.md | 26 +++- RELEASING.md | 3 +- engdocs/RELEASE-STABILITY-GATE.md | 2 +- engdocs/TESTING.md | 112 ++++++++++++++++-- 7 files changed, 136 insertions(+), 17 deletions(-) diff --git a/.claude/skills/beads-docs/SKILL.md b/.claude/skills/beads-docs/SKILL.md index 52bab8362..a5454d2ea 100644 --- a/.claude/skills/beads-docs/SKILL.md +++ b/.claude/skills/beads-docs/SKILL.md @@ -201,7 +201,8 @@ document what the pinned release does, never main-only features. See ## 9. Verify before you call it done Run the gates in [references/verification.md](references/verification.md). -The short list: `go test ./test/docsync` (nav↔file sync + link conventions), +The short list: `bazel test //test/docsync:docsync_test --config=ci` +(nav↔file sync + link conventions), `./scripts/generate-cli-docs.sh --check` (generated docs fresh), `./scripts/check-doc-freshness.sh` (Last reviewed markers), and a live preview with `make docs-dev` (or `./mint.sh dev`) at `localhost:3000`. diff --git a/.claude/skills/beads-docs/references/simplification.md b/.claude/skills/beads-docs/references/simplification.md index 5152fd2b5..916834ca5 100644 --- a/.claude/skills/beads-docs/references/simplification.md +++ b/.claude/skills/beads-docs/references/simplification.md @@ -39,8 +39,9 @@ don't measure that work by the word delta. *not* simplifying — it buries the real change in the diff. 4. **Loss-check** (below). 5. **Fact-check** (below). -6. **Gates:** `go test ./test/docsync`, every fence parses, no body H1, no - HTML comments (MDX), preview with `make docs-dev`. +6. **Gates:** `bazel test //test/docsync:docsync_test --config=ci`, every + fence parses, no body H1, no HTML comments (MDX), preview with + `make docs-dev`. 7. **Preview, then commit on approval** — one page or section per batch, so the reviewer can steer before a cut compounds across the section. diff --git a/.claude/skills/beads-docs/references/verification.md b/.claude/skills/beads-docs/references/verification.md index 559bfb6f3..44aabca76 100644 --- a/.claude/skills/beads-docs/references/verification.md +++ b/.claude/skills/beads-docs/references/verification.md @@ -8,7 +8,7 @@ Run these before considering docs work complete. Ordered cheapest-first. # 1. Docs sync: docs.json nav <-> file consistency, link conventions # (root-relative extensionless in docs/, exact paths in engdocs/ and # curated root files), orphan detection (only CLI_REFERENCE.md is exempt). -go test ./test/docsync +bazel test //test/docsync:docsync_test --config=ci # 2. Generated CLI docs freshness: regenerates from the live command tree # and diffs docs/CLI_REFERENCE.md, docs/cli-reference/, and the CLI diff --git a/AGENTS.md b/AGENTS.md index 1a450309e..e908a5070 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -126,9 +126,29 @@ make install # build and install bd to ~/.local/bin (canonical) certificate `--config=remote-exec`, agent hosts whose `~/.bazelrc` names the executor neither. Make yours the default with a `build --config=...` line in the gitignored `.bazelrc.local`, or pass `BAZEL_FLAGS=...` to make. -- `go test` (`./scripts/test.sh`, `make test-go`) is an inner-loop - convenience only. CI does not enforce it, and it skips nogo, gofmt and the - guards; finish with `make test`. +- Agents run tests only with `bazel test`, also while iterating on one + behavior: + + ```bash + bazel test //cmd/bd:bd_test --config=ci --test_filter='^TestName$' --test_output=all + ``` + + A target also passes when the test never ran (a filter that matches + nothing, a skip, a file the configuration does not compile), so count a run + only when its log shows `--- PASS: TestName` and no `--- SKIP` for the test + or a `TestName/...` subtest: Go prints `--- PASS` for a test whose every + subtest skipped. "Check That the Test Ran" in + [engdocs/TESTING.md](engdocs/TESTING.md) names the tests `--config=ci` does + not run (Dolt-backed, `-short`-gated, integration-tagged, `TestEmbedded*` + and embedded-Dolt, files `BUILD.bazel` does not list) and the way into + each. An unchanged target returns its cached result, which is the same + evidence. Finish with `make test`. +- Agents do not run `go test` (`./scripts/test.sh`, `make test-go`): CI does + not enforce it, it skips nogo, gofmt and the guards, and agent hosts may + refuse it. It stays a convenience for humans working offline. The exception + is regenerating golden files (`make corpus-regen`, `make githooks-regen`, + `BD_UPDATE_GOLDEN=1`), which Bazel cannot do: see "Regenerating Golden + Files" in [engdocs/TESTING.md](engdocs/TESTING.md). - **Do NOT** use `go build -o bd ./cmd/bd`, `go install ./cmd/bd`, or raw `go run ./cmd/bd ...`: they bypass the canonical build path, leave stale binaries, and raw `go run` misses the `gms_pure_go` tag. Use `make install`, diff --git a/RELEASING.md b/RELEASING.md index e05b73072..50acb6497 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -145,7 +145,8 @@ Before starting a release: - [ ] All tests passing (`make test`, the Bazel test lane) - [ ] npm package tests passing (`cd npm-package && npm run test:all`) - [ ] **Upgrade smoke tests pass** (`make test-upgrade`) — see [Release Stability Gate](engdocs/RELEASE-STABILITY-GATE.md) -- [ ] **Regression tests pass** (`make test-regression`) +- [ ] **Regression tests pass** + (`bazel test //tests/regression:regression_test --config=doltserver-cmd`) - [ ] **Every release target cross-compiles** — see [Cross-compile before tagging](#cross-compile-before-tagging). PR CI builds them only with cgo off, and a target that fails at tag time diff --git a/engdocs/RELEASE-STABILITY-GATE.md b/engdocs/RELEASE-STABILITY-GATE.md index af4d4636a..08af1a3ea 100644 --- a/engdocs/RELEASE-STABILITY-GATE.md +++ b/engdocs/RELEASE-STABILITY-GATE.md @@ -49,7 +49,7 @@ requires: - [ ] Upgrade smoke tests pass (`make test-upgrade`) - [ ] Breaking changes documented in CHANGELOG.md with migration steps - [ ] If config/schema migration is involved: recovery steps documented -- [ ] Regression tests pass (`make test-regression`) +- [ ] Regression tests pass (`bazel test //tests/regression:regression_test --config=doltserver-cmd`) ## Sign-off diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index d92a75a2e..c8f2cf077 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -96,17 +96,108 @@ a unit test: use the real boundary when the defect could live there. | Need | Run | When | |---|---|---| | Docs-only validation | `git diff --check` and `make check-docs` | For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full suite merely because a Markdown file changed. | -| Focused red/green loop | `bazel test //path/to/package:package_test --config=ci --test_filter='^TestExactName$'`, or the plain-Go `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...` | While writing or fixing one behavior. The `go test` loop is fine here; the Bazel targets below are the gate. | +| Focused red/green loop | `bazel test //path/to/package:package_test --config=ci --test_filter='^TestExactName$' --test_output=all` | While writing or fixing one behavior. Agents use only this form; an unchanged target returns its cached result. The run counts only when its log shows `--- PASS: TestExactName` and no `--- SKIP` for it or a subtest (see "Check That the Test Ran" below). Humans working offline may use the plain-Go `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...`, which is not the gate. | | Affected-package confidence | `bazel test //path/to/package/... --config=ci` | After the focused test passes; include directly affected neighbors when their contract changed. | | Final gate | `make test` | Once after focused work on Go code is green: the whole test lane, mostly cache hits. | | Another lane's risk | that lane's command from the table above | When the change touches what that lane covers (integration-tagged files, the Dolt server path, embedded Dolt, pure-Go builds). | -| Named CI wrapper | `make ci-pr-core` or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. | +| Named CI wrapper | `make ci-pr-lint`, or `make ci-pr-core` (plain `go test`, so not for agents: `make test` runs PR Core's selection) | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run them routinely for every edit. | | Hook shims against real timeout implementations | `bazel test //tests/hook_timeout_backends:hook_timeout_backends_test` | After changing the hook generator in `cmd/bd/hooks.go` (then `make githooks-regen`) or anything under `.githooks/`. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox `timeout` — the multicalls also installed as `gtimeout` alone — with and without Perl, under dash, bash and busybox ash (240 cases). About one deadline of wall time; needs no Go build. The PR-core lane runs it on every PR. | Do not replace the focused loop with repeated full-suite runs. Run the final `make test` once the affected tests are green. For docs-only changes, use the docs, link, and diff checks instead. +### Check That the Test Ran + +Bazel reports a target as passed when the test it was asked for never ran: +a `--test_filter` that matches nothing, a test that skips, and a test file +the configuration does not compile all end in `1 test passes`. With +`--test_output=all` the log lists each test and subtest (the lane configs set +`GO_TEST_WRAP_TESTV=1`, and a cached result prints its log too), so count a +run only when it shows `--- PASS: TestExactName` and no `--- SKIP` for the +test or a `TestExactName/...` subtest: Go prints `--- PASS` for a test whose +every subtest skipped. When a subtest skips by design (one for another +platform, say), filter on the subtests that cover your change +(`--test_filter='^TestExactName$/^SubtestName$'`). A sharded target runs the +test in one shard, and its other shards print `testing: warning: no tests to +run`; that warning from every shard means the test did not run. Under +`--config=ci` these tests do not run: + +- **Dolt-backed tests** skip (`BEADS_TEST_SKIP=dolt`). Clear the skip and + select the `local` backend, or run the test's Dolt lane from the table + above: + + ```bash + bazel test //internal/storage/dolt:dolt_test --config=ci --test_filter='^TestExactName$' --test_output=all \ + --test_env=BEADS_TEST_SKIP= --test_env=BEADS_TEST_DOLT_SERVER=local --test_env=BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 + ``` + +- **Tests gated on `testing.Short()`** skip (`-test.short`). Add + `--test_arg=-test.short=false`. +- **Tests in `integration`-tagged files** are not compiled. Use + `--config=integration` in place of `--config=ci`; it skips Dolt-backed + tests too. An integration-tagged Dolt test of cmd/bd, tests/regression or + internal/storage/dolt runs in that package's Dolt-server lane, whose target + sets the environment the suite needs: `//cmd/bd:bd_dolt_server_test` and + `//tests/regression:regression_test` with `--config=doltserver-cmd`, + `//internal/storage/dolt:dolt_server_full_test` with + `--config=doltserver-integration`; select the test as described below. In + any other package no lane runs such a test with Dolt: add the Dolt flags + above to `--config=integration`, and when it fails, check whether it fails + the same way on the base commit before you blame your change. +- **`TestEmbedded*` and embedded-Dolt tests**: `--config=ci` skips + `TestEmbedded*` by name (`-test.skip=^TestEmbedded`), so add + `--test_arg=-test.skip=`. A test that needs embedded Dolt, whatever its + name, still skips unless `BEADS_TEST_EMBEDDED_DOLT=1` (its `--- SKIP` says + so), which only the embedded lane's targets set: run it in its package's + `embedded`-tagged target with `--config=embedded`. For one cmd/bd test: + + ```bash + bazel test //cmd/bd:bd_embedded_test --config=embedded --test_sharding_strategy=disabled \ + --test_arg='-test.run=^TestEmbeddedExactName$' --test_output=all + ``` + + For one internal/storage/embeddeddolt test, name + `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` instead. +- **Tests in a file `BUILD.bazel` does not list** are not compiled. Run + `make bazel-sync` after adding a test file. + +Lane targets that are `sh_test`s run the test binary through a wrapper or +shard script, which can pass its own `-test.run` and so override +`--test_filter`. Select the test with a trailing +`--test_arg='-test.run=^TestExactName$'` instead (the last one wins), with +`--test_sharding_strategy=disabled` so that it runs once. + +A test can need more than one of these (a `-short`-gated Dolt test needs the +Dolt flags and `--test_arg=-test.short=false`); combine them and check the +log again. A package's main test target is `//:_test` +(the root package's is `//:beads_test`). A package can split its test files +over more targets (`//scripts` does), so run the target whose `srcs` in +`BUILD.bazel` list the test's file; the lanes in the table above add variants +of some targets. One `--test_filter` applies to every target an invocation +names. + +### Regenerating Golden Files + +Golden-file tests compare output with files checked into the repository and +rewrite them in an update mode: + +- `make corpus-regen`: the `--json` contract corpus under + `cmd/bd/protocol/testdata/corpus/`. +- `make githooks-regen`: the managed sections of the tracked `.githooks/*`. +- The `BD_UPDATE_GOLDEN=1 … go test` command in `cmd/bd/AGENTS.md`: the + command-tree and `--help` goldens. +- Any other golden test: a `go test` run of its package and test with the + update switch its failure message names (`BD_UPDATE_GOLDEN=1`, + `-update-ledger-doc`). + +These are plain `go test` runs, and the one place agents run `go test`: +under Bazel a test runs from its runfiles tree, not your checkout, so it +cannot rewrite the files, and `bazel test` of the same test only reports the +drift. Review the regenerated diff, then confirm with `bazel test`. Where an +agent host refuses `go test` even for these, leave the regeneration to a +maintainer and say so in the PR. + ### Git Hooks `make install` points `core.hooksPath` at `.githooks`. The pre-commit hook @@ -121,11 +212,15 @@ the suite CI gates on. ## Plain `go test` (Inner Loop) -`./scripts/test.sh` is the plain-Go runner behind `make test-go`. It sources -`.buildflags`, creates an isolated test environment, applies `.test-skip`, and -defaults to a **25m per-package** Go-test timeout. The timeout is a hang -backstop, not a target runtime. Override it only when diagnosing a legitimate -slow path: +`./scripts/test.sh` is the plain-Go runner behind `make test-go`. It and +this section's other plain-`go test` targets are for humans working offline; +agents run `bazel test` (see "Choose the Smallest Useful Test" above). The +rules in "Test Environment and Readiness" below (the Dolt backends and their +Bazel lanes, test fixtures, `testing.Short()`) apply to every test run. The +runner sources `.buildflags`, creates an isolated test environment, applies +`.test-skip`, and defaults to a **25m per-package** Go-test timeout. The +timeout is a hang backstop, not a target runtime. Override it only when +diagnosing a legitimate slow path: ```bash TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/... @@ -199,7 +294,8 @@ Under Bazel, `bazel test //... --config=doltserver` runs the Dolt-backed domain, uow, tracker, doctor/fix, protocol and testutil suites on the `local` backend (the `dolt-server` targets); they need no docker and execute remotely with `--config=remote-exec`. No Bazel target uses the `container` -backend; run `go test` with docker and the pulled image to exercise it. +backend; only a human's plain `go test` with docker and the pulled image +exercises it. PR Risk's heavier server tiers have configs of their own, run by bazel.yml only with remote execution, each in a job of its own (`bazel-proxied`, which in mode `remote` is folded with `bazel-doltserver` into