docs(agents): bazel test is authoritative; agents never run go test (#7695)

* docs(agents): bazel test is authoritative; agents never run go test

Agents run tests only with `bazel test`, also while iterating on one
behavior: AGENTS.md shows the focused form
(`bazel test //cmd/bd:bd_test --config=ci --test_filter='^TestName$'`), the
target naming, that one --test_filter covers every target, that a cached
result for unchanged inputs is the same evidence, and that a new test file
needs `make bazel-sync` before Bazel runs it. `go test` (`./scripts/test.sh`,
`make test-go`) stays a convenience for humans working offline. TESTING.md's
focused red/green row and its plain-go-test section say the same.

Why: on the maintainer-city factory host, seats' own uncached `go test`
reruns of the same packages are the largest tool cost in adopt-pr, while
`bazel test` there executes on rbe-west and returns an unchanged target's
cached result in seconds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(agents): check that a focused test ran; exempt golden regeneration

Review fixes for #7695 (iterations 1 and 2).

The focused form exits 0 with "1 test passes" when the test never ran: a
filter that matches nothing, an integration-tagged file --config=ci does not
compile, the TestEmbedded* name skip, a -short skip, and a parent test whose
every subtest skipped (Go still prints "--- PASS" for the parent). It now
adds --test_output=all, and a run counts only when its log shows
"--- PASS: <TestName>" and no "--- SKIP" for the test or a <TestName>/...
subtest; a subtest that skips by design is left out with a subtest filter.

engdocs/TESTING.md gains "Check That the Test Ran"; AGENTS.md links there
instead of restating it. The section names the tests --config=ci does not
run and the way into each:
- Dolt-backed: the three --test_env flags, or the test's Dolt lane.
- -short-gated: --test_arg=-test.short=false.
- integration-tagged: --config=integration; the Dolt-backed ones of cmd/bd,
  tests/regression and internal/storage/dolt run in their Dolt-server lane
  target, whose env the suite needs. No lane runs those of other packages
  with Dolt, so a failure there is compared with the base commit first.
- TestEmbedded*: --test_arg=-test.skip= lifts the name skip; a test that
  then still skips for BEADS_TEST_EMBEDDED_DOLT=1 runs in its package's
  embedded-tagged target under --config=embedded (the cmd/bd and
  internal/storage/embeddeddolt targets are named).
- files BUILD.bazel does not list: make bazel-sync.
A lane's sh_test can hand the test binary its own -test.run, which
overrides --test_filter, so a trailing --test_arg=-test.run with sharding
disabled selects the test there. In a sharded target the other shards print
"testing: warning: no tests to run". The section also gives the
target-naming rule.

Golden regeneration (make corpus-regen, make githooks-regen,
BD_UPDATE_GOLDEN=1 ... go test, -update-ledger-doc) writes the source tree,
which a Bazel test cannot do: it runs from its runfiles tree, and on
rbe-west update mode fails with "permission denied". AGENTS.md and the new
"Regenerating Golden Files" section exempt these go test runs; where a host
refuses go test even for them, a maintainer regenerates. The factory seat
shim (gastownhall/workflows#102, scripts/seat-bin/go) still refuses update
mode and needs the same carve-out there.

Also: the plain-go-test opener scopes "humans working offline" to the
runner and its go test targets, so the Dolt-backend, fixture and
testing.Short() rules still bind agents; make ci-pr-core is marked plain go
test; the beads-docs skill's docsync gate is the Bazel target; RELEASING.md
and engdocs/RELEASE-STABILITY-GATE.md name the regression suite's Bazel
target (//tests/regression:regression_test under --config=doltserver-cmd)
in place of make test-regression, which is plain go test.

Measured on rbe-west (exit 0 unless noted). Iteration 1, at c4d5409895:
- a mistyped filter, and //internal/compact TestNew (integration-tagged)
  under --config=ci: "testing: warning: no tests to run"
- TestNew under --config=integration: "--- SKIP" (BEADS_TEST_SKIP=dolt);
  with the Dolt flags added it runs (and fails, exit 3, on the local
  backend: its store finds no "beads" database; outside this PR)
- TestConcurrentHeartbeatReclaimClose with the Dolt flags: "--- SKIP"
  (short mode); plus --test_arg=-test.short=false: "--- PASS"
- //cmd/bd:bd_embedded_test, sharding disabled: --test_filter alone ran
  TestEmbeddedUpdateConcurrent (manifest shard 1's test); the trailing
  -test.run ran "--- PASS: TestEmbeddedShowBasicsAndJSON"
- //scripts:go_test_sources_test
  TestEmbeddedCmdShardManifestBareWriteIsIdempotent: --config=ci "no tests
  to run"; plus --test_arg=-test.skip=: "--- PASS"
Iteration 2, at b1d25543aa (the same Go inputs):
- //internal/storage/dolt:dolt_test TestMoveIssuePersistenceInTxTransitions
  under --config=ci: "--- PASS" for the parent and "--- SKIP" for all 6
  subtests (BEADS_TEST_SKIP=dolt), "1 test passes"; with the Dolt flags and
  --test_filter='^TestMoveIssuePersistenceInTxTransitions$/^persistent-to-ephemeral$'
  "--- PASS" for the parent and that subtest only, no "--- SKIP"
- //internal/storage/embeddeddolt:embeddeddolt_test
  TestEmbeddedRemoteMigrateGate_BlocksReopen under --config=ci plus
  --test_arg=-test.skip=: "--- SKIP" (BEADS_TEST_EMBEDDED_DOLT unset);
  //internal/storage/embeddeddolt:embeddeddolt_embedded_test under
  --config=embedded, sharding disabled, trailing -test.run: "--- PASS"
  (the shard script listed 433 tests; only the named one ran)
- --config=ci plus --test_arg=-test.skip=: "--- PASS" for //tools/nogo
  TestEmbeddedConfigIsTheRepositoryConfig and //internal/tracker
  TestEmbeddedDoltExternalRefChangedAfterUsesHistoryFastPath
- sharding disabled, trailing -test.run: "--- PASS" for
  //internal/storage/dolt:dolt_server_full_test
  TestWispDeleteCascade_TransactionDeleteIssue under
  --config=doltserver-integration, and for //cmd/bd:bd_dolt_server_test
  TestCLI_ReopenClearsDeferUntil under --config=doltserver-cmd
- make check-docs: exit 0 (docsync: 10 test cases ran, none skipped)

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
gascity-eddie[bot] 2026-10-19 14:04:30 +00:00 • committed by GitHub
commit 4f9174e6ba
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 136 additions and 17 deletions

View file

@ -201,7 +201,8 @@ document what the pinned release does, never main-only features. See
## 9. Verify before you call it done
Run the gates in [references/verification.md](references/verification.md).
The short list: `go test ./test/docsync` (nav↔file sync + link conventions),
The short list: `bazel test //test/docsync:docsync_test --config=ci`
(nav↔file sync + link conventions),
`./scripts/generate-cli-docs.sh --check` (generated docs fresh),
`./scripts/check-doc-freshness.sh` (Last reviewed markers), and a live
preview with `make docs-dev` (or `./mint.sh dev`) at `localhost:3000`.

View file

@ -39,8 +39,9 @@ don't measure that work by the word delta.
*not* simplifying — it buries the real change in the diff.
4. **Loss-check** (below).
5. **Fact-check** (below).
6. **Gates:** `go test ./test/docsync`, every fence parses, no body H1, no
HTML comments (MDX), preview with `make docs-dev`.
6. **Gates:** `bazel test //test/docsync:docsync_test --config=ci`, every
fence parses, no body H1, no HTML comments (MDX), preview with
`make docs-dev`.
7. **Preview, then commit on approval** — one page or section per batch, so
the reviewer can steer before a cut compounds across the section.

View file

@ -8,7 +8,7 @@ Run these before considering docs work complete. Ordered cheapest-first.
# 1. Docs sync: docs.json nav <-> file consistency, link conventions
# (root-relative extensionless in docs/, exact paths in engdocs/ and
# curated root files), orphan detection (only CLI_REFERENCE.md is exempt).
go test ./test/docsync
bazel test //test/docsync:docsync_test --config=ci
# 2. Generated CLI docs freshness: regenerates from the live command tree
# and diffs docs/CLI_REFERENCE.md, docs/cli-reference/, and the CLI

View file

@ -126,9 +126,29 @@ make install # build and install bd to ~/.local/bin (canonical)
certificate `--config=remote-exec`, agent hosts whose `~/.bazelrc` names the
executor neither. Make yours the default with a `build --config=...` line in
the gitignored `.bazelrc.local`, or pass `BAZEL_FLAGS=...` to make.
- `go test` (`./scripts/test.sh`, `make test-go`) is an inner-loop
convenience only. CI does not enforce it, and it skips nogo, gofmt and the
guards; finish with `make test`.
- Agents run tests only with `bazel test`, also while iterating on one
behavior:
```bash
bazel test //cmd/bd:bd_test --config=ci --test_filter='^TestName$' --test_output=all
```
A target also passes when the test never ran (a filter that matches
nothing, a skip, a file the configuration does not compile), so count a run
only when its log shows `--- PASS: TestName` and no `--- SKIP` for the test
or a `TestName/...` subtest: Go prints `--- PASS` for a test whose every
subtest skipped. "Check That the Test Ran" in
[engdocs/TESTING.md](engdocs/TESTING.md) names the tests `--config=ci` does
not run (Dolt-backed, `-short`-gated, integration-tagged, `TestEmbedded*`
and embedded-Dolt, files `BUILD.bazel` does not list) and the way into
each. An unchanged target returns its cached result, which is the same
evidence. Finish with `make test`.
- Agents do not run `go test` (`./scripts/test.sh`, `make test-go`): CI does
not enforce it, it skips nogo, gofmt and the guards, and agent hosts may
refuse it. It stays a convenience for humans working offline. The exception
is regenerating golden files (`make corpus-regen`, `make githooks-regen`,
`BD_UPDATE_GOLDEN=1`), which Bazel cannot do: see "Regenerating Golden
Files" in [engdocs/TESTING.md](engdocs/TESTING.md).
- **Do NOT** use `go build -o bd ./cmd/bd`, `go install ./cmd/bd`, or raw
`go run ./cmd/bd ...`: they bypass the canonical build path, leave stale
binaries, and raw `go run` misses the `gms_pure_go` tag. Use `make install`,

View file

@ -145,7 +145,8 @@ Before starting a release:
- [ ] All tests passing (`make test`, the Bazel test lane)
- [ ] npm package tests passing (`cd npm-package && npm run test:all`)
- [ ] **Upgrade smoke tests pass** (`make test-upgrade`) — see [Release Stability Gate](engdocs/RELEASE-STABILITY-GATE.md)
- [ ] **Regression tests pass** (`make test-regression`)
- [ ] **Regression tests pass**
(`bazel test //tests/regression:regression_test --config=doltserver-cmd`)
- [ ] **Every release target cross-compiles** — see
[Cross-compile before tagging](#cross-compile-before-tagging). PR CI
builds them only with cgo off, and a target that fails at tag time

View file

@ -49,7 +49,7 @@ requires:
- [ ] Upgrade smoke tests pass (`make test-upgrade`)
- [ ] Breaking changes documented in CHANGELOG.md with migration steps
- [ ] If config/schema migration is involved: recovery steps documented
- [ ] Regression tests pass (`make test-regression`)
- [ ] Regression tests pass (`bazel test //tests/regression:regression_test --config=doltserver-cmd`)
## Sign-off

View file

@ -96,17 +96,108 @@ a unit test: use the real boundary when the defect could live there.
| Need | Run | When |
|---|---|---|
| Docs-only validation | `git diff --check` and `make check-docs` | For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full suite merely because a Markdown file changed. |
| Focused red/green loop | `bazel test //path/to/package:package_test --config=ci --test_filter='^TestExactName$'`, or the plain-Go `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...` | While writing or fixing one behavior. The `go test` loop is fine here; the Bazel targets below are the gate. |
| Focused red/green loop | `bazel test //path/to/package:package_test --config=ci --test_filter='^TestExactName$' --test_output=all` | While writing or fixing one behavior. Agents use only this form; an unchanged target returns its cached result. The run counts only when its log shows `--- PASS: TestExactName` and no `--- SKIP` for it or a subtest (see "Check That the Test Ran" below). Humans working offline may use the plain-Go `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...`, which is not the gate. |
| Affected-package confidence | `bazel test //path/to/package/... --config=ci` | After the focused test passes; include directly affected neighbors when their contract changed. |
| Final gate | `make test` | Once after focused work on Go code is green: the whole test lane, mostly cache hits. |
| Another lane's risk | that lane's command from the table above | When the change touches what that lane covers (integration-tagged files, the Dolt server path, embedded Dolt, pure-Go builds). |
| Named CI wrapper | `make ci-pr-core` or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. |
| Named CI wrapper | `make ci-pr-lint`, or `make ci-pr-core` (plain `go test`, so not for agents: `make test` runs PR Core's selection) | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run them routinely for every edit. |
| Hook shims against real timeout implementations | `bazel test //tests/hook_timeout_backends:hook_timeout_backends_test` | After changing the hook generator in `cmd/bd/hooks.go` (then `make githooks-regen`) or anything under `.githooks/`. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox `timeout` — the multicalls also installed as `gtimeout` alone — with and without Perl, under dash, bash and busybox ash (240 cases). About one deadline of wall time; needs no Go build. The PR-core lane runs it on every PR. |
Do not replace the focused loop with repeated full-suite runs. Run the final
`make test` once the affected tests are green. For docs-only changes, use
the docs, link, and diff checks instead.
### Check That the Test Ran
Bazel reports a target as passed when the test it was asked for never ran:
a `--test_filter` that matches nothing, a test that skips, and a test file
the configuration does not compile all end in `1 test passes`. With
`--test_output=all` the log lists each test and subtest (the lane configs set
`GO_TEST_WRAP_TESTV=1`, and a cached result prints its log too), so count a
run only when it shows `--- PASS: TestExactName` and no `--- SKIP` for the
test or a `TestExactName/...` subtest: Go prints `--- PASS` for a test whose
every subtest skipped. When a subtest skips by design (one for another
platform, say), filter on the subtests that cover your change
(`--test_filter='^TestExactName$/^SubtestName$'`). A sharded target runs the
test in one shard, and its other shards print `testing: warning: no tests to
run`; that warning from every shard means the test did not run. Under
`--config=ci` these tests do not run:
- **Dolt-backed tests** skip (`BEADS_TEST_SKIP=dolt`). Clear the skip and
select the `local` backend, or run the test's Dolt lane from the table
above:
```bash
bazel test //internal/storage/dolt:dolt_test --config=ci --test_filter='^TestExactName$' --test_output=all \
--test_env=BEADS_TEST_SKIP= --test_env=BEADS_TEST_DOLT_SERVER=local --test_env=BEADS_TEST_REQUIRE_DOLT_CONTAINER=1
```
- **Tests gated on `testing.Short()`** skip (`-test.short`). Add
`--test_arg=-test.short=false`.
- **Tests in `integration`-tagged files** are not compiled. Use
`--config=integration` in place of `--config=ci`; it skips Dolt-backed
tests too. An integration-tagged Dolt test of cmd/bd, tests/regression or
internal/storage/dolt runs in that package's Dolt-server lane, whose target
sets the environment the suite needs: `//cmd/bd:bd_dolt_server_test` and
`//tests/regression:regression_test` with `--config=doltserver-cmd`,
`//internal/storage/dolt:dolt_server_full_test` with
`--config=doltserver-integration`; select the test as described below. In
any other package no lane runs such a test with Dolt: add the Dolt flags
above to `--config=integration`, and when it fails, check whether it fails
the same way on the base commit before you blame your change.
- **`TestEmbedded*` and embedded-Dolt tests**: `--config=ci` skips
`TestEmbedded*` by name (`-test.skip=^TestEmbedded`), so add
`--test_arg=-test.skip=`. A test that needs embedded Dolt, whatever its
name, still skips unless `BEADS_TEST_EMBEDDED_DOLT=1` (its `--- SKIP` says
so), which only the embedded lane's targets set: run it in its package's
`embedded`-tagged target with `--config=embedded`. For one cmd/bd test:
```bash
bazel test //cmd/bd:bd_embedded_test --config=embedded --test_sharding_strategy=disabled \
--test_arg='-test.run=^TestEmbeddedExactName$' --test_output=all
```
For one internal/storage/embeddeddolt test, name
`//internal/storage/embeddeddolt:embeddeddolt_embedded_test` instead.
- **Tests in a file `BUILD.bazel` does not list** are not compiled. Run
`make bazel-sync` after adding a test file.
Lane targets that are `sh_test`s run the test binary through a wrapper or
shard script, which can pass its own `-test.run` and so override
`--test_filter`. Select the test with a trailing
`--test_arg='-test.run=^TestExactName$'` instead (the last one wins), with
`--test_sharding_strategy=disabled` so that it runs once.
A test can need more than one of these (a `-short`-gated Dolt test needs the
Dolt flags and `--test_arg=-test.short=false`); combine them and check the
log again. A package's main test target is `//<dir>:<last dir element>_test`
(the root package's is `//:beads_test`). A package can split its test files
over more targets (`//scripts` does), so run the target whose `srcs` in
`BUILD.bazel` list the test's file; the lanes in the table above add variants
of some targets. One `--test_filter` applies to every target an invocation
names.
### Regenerating Golden Files
Golden-file tests compare output with files checked into the repository and
rewrite them in an update mode:
- `make corpus-regen`: the `--json` contract corpus under
`cmd/bd/protocol/testdata/corpus/`.
- `make githooks-regen`: the managed sections of the tracked `.githooks/*`.
- The `BD_UPDATE_GOLDEN=1 … go test` command in `cmd/bd/AGENTS.md`: the
command-tree and `--help` goldens.
- Any other golden test: a `go test` run of its package and test with the
update switch its failure message names (`BD_UPDATE_GOLDEN=1`,
`-update-ledger-doc`).
These are plain `go test` runs, and the one place agents run `go test`:
under Bazel a test runs from its runfiles tree, not your checkout, so it
cannot rewrite the files, and `bazel test` of the same test only reports the
drift. Review the regenerated diff, then confirm with `bazel test`. Where an
agent host refuses `go test` even for these, leave the regeneration to a
maintainer and say so in the PR.
### Git Hooks
`make install` points `core.hooksPath` at `.githooks`. The pre-commit hook
@ -121,11 +212,15 @@ the suite CI gates on.
## Plain `go test` (Inner Loop)
`./scripts/test.sh` is the plain-Go runner behind `make test-go`. It sources
`.buildflags`, creates an isolated test environment, applies `.test-skip`, and
defaults to a **25m per-package** Go-test timeout. The timeout is a hang
backstop, not a target runtime. Override it only when diagnosing a legitimate
slow path:
`./scripts/test.sh` is the plain-Go runner behind `make test-go`. It and
this section's other plain-`go test` targets are for humans working offline;
agents run `bazel test` (see "Choose the Smallest Useful Test" above). The
rules in "Test Environment and Readiness" below (the Dolt backends and their
Bazel lanes, test fixtures, `testing.Short()`) apply to every test run. The
runner sources `.buildflags`, creates an isolated test environment, applies
`.test-skip`, and defaults to a **25m per-package** Go-test timeout. The
timeout is a hang backstop, not a target runtime. Override it only when
diagnosing a legitimate slow path:
```bash
TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/...
@ -199,7 +294,8 @@ Under Bazel, `bazel test //... --config=doltserver` runs the Dolt-backed
domain, uow, tracker, doctor/fix, protocol and testutil suites on the `local`
backend (the `dolt-server` targets); they need no docker and execute
remotely with `--config=remote-exec`. No Bazel target uses the `container`
backend; run `go test` with docker and the pulled image to exercise it.
backend; only a human's plain `go test` with docker and the pulled image
exercises it.
PR Risk's heavier server tiers have configs of their own, run by bazel.yml
only with remote execution, each in a job of its own (`bazel-proxied`,
which in mode `remote` is folded with `bazel-doltserver` into