diff --git a/.claude/skills/beads-docs/SKILL.md b/.claude/skills/beads-docs/SKILL.md index 52bab8362..a5454d2ea 100644 --- a/.claude/skills/beads-docs/SKILL.md +++ b/.claude/skills/beads-docs/SKILL.md @@ -201,7 +201,8 @@ document what the pinned release does, never main-only features. See ## 9. Verify before you call it done Run the gates in [references/verification.md](references/verification.md). -The short list: `go test ./test/docsync` (nav↔file sync + link conventions), +The short list: `bazel test //test/docsync:docsync_test --config=ci` +(nav↔file sync + link conventions), `./scripts/generate-cli-docs.sh --check` (generated docs fresh), `./scripts/check-doc-freshness.sh` (Last reviewed markers), and a live preview with `make docs-dev` (or `./mint.sh dev`) at `localhost:3000`. diff --git a/.claude/skills/beads-docs/references/simplification.md b/.claude/skills/beads-docs/references/simplification.md index 5152fd2b5..916834ca5 100644 --- a/.claude/skills/beads-docs/references/simplification.md +++ b/.claude/skills/beads-docs/references/simplification.md @@ -39,8 +39,9 @@ don't measure that work by the word delta. *not* simplifying — it buries the real change in the diff. 4. **Loss-check** (below). 5. **Fact-check** (below). -6. **Gates:** `go test ./test/docsync`, every fence parses, no body H1, no - HTML comments (MDX), preview with `make docs-dev`. +6. **Gates:** `bazel test //test/docsync:docsync_test --config=ci`, every + fence parses, no body H1, no HTML comments (MDX), preview with + `make docs-dev`. 7. **Preview, then commit on approval** — one page or section per batch, so the reviewer can steer before a cut compounds across the section. diff --git a/.claude/skills/beads-docs/references/verification.md b/.claude/skills/beads-docs/references/verification.md index 559bfb6f3..44aabca76 100644 --- a/.claude/skills/beads-docs/references/verification.md +++ b/.claude/skills/beads-docs/references/verification.md @@ -8,7 +8,7 @@ Run these before considering docs work complete. Ordered cheapest-first. # 1. Docs sync: docs.json nav <-> file consistency, link conventions # (root-relative extensionless in docs/, exact paths in engdocs/ and # curated root files), orphan detection (only CLI_REFERENCE.md is exempt). -go test ./test/docsync +bazel test //test/docsync:docsync_test --config=ci # 2. Generated CLI docs freshness: regenerates from the live command tree # and diffs docs/CLI_REFERENCE.md, docs/cli-reference/, and the CLI diff --git a/AGENTS.md b/AGENTS.md index 1a450309e..e908a5070 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -126,9 +126,29 @@ make install # build and install bd to ~/.local/bin (canonical) certificate `--config=remote-exec`, agent hosts whose `~/.bazelrc` names the executor neither. Make yours the default with a `build --config=...` line in the gitignored `.bazelrc.local`, or pass `BAZEL_FLAGS=...` to make. -- `go test` (`./scripts/test.sh`, `make test-go`) is an inner-loop - convenience only. CI does not enforce it, and it skips nogo, gofmt and the - guards; finish with `make test`. +- Agents run tests only with `bazel test`, also while iterating on one + behavior: + + ```bash + bazel test //cmd/bd:bd_test --config=ci --test_filter='^TestName$' --test_output=all + ``` + + A target also passes when the test never ran (a filter that matches + nothing, a skip, a file the configuration does not compile), so count a run + only when its log shows `--- PASS: TestName` and no `--- SKIP` for the test + or a `TestName/...` subtest: Go prints `--- PASS` for a test whose every + subtest skipped. "Check That the Test Ran" in + [engdocs/TESTING.md](engdocs/TESTING.md) names the tests `--config=ci` does + not run (Dolt-backed, `-short`-gated, integration-tagged, `TestEmbedded*` + and embedded-Dolt, files `BUILD.bazel` does not list) and the way into + each. An unchanged target returns its cached result, which is the same + evidence. Finish with `make test`. +- Agents do not run `go test` (`./scripts/test.sh`, `make test-go`): CI does + not enforce it, it skips nogo, gofmt and the guards, and agent hosts may + refuse it. It stays a convenience for humans working offline. The exception + is regenerating golden files (`make corpus-regen`, `make githooks-regen`, + `BD_UPDATE_GOLDEN=1`), which Bazel cannot do: see "Regenerating Golden + Files" in [engdocs/TESTING.md](engdocs/TESTING.md). - **Do NOT** use `go build -o bd ./cmd/bd`, `go install ./cmd/bd`, or raw `go run ./cmd/bd ...`: they bypass the canonical build path, leave stale binaries, and raw `go run` misses the `gms_pure_go` tag. Use `make install`, diff --git a/RELEASING.md b/RELEASING.md index e05b73072..50acb6497 100644 --- a/RELEASING.md +++ b/RELEASING.md @@ -145,7 +145,8 @@ Before starting a release: - [ ] All tests passing (`make test`, the Bazel test lane) - [ ] npm package tests passing (`cd npm-package && npm run test:all`) - [ ] **Upgrade smoke tests pass** (`make test-upgrade`) — see [Release Stability Gate](engdocs/RELEASE-STABILITY-GATE.md) -- [ ] **Regression tests pass** (`make test-regression`) +- [ ] **Regression tests pass** + (`bazel test //tests/regression:regression_test --config=doltserver-cmd`) - [ ] **Every release target cross-compiles** — see [Cross-compile before tagging](#cross-compile-before-tagging). PR CI builds them only with cgo off, and a target that fails at tag time diff --git a/engdocs/RELEASE-STABILITY-GATE.md b/engdocs/RELEASE-STABILITY-GATE.md index af4d4636a..08af1a3ea 100644 --- a/engdocs/RELEASE-STABILITY-GATE.md +++ b/engdocs/RELEASE-STABILITY-GATE.md @@ -49,7 +49,7 @@ requires: - [ ] Upgrade smoke tests pass (`make test-upgrade`) - [ ] Breaking changes documented in CHANGELOG.md with migration steps - [ ] If config/schema migration is involved: recovery steps documented -- [ ] Regression tests pass (`make test-regression`) +- [ ] Regression tests pass (`bazel test //tests/regression:regression_test --config=doltserver-cmd`) ## Sign-off diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index d92a75a2e..c8f2cf077 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -96,17 +96,108 @@ a unit test: use the real boundary when the defect could live there. | Need | Run | When | |---|---|---| | Docs-only validation | `git diff --check` and `make check-docs` | For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full suite merely because a Markdown file changed. | -| Focused red/green loop | `bazel test //path/to/package:package_test --config=ci --test_filter='^TestExactName$'`, or the plain-Go `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...` | While writing or fixing one behavior. The `go test` loop is fine here; the Bazel targets below are the gate. | +| Focused red/green loop | `bazel test //path/to/package:package_test --config=ci --test_filter='^TestExactName$' --test_output=all` | While writing or fixing one behavior. Agents use only this form; an unchanged target returns its cached result. The run counts only when its log shows `--- PASS: TestExactName` and no `--- SKIP` for it or a subtest (see "Check That the Test Ran" below). Humans working offline may use the plain-Go `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...`, which is not the gate. | | Affected-package confidence | `bazel test //path/to/package/... --config=ci` | After the focused test passes; include directly affected neighbors when their contract changed. | | Final gate | `make test` | Once after focused work on Go code is green: the whole test lane, mostly cache hits. | | Another lane's risk | that lane's command from the table above | When the change touches what that lane covers (integration-tagged files, the Dolt server path, embedded Dolt, pure-Go builds). | -| Named CI wrapper | `make ci-pr-core` or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. | +| Named CI wrapper | `make ci-pr-lint`, or `make ci-pr-core` (plain `go test`, so not for agents: `make test` runs PR Core's selection) | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run them routinely for every edit. | | Hook shims against real timeout implementations | `bazel test //tests/hook_timeout_backends:hook_timeout_backends_test` | After changing the hook generator in `cmd/bd/hooks.go` (then `make githooks-regen`) or anything under `.githooks/`. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox `timeout` — the multicalls also installed as `gtimeout` alone — with and without Perl, under dash, bash and busybox ash (240 cases). About one deadline of wall time; needs no Go build. The PR-core lane runs it on every PR. | Do not replace the focused loop with repeated full-suite runs. Run the final `make test` once the affected tests are green. For docs-only changes, use the docs, link, and diff checks instead. +### Check That the Test Ran + +Bazel reports a target as passed when the test it was asked for never ran: +a `--test_filter` that matches nothing, a test that skips, and a test file +the configuration does not compile all end in `1 test passes`. With +`--test_output=all` the log lists each test and subtest (the lane configs set +`GO_TEST_WRAP_TESTV=1`, and a cached result prints its log too), so count a +run only when it shows `--- PASS: TestExactName` and no `--- SKIP` for the +test or a `TestExactName/...` subtest: Go prints `--- PASS` for a test whose +every subtest skipped. When a subtest skips by design (one for another +platform, say), filter on the subtests that cover your change +(`--test_filter='^TestExactName$/^SubtestName$'`). A sharded target runs the +test in one shard, and its other shards print `testing: warning: no tests to +run`; that warning from every shard means the test did not run. Under +`--config=ci` these tests do not run: + +- **Dolt-backed tests** skip (`BEADS_TEST_SKIP=dolt`). Clear the skip and + select the `local` backend, or run the test's Dolt lane from the table + above: + + ```bash + bazel test //internal/storage/dolt:dolt_test --config=ci --test_filter='^TestExactName$' --test_output=all \ + --test_env=BEADS_TEST_SKIP= --test_env=BEADS_TEST_DOLT_SERVER=local --test_env=BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 + ``` + +- **Tests gated on `testing.Short()`** skip (`-test.short`). Add + `--test_arg=-test.short=false`. +- **Tests in `integration`-tagged files** are not compiled. Use + `--config=integration` in place of `--config=ci`; it skips Dolt-backed + tests too. An integration-tagged Dolt test of cmd/bd, tests/regression or + internal/storage/dolt runs in that package's Dolt-server lane, whose target + sets the environment the suite needs: `//cmd/bd:bd_dolt_server_test` and + `//tests/regression:regression_test` with `--config=doltserver-cmd`, + `//internal/storage/dolt:dolt_server_full_test` with + `--config=doltserver-integration`; select the test as described below. In + any other package no lane runs such a test with Dolt: add the Dolt flags + above to `--config=integration`, and when it fails, check whether it fails + the same way on the base commit before you blame your change. +- **`TestEmbedded*` and embedded-Dolt tests**: `--config=ci` skips + `TestEmbedded*` by name (`-test.skip=^TestEmbedded`), so add + `--test_arg=-test.skip=`. A test that needs embedded Dolt, whatever its + name, still skips unless `BEADS_TEST_EMBEDDED_DOLT=1` (its `--- SKIP` says + so), which only the embedded lane's targets set: run it in its package's + `embedded`-tagged target with `--config=embedded`. For one cmd/bd test: + + ```bash + bazel test //cmd/bd:bd_embedded_test --config=embedded --test_sharding_strategy=disabled \ + --test_arg='-test.run=^TestEmbeddedExactName$' --test_output=all + ``` + + For one internal/storage/embeddeddolt test, name + `//internal/storage/embeddeddolt:embeddeddolt_embedded_test` instead. +- **Tests in a file `BUILD.bazel` does not list** are not compiled. Run + `make bazel-sync` after adding a test file. + +Lane targets that are `sh_test`s run the test binary through a wrapper or +shard script, which can pass its own `-test.run` and so override +`--test_filter`. Select the test with a trailing +`--test_arg='-test.run=^TestExactName$'` instead (the last one wins), with +`--test_sharding_strategy=disabled` so that it runs once. + +A test can need more than one of these (a `-short`-gated Dolt test needs the +Dolt flags and `--test_arg=-test.short=false`); combine them and check the +log again. A package's main test target is `//:_test` +(the root package's is `//:beads_test`). A package can split its test files +over more targets (`//scripts` does), so run the target whose `srcs` in +`BUILD.bazel` list the test's file; the lanes in the table above add variants +of some targets. One `--test_filter` applies to every target an invocation +names. + +### Regenerating Golden Files + +Golden-file tests compare output with files checked into the repository and +rewrite them in an update mode: + +- `make corpus-regen`: the `--json` contract corpus under + `cmd/bd/protocol/testdata/corpus/`. +- `make githooks-regen`: the managed sections of the tracked `.githooks/*`. +- The `BD_UPDATE_GOLDEN=1 … go test` command in `cmd/bd/AGENTS.md`: the + command-tree and `--help` goldens. +- Any other golden test: a `go test` run of its package and test with the + update switch its failure message names (`BD_UPDATE_GOLDEN=1`, + `-update-ledger-doc`). + +These are plain `go test` runs, and the one place agents run `go test`: +under Bazel a test runs from its runfiles tree, not your checkout, so it +cannot rewrite the files, and `bazel test` of the same test only reports the +drift. Review the regenerated diff, then confirm with `bazel test`. Where an +agent host refuses `go test` even for these, leave the regeneration to a +maintainer and say so in the PR. + ### Git Hooks `make install` points `core.hooksPath` at `.githooks`. The pre-commit hook @@ -121,11 +212,15 @@ the suite CI gates on. ## Plain `go test` (Inner Loop) -`./scripts/test.sh` is the plain-Go runner behind `make test-go`. It sources -`.buildflags`, creates an isolated test environment, applies `.test-skip`, and -defaults to a **25m per-package** Go-test timeout. The timeout is a hang -backstop, not a target runtime. Override it only when diagnosing a legitimate -slow path: +`./scripts/test.sh` is the plain-Go runner behind `make test-go`. It and +this section's other plain-`go test` targets are for humans working offline; +agents run `bazel test` (see "Choose the Smallest Useful Test" above). The +rules in "Test Environment and Readiness" below (the Dolt backends and their +Bazel lanes, test fixtures, `testing.Short()`) apply to every test run. The +runner sources `.buildflags`, creates an isolated test environment, applies +`.test-skip`, and defaults to a **25m per-package** Go-test timeout. The +timeout is a hang backstop, not a target runtime. Override it only when +diagnosing a legitimate slow path: ```bash TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/... @@ -199,7 +294,8 @@ Under Bazel, `bazel test //... --config=doltserver` runs the Dolt-backed domain, uow, tracker, doctor/fix, protocol and testutil suites on the `local` backend (the `dolt-server` targets); they need no docker and execute remotely with `--config=remote-exec`. No Bazel target uses the `container` -backend; run `go test` with docker and the pulled image to exercise it. +backend; only a human's plain `go test` with docker and the pulled image +exercises it. PR Risk's heavier server tiers have configs of their own, run by bazel.yml only with remote execution, each in a job of its own (`bazel-proxied`, which in mode `remote` is folded with `bazel-doltserver` into