beads/engdocs/TESTING.md
Julian Knutsen e7138e93af
ci(bazel): server-Dolt storage and proxied tiers on remote execution (#6940)
* test(integration): let SubprocessRunner reuse a prebuilt test binary

BEADS_TEST_SUBPROCESS_BINARY hands SubprocessRunner an already-built test
binary of the package instead of running `go test -c`, which no Bazel
action can do offline (it needs the module source tree and module cache).
A Bazel target passes the running go_test itself: the helper tests
(TestHelperSchemaInit, TestHelperMultiStore) live in the same package.
Unset, behavior is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(dolt): keep a lane's REQUIRE switch out of the leak-probe child

TestSuiteFixtureLeakFailsOwningSuite re-runs the suite with
BEADS_TEST_SKIP=dolt on purpose. When the parent runs under
BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 (a fail-closed server lane), the child
inherited it and TestMain exited FATAL before the leak sweep the test
checks. Clear it in the child like the other suite-scoped variables.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(bazel): proxied-server and server-Dolt storage tiers (D6d, D6e)

PR Risk's "Test (Proxied Dolt Cmd N/15)" (also main.yml's) and "Test
(Server Dolt Conformance)" / "Test (Server Dolt Full Suite N/16)" as
Bazel variants against local dolt sql-servers (internal/testutil local
backend, pinned dolt from the test wrapper), fail-closed like the
dolt-server lane:

- //cmd/bd:bd_proxied_test: 15 shards; shard k runs the job's
  proxied-test-shard.sh k+1 15 through go_test_manifest_shard.sh over the
  race bd_test, non-race bd_for_tests as the subprocess bd,
  BEADS_TEST_PROXIED_SERVER=1, GOMAXPROCS=4. Tag dolt-server-proxied,
  --config=doltserver-proxied (the doltserver lane's build).
- //internal/storage/dolt:dolt_server_{conformance,full}_test over
  dolt_race_off (go_test_race_off of dolt_test: the jobs' binary is not
  race); the full suite runs server-storage-test-shard.sh k+1 16 with
  BEADS_TEST_ENV_RUN_DOLT=1 and hands SubprocessRunner its own binary.
  Tag dolt-server-integration, --config=doltserver-integration, whose
  build flags are exactly --config=integration's, so the tier shares that
  lane's compile (race-off deps land where bd_for_tests already is)
  instead of compiling the tagged graph a third time.
- dolt_test gains //internal/storage/schema:migrations as data (the
  PR4107/migration replay tests read ../schema/migrations).
- test_env.sh: BEADS_TEST_GIT_IDENTITY=1 writes the jobs' global git
  identity ("Configure Git"); the git-remote tests commit in fresh repos.
- Both tags join the taxonomy and bazelPRCoreExcludedTags, so prcore/ci
  and integration exclude them.

Per-shard names and statuses equal a replay of the jobs' commands
(container backend) for all 32 jobs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): run the proxied and server storage tiers remotely (advisory)

bazel-doltserver gains a second, remote-only step, `bazel test //...
--config=doltserver-proxied` (15 shards of ~5-10 min with a busy dolt
sql-server each do not fit a GitHub-hosted runner; forks and rbe=off keep
the local lane unchanged). bazel-integration gains `bazel test //...
--config=doltserver-integration`, which shares its build flags, so the
server storage tier compiles nothing the integration step has not. Both
steps write their own BEP, run check_testcases.py on it, run even when the
job's first test step failed, and upload logs on failure.

Policy (TestBazelDoltServerTiersMirrorPRRisk): each variant runs its jobs'
shard script with their shard total and has script + manifest in data;
the conformance variant passes the job's flags; the jobs' race/non-race
binaries and switches (BEADS_TEST_PROXIED_SERVER, BEADS_TEST_ENV_RUN_DOLT)
match; doltserver-integration's tags and race equal --config=integration's;
the steps are pinned to their jobs. checkDoltServerRules now holds all
three dolt-server lane tags to the local-backend, fail-closed,
remote-capable rules.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(bazel): run the proxied and server storage tiers in jobs of their own

Review of D6d/e: as steps of bazel-doltserver and bazel-integration the
tiers inherited those jobs' gating (a caller switching the integration
lane off would drop PR Risk's PR-time server tier; a gate requiring the
dolt-server lane would silently require the proxied tier too) and
lengthened their critical paths.

bazel-proxied (--config=doltserver-proxied, 15 shards) and
bazel-server-storage (--config=doltserver-integration: conformance + 16
shards) are remote-only jobs that need only rbe, each with its own BEP,
check_testcases.py step and log upload, 25-minute test step inside a
30-minute job. bazel-doltserver and bazel-integration are back to their
previous content, including the dolt-server step's 80-minute timeout for
local runs. The server tier shares bazel-integration's compiles through
the remote cache (same build flags).

The proxied tier mirrors PR Risk's race bd-cmd-test; main.yml's twin
job runs build-artifacts' non-race one, so Bazel is the stricter of the
two. Comments now say so, and the policy test pins main.yml's build so a
change there is noticed. Policy: the new jobs are pinned (command,
config, remote-only if, BEP + testcase steps, timeouts, log upload), and
no tier step may run in bazel-integration or bazel-doltserver.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 23:01:40 -07:00

11 KiB

Testing Guide

TESTING.md is the single authority for test commands, test selection, and test design in this repository. Use the commands here for local work. For the exact command behind a current CI check, inspect its workflow and corresponding Makefile target; the CI audit and cleanup plan are dated maintainer context.

Choose the Smallest Useful Test

Test at the lowest seam that can fail for the user-visible reason. Add a higher tier only when it covers a distinct risk that the lower tier cannot show: integration wiring, a real persistence property, a process boundary, or an external contract.

This keeps feedback fast and failures readable. It does not mean every test is a unit test: use the real boundary when the defect could live there.

Need Run When
Docs-only validation git diff --check, go test -tags=gms_pure_go ./test/docsync, and ./scripts/check-doc-freshness.sh For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full Go suite merely because a Markdown file changed.
Focused red/green loop ./scripts/test.sh -run '^TestExactName$' ./path/to/package/... While writing or fixing one behavior.
Affected-package confidence ./scripts/test.sh ./path/to/package/... After the focused test passes; include directly affected neighbors when their contract changed.
Final Go baseline make test Once after focused work on Go code is green. It applies the normal local build flags, coverage, and local skip handling.
Named CI wrapper make ci-pr-core, make ci-pr-policy, or make ci-pr-lint Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit.
Hook shims against real timeout implementations nix flake check -L (or nix build .#checks.<system>.hook-timeout-backends -L) After changing the hook generator in cmd/bd/hooks.go (then make githooks-regen) or anything under .githooks/. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox timeout — the multicalls also installed as gtimeout alone — with and without Perl, under dash, bash and busybox ash. About one deadline of wall time; needs no Go build.

Do not replace the focused loop with repeated full-suite runs. Run the final make test once the affected Go tests are green. For docs-only changes, use the docs, link, and diff checks instead.

Commands and Local Environment

./scripts/test.sh is the normal runner. It sources .buildflags, creates an isolated test environment, applies .test-skip, and defaults to a 25m per-package Go-test timeout. The timeout is a hang backstop, not a target runtime. Override it only when diagnosing a legitimate slow path:

TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/...
TEST_VERBOSE=1 ./scripts/test.sh ./cmd/bd/...
TEST_RUN='^TestExactName$' ./scripts/test.sh ./cmd/bd/...

# Equivalent command-line options.
./scripts/test.sh -v -run '^TestExactName$' ./cmd/bd/...
./scripts/test.sh -timeout 30m ./cmd/bd/...

Use the opt-in ICU regex path only when the change requires it:

make test-icu-path

It is maintainer-only and not part of normal validation. make test-full-cgo and ./scripts/test-cgo.sh remain deprecated compatibility aliases.

Use a named specialized target only when its risk is in scope:

make test-regression
make test-upgrade
make test-cross-version
make test-migration

For a failing GitHub Actions check, follow the current workflow and its Makefile target when exact reproduction matters. The local runner and CI intentionally have different contracts in some cases.

Test Environment and Readiness

The runner isolates HOME, Git configuration, and Dolt state. By default its test environment adds dolt to BEADS_TEST_SKIP; set BEADS_TEST_ENV_RUN_DOLT=1 only when deliberately exercising the Dolt path and its prerequisites are available. Do not make ordinary tests depend on a developer's database, daemon, global Git configuration, or filesystem state.

To skip an optional service explicitly, use the existing skip mechanism:

BEADS_TEST_SKIP=dolt ./scripts/test.sh ./...

Tests that need a Dolt SQL server get one from internal/testutil (EnsureDoltContainerForTestMain, RequireDoltContainer, StartIsolatedDoltContainer[Handle], NewContainerProvider). Two backends sit behind that API, selected by BEADS_TEST_DOLT_SERVER:

  • container: the dolthub/dolt-sql-server image through testcontainers (needs docker and the pulled image). The default under plain go test.
  • local: a dolt sql-server started by the test process from the pinned dolt CLI (BEADS_TEST_DOLT_BINARY, else dolt on PATH; it must be the image's version). No docker. Used only when explicitly selected, under go test and bazel test alike (a Bazel target's env, or --test_env=BEADS_TEST_DOLT_SERVER=local); unset means container, which in a Bazel action without docker keeps the usual skip.

BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 turns an unavailable backend into a failure (per test and in every TestMain) instead of a skip; lanes that exist to run the Dolt suites set it.

Under Bazel, bazel test //... --config=doltserver runs the Dolt-backed suites of pr.yml's "Test (storage domain + uow)" and "Contract corpus" jobs on the local backend (the dolt-server targets); they need no docker and execute remotely with --config=remote-exec. --config=docker runs the same suites on the container backend (host docker) as the A/B control. PR Risk's heavier server tiers have configs of their own, run by bazel.yml only with remote execution, each in a job of its own (bazel-proxied, bazel-server-storage): --config=doltserver-proxied is the proxied-server cmd/bd tier ("Test (Proxied Dolt Cmd N/15)", //cmd/bd:bd_proxied_test), and --config=doltserver-integration the server-Dolt storage tier ("Test (Server Dolt Conformance)", "Test (Server Dolt Full Suite N/16)", //internal/storage/dolt:dolt_server_*_test), which builds with the integration tag like --config=integration. Each shard runs its CI job's shard script, so Bazel shard k runs the tests of job k+1.

Tests that need a temporary repository or store should use t.TempDir() and t.Cleanup(). Temporary repositories must set a repository-local hooks path; do not inherit the developer's global hooks configuration.

In cmd/bd, fresh-workspace command fixtures should call isolateBeadsDirForTest(t) before setup or dispatch. It clears inherited BEADS_DIR and restores that variable exactly at cleanup, even after raw command-dispatch mutations. The TestMain reset only isolates startup. These fixtures must not use t.Parallel(). Tests intentionally selecting a workspace should use t.Setenv("BEADS_DIR", ...); initConfigForTest and ensureCleanGlobalState preserve that selection.

For manual CLI experiments, run both initialization and subsequent commands from a disposable working directory:

beads_manual_dir="$(mktemp -d)"
(
  set -e
  cd "$beads_manual_dir"
  bd init --quiet --prefix test --skip-hooks --skip-agents
  bd create "Test issue" -p 1
)
rm -rf -- "$beads_manual_dir"

BEADS_DB selects a database for database-opening commands, but it does not by itself redirect bd init workspace setup. Never run a manual bd init from a production workspace merely because BEADS_DB points elsewhere.

testing.Short() is for genuine runtime, stress, or large-fixture skips. It is not a substitute for declaring an integration, end-to-end, API, Docker, or external-dependency boundary. Keep new uses within the repository policy:

make check-testing-short

Test Design

Seams, Scenarios, and Doubles

Write one scenario at the smallest seam that demonstrates the behavior. Cover the boundary or failure mode that changes the user result; use table-driven subtests when examples share setup. Do not repeat the same scenario through a helper, every caller, and the CLI merely because all are available.

A recording double or fake should be narrow: model only the calls, inputs, outputs, and failures the test needs. It should not recreate a storage engine, process manager, or another subsystem just to make a unit test look realistic.

A behavioral fake is different. If it stands in for a contract shared by multiple production implementations, give it the same semantic-conformance suite as those implementations. That shared suite defines observable behavior; it prevents the fake from teaching callers a contract production code does not honor.

Semantic conformance asks whether an implementation produces the promised results, errors, and state transitions for the same operation. Persistence conformance asks whether the real persistence boundary preserves its required durability, transaction, migration, and recovery properties. They answer different questions. Do not claim backend parity unless a stated contract and its conformance suite establish it.

Tier Admission

An integration test belongs above the unit seam only when it covers a distinct boundary that a narrow double cannot prove, such as configuration wiring, a real filesystem or Git interaction, a subprocess protocol, or persistence behavior.

An end-to-end test is admitted only when all of these are true:

  1. The failure would be user-visible.
  2. A real process, setup, or wiring boundary owns a distinct failure risk that no lower seam can prove.
  3. Lower-tier tests cover the underlying behavior where practical, leaving the end-to-end test focused on that boundary.
  4. No existing end-to-end test already covers the same boundary risk.

Lower-tier coverage of the same user journey does not disqualify the end-to-end test; duplicate coverage of the same boundary risk does. State that risk in the test name or nearby documentation.

Avoid Incidental Complexity

Avoid these patterns unless they are the behavior under test:

  • Duplicate scenarios at several layers.
  • Global-state reset choreography. Prefer per-test state and cleanup.
  • Subprocesses, listeners, sleeps, or real-store setup for a unit-level claim.
  • Assertions on private implementation shape when observable behavior is the contract.
  • Performance claims without a repeatable measurement and stated workload.

Sleeps, listeners, and real stores are appropriate when the test is specifically about timing, lifecycle, protocol, or persistence. Keep the setup scoped and make that reason apparent.

Failures, Skips, and Review

.test-skip is a local, temporary exception list. If an unrelated failure is already listed, report it rather than silently broadening the skip. Before adding a new skip, record the issue it tracks and remove the skip when the underlying failure is fixed.

Before opening a PR:

  1. For docs-only changes, run the applicable docs, link, freshness, and diff checks; do not run the full Go suite by default.
  2. For Go code, keep the focused and affected-package tests green, then run one final make test.
  3. Run only the named CI wrapper, specialized target, or risk gate required by the changed surface, or the one needed to reproduce a CI result.

For historical CI inventory and maintainer planning context, consult CI_TEST_SURFACE_AUDIT.md and CI_CLEANUP_PLAN.md. For current commands, use the workflow files and Makefile; these context documents are not a second testing guide or a live CI inventory.