* test(cmd/bd): reset cobra flags between in-process runs; keep them out of parallel tests
runBDInProcess and runBDInProcessAllowError execute the package-level
rootCmd tree again and again in one process, and cobra never resets a
flag between Execute calls: a value one invocation parsed (create's
--title, --labels, list's --status, ...) and its Changed bit stayed on the
FlagSet, so the next invocation saw it, and a slice flag that had been
set appended to the stale value instead of replacing it. With a Dolt test
server (the only setup in which these tests run) that failed about 20
TestCLI_* tests depending on suite order.
TestMain now snapshots every flag of the tree (snapshotCommandFlags,
after all init()s registered them, before any test runs) and both runners
put the tree back before and after each run (resetCommandFlags): the
pointee of each flag.Value is restored (for pflag's scalar types that is
the bound variable; for slice types it also clears the internal "changed"
bit, so the next Set replaces), slice contents via SliceValue.Replace,
and Changed=false. TestResetCommandFlagsRestoresEveryFlag sets every flag
of the real tree and checks the reset. The ad-hoc --title reset in
TestCLI_CreateRejectsEmptyTitle_ProxiedServerMode is no longer needed.
Seven TestCLI_Comment* tests called t.Parallel() around the runners.
inProcessMutex serializes the runners only against each other, so their
chdir, stdio swap, env and config re-initialization raced every other
parallel test (data races on config globals; TestCobraParallelPolicyGuard
failing to read its own source files after a concurrent chdir). They run
serially now, and the guard treats the two runners like rootCmd.Execute.
Also two stale TestCLI_Create* cases: DryRunWithRigPrefix exercised a
create --prefix flag that no longer exists, and the flag-like-title cases
passed "--help" or "--foo-bar" where cobra parses them as flags; they now
pass the title after "--", the only way it reaches create's own guard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(cmd/bd): fix the ADO round-trip and bulk-delete tests that rotted unseen
Both need the integration tag and a Dolt test server, which no CI lane
gave cmd/bd (BEADS_TEST_SKIP=dolt everywhere), so they failed unnoticed:
- TestADORoundTrip* stored ado.pat in the database. ado.pat is a secret
read only from config.yaml or AZURE_DEVOPS_PAT (config.IsYamlOnlyKey),
so the tracker found no PAT. The fixture sets AZURE_DEVOPS_PAT.
- TestBulkDeleteNoResurrection expected GetIssue to return (nil, nil) for
a deleted issue; the Dolt store reports storage.ErrNotFound.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(cmd/bd): fix parity, completion and notion tests that only fail with a Dolt server
These are Dolt-gated (integration tag + test server), so no CI lane ran
them and they rotted:
- TestParityCreate*/TestParityReopenErrorExits1Silently: bd create
auto-routes via routing.DetectUserRole("."), which warns "beads.role
not configured (GH#2950)" on stderr when the working directory's git
config has no beads.role. The suite pins the configured case, so the
parity env now sets beads.role=maintainer in a test-owned global git
config ($XDG_CONFIG_HOME/git/config; the role lookup scrubs every
GIT_CONFIG_* variable, so those cannot be used).
- TestIssueIDCompletion: issueIDCompletion resolves a database path
before looking at the open store and returns nothing when there is
none; the test now sets dbPath to its store's path.
- TestIssueIDCompletion_UsesWorktreeFallbackWhenStoreNil and
TestGetNotionConfigReadsDBPathWhenStoreUnset: the code under test
reopens the store from metadata.json, which names a database but not
the per-test branch of the shared database that newTestStore uses, so
the reopened store saw none of the test's rows. They now use their own
database (newTestStoreIsolatedDB).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(cmd/bd): fix stale doctor/e2e/autocommit/readonly Dolt-gated tests
These need the integration tag and a Dolt test server, which no CI lane
gave cmd/bd, so they rotted unseen:
- TestE2E_DoctorFixMetadataRoundtrip, TestE2E_MigrateDoltMetadata,
TestDoctorCheckHealthReportsVersionMismatchOnRepoLocalPort: bd init now
defaults to embedded mode, where `bd sql` and `bd dolt start` are
refused. Init with --server on a repo-local server of their own
(repoLocalServerEnv: the suite's shared-server port vars cleared, server
stopped at cleanup) and start it explicitly, since BEADS_TEST_MODE=1 never
auto-starts one. MigrateDoltMetadata's DELETE also needed `key` quoted.
- TestDoctor{Conventions,Pollution}MaxRows_EnvOnly_Exits2: bdRunRaw drops
every BEADS_* variable, so the subprocess hit dolt.New's test-database
firewall (exit 1) before the row cap; pass BEADS_TEST_SERVER=1.
- TestDoltAutoCommit_On_WritesAdvanceHead: the embedded engine keeps its
data under .beads/embeddeddolt, not .beads/dolt; findDoltRepoDir looks
in both.
- TestValidateCheck_AllClean: bd doctor validate runs five checks since
the cross-table duplicates check was added.
- TestConfigValidateReadOnlyIsHermetic: readonly commands take a shared
workspace gate, whose lock file (.beads/dolt.gate.lock) workspacegate
creates on first use by design; compare the .beads tree without gate
lock files.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(cmd/bd): undo each in-process execution's env and storage-mode leaks
Every in-process run of the bd command tree (runBDInProcess and the many
tests that call rootCmd.Execute directly) leaves process state behind:
PersistentPreRun exports the selected workspace as BEADS_DIR, loads its
.beads/.env, and records the workspace's mode in serverMode /
proxiedServerMode. With a Dolt test server, later tests in the same shard
then resolved a deleted fixture ("no active beads workspace found",
"migration sidecar is missing"), ran subprocess bd against the wrong
workspace ("recorded as proxied-server in .beads/metadata.json"), or took
the proxied path in RunE-level tests ("proxied-server UOW provider not
initialized", a nil-UOW panic in reopenProxiedResolve that killed a shard).
TestMain now registers a cobra initializer/finalizer pair
(installExecuteIsolation) that snapshots the environment and the two mode
globals when an execution starts and restores them when it ends; the
runners' restoreEnvSnapshot shares the helper. TestDoltConfigEnvironmentOverrides
used os.Setenv + defer os.Unsetenv, which deleted the suite's
BEADS_DOLT_SERVER_PORT (the test server's port) for every later test; it
uses t.Setenv now.
Full integration-tagged cmd/bd suite against a hermetic dolt (16 shards):
18 order-dependent failures and the shard panic gone; the 27 remaining
failures all fail alone too.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(init): keep --quiet quiet through the Claude/Codex/Cursor auto-setup
`bd init --quiet` ("Suppress output") still printed the project
integration installers' progress to stdout ("Installing Claude hooks for
this project...", "Claude Code integration installed", ...): the
installers write to os.Stdout unconditionally. Point stdout at the null
device around them while quiet; their failures keep going to stderr as
before. Found by TestInitCommand's quiet case, which asserts no output and
had not run in CI (Dolt-gated).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(init): bring the Dolt-gated init tests up to the current init contract
These tests need a Dolt test server, so no CI lane ran them and they
drifted from the product:
- Embedded is init's default mode, so the server-mode assertions
(.beads/dolt, opening the database on the test server, dolt_mode
"server" in metadata.json) now pass --server (TestInitCommand,
TestInitAlreadyInitialized, TestInitRedirect, TestInitBEADS_DIR,
TestInitDoltMetadata, TestInitDoltMetadataNoGit, TestInitDatabaseFlag's
metadata subtests). The in-process ones reset rootCmd's flags around
themselves so --server cannot leak into later in-process inits.
- Each server init gets its own --database: the suite shares one test
server, and a "test" database another test created belongs to another
project (init refuses: PROJECT IDENTITY MISMATCH).
- The marker tests seed a current-era version witness: the legacy-upgrade
guard refuses a non-empty .beads/dolt without one.
- The redirect test provisions the canonical target with config.yaml: bd
ignores a redirect to a directory with no project files (#4692).
- Shared-server inits get a free port of their own
(sharedServerTestEnv): the inherited BEADS_DOLT_SERVER_PORT is the
suite's test server, which a shared-server start refuses as another
project's, and the default shared port is 3308, a developer's live
server.
- --shared-server's config.yaml is now nested YAML (dolt: shared-server:
true); parse it instead of matching the old flat line.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(cmd/bd): run the Dolt-gated tests that skipped on a bare dolt.New
setupTestDB (config, kv and Linear round-trip tests), setupGatedTestDB
(mol ready --gated) and the rename-prefix and bd sql tests opened stores
with dolt.New(&dolt.Config{Path: ...}): no server port, no
CreateIfMissing, so against a running test server they got "database ...
not found" and skipped as "Dolt server not available". They now use the
suite's newTestStore helpers (skip only when no test server runs), and
what that exposed is fixed test-side:
- TestCustomStatusConfig: custom_statuses is read back ORDER BY name;
look statuses up by name / expect name order.
- TestLinearRoundTripCoreFields, TestLinearPullMilestonesCreatesEpicHierarchy:
linear.api_key is a yaml-only secret (LINEAR_API_KEY), push now needs an
explicit linear.state_map, and batch push uses TeamLabels and
issueBatchCreate, which the mock server now answers.
- TestSqlCommand: bd sql refuses in the default embedded mode; set
serverMode for the server-backed store, and stop discarding RunE errors.
- TestE2E_AutoStartedRepoLocalServerPersistsAcrossCommands skipped on
this process's embedded mode although every bd it runs is a subprocess
on a --server workspace; gate on the dolt binary instead and stop the
auto-started server at cleanup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(cmd/bd): no fixed "unused" port, no swallowed JSON, no leaked front-door server
Three more things the Dolt-backed cmd/bd run exposed:
- TestTestServerConnection, TestSharedServerCfgNilHonorsSharedServer and
the two TestLegacyGuard* e2e tests pointed bd at 127.0.0.1:59999 as a
port nothing listens on. Any process on the host may hold it; a
listener that accepts and resets turns the expected "connection
refused" / "Dolt server unreachable" into another error. They take a
just-freed port from closedLoopbackPort (testutil.FindFreePort) now.
- TestDoltSetConfigJSONOutput and TestDoltSetConfigWithUpdateConfig
created .beads 0755, so bd's permissions warning landed in the JSON
under test and the tests skipped ("output not pure JSON") every run.
The fixture uses 0700, and non-JSON output fails instead of skipping.
- TestMigrateDoltModeFrontDoor's closing show/list auto-start a
repo-local server after the reverse migration; nothing stopped it, so
the suite's leak check failed (when the opt-in
BEADS_TEST_MIGRATION_FRONTDOOR suite runs). It stops it at cleanup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ci(bazel): advisory cmd/bd Dolt-server tier (bazel-cmd-dolt)
No CI lane ran cmd/bd's Dolt-gated tests (TestCLI_*, the init and
store-backed suites, the doctor/e2e tests): main.yml's integration shards,
nightly's Full Test Suite, and Bazel's --config=prcore/ci and
--config=integration all pass BEADS_TEST_SKIP=dolt, so cmd/bd's TestMain
never starts a test server and those tests skip; the proxied and embedded
tiers run only their manifests' tests. They rotted unseen.
//cmd/bd:bd_dolt_server_test (tag dolt-server-cmd) runs the whole
integration-tagged cmd/bd suite, the integration build's bd_test, against
a hermetic dolt sql-server: BEADS_TEST_DOLT_SERVER=local starts it from
the pinned dolt in the test wrapper's runfiles, and
BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 fails the target instead of skipping
into a cached pass. No test selection: the Go binary shards itself over
every top-level test (32 shards), so each runs exactly once, and
check_testcases.py fails a shard that ran none. --config=doltserver-cmd
has exactly --config=integration's build flags, so the two lanes share
bd_test's compile; prcore/ci and integration exclude the tag.
bazel.yml's new bazel-cmd-dolt job runs it exactly where bazel-integration
runs (same if: remotely on the farm for same-repo PRs, fork-ro/fork-rw,
and locally with the read-only cache in mode cache for forks). It is
advisory: job-level continue-on-error keeps a failure out of the call's
aggregate (BAZEL), and pr.yml's ci-gate reads its result (BAZEL_CMD_DOLT)
but requires it only while the committed top-level flag
BAZEL_CMD_DOLT_REQUIRED is "true" (now "false"; flipping it is the one
change needed). bazel-gate.sh accepts its skip in modes skip and local.
Policy tests: a third lane category in TestBazelLaneIsGatedAlongsideLegacy
(bazelFlagGatedLanes: gate env wired, never statically required, added by
the run script exactly when the pinned flag is "true"), the gate
simulation over every mode with the flag off (advisory: nothing it
reports reddens the gate) and on (exactly a gated lane), the cache-mode
and execution-mode checks, the tag taxonomy and prcore/integration
exclusions, and TestBazelCmdDoltJob (job steps, timeouts, .bazelrc
config equal to integration's build with no dolt skip or test selection,
the target's dolt-server rules, and bd_test's env repeated).
Measured locally (all fixes in this series, 32 shards, 16 at a time on a
shared host): 3267 top-level tests, all shards pass, max shard 313s (one
5001-issue export test, 214s, sets it), mean 100s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(init): hand the project installers a writer instead of swapping os.Stdout
The --quiet fix earlier in this series pointed the process-global
os.Stdout at the null device around the Claude/Codex/Cursor installers.
That works, but it mutates a global from inside a cobra Run: anything else
writing to stdout meanwhile (a goroutine, a future concurrent installer)
would vanish, and a concurrent read of os.Stdout is a data race.
The installers already write through env structs with stdout/stderr
fields. setup.ProjectInstallOutput carries the two writers, and
InstallClaudeProjectTo / InstallCodexProjectTo / InstallCursorProjectTo
use them (the old entry points keep the process streams). init passes
QuietProjectInstallOutput() under --quiet: progress discarded, error lines
still on stderr. The proxied-server init path (domain adapter
InstallClaudeProject now takes quiet) does the same, so its
"Installing Claude hooks..." is quiet too.
Consistent error policy under --quiet: errors still reach stderr. init's
own "Warning: failed to setup <installer>" lines were suppressed under
--quiet while the installers' "Error:" lines were not, and some installer
failures (Cursor rules) print nothing themselves; the warnings now print
whether or not quiet, like the installers' errors.
Coverage in required lanes: TestProjectInstallersReportToTheGivenOutput
(cmd/bd/setup) and TestEmbeddedInitA/basic, which now asserts that
`bd init --quiet` with agents enabled installs the Claude hooks and prints
none of the installers' progress (verified to fail with the quiet output
disabled).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(export): exercise the incremental-export fallback with a small threshold
TestTryIncrementalExport_ThresholdExceededFallsBack seeded
incrementalExportThreshold+1 = 5001 issues to cross the threshold. Batch
create is one transaction but still several statements per issue, so under
-race against the shared test server it took ~214 s, the longest test in
cmd/bd's Dolt-backed suite by far. The behaviour under test is only
"more changed issues than the threshold -> full export fallback".
incrementalExportThreshold becomes a var (same value), and the test, which
is not parallel, sets it to 20 and seeds 21 issues: 214 s -> 0.6 s.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(cmd/bd): tighten the flag-reset self-test, guards and canaries
Review follow-ups:
- TestResetCommandFlagsRestoresEveryFlag no longer skips a flag it cannot
set: a flag type without a sample value, a Set error, or a non-pointer
Value (which resetCommandFlags could not restore) is an error, so a new
flag type cannot go untested. resetCommandFlags documents that it also
resets flag-bound package globals.
- TestCobraParallelPolicyGuard treats .ExecuteC/.ExecuteContext/
.ExecuteContextC like .Execute (the per-execution isolation writes the
mode globals; a parallel execution would race).
- TestConfigValidateReadOnlyIsHermetic ignores only the gate a readonly
command takes (.beads/dolt.gate.lock and its .info sidecar), not any
"*.gate.lock*" file, so a new unexpected gate still trips the canary.
- TestCLI_CreateRejectsFlagLikeTitles: a bare `bd create --foo-bar` must
fail and create no issue (bd-2c0's intent), next to the "--" cases.
- TestCustomStatusConfig cites why statuses come back in name order: the
normalized custom_statuses table (#2961, 8537d943b) has no ordinal and is
read ORDER BY name, like custom_types; the test predates it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ci(bazel): run the cmd/bd Dolt-server tier in 16 shards
Each shard pays ~30-40 s of fixed cost (TestMain plus its own dolt
sql-server), so 32 shards spent roughly 40% of the lane's executor time on
setup and added 32 farm actions to every PR touching cmd/bd. With the
5001-issue export test cut to under a second, 16 shards measure (locally,
--nocache_test_results): max 186 s, mean 137 s, 2,187 executor-seconds
(vs ~2,800-3,200 at 32), all 3,267 top-level tests, every shard passing.
TestBazelCmdDoltJob pins the count; bazel.yml's timing comment is
restated (cache mode ~20-30 minutes).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
18 KiB
Testing Guide
TESTING.md is the single authority for test commands, test selection, and
test design in this repository. Use the commands here for local work. For the
exact command behind a current CI check, inspect its workflow and corresponding
Makefile target; the CI audit and cleanup plan are dated maintainer context.
Choose the Smallest Useful Test
Test at the lowest seam that can fail for the user-visible reason. Add a higher tier only when it covers a distinct risk that the lower tier cannot show: integration wiring, a real persistence property, a process boundary, or an external contract.
This keeps feedback fast and failures readable. It does not mean every test is a unit test: use the real boundary when the defect could live there.
| Need | Run | When |
|---|---|---|
| Docs-only validation | git diff --check, go test -tags=gms_pure_go ./test/docsync, and ./scripts/check-doc-freshness.sh |
For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full Go suite merely because a Markdown file changed. |
| Focused red/green loop | ./scripts/test.sh -run '^TestExactName$' ./path/to/package/... |
While writing or fixing one behavior. |
| Affected-package confidence | ./scripts/test.sh ./path/to/package/... |
After the focused test passes; include directly affected neighbors when their contract changed. |
| Final Go baseline | make test |
Once after focused work on Go code is green. It applies the normal local build flags, coverage, and local skip handling. |
| Named CI wrapper | make ci-pr-core, make ci-pr-policy, or make ci-pr-lint |
Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. |
| Hook shims against real timeout implementations | nix flake check -L (or nix build .#checks.<system>.hook-timeout-backends -L) |
After changing the hook generator in cmd/bd/hooks.go (then make githooks-regen) or anything under .githooks/. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox timeout — the multicalls also installed as gtimeout alone — with and without Perl, under dash, bash and busybox ash. About one deadline of wall time; needs no Go build. |
Do not replace the focused loop with repeated full-suite runs. Run the final
make test once the affected Go tests are green. For docs-only changes, use
the docs, link, and diff checks instead.
Commands and Local Environment
./scripts/test.sh is the normal runner. It sources .buildflags, creates an
isolated test environment, applies .test-skip, and defaults to a 25m
per-package Go-test timeout. The timeout is a hang backstop, not a target
runtime. Override it only when diagnosing a legitimate slow path:
TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/...
TEST_VERBOSE=1 ./scripts/test.sh ./cmd/bd/...
TEST_RUN='^TestExactName$' ./scripts/test.sh ./cmd/bd/...
# Equivalent command-line options.
./scripts/test.sh -v -run '^TestExactName$' ./cmd/bd/...
./scripts/test.sh -timeout 30m ./cmd/bd/...
Use the opt-in ICU regex path only when the change requires it:
make test-icu-path
It is maintainer-only and not part of normal validation. make test-full-cgo
and ./scripts/test-cgo.sh remain deprecated compatibility aliases.
Use a named specialized target only when its risk is in scope:
make test-regression
make test-upgrade
make test-cross-version
make test-migration
For a failing GitHub Actions check, follow the current workflow and its
Makefile target when exact reproduction matters. The local runner and CI
intentionally have different contracts in some cases.
Test Environment and Readiness
The runner isolates HOME, Git configuration, and Dolt state. By default its
test environment adds dolt to BEADS_TEST_SKIP; set
BEADS_TEST_ENV_RUN_DOLT=1 only when deliberately exercising the Dolt path
and its prerequisites are available. Do not make ordinary tests depend on a
developer's database, daemon, global Git configuration, or filesystem state.
To skip an optional service explicitly, use the existing skip mechanism:
BEADS_TEST_SKIP=dolt ./scripts/test.sh ./...
Tests that need a Dolt SQL server get one from internal/testutil
(EnsureDoltContainerForTestMain, RequireDoltContainer,
StartIsolatedDoltContainer[Handle], NewContainerProvider). Two backends
sit behind that API, selected by BEADS_TEST_DOLT_SERVER:
container: thedolthub/dolt-sql-serverimage through testcontainers (needs docker and the pulled image). The default under plaingo test.local: adolt sql-serverstarted by the test process from the pinned dolt CLI (BEADS_TEST_DOLT_BINARY, elsedoltonPATH; it must be the image's version). No docker. Used only when explicitly selected, undergo testandbazel testalike (a Bazel target'senv, or--test_env=BEADS_TEST_DOLT_SERVER=local); unset meanscontainer, which in a Bazel action without docker keeps the usual skip.
BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 turns an unavailable backend into a
failure (per test and in every TestMain) instead of a skip; lanes that
exist to run the Dolt suites set it. BEADS_TEST_REQUIRE_SOCAT=1 does the
same for the proxied subtests that bridge an external endpoint with socat
(external-unix, the outage/reconnect matrix); //cmd/bd:bd_proxied_test
sets it, and the legacy GitHub proxied jobs, which have no socat, do not.
Under Bazel, bazel test //... --config=doltserver runs the Dolt-backed
suites of pr.yml's "Test (storage domain + uow)" and "Contract corpus" jobs
on the local backend (the dolt-server targets); they need no docker and
execute remotely with --config=remote-exec. --config=docker runs the same
suites on the container backend (host docker) as the A/B control.
PR Risk's heavier server tiers have configs of their own, run by bazel.yml
only with remote execution, each in a job of its own (bazel-proxied,
bazel-server-storage): --config=doltserver-proxied is the
proxied-server cmd/bd tier ("Test (Proxied Dolt Cmd N/15)",
//cmd/bd:bd_proxied_test), and --config=doltserver-integration the
server-Dolt storage tier ("Test (Server Dolt Conformance)", "Test (Server
Dolt Full Suite N/16)", //internal/storage/dolt:dolt_server_*_test), which
builds with the integration tag like --config=integration. Each shard runs
its CI job's shard script, so for --config=doltserver-integration Bazel
shard k runs the tests of job k+1 (both split the manifest's 16-shard block
the same way). --config=doltserver-proxied's bd_proxied_test instead
runs the manifest's own 30-shard block — bin-packed by measured duration,
not the legacy jobs' 15-shard, bd-init-cost-proxy block — so shard k there
is not job k+1's tests; it is a different split of the same tests.
--config=doltserver-cmd (//cmd/bd:bd_dolt_server_test, bazel.yml's
advisory bazel-cmd-dolt job) runs the whole integration-tagged cmd/bd
suite on the local backend: the Dolt-gated cmd/bd tests (TestCLI_*, the
init and store-backed suites) that every other lane skips with
BEADS_TEST_SKIP=dolt or leaves out of its manifest. It shares
--config=integration's build, passes the binary no test selection (the Go
binary shards itself over every top-level test, 16 shards), and runs where
the integration lane runs (remote, or with the read-only cache). pr.yml's
gate requires it once BAZEL_CMD_DOLT_REQUIRED is "true". Locally:
bazel test //cmd/bd:bd_dolt_server_test --config=doltserver-cmd.
An ambient BEADS_DOLT_SERVER_PORT or BEADS_DOLT_PORT is never honored by
the suites that call testutil.EnsureDoltContainerForTestMain. When a test
container is started, that container's port overwrites both variables; when one
cannot be started -- for any reason, including BEADS_TEST_SKIP=dolt -- both
are cleared, so no environment-named server can be resolved. Point a test run
at a specific Dolt server by starting a container for it, not by exporting a
port.
The one sanctioned way to hand these suites a server you started yourself is
./scripts/test.sh with BEADS_TEST_SHARED_SERVER=1: it starts one
dolt sql-server, exports its port as BEADS_DOLT_PORT, and marks it by
exporting BEADS_TEST_SHARED_DOLT_SERVER set to that same port number. It
starts nothing if either port variable is still set when it gets there. The
runner's test environment clears both first, so that only happens when that
isolation is skipped (for example BEADS_TEST_ENV_DISABLE=1), and the script
then says so on stderr. The marker -- set only by that script, only for a
port it allocated -- is what the helper treats as container-equivalent, and
only for a variable holding exactly the port it names: that variable survives
the clearing above, and any other port variable is still cleared. A container
still wins where one can be started; the shared server is the Docker-less path.
Clearing the environment variables closes the channel the ambient port
travelled on; it does not make a port unresolvable in general. Resolution
continues into the file chain (.beads/dolt-server.port, config.yaml,
metadata.json), and internal/storage/dolt's production-port detection is
narrower than that resolution -- see be-rl6tm, which tracks the remaining
gap.
Tests that need a temporary repository or store should use t.TempDir() and
t.Cleanup(). Temporary repositories must set a repository-local hooks path;
do not inherit the developer's global hooks configuration.
In cmd/bd, fresh-workspace command fixtures should call
isolateBeadsDirForTest(t) before setup or dispatch. It clears inherited
BEADS_DIR and restores that variable exactly at cleanup, even after raw
command-dispatch mutations. The TestMain reset only isolates startup.
These fixtures must not use t.Parallel(). Tests intentionally selecting a
workspace should use t.Setenv("BEADS_DIR", ...); initConfigForTest and
ensureCleanGlobalState preserve that selection.
For manual CLI experiments, run both initialization and subsequent commands from a disposable working directory:
beads_manual_dir="$(mktemp -d)"
(
set -e
cd "$beads_manual_dir"
bd init --quiet --prefix test --skip-hooks --skip-agents
bd create "Test issue" -p 1
)
rm -rf -- "$beads_manual_dir"
BEADS_DB selects a database for database-opening commands, but it does not by
itself redirect bd init workspace setup. Never run a manual bd init from a
production workspace merely because BEADS_DB points elsewhere.
testing.Short() is for genuine runtime, stress, or large-fixture skips. It
is not a substitute for declaring an integration, end-to-end, API, Docker, or
external-dependency boundary. Keep new uses within the repository policy:
make check-testing-short
Dolt Container Tests (podman-rootless)
Anything that does not carry BEADS_TEST_SKIP=dolt — including
BEADS_TEST_ENV_RUN_DOLT=1 and a bare go test — reaches a real
dolt sql-server: by default through testcontainers-go where a container
runtime and the pinned image are present (otherwise those suites self-skip),
and from the local dolt CLI in the suites that use
testutil.RequireDoltBinary. Lanes that set
BEADS_TEST_REQUIRE_DOLT_CONTAINER=1 fail instead of skipping. Two limitations
of the containerized path are worth recognizing before reading a failure as a
product bug. Neither failure mode manifests in production or on GitHub Actions:
CI exercises this same containerized path green on every risk-tier PR, and
.github/workflows/pr-risk.yml pulls the pinned image precisely to defeat the
self-skip. A hang there is a real bug, not this section's subject.
Migration 0032 hangs over the wire protocol. Migration 0032
(drop_schema_migrations_applied_at) hangs indefinitely when applied through
the containerized sql-server. How it surfaces depends on the caller's context.
A go test ./internal/storage/uow/... -count=1 with no BEADS_TEST_SKIP
passes context.Background(), so it sits in initSchema until the package
timeout rather than failing — that is the shape you will normally see, with no
deadline to cut it short. A caller that does supply one gets the hang reported
by the server-mode store open (newServerMode in
internal/storage/dolt/store.go) instead:
failed to initialize schema: context deadline exceeded
The cause is the podman-rootless container port-forwarding path, not migration
0032's SQL: the identical statement completes normally through the embedded/CLI
engine, and against a bare-host-process dolt sql-server matching the deployed
shape. Evidence chain in be-j3szz. Use BEADS_TEST_SKIP=dolt unless the
container path is what you are testing.
Disabling Ryuk removes the container safety net. This repository sets no
TESTCONTAINERS_RYUK_DISABLED, so CI runs with Ryuk — testcontainers-go's
orphan-reaper sidecar — enabled. The podman-rootless flow generally requires
turning it off by hand (TESTCONTAINERS_RYUK_DISABLED=true), because under
rootless podman Ryuk frequently cannot start: it wants the runtime socket. See
be-w3n2m. With Ryuk off, nothing reaps a container whose test process exited
without running its cleanup — os.Exit reached before a deferred
TerminateDoltContainer, for instance. be-5kkk6 records the cost: 101 leaked
containers, and an exhausted swap. When running with Ryuk disabled, keep
teardown on the normal return path using the testMainInner pattern
(beads_test.go), and check for strays afterwards. Carry the rootless socket
this flow runs on — a bare docker ps talks to the CLI's default endpoint and
reports a false all-clear — and read the tag from the pin rather than copying
it, so the command cannot drift when the pin moves. An empty read would be the
same false all-clear (ancestor= matches nothing), so the command refuses to
run without a tag:
dolt_image=$(sed -n 's/.*DoltDockerImage = "\(.*\)".*/\1/p' \
"$(git rev-parse --show-toplevel)/internal/testutil/testdoltcommon.go")
DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock \
docker ps -a --filter "ancestor=${dolt_image:?could not read the Dolt image pin}"
Test Design
Seams, Scenarios, and Doubles
Write one scenario at the smallest seam that demonstrates the behavior. Cover the boundary or failure mode that changes the user result; use table-driven subtests when examples share setup. Do not repeat the same scenario through a helper, every caller, and the CLI merely because all are available.
A recording double or fake should be narrow: model only the calls, inputs, outputs, and failures the test needs. It should not recreate a storage engine, process manager, or another subsystem just to make a unit test look realistic.
A behavioral fake is different. If it stands in for a contract shared by multiple production implementations, give it the same semantic-conformance suite as those implementations. That shared suite defines observable behavior; it prevents the fake from teaching callers a contract production code does not honor.
Semantic conformance asks whether an implementation produces the promised results, errors, and state transitions for the same operation. Persistence conformance asks whether the real persistence boundary preserves its required durability, transaction, migration, and recovery properties. They answer different questions. Do not claim backend parity unless a stated contract and its conformance suite establish it.
Tier Admission
An integration test belongs above the unit seam only when it covers a distinct boundary that a narrow double cannot prove, such as configuration wiring, a real filesystem or Git interaction, a subprocess protocol, or persistence behavior.
An end-to-end test is admitted only when all of these are true:
- The failure would be user-visible.
- A real process, setup, or wiring boundary owns a distinct failure risk that no lower seam can prove.
- Lower-tier tests cover the underlying behavior where practical, leaving the end-to-end test focused on that boundary.
- No existing end-to-end test already covers the same boundary risk.
Lower-tier coverage of the same user journey does not disqualify the end-to-end test; duplicate coverage of the same boundary risk does. State that risk in the test name or nearby documentation.
Avoid Incidental Complexity
Avoid these patterns unless they are the behavior under test:
- Duplicate scenarios at several layers.
- Global-state reset choreography. Prefer per-test state and cleanup.
- Subprocesses, listeners, sleeps, or real-store setup for a unit-level claim.
- Assertions on private implementation shape when observable behavior is the contract.
- Performance claims without a repeatable measurement and stated workload.
Sleeps, listeners, and real stores are appropriate when the test is specifically about timing, lifecycle, protocol, or persistence. Keep the setup scoped and make that reason apparent.
Failures, Skips, and Review
.test-skip is a local, temporary exception list. If an unrelated failure is
already listed, report it rather than silently broadening the skip. Before
adding a new skip, record the issue it tracks and remove the skip when the
underlying failure is fixed.
Before opening a PR:
- For docs-only changes, run the applicable docs, link, freshness, and diff checks; do not run the full Go suite by default.
- For Go code, keep the focused and affected-package tests green, then run one
final
make test. - Run only the named CI wrapper, specialized target, or risk gate required by the changed surface, or the one needed to reproduce a CI result.
For historical CI inventory and maintainer planning context, consult
CI_TEST_SURFACE_AUDIT.md and
CI_CLEANUP_PLAN.md. For current commands, use the
workflow files and Makefile; these context documents are not a second testing
guide or a live CI inventory.