beads/engdocs/TESTING.md
Tomasz Feliksik 33bed587e0
Accept uutils coreutils timeout in generated hooks, and check the shims against real implementations (#6561)
* Accept uutils coreutils timeout in generated hooks

The generated hook selects a timeout helper only when its --version
banner starts with "timeout (GNU coreutils) ". uutils coreutils reports
"timeout (uutils coreutils) " with the same command line and the same
exit 124 on expiry, so hosts that ship it (Ubuntu 25.10+, NixOS with
uutils-coreutils-noprefix) silently fell through to the Perl alarm, or
to the unbounded direct run when Perl is absent. Follow-up from #5522,
tracked in #5541.

- Widen the identity allowlist to uutils; the match stays by banner so
  Windows timeout.exe is still never probed (#5503).
- Add a backend-selection case using the GNU stub with the uutils
  banner: fails on the old shim (selects gtimeout), passes with the
  widening.
- Regenerate the managed sections in the tracked .githooks/* so
  TestTrackedManagedHookSectionsMatchGenerator stays green.
- Update the hook-timeout passages in docs/reference/git-integration.md
  and troubleshooting.md, which described the probe as GNU-only.

Verified on NixOS with uutils 0.10.0 as /run/current-system/sw/bin/timeout:
with Perl removed from PATH and a bd that hangs, the old shim runs
"without timeout" for the full duration; the new shim expires at
BEADS_HOOK_TIMEOUT and continues.

Agent-Signature: claude-code-claude-opus-5-unknown-reasoning on behalf of Tomasz Feliksik

* Regenerate tracked .githooks managed sections with make githooks-regen

TestTrackedManagedHookSectionsMatchGenerator holds the managed section in
each tracked .githooks/* byte-equal to the cmd/bd/hooks.go generator, but
nothing regenerated them: after a generator change a contributor had to
splice the new section in by hand, and the failure only dumped both
texts. The v1.2.0 bump reddened main this way (ce20a64bb), and #6561 shipped
its first revision with the five hooks stale.

- BD_UPDATE_GOLDEN=1 makes the test rewrite the section in place, keeping
  everything around the markers (the project's own hook logic) untouched.
  Same switch the package's events golden already uses.
- `make githooks-regen` wraps it, next to `corpus-regen`.
- The drift failure now names that command.
- The exactly-one-section check runs before any rewrite.

Verified: drifting one byte inside .githooks/pre-push fails the test with
the hint; `make githooks-regen` restores the file to the same blob SHA;
the test is green again.

Agent-Signature: claude-code-claude-opus-5-unknown-reasoning on behalf of Tomasz Feliksik

* Check the tracked hook shims against real timeout implementations

The Go tests pin the shim's helper selection with scripted stand-ins, so
they can only encode what the author believes each real `timeout` does.
That is where the last two defects lived: uutils prints a banner the
allowlist did not know (#5541), and busybox — which #5541 suggests
adding — exits 143 on expiry rather than 124, so allowlisting it would
turn every deadline into a failed hook and a blocked push.

checks.nix adds a flake check that runs the tracked .githooks/* managed
sections (held equal to the generator by
TestTrackedManagedHookSectionsMatchGenerator) against the binaries
themselves: GNU coreutils, uutils, busybox and toybox `timeout`, each
with and without perl, under dash, bash and busybox ash, for all five
hooks — 150 cases. Expectations come from a model of the contract (the
allowlist as data), so widening it is one line here and the binaries
either agree or they do not.

A fake bd on the shim's restricted PATH blocks on a held-open fifo until
a helper kills it, records the signal and re-raises it, so each case
asserts the backend that ran (TERM from timeout, ALRM from perl's alarm)
and not merely that the shim finished. No clock: a helper that fails to
kill the fake trips the harness's own `timeout 10`. Every case asserts
exit 0, since anything else blocks the user's push. Reading .githooks/*
instead of building bd keeps it at ~1 s and off the Go build's critical
path; CI's test-nix job runs it after its existing help-text smoke test.

Mutation-checked: restoring main's GNU-only sections fails all 30 uutils
cases (15 unbounded, 15 on the perl backend); allowlisting busybox with
a loosened probe fails all 30 busybox cases with hook exit 143.

Agent-Signature: claude-code-claude-opus-5-unknown-reasoning on behalf of Tomasz Feliksik

* Run the hook-timeout check under the gtimeout candidate name too

The shim probes `timeout` and then `gtimeout`, but the matrix only ever
put a binary named `timeout` on PATH, so the second candidate was never
run against a real implementation. Whether a helper is accepted under
that name depends on what its banner does with argv[0].

Measured: the uutils multicall dispatches on the argv[0] SUFFIX
(src/bin/coreutils.rs, `binary_as_util.ends_with(util)`) and then prints
the canonical utility name — invoked as gtimeout it reports
"timeout (uutils coreutils) 0.8.0" (this flake's nixos-25.11 pin; 0.11.0
identical), which the allowlist accepts. busybox and toybox know no
gtimeout applet and fail the probe outright, rejected as before. GNU is
immune by construction: Homebrew's gtimeout is a separate binary whose
banner is the fixed "timeout (GNU coreutils) 9.11"; nixpkgs' coreutils
is a single-binary multicall that refuses a foreign name ("unknown
program 'gtimeout'"), so it gets no variant here.

Each multicall is now also installed on a case PATH as `gtimeout` ALONE
(`installs`, derived from `implementations`; `backendFor` stays keyed on
the implementation, so the model's claim is "uutils is allowlisted
whatever it is called"). 240 cases on Linux, 80 on darwin (no busybox or
toybox there, and no busybox-ash shell). The shadowed
`out=` inside run_case is `case_out` now: it only survived because every
case is a backgrounded subshell, and a de-backgrounded case would have
died on the final `touch "$out"`.

Mutation-checked: installing the variant as `xtimeout` (a name the shim
never probes) fails exactly the 30 uutils-as-gtimeout cases — 15 with
nothing killing the fake bd, 15 on the perl backend (ALRM) instead of
coreutils; restored, 240 of 240 pass. The busybox/toybox variants cannot
distinguish (their expectation is rejection either way); their weight is
that the probe does not blow up on a foreign name.

* Run the flake checks on main too, by nix flake check; docs wording

pr-risk.yml's test-nix job ran the hook-timeout check on PRs only; the
byte-identical job in main.yml never got the step, so post-merge main
never ran it. Both jobs now run `nix flake check -L`: no hardcoded
`x86_64-linux` (the step beside it, `nix run .#default`, was already
system-agnostic), and a check added to checks.nix runs without a
workflow edit. checks.aarch64-darwin and .x86_64-darwin evaluate.

Measured on ubuntu-latest with a cold Nix store (fork run 35700791032,
this tree, 240 cases): `nix flake check -L` 3.6 s including substitution
of uutils, busybox, toybox, dash and perl; 0.08 s warm — beside the
4 min 9 s the existing `nix run .#default` step spends building bd in
the same job. The 10-minute timeout is not in question.

git-integration.md: on Git for Windows the preferred deadline backend is
"a coreutils timeout (GNU or uutils)", not GNU alone, since both now
qualify. troubleshooting.md: reflow a ragged paragraph.

* review: close the duplicated contracts around the uutils timeout accept

Maintainer review fixes for #6561. No change to the shim's behavior or to the
generated/tracked hook sections.

- cmd/bd/hooks_timeout_process_test.go: the uutils fixture was built with
  strings.Replace, which returns its input unchanged when the pattern is
  absent, so a later edit to the GNU stub's banner would silently degenerate
  the uutils case into a second copy of the GNU one — still green, no longer
  testing the new accept. Hoist the derivation and fail loudly when the banner
  no longer matches.
- cmd/bd/hooks_timeout_process_test.go: findHookProcessGNUTimeout held a second,
  un-widened copy of the identity allowlist, so on exactly the hosts this PR
  exists for (uutils timeout, no gtimeout) the only Go test that drives a real
  timeout binary skipped with a reason the fix had just made false. Widen the
  probe's case, rename it findHookProcessCoreutilsTimeout, and reword the skip
  and the subtest. Measured against stub PATHs: uutils goes rc=1 -> rc=0, GNU
  stays rc=0, a busybox banner stays rc=1.
- cmd/bd/init_hooks_test.go: TestGenerateHookSection_Timeout asserted only the
  GNU alternative, so deleting the new uutils one from the generator left the
  structural contract green (verified: it passes under that mutation before
  this change, and reports "section missing uutils coreutils identity check"
  after). Add the companion assertion.
- cmd/bd/init_hooks_test.go, Makefile: the regen arm rewrites tracked source
  files, so give it its own BD_UPDATE_HOOKS_GOLDEN rather than the package's
  shared BD_UPDATE_GOLDEN — an unscoped `BD_UPDATE_GOLDEN=1 go test ./cmd/bd/`
  used to turn genuine hook drift into a silent working-tree rewrite that
  reported PASS.
- cmd/bd/init_hooks_test.go: locate the managed section by marker prefix
  instead of the versioned marker line. Tracked hooks drift both from generator
  content and from the release version stamp; the exact-line lookup made the
  second class fatal ("tracked hook missing ...") before the regen arm was
  reachable, so `make githooks-regen` failed at exactly the moment a release
  engineer reaches for it. The versioned markers are still verified — they are
  part of the byte compare.
- checks.nix: the `rc -eq 124` safety-net verdict discarded both $case_out and
  the already-captured $signal, so the one failure mode that is intermittent
  and not locally reproducible was also the only one with no diagnostics.
  Report them like the `rc -ne 0` branch below it.
- docs/reference/troubleshooting.md, docs/reference/git-integration.md: mirror
  the coreutils wording across both docs this PR touches, and generalize the
  re-install guidance in both. Managed sections are never reported Outdated
  (getHookVersion returns IsShim for them and computeHookStatuses gates on
  !IsShim), so nothing tells an existing install it is stale; the sentence was
  still scoped to the older name-only check (#5503).
- CHANGELOG.md: add the [Unreleased] entry for #5541, worded to the actual
  scope — with Perl installed the shim still reached the Perl alarm arm, so the
  lost deadline bit uutils hosts without Perl — say that existing installs must
  run `bd hooks install` once, and drop the claim that stock NixOS ships uutils
  (this PR's own checks.nix takes GNU from pkgs.coreutils).

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-27 03:38:45 -07:00

9.4 KiB

Testing Guide

TESTING.md is the single authority for test commands, test selection, and test design in this repository. Use the commands here for local work. For the exact command behind a current CI check, inspect its workflow and corresponding Makefile target; the CI audit and cleanup plan are dated maintainer context.

Choose the Smallest Useful Test

Test at the lowest seam that can fail for the user-visible reason. Add a higher tier only when it covers a distinct risk that the lower tier cannot show: integration wiring, a real persistence property, a process boundary, or an external contract.

This keeps feedback fast and failures readable. It does not mean every test is a unit test: use the real boundary when the defect could live there.

Need Run When
Docs-only validation git diff --check, go test -tags=gms_pure_go ./test/docsync, and ./scripts/check-doc-freshness.sh For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full Go suite merely because a Markdown file changed.
Focused red/green loop ./scripts/test.sh -run '^TestExactName$' ./path/to/package/... While writing or fixing one behavior.
Affected-package confidence ./scripts/test.sh ./path/to/package/... After the focused test passes; include directly affected neighbors when their contract changed.
Final Go baseline make test Once after focused work on Go code is green. It applies the normal local build flags, coverage, and local skip handling.
Named CI wrapper make ci-pr-core, make ci-pr-policy, or make ci-pr-lint Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit.
Hook shims against real timeout implementations nix flake check -L (or nix build .#checks.<system>.hook-timeout-backends -L) After changing the hook generator in cmd/bd/hooks.go (then make githooks-regen) or anything under .githooks/. Runs the tracked managed sections against GNU coreutils, uutils, busybox and toybox timeout — the multicalls also installed as gtimeout alone — with and without Perl, under dash, bash and busybox ash. About one deadline of wall time; needs no Go build.

Do not replace the focused loop with repeated full-suite runs. Run the final make test once the affected Go tests are green. For docs-only changes, use the docs, link, and diff checks instead.

Commands and Local Environment

./scripts/test.sh is the normal runner. It sources .buildflags, creates an isolated test environment, applies .test-skip, and defaults to a 25m per-package Go-test timeout. The timeout is a hang backstop, not a target runtime. Override it only when diagnosing a legitimate slow path:

TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/...
TEST_VERBOSE=1 ./scripts/test.sh ./cmd/bd/...
TEST_RUN='^TestExactName$' ./scripts/test.sh ./cmd/bd/...

# Equivalent command-line options.
./scripts/test.sh -v -run '^TestExactName$' ./cmd/bd/...
./scripts/test.sh -timeout 30m ./cmd/bd/...

Use the opt-in ICU regex path only when the change requires it:

make test-icu-path

It is maintainer-only and not part of normal validation. make test-full-cgo and ./scripts/test-cgo.sh remain deprecated compatibility aliases.

Use a named specialized target only when its risk is in scope:

make test-regression
make test-upgrade
make test-cross-version
make test-migration

For a failing GitHub Actions check, follow the current workflow and its Makefile target when exact reproduction matters. The local runner and CI intentionally have different contracts in some cases.

Test Environment and Readiness

The runner isolates HOME, Git configuration, and Dolt state. By default its test environment adds dolt to BEADS_TEST_SKIP; set BEADS_TEST_ENV_RUN_DOLT=1 only when deliberately exercising the Dolt path and its prerequisites are available. Do not make ordinary tests depend on a developer's database, daemon, global Git configuration, or filesystem state.

To skip an optional service explicitly, use the existing skip mechanism:

BEADS_TEST_SKIP=dolt ./scripts/test.sh ./...

Tests that need a temporary repository or store should use t.TempDir() and t.Cleanup(). Temporary repositories must set a repository-local hooks path; do not inherit the developer's global hooks configuration.

In cmd/bd, fresh-workspace command fixtures should call isolateBeadsDirForTest(t) before setup or dispatch. It clears inherited BEADS_DIR and restores that variable exactly at cleanup, even after raw command-dispatch mutations. The TestMain reset only isolates startup. These fixtures must not use t.Parallel(). Tests intentionally selecting a workspace should use t.Setenv("BEADS_DIR", ...); initConfigForTest and ensureCleanGlobalState preserve that selection.

For manual CLI experiments, run both initialization and subsequent commands from a disposable working directory:

beads_manual_dir="$(mktemp -d)"
(
  set -e
  cd "$beads_manual_dir"
  bd init --quiet --prefix test --skip-hooks --skip-agents
  bd create "Test issue" -p 1
)
rm -rf -- "$beads_manual_dir"

BEADS_DB selects a database for database-opening commands, but it does not by itself redirect bd init workspace setup. Never run a manual bd init from a production workspace merely because BEADS_DB points elsewhere.

testing.Short() is for genuine runtime, stress, or large-fixture skips. It is not a substitute for declaring an integration, end-to-end, API, Docker, or external-dependency boundary. Keep new uses within the repository policy:

make check-testing-short

Test Design

Seams, Scenarios, and Doubles

Write one scenario at the smallest seam that demonstrates the behavior. Cover the boundary or failure mode that changes the user result; use table-driven subtests when examples share setup. Do not repeat the same scenario through a helper, every caller, and the CLI merely because all are available.

A recording double or fake should be narrow: model only the calls, inputs, outputs, and failures the test needs. It should not recreate a storage engine, process manager, or another subsystem just to make a unit test look realistic.

A behavioral fake is different. If it stands in for a contract shared by multiple production implementations, give it the same semantic-conformance suite as those implementations. That shared suite defines observable behavior; it prevents the fake from teaching callers a contract production code does not honor.

Semantic conformance asks whether an implementation produces the promised results, errors, and state transitions for the same operation. Persistence conformance asks whether the real persistence boundary preserves its required durability, transaction, migration, and recovery properties. They answer different questions. Do not claim backend parity unless a stated contract and its conformance suite establish it.

Tier Admission

An integration test belongs above the unit seam only when it covers a distinct boundary that a narrow double cannot prove, such as configuration wiring, a real filesystem or Git interaction, a subprocess protocol, or persistence behavior.

An end-to-end test is admitted only when all of these are true:

  1. The failure would be user-visible.
  2. A real process, setup, or wiring boundary owns a distinct failure risk that no lower seam can prove.
  3. Lower-tier tests cover the underlying behavior where practical, leaving the end-to-end test focused on that boundary.
  4. No existing end-to-end test already covers the same boundary risk.

Lower-tier coverage of the same user journey does not disqualify the end-to-end test; duplicate coverage of the same boundary risk does. State that risk in the test name or nearby documentation.

Avoid Incidental Complexity

Avoid these patterns unless they are the behavior under test:

  • Duplicate scenarios at several layers.
  • Global-state reset choreography. Prefer per-test state and cleanup.
  • Subprocesses, listeners, sleeps, or real-store setup for a unit-level claim.
  • Assertions on private implementation shape when observable behavior is the contract.
  • Performance claims without a repeatable measurement and stated workload.

Sleeps, listeners, and real stores are appropriate when the test is specifically about timing, lifecycle, protocol, or persistence. Keep the setup scoped and make that reason apparent.

Failures, Skips, and Review

.test-skip is a local, temporary exception list. If an unrelated failure is already listed, report it rather than silently broadening the skip. Before adding a new skip, record the issue it tracks and remove the skip when the underlying failure is fixed.

Before opening a PR:

  1. For docs-only changes, run the applicable docs, link, freshness, and diff checks; do not run the full Go suite by default.
  2. For Go code, keep the focused and affected-package tests green, then run one final make test.
  3. Run only the named CI wrapper, specialized target, or risk gate required by the changed surface, or the one needed to reproduce a CI result.

For historical CI inventory and maintainer planning context, consult CI_TEST_SURFACE_AUDIT.md and CI_CLEANUP_PLAN.md. For current commands, use the workflow files and Makefile; these context documents are not a second testing guide or a live CI inventory.