docs(test): make TESTING.md canonical (bd-mrs20) (#5226)

Consolidate test execution and design guidance around the smallest trustworthy seam, narrow doubles, semantic versus persistence conformance, strict E2E admission, and proportional verification. Convert the former philosophy documents to compatibility pointers.

Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot

Co-authored-by: CI Bot <ci@beads.test>
This commit is contained in:
Julian Knutsen 2026-08-01 02:30:21 -07:00 • committed by GitHub
commit e9a12ed3b1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 195 additions and 564 deletions

View file

@ -88,16 +88,10 @@ echo 'Description with `backticks` and "quotes"' | bd create "Title" --descripti
echo 'Updated text' | bd update <id> --description=-
```
## Testing Commands (No Ambiguity)
## Testing
- Default local test command: `make test` (or `./scripts/test.sh`).
- Opt-in ICU regex path: `make test-icu-path` (or `./scripts/test-icu-path.sh ./...`).
- This ICU path is maintainer-only and not part of normal validation; `make test-full-cgo` and `./scripts/test-cgo.sh` are deprecated aliases.
- For package- or test-scoped shipped-config CGO runs, prefer:
```bash
CGO_ENABLED=1 go test -tags gms_pure_go ./cmd/bd/...
CGO_ENABLED=1 go test -tags gms_pure_go -run '^TestName$' ./cmd/bd/...
```
Use [engdocs/TESTING.md](engdocs/TESTING.md) for the canonical commands,
test-design guidance, and PR-readiness gates.
## Non-Interactive Shell Commands

View file

@ -10,7 +10,7 @@ This document contains detailed operational instructions for AI agents working o
- **Go version**: see `go.mod` for the required version (currently 1.26+)
- **Linting**: `golangci-lint run ./...` (baseline warnings documented in [engdocs/LINTING.md](engdocs/LINTING.md))
- **Testing**: All new features need tests (`make test` for the normal local/CI path, `make test-icu-path` only when intentionally exercising the opt-in ICU regex path)
- **Testing**: All new features need tests; use [engdocs/TESTING.md](engdocs/TESTING.md) for command selection, test design, and PR readiness.
- **Documentation**: Update relevant .md files
### File Organization
@ -30,17 +30,26 @@ beads/
**IMPORTANT:** Never pollute the production database with test issues!
**For manual testing**, use the `BEADS_DB` environment variable to point to a temporary database:
For test commands, tier selection, and test-design guidance, use the canonical
[engdocs/TESTING.md](engdocs/TESTING.md).
**For manual testing**, keep initialization and all experiments in a disposable
working directory. This safety example supplements the canonical testing guide:
```bash
# Create test issues in isolated database
BEADS_DB=/tmp/test.db bd init --quiet --prefix test
BEADS_DB=/tmp/test.db bd create "Test issue" -p 1
# Or for quick testing
BEADS_DB=/tmp/test.db bd create "Test feature" -p 1
beads_manual_dir="$(mktemp -d)"
(
set -e
cd "$beads_manual_dir"
bd init --quiet --prefix test --skip-hooks --skip-agents
bd create "Test issue" -p 1
)
rm -rf -- "$beads_manual_dir"
```
`BEADS_DB` alone does not redirect `bd init` workspace setup. Do not run manual
initialization from a production workspace even when selecting another database.
**For automated tests**, use `t.TempDir()` in Go tests:
```go
@ -61,7 +70,9 @@ git config core.hooksPath .git/hooks
Do not rely on the developer's global git config. Global `core.hooksPath` can leak
into temp repos and produce flaky test behavior.
**Warning:** bd will warn you when creating issues with "Test" prefix in the production database. Always use `BEADS_DB` for manual testing.
**Warning:** bd will warn you when creating issues with a "Test" prefix in the
production database. The disposable working directory is the isolation
boundary for manual initialization and experiments.
**Tmpfs hosts:** the `cmd/bd` test suite creates an isolated `$HOME` and several
test binaries under `$TMPDIR`. They are normally cleaned by the test process,
@ -71,9 +82,9 @@ test runs if `du -sh /tmp/beads-* /tmp/bd-*` shows accumulation. See bd-3q2u.
### Before Committing
1. **Run tests**: `make test` (or `./scripts/test.sh`)
- Only if intentionally exercising the ICU regex path: `make test-icu-path`
2. **Run linter**: `golangci-lint run ./...` (ignore baseline warnings)
1. **Run tests**: follow [engdocs/TESTING.md](engdocs/TESTING.md).
2. **Run linter when its code surface changed**: `golangci-lint run ./...`
(ignore baseline warnings)
3. **Update docs**: If you changed behavior, update README.md or other docs
4. **Commit**: With git hooks installed (`bd hooks install`), Dolt changes are auto-committed
@ -375,28 +386,21 @@ case types.StatusClosed:
4. Link from `examples/README.md`
5. Mention in main README.md
## Building and Testing
## Building
```bash
# Build and install bd to ~/.local/bin (the canonical location)
make install
# Test (local baseline)
make test
# Optional ICU regex path smoke (maintainer-only, not normal validation)
make test-icu-path
# Coverage run
go test -tags gms_pure_go -coverprofile=coverage.out ./...
go tool cover -html=coverage.out
# Verify installed binary
bd init --prefix test
bd create "Test issue" -p 1
bd ready
```
For testing commands, test design, and PR-readiness gates, use
[engdocs/TESTING.md](engdocs/TESTING.md).
> **WARNING**: Do NOT use `go build -o bd ./cmd/bd`, `go install ./cmd/bd`,
> or raw `go run ./cmd/bd ...`.
> These bypass the canonical build path, can create stale binaries in the

View file

@ -44,20 +44,10 @@ beads/
## Running Tests
```bash
# Run all tests (recommended — uses correct build tags)
make test
# Run tests with coverage
go test -tags gms_pure_go -v -coverprofile=coverage.out ./...
go tool cover -html=coverage.out
# Run specific package tests
go test -tags gms_pure_go ./internal/storage/dolt/ -v
# Run tests with race detection
go test -tags gms_pure_go -race ./...
```
Use the canonical [testing guide](engdocs/TESTING.md) to choose focused tests,
the proportional validation budget, and any applicable CI wrapper. The setup
and safety notes in this file supplement that guide; they do not define a
second test policy.
## Code Style
@ -139,22 +129,19 @@ If you are contributing code that involves AI decision-making or orchestration,
## Testing Guidelines
For how to run tests, see [engdocs/TESTING.md](engdocs/TESTING.md). For what to
test and why (the test pyramid and tiering we follow), see
[engdocs/TESTING_PHILOSOPHY.md](engdocs/TESTING_PHILOSOPHY.md).
For test commands, test design, and PR-readiness gates, see the canonical
[engdocs/TESTING.md](engdocs/TESTING.md).
### Before Opening a PR
- Run `make test` (or `./scripts/test.sh`) locally and make sure it passes.
- Add tests for new functionality; extend existing tests when fixing bugs.
- Write table-driven tests for multiple scenarios, use descriptive test
names, use `t.Run()` for subtests, and clean up resources (database
files, etc.) in test teardown.
- Follow the proportional validation budget in
[engdocs/TESTING.md](engdocs/TESTING.md): docs-only changes use docs checks;
Go changes use focused and affected-package tests plus one final `make test`.
- If you hit a test failure unrelated to your change, don't silently skip
it -- check `.test-skip` and file an issue if it's not already tracked
(see [engdocs/TESTING.md](engdocs/TESTING.md#known-broken-tests)).
- Ensure CI passes (`make ci-pr-core`, `make ci-pr-policy`, `make
ci-pr-lint`) before requesting review.
(see [engdocs/TESTING.md](engdocs/TESTING.md#failures-skips-and-review)).
- Run a named CI wrapper only when its risk or surface is affected, or when
reproducing that CI check.
- If your change touches ICU or build tags, see
[engdocs/ICU-POLICY.md](engdocs/ICU-POLICY.md) for the policy and rationale.

View file

@ -61,19 +61,18 @@ See `internal/types/types.go`:
## Development Command Source
Use [AGENT_INSTRUCTIONS.md](../AGENT_INSTRUCTIONS.md#testing-commands-no-ambiguity)
for the current command policy. This file should not duplicate command matrices
or version-management workflows.
Use the canonical [TESTING.md](TESTING.md) for test commands, test design, and
PR-readiness gates. This file should not duplicate command matrices or
version-management workflows.
> **Do NOT** use `go build -o bd` or `go install` directly — they create
> stale binaries that shadow `~/.local/bin/bd`. Always use `make install`.
## Testing Philosophy
## Testing
Testing guidance lives in [TESTING.md](TESTING.md) and
[TESTING_PHILOSOPHY.md](TESTING_PHILOSOPHY.md). Architecture-specific notes for
Claude are limited to where tests touch agent setup, hooks, or instruction-file
generation.
Testing guidance lives in [TESTING.md](TESTING.md). Architecture-specific notes
for Claude are limited to where tests touch agent setup, hooks, or
instruction-file generation.
## Important Notes

View file

@ -58,9 +58,9 @@ Follow-up automation should replace marker-only checks with generated or
| `CLAUDE_INTEGRATION.md` | Keep | Design/user guide for Claude setup; paired with `SETUP.md`. |
| `CLAUDE.md` | Revise | Kept as architecture orientation only; command/workflow duplication was reduced in favour of root `AGENTS.md` and `AGENT_INSTRUCTIONS.md`. |
| `CLI_REFERENCE.md` | Keep/generated | Generated command reference from live help output. |
| `CI_CLEANUP_PLAN.md` | Keep with freshness | Accepted CI tier decisions, package gate policy, measurement plan, and implementation order. |
| `CI_CLEANUP_PLAN.md` | Keep as maintainer context | Dated CI policy, measurements, and implementation roadmap; current commands live in workflows and the `Makefile`. |
| `CI_REQUIRED_CHECK_TOPOLOGY.md` | Keep | Design note for the required-check aggregate gate topology; explicitly flags pre-rollout status and gates branch-protection changes on the new checks passing first. |
| `CI_TEST_SURFACE_AUDIT.md` | Keep with freshness | Snapshot of local validation commands, GitHub Actions coverage, gaps, and CI cleanup roadmap. |
| `CI_TEST_SURFACE_AUDIT.md` | Keep as historical snapshot | Dated test-surface inventory; current commands live in workflows and the `Makefile`. |
| `plugins/beads/skills/beads/resources/CLI_REFERENCE.md` | Keep pointer | Plugin skill resource intentionally points at live CLI sources to avoid duplicate generated command snapshots. |
| `CODEX_INTEGRATION.md` | Keep | User-facing Codex integration guide. |
| `COLLISION_MATH.md` | Keep | Mathematical background; low product drift. |
@ -101,7 +101,7 @@ Follow-up automation should replace marker-only checks with generated or
| `PROJECT_CHARTER.md` | Keep | Canonical product scope and boundary policy. |
| `PROTECTED_BRANCHES.md` | Keep | User-facing protected-branch workflow. |
| `QUICKSTART.md` | Keep pointer | Short pointer to website quickstart; low drift. |
| `README_TESTING.md` | Consolidated pointer | Old duplicate staged; active path points to `TESTING.md` and `TESTING_PHILOSOPHY.md`. |
| `README_TESTING.md` | Compatibility pointer | Points to the sole canonical testing guide, `TESTING.md`. |
| `RECOVERY.md` | Keep with freshness | Runtime recovery playbooks; marker tied to init safety constants/tests. |
| `RELATED_PROJECTS.md` | Keep | Curated list of adjacent, non-integration projects; companion to `COMMUNITY_TOOLS.md`. |
| `RELEASE-STABILITY-GATE.md` | Keep | Maintainer release gate policy. |
@ -116,8 +116,8 @@ Follow-up automation should replace marker-only checks with generated or
| `superpowers/specs/2026-05-03-unclaim-command-design.md` | Removed | PR-passenger design-spec artifact paired with the plans file above; same removal rationale, content stays in git history. |
| `SYNC_CONCEPTS.md` | Keep/canonical | Sanctioned website-mirror source for the sync conceptual model (Dolt source of truth, wire format, JSONL export, anti-patterns); listed in `deploy-docs.yml` and linked from `AGENTS.md`, `README.md`, `ARCHITECTURE.md`, `DOLT.md`, `GIT_INTEGRATION.md`, and `FAQ.md`. |
| `SYNC_SETUP.md` | Revise | Links now point at canonical `DOLT.md`. |
| `TESTING.md` | Revise/canonical | Canonical test-running guide; stale line/test-count and stale skip entry removed. |
| `TESTING_PHILOSOPHY.md` | Keep | Canonical test-design guidance; duplicate/historical links removed. |
| `TESTING.md` | Keep/canonical | Sole authority for test commands, test design, and PR-readiness guidance. |
| `TESTING_PHILOSOPHY.md` | Compatibility pointer | Points to the sole canonical testing guide, `TESTING.md`. |
| `TODO.md` | Keep | Behaviour/user guide for `bd todo`. |
| `TROUBLESHOOTING.md` | Keep | User-facing recovery guide; debug/env tables need freshness review when edited. |
| `UI_PHILOSOPHY.md` | Keep | Design philosophy. |

View file

@ -1,9 +1,3 @@
# Testing Strategy
# Testing
The active testing workflow lives in [TESTING.md](TESTING.md).
Testing rationale and test-design guidance live in
[TESTING_PHILOSOPHY.md](TESTING_PHILOSOPHY.md).
The previous duplicate strategy note has been staged at
[staged-for-removal/README_TESTING.md](staged-for-removal/README_TESTING.md).
This compatibility page points to the canonical [Testing Guide](TESTING.md).

View file

@ -1,286 +1,194 @@
# Testing Guide
## Overview
`TESTING.md` is the single authority for test commands, test selection, and
test design in this repository. Use the commands here for local work. For the
exact command behind a current CI check, inspect its workflow and corresponding
`Makefile` target; the CI audit and cleanup plan are dated maintainer context.
The beads project uses Go tests plus repository wrapper scripts. Prefer the
wrapper scripts for local validation because they apply the repository's normal
local build flags, skip policy, and timeout policy. The current GitHub Actions
PR contract still runs direct `go test` commands; the CI cleanup plan will move
that contract behind dedicated `scripts/ci/*` wrappers.
## Choose the Smallest Useful Test
## Test Performance
Test at the lowest seam that can fail for the user-visible reason. Add a
higher tier only when it covers a distinct risk that the lower tier cannot
show: integration wiring, a real persistence property, a process boundary, or
an external contract.
- Go compilation dominates full-suite runtime.
- Target package/test runs are usually the fastest way to validate focused changes.
- Docker-backed Dolt integration tests auto-detect prerequisites and skip when unavailable.
This keeps feedback fast and failures readable. It does not mean every test is
a unit test: use the real boundary when the defect could live there.
## Running Tests
| Need | Run | When |
|---|---|---|
| Docs-only validation | `git diff --check`, `go test -tags=gms_pure_go ./test/docsync`, and `./scripts/check-doc-freshness.sh` | For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full Go suite merely because a Markdown file changed. |
| Focused red/green loop | `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...` | While writing or fixing one behavior. |
| Affected-package confidence | `./scripts/test.sh ./path/to/package/...` | After the focused test passes; include directly affected neighbors when their contract changed. |
| Final Go baseline | `make test` | Once after focused work on Go code is green. It applies the normal local build flags, coverage, and local skip handling. |
| Named CI wrapper | `make ci-pr-core`, `make ci-pr-policy`, or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. |
### Quick Start
Do not replace the focused loop with repeated full-suite runs. Run the final
`make test` once the affected Go tests are green. For docs-only changes, use
the docs, link, and diff checks instead.
## Commands and Local Environment
`./scripts/test.sh` is the normal runner. It sources `.buildflags`, creates an
isolated test environment, applies `.test-skip`, and defaults to a **25m
per-package** Go-test timeout. The timeout is a hang backstop, not a target
runtime. Override it only when diagnosing a legitimate slow path:
```bash
# Run all tests (auto-skips known broken tests)
make test
TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/...
TEST_VERBOSE=1 ./scripts/test.sh ./cmd/bd/...
TEST_RUN='^TestExactName$' ./scripts/test.sh ./cmd/bd/...
# Or directly:
./scripts/test.sh
# Equivalent command-line options.
./scripts/test.sh -v -run '^TestExactName$' ./cmd/bd/...
./scripts/test.sh -timeout 30m ./cmd/bd/...
```
# Run opt-in ICU regex path tests (maintainer-only, not normal validation)
Use the opt-in ICU regex path only when the change requires it:
```bash
make test-icu-path
# Run specific package
./scripts/test.sh ./cmd/bd/...
# Run specific test pattern
./scripts/test.sh -run TestCreate ./cmd/bd/...
# Verbose output
./scripts/test.sh -v
```
### Environment Variables
It is maintainer-only and not part of normal validation. `make test-full-cgo`
and `./scripts/test-cgo.sh` remain deprecated compatibility aliases.
Use a named specialized target only when its risk is in scope:
```bash
# Set custom timeout (default: 3m)
TEST_TIMEOUT=5m ./scripts/test.sh
# Enable verbose output
TEST_VERBOSE=1 ./scripts/test.sh
# Run specific pattern
TEST_RUN=TestCreate ./scripts/test.sh
make test-regression
make test-upgrade
make test-cross-version
make test-migration
```
### Docker (Dolt Integration Tests)
For a failing GitHub Actions check, follow the current workflow and its
`Makefile` target when exact reproduction matters. The local runner and CI
intentionally have different contracts in some cases.
Dolt integration tests require Docker with the exact Dolt image cached locally.
Tests auto-detect the environment and skip gracefully — no manual configuration
needed.
### Test Environment and Readiness
#### Readiness states
The runner isolates `HOME`, Git configuration, and Dolt state. By default its
test environment adds `dolt` to `BEADS_TEST_SKIP`; set
`BEADS_TEST_ENV_RUN_DOLT=1` only when deliberately exercising the Dolt path
and its prerequisites are available. Do not make ordinary tests depend on a
developer's database, daemon, global Git configuration, or filesystem state.
```csv
State,Condition,Behavior
doltSkipped,BEADS_TEST_SKIP contains "dolt",Silent skip (no warning)
doltNoDocker,Docker daemon not reachable,WARN + skip
doltNoImage,No Dolt image at all,WARN + skip with pull instruction
doltWrongVersion,Image repo cached but wrong tag,WARN + skip with pull instruction
doltReady,Exact image cached and Docker running,Run tests
```
States are checked once per test binary and cached. Order of evaluation:
`BEADS_TEST_SKIP` → Docker availability → exact image → any image version.
#### Skipping Dolt tests explicitly
Set `BEADS_TEST_SKIP` to opt out without Docker overhead (~1s `docker info`):
To skip an optional service explicitly, use the existing skip mechanism:
```bash
# Skip Dolt tests silently
BEADS_TEST_SKIP=dolt ./scripts/test.sh
# Skip multiple services (comma-separated)
BEADS_TEST_SKIP=dolt,slow ./scripts/test.sh
BEADS_TEST_SKIP=dolt ./scripts/test.sh ./...
```
### Short Mode and Test Boundaries
Tests that need a temporary repository or store should use `t.TempDir()` and
`t.Cleanup()`. Temporary repositories must set a repository-local hooks path;
do not inherit the developer's global hooks configuration.
`testing.Short()` is reserved for true runtime, stress, and large-fixture skips.
It must not be used as an implicit integration, e2e, API, Docker, or external
dependency boundary.
Use these mechanisms instead:
- `//go:build integration` or `//go:build e2e` for named suites.
- Environment readiness checks such as `BEADS_TEST_SKIP=dolt`,
`BEADS_TEST_EMBEDDED_DOLT=1`, or required API-key checks.
- Named wrappers such as `make ci-pr-core`, the main integration shards, and the
package gate wrappers.
Run `make check-testing-short` to verify that new `testing.Short()` usage stays
within the approved runtime/stress/large-fixture allowlist. The PR policy wrapper
runs the same check.
#### Enabling Dolt tests
For manual CLI experiments, run both initialization and subsequent commands
from a disposable working directory:
```bash
# Pull the exact Dolt image to enable integration tests
docker pull dolthub/dolt-sql-server:2.2.0
# Point tests at an existing Dolt server (skips container startup)
BEADS_DOLT_PORT=3308 ./scripts/test.sh
beads_manual_dir="$(mktemp -d)"
(
set -e
cd "$beads_manual_dir"
bd init --quiet --prefix test --skip-hooks --skip-agents
bd create "Test issue" -p 1
)
rm -rf -- "$beads_manual_dir"
```
`BEADS_DOLT_PORT` — when set, tests reuse the server at that port instead of
starting a container. Port 3307 is hardcoded as production and always rejected.
`BEADS_DB` selects a database for database-opening commands, but it does not by
itself redirect `bd init` workspace setup. Never run a manual `bd init` from a
production workspace merely because `BEADS_DB` points elsewhere.
### Advanced Usage
`testing.Short()` is for genuine runtime, stress, or large-fixture skips. It
is not a substitute for declaring an integration, end-to-end, API, Docker, or
external-dependency boundary. Keep new uses within the repository policy:
```bash
# Skip additional tests beyond .test-skip
./scripts/test.sh -skip SomeSlowTest
# Run with custom timeout
./scripts/test.sh -timeout 5m
# Combine flags
./scripts/test.sh -v -run TestCreate ./internal/beads/...
make check-testing-short
```
## Known Broken Tests
## Test Design
Tests in `.test-skip` are automatically skipped by `scripts/test.sh`.
### Seams, Scenarios, and Doubles
At the time of this review, `.test-skip` contains only comments and no active
test-name patterns. Treat any new skip as a temporary exception: file the
upstream issue first, record it in `.test-skip`, and remove the skip when the
test is fixed.
Write one scenario at the smallest seam that demonstrates the behavior. Cover
the boundary or failure mode that changes the user result; use table-driven
subtests when examples share setup. Do not repeat the same scenario through a
helper, every caller, and the CLI merely because all are available.
## For Claude Code / AI Agents
A recording double or fake should be narrow: model only the calls, inputs,
outputs, and failures the test needs. It should not recreate a storage engine,
process manager, or another subsystem just to make a unit test look realistic.
When running tests during development:
A behavioral fake is different. If it stands in for a contract shared by
multiple production implementations, give it the same semantic-conformance
suite as those implementations. That shared suite defines observable behavior;
it prevents the fake from teaching callers a contract production code does not
honor.
### Best Practices
Semantic conformance asks whether an implementation produces the promised
results, errors, and state transitions for the same operation. Persistence
conformance asks whether the real persistence boundary preserves its required
durability, transaction, migration, and recovery properties. They answer
different questions. Do not claim backend parity unless a stated contract and
its conformance suite establish it.
1. **Use the test script:** Always use `./scripts/test.sh` instead of `go test` directly
- Automatically skips known broken tests
- Uses appropriate timeouts
- Matches local default validation; use future `scripts/ci/*` wrappers when
reproducing exact CI jobs
- Only if intentionally exercising the ICU regex path, use `./scripts/test-icu-path.sh` (or deprecated `make test-full-cgo`)
### Tier Admission
2. **Target specific tests when possible:**
```bash
# Instead of running everything:
./scripts/test.sh
An integration test belongs above the unit seam only when it covers a distinct
boundary that a narrow double cannot prove, such as configuration wiring, a
real filesystem or Git interaction, a subprocess protocol, or persistence
behavior.
# Run just what you changed:
./scripts/test.sh -run TestSpecificFeature ./cmd/bd/...
```
An end-to-end test is admitted only when all of these are true:
3. **Compilation is the bottleneck:**
- The 180-second compilation time dominates
- Individual tests are fast
- Use `-run` to avoid recompiling unnecessarily
1. The failure would be user-visible.
2. A real process, setup, or wiring boundary owns a distinct failure risk that
no lower seam can prove.
3. Lower-tier tests cover the underlying behavior where practical, leaving the
end-to-end test focused on that boundary.
4. No existing end-to-end test already covers the same boundary risk.
4. **Check for new failures:**
```bash
# If you see a new failure, check if it's known:
cat .test-skip
```
Lower-tier coverage of the same user journey does not disqualify the
end-to-end test; duplicate coverage of the same boundary risk does. State that
risk in the test name or nearby documentation.
### Adding Tests to Skip List
### Avoid Incidental Complexity
If you discover a broken test:
Avoid these patterns unless they are the behavior under test:
1. File a GitHub issue documenting the problem
2. Add to `.test-skip`:
```bash
# Issue #NNN: Brief description
TestNameToSkip
```
3. Tests in `.test-skip` support regex patterns
- Duplicate scenarios at several layers.
- Global-state reset choreography. Prefer per-test state and cleanup.
- Subprocesses, listeners, sleeps, or real-store setup for a unit-level claim.
- Assertions on private implementation shape when observable behavior is the
contract.
- Performance claims without a repeatable measurement and stated workload.
## Test Organization
Sleeps, listeners, and real stores are appropriate when the test is specifically
about timing, lifecycle, protocol, or persistence. Keep the setup scoped and
make that reason apparent.
### Slowest Tests (>0.05s)
## Failures, Skips, and Review
The top slow tests in cmd/bd:
- `TestDoctorWithBeadsDir` (1.68s) - Only significantly slow test
- `TestFlushManagerDebouncing` (0.21s)
- `TestDebouncer_*` tests (0.06-0.12s each) - Intentional sleeps for concurrency testing
- `TestMultiWorkspaceDeletionSync` (0.12s)
`.test-skip` is a local, temporary exception list. If an unrelated failure is
already listed, report it rather than silently broadening the skip. Before
adding a new skip, record the issue it tracks and remove the skip when the
underlying failure is fixed.
Most tests are <0.01s and very fast.
Before opening a PR:
### Package Structure
1. For docs-only changes, run the applicable docs, link, freshness, and diff
checks; do not run the full Go suite by default.
2. For Go code, keep the focused and affected-package tests green, then run one
final `make test`.
3. Run only the named CI wrapper, specialized target, or risk gate required by
the changed surface, or the one needed to reproduce a CI result.
```
cmd/bd/ - Main CLI tests (82 test files, most of the suite)
internal/beads/ - Core beads library tests
internal/storage/ - Storage backend tests (Dolt embedded/server, conformance)
internal/rpc/ - RPC protocol tests
internal/*/ - Various internal package tests
```
## Continuous Integration
The current CI workflow does not yet call the `scripts/ci/*` wrappers for every
job. Until workflow migration is complete, use the command documented in the
failing workflow when reproducing an existing status check exactly.
Use `scripts/test.sh` for local default validation and targeted development
runs.
Use the CI wrappers for the accepted target PR contracts:
```bash
make ci-pr-core
make ci-pr-policy
make ci-pr-lint
```
Use the package gate wrappers when touching package or docs surfaces:
```bash
make ci-package-mcp
make ci-package-npm
```
### Coverage Signal Policy
PR confidence is based on behavior checks, not raw coverage percentage.
- Treat Codecov percentages as informational trend data.
- Prefer focused tests for risky paths (storage, sync/git, migrations, state
transitions, and corruption/integrity handling) over broad line-coverage
churn.
- Add or extend at least one targeted regression test when fixing risky logic.
- Do not block a change solely on overall coverage movement when behavioral
checks are strong.
For the current CI/test-surface inventory and cleanup roadmap, see
[CI_TEST_SURFACE_AUDIT.md](CI_TEST_SURFACE_AUDIT.md). That audit documents
where local commands and GitHub Actions currently diverge before the CI cleanup
work starts changing workflow behavior.
For accepted CI tier decisions and the implementation order, see
[CI_CLEANUP_PLAN.md](CI_CLEANUP_PLAN.md).
## Debugging Test Failures
### Get detailed output
```bash
./scripts/test.sh -v ./path/to/package/...
```
### Run a single test
```bash
./scripts/test.sh -run '^TestExactName$' ./cmd/bd/...
```
### Check which tests are being skipped
```bash
./scripts/test.sh 2>&1 | head -5
```
Output shows:
```
Running: go test -timeout 3m -skip TestFoo|TestBar ./...
Skipping: TestFoo|TestBar
```
## Contributing
When adding new tests:
1. Keep tests fast (<0.1s if possible)
2. Use `t.Parallel()` for independent tests
3. Clean up resources in `t.Cleanup()` or `defer`
4. Avoid sleeps unless testing concurrency
When tests break:
1. Fix them if possible
2. If unfixable right now, file an issue and add to `.test-skip`
3. Document the issue in `.test-skip` with issue number
For historical CI inventory and maintainer planning context, consult
[CI_TEST_SURFACE_AUDIT.md](CI_TEST_SURFACE_AUDIT.md) and
[CI_CLEANUP_PLAN.md](CI_CLEANUP_PLAN.md). For current commands, use the
workflow files and `Makefile`; these context documents are not a second testing
guide or a live CI inventory.

View file

@ -1,259 +1,4 @@
# Testing Philosophy
This document covers **what to test** and **what not to test**. For how to run tests, see [TESTING.md](TESTING.md).
## The Test Pyramid
```
┌─────────────────┐
│ E2E Tests │ ← PR/Deploy only (slow, expensive)
│ ~5% of tests │
└────────┬────────┘
│
┌──────────────┴──────────────┐
│ Integration Tests │ ← PR gate (moderate)
│ ~15% of tests │
└──────────────┬──────────────┘
│
┌────────────────────────┴────────────────────────┐
│ Unit Tests (Fast) │ ← Every save/commit
│ ~80% of tests │
└─────────────────────────────────────────────────┘
```
### Tier 1: Fast Tests (< 5 seconds total)
**When**: Every file save, pre-commit hooks, continuous during development
- Pure function tests (no I/O)
- In-memory data structure tests
- Business logic validation
- Mock all external dependencies
**In beads**: Core logic tests using `newTestStore()` with in-memory SQLite
### Tier 2: Integration Tests (< 30 seconds)
**When**: Pre-push, PR checks
- Real file system operations
- Git operations with temp repos
- Config file parsing
- CLI argument handling
**In beads**: Tests tagged with `//go:build integration`, server mode tests
### Tier 3: E2E / Smoke Tests (1-5 minutes)
**When**: PR merge, pre-deploy, nightly
- Full `bd init` → `bd doctor` → `bd doctor --fix` workflow
- Real API calls (to staging)
- Cross-platform verification
---
## What Makes a Test "Right"
A good test:
1. **Catches a bug you'd actually ship** - not theoretical edge cases
2. **Documents expected behavior** - serves as living documentation
3. **Runs fast enough to not skip** - slow tests get disabled
4. **Isn't duplicated elsewhere** - tests one thing, one way
---
## What to Test (Priority Matrix)
| Priority | What | Why | Examples in beads |
|----------|------|-----|-------------------|
| **High** | Core business logic | This is what users depend on | `sync`, `doctor`, `export`, `import` |
| **High** | Error paths that could corrupt data | Data loss is catastrophic | Config handling, git operations, database integrity |
| **Medium** | Edge cases from production bugs | Discovered through real issues | Orphan handling, ID collision detection |
| **Low** | Display/formatting | Visual output, can be manually verified | Table formatting, color output |
---
## What NOT to Test Extensively
### Simple utility functions
Trust the language. Don't test that `strings.TrimSpace` works.
### Every permutation of inputs
Use table-driven tests with representative cases instead of exhaustive permutations.
```go
// BAD: 10 separate test functions
func TestPriority0(t *testing.T) { ... }
func TestPriority1(t *testing.T) { ... }
func TestPriority2(t *testing.T) { ... }
// GOOD: One table-driven test
func TestPriorityMapping(t *testing.T) {
cases := []struct{ in, want int }{
{0, 4}, {1, 0}, {5, 3}, // includes boundary
}
for _, tc := range cases {
t.Run(fmt.Sprintf("priority_%d", tc.in), func(t *testing.T) {
got := mapPriority(tc.in)
if got != tc.want { t.Errorf(...) }
})
}
}
```
### Obvious behavior
Don't test "if file exists, return true" - trust the implementation.
### Same logic through different entry points
If you test a function directly, don't also test it through every caller.
---
## Anti-Patterns to Avoid
### 1. Trivial Assertions
Testing obvious happy paths that would pass with trivial implementations.
```go
// BAD: What bug would this catch?
func TestValidateBeadsWorkspace(t *testing.T) {
dir := setupTestWorkspace(t)
if err := validateBeadsWorkspace(dir); err != nil {
t.Errorf("expected no error, got: %v", err)
}
}
// GOOD: Test the interesting error cases
func TestValidateBeadsWorkspace(t *testing.T) {
cases := []struct{
name string
setup func(t *testing.T) string
wantErr string
}{
{"missing .beads dir", setupNoBeadsDir, "not a beads workspace"},
{"corrupted db", setupCorruptDB, "database is corrupted"},
{"permission denied", setupNoReadAccess, "permission denied"},
}
// ...
}
```
### 2. Duplicate Error Path Testing
Testing the same logic multiple ways instead of once with table-driven tests.
```go
// BAD: Repetitive individual assertions
if config.PriorityMap["0"] != 4 { t.Errorf(...) }
if config.PriorityMap["1"] != 0 { t.Errorf(...) }
if config.PriorityMap["2"] != 1 { t.Errorf(...) }
// GOOD: Table-driven
for k, want := range expectedMap {
if got := config.PriorityMap[k]; got != want {
t.Errorf("PriorityMap[%q] = %d, want %d", k, got, want)
}
}
```
### 3. I/O Heavy Tests Without Mocking
Unit tests that execute real commands or heavy I/O when they could mock.
```go
// BAD: Actually executes external commands in unit test
func TestServerFix(t *testing.T) {
exec.Command("bd", "dolt", "stop").Run()
// ...
}
// GOOD: Mock the execution or use integration test tag
func TestServerFix(t *testing.T) {
executor := &mockExecutor{}
fix := NewServerFix(executor)
// ...
}
```
### 4. Testing Implementation, Not Behavior
Tests that break when you refactor, even though behavior is unchanged.
```go
// BAD: Tests internal state
if len(server.connectionPool) != 3 { t.Error(...) }
// GOOD: Tests observable behavior
if resp, err := server.HandleRequest(req); err != nil { t.Error(...) }
```
### 5. Missing Boundary Tests
Testing known-good values but not boundaries and invalid inputs.
```go
// BAD: Only tests middle values
TestPriority(1) // works
TestPriority(2) // works
// GOOD: Tests boundaries and invalid
TestPriority(-1) // invalid - expect error
TestPriority(0) // boundary - min valid
TestPriority(4) // boundary - max valid
TestPriority(5) // boundary - first invalid
```
---
## Target Metrics
| Metric | Target | Current (beads) | Status |
|--------|--------|-----------------|--------|
| Test-to-code ratio | 0.5:1 - 1.5:1 | 0.85:1 | Healthy |
| Fast test suite | < 5 seconds | 3.8 seconds | Good |
| Integration tests | < 30 seconds | ~15 seconds | Good |
| Compilation overhead | Minimize | 180 seconds | Bottleneck |
### Interpretation
- **0.5:1** - Light coverage, fast iteration (acceptable for utilities)
- **1:1** - Solid coverage for most projects (our target)
- **1.5:1** - Heavy coverage for critical systems
- **2:1+** - Over-engineered, maintenance burden
---
## Beads-Specific Guidance
### Well-Covered (Maintain)
| Area | Why It's Well-Tested |
|------|---------------------|
| Sync/Export/Import | Data integrity critical - comprehensive edge cases |
| SQLite transactions | Rollback safety, atomicity guarantees |
| Merge operations | Dolt-native cell-level merge |
| Database locking | Prevents corruption from multiple instances |
### Needs Attention
| Area | Gap | Priority |
|------|-----|----------|
| Server lifecycle | Shutdown/signal handling | Medium |
| Concurrent operations | Stress testing under load | Medium |
| Boundary validation | Edge inputs in mapping functions | Low |
### Skip These
- Display formatting tests (manually verify)
- Simple getters/setters
- Tests that duplicate SQLite's guarantees
---
## Related Docs
- [TESTING.md](TESTING.md) - How to run tests
- [README_TESTING.md](README_TESTING.md) - Stable pointer to the active testing docs
This compatibility page points to the canonical [Testing Guide](TESTING.md),
which covers test design, test selection, and commands.