diff --git a/AGENTS.md b/AGENTS.md index c9215a212..6e34e55a1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,16 +88,10 @@ echo 'Description with `backticks` and "quotes"' | bd create "Title" --descripti echo 'Updated text' | bd update --description=- ``` -## Testing Commands (No Ambiguity) +## Testing -- Default local test command: `make test` (or `./scripts/test.sh`). -- Opt-in ICU regex path: `make test-icu-path` (or `./scripts/test-icu-path.sh ./...`). -- This ICU path is maintainer-only and not part of normal validation; `make test-full-cgo` and `./scripts/test-cgo.sh` are deprecated aliases. -- For package- or test-scoped shipped-config CGO runs, prefer: -```bash -CGO_ENABLED=1 go test -tags gms_pure_go ./cmd/bd/... -CGO_ENABLED=1 go test -tags gms_pure_go -run '^TestName$' ./cmd/bd/... -``` +Use [engdocs/TESTING.md](engdocs/TESTING.md) for the canonical commands, +test-design guidance, and PR-readiness gates. ## Non-Interactive Shell Commands diff --git a/AGENT_INSTRUCTIONS.md b/AGENT_INSTRUCTIONS.md index 27fd62dee..fef57939b 100644 --- a/AGENT_INSTRUCTIONS.md +++ b/AGENT_INSTRUCTIONS.md @@ -10,7 +10,7 @@ This document contains detailed operational instructions for AI agents working o - **Go version**: see `go.mod` for the required version (currently 1.26+) - **Linting**: `golangci-lint run ./...` (baseline warnings documented in [engdocs/LINTING.md](engdocs/LINTING.md)) -- **Testing**: All new features need tests (`make test` for the normal local/CI path, `make test-icu-path` only when intentionally exercising the opt-in ICU regex path) +- **Testing**: All new features need tests; use [engdocs/TESTING.md](engdocs/TESTING.md) for command selection, test design, and PR readiness. - **Documentation**: Update relevant .md files ### File Organization @@ -30,17 +30,26 @@ beads/ **IMPORTANT:** Never pollute the production database with test issues! -**For manual testing**, use the `BEADS_DB` environment variable to point to a temporary database: +For test commands, tier selection, and test-design guidance, use the canonical +[engdocs/TESTING.md](engdocs/TESTING.md). + +**For manual testing**, keep initialization and all experiments in a disposable +working directory. This safety example supplements the canonical testing guide: ```bash -# Create test issues in isolated database -BEADS_DB=/tmp/test.db bd init --quiet --prefix test -BEADS_DB=/tmp/test.db bd create "Test issue" -p 1 - -# Or for quick testing -BEADS_DB=/tmp/test.db bd create "Test feature" -p 1 +beads_manual_dir="$(mktemp -d)" +( + set -e + cd "$beads_manual_dir" + bd init --quiet --prefix test --skip-hooks --skip-agents + bd create "Test issue" -p 1 +) +rm -rf -- "$beads_manual_dir" ``` +`BEADS_DB` alone does not redirect `bd init` workspace setup. Do not run manual +initialization from a production workspace even when selecting another database. + **For automated tests**, use `t.TempDir()` in Go tests: ```go @@ -61,7 +70,9 @@ git config core.hooksPath .git/hooks Do not rely on the developer's global git config. Global `core.hooksPath` can leak into temp repos and produce flaky test behavior. -**Warning:** bd will warn you when creating issues with "Test" prefix in the production database. Always use `BEADS_DB` for manual testing. +**Warning:** bd will warn you when creating issues with a "Test" prefix in the +production database. The disposable working directory is the isolation +boundary for manual initialization and experiments. **Tmpfs hosts:** the `cmd/bd` test suite creates an isolated `$HOME` and several test binaries under `$TMPDIR`. They are normally cleaned by the test process, @@ -71,9 +82,9 @@ test runs if `du -sh /tmp/beads-* /tmp/bd-*` shows accumulation. See bd-3q2u. ### Before Committing -1. **Run tests**: `make test` (or `./scripts/test.sh`) - - Only if intentionally exercising the ICU regex path: `make test-icu-path` -2. **Run linter**: `golangci-lint run ./...` (ignore baseline warnings) +1. **Run tests**: follow [engdocs/TESTING.md](engdocs/TESTING.md). +2. **Run linter when its code surface changed**: `golangci-lint run ./...` + (ignore baseline warnings) 3. **Update docs**: If you changed behavior, update README.md or other docs 4. **Commit**: With git hooks installed (`bd hooks install`), Dolt changes are auto-committed @@ -375,28 +386,21 @@ case types.StatusClosed: 4. Link from `examples/README.md` 5. Mention in main README.md -## Building and Testing +## Building ```bash # Build and install bd to ~/.local/bin (the canonical location) make install -# Test (local baseline) -make test - -# Optional ICU regex path smoke (maintainer-only, not normal validation) -make test-icu-path - -# Coverage run -go test -tags gms_pure_go -coverprofile=coverage.out ./... -go tool cover -html=coverage.out - # Verify installed binary bd init --prefix test bd create "Test issue" -p 1 bd ready ``` +For testing commands, test design, and PR-readiness gates, use +[engdocs/TESTING.md](engdocs/TESTING.md). + > **WARNING**: Do NOT use `go build -o bd ./cmd/bd`, `go install ./cmd/bd`, > or raw `go run ./cmd/bd ...`. > These bypass the canonical build path, can create stale binaries in the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 216fc87ea..e02a1c39d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -44,20 +44,10 @@ beads/ ## Running Tests -```bash -# Run all tests (recommended — uses correct build tags) -make test - -# Run tests with coverage -go test -tags gms_pure_go -v -coverprofile=coverage.out ./... -go tool cover -html=coverage.out - -# Run specific package tests -go test -tags gms_pure_go ./internal/storage/dolt/ -v - -# Run tests with race detection -go test -tags gms_pure_go -race ./... -``` +Use the canonical [testing guide](engdocs/TESTING.md) to choose focused tests, +the proportional validation budget, and any applicable CI wrapper. The setup +and safety notes in this file supplement that guide; they do not define a +second test policy. ## Code Style @@ -139,22 +129,19 @@ If you are contributing code that involves AI decision-making or orchestration, ## Testing Guidelines -For how to run tests, see [engdocs/TESTING.md](engdocs/TESTING.md). For what to -test and why (the test pyramid and tiering we follow), see -[engdocs/TESTING_PHILOSOPHY.md](engdocs/TESTING_PHILOSOPHY.md). +For test commands, test design, and PR-readiness gates, see the canonical +[engdocs/TESTING.md](engdocs/TESTING.md). ### Before Opening a PR -- Run `make test` (or `./scripts/test.sh`) locally and make sure it passes. -- Add tests for new functionality; extend existing tests when fixing bugs. -- Write table-driven tests for multiple scenarios, use descriptive test - names, use `t.Run()` for subtests, and clean up resources (database - files, etc.) in test teardown. +- Follow the proportional validation budget in + [engdocs/TESTING.md](engdocs/TESTING.md): docs-only changes use docs checks; + Go changes use focused and affected-package tests plus one final `make test`. - If you hit a test failure unrelated to your change, don't silently skip it -- check `.test-skip` and file an issue if it's not already tracked - (see [engdocs/TESTING.md](engdocs/TESTING.md#known-broken-tests)). -- Ensure CI passes (`make ci-pr-core`, `make ci-pr-policy`, `make - ci-pr-lint`) before requesting review. + (see [engdocs/TESTING.md](engdocs/TESTING.md#failures-skips-and-review)). +- Run a named CI wrapper only when its risk or surface is affected, or when + reproducing that CI check. - If your change touches ICU or build tags, see [engdocs/ICU-POLICY.md](engdocs/ICU-POLICY.md) for the policy and rationale. diff --git a/engdocs/CLAUDE.md b/engdocs/CLAUDE.md index df2b145e9..c59967dd1 100644 --- a/engdocs/CLAUDE.md +++ b/engdocs/CLAUDE.md @@ -61,19 +61,18 @@ See `internal/types/types.go`: ## Development Command Source -Use [AGENT_INSTRUCTIONS.md](../AGENT_INSTRUCTIONS.md#testing-commands-no-ambiguity) -for the current command policy. This file should not duplicate command matrices -or version-management workflows. +Use the canonical [TESTING.md](TESTING.md) for test commands, test design, and +PR-readiness gates. This file should not duplicate command matrices or +version-management workflows. > **Do NOT** use `go build -o bd` or `go install` directly — they create > stale binaries that shadow `~/.local/bin/bd`. Always use `make install`. -## Testing Philosophy +## Testing -Testing guidance lives in [TESTING.md](TESTING.md) and -[TESTING_PHILOSOPHY.md](TESTING_PHILOSOPHY.md). Architecture-specific notes for -Claude are limited to where tests touch agent setup, hooks, or instruction-file -generation. +Testing guidance lives in [TESTING.md](TESTING.md). Architecture-specific notes +for Claude are limited to where tests touch agent setup, hooks, or +instruction-file generation. ## Important Notes diff --git a/engdocs/DOC_INVENTORY.md b/engdocs/DOC_INVENTORY.md index 633be7bc2..c7e7d1ee1 100644 --- a/engdocs/DOC_INVENTORY.md +++ b/engdocs/DOC_INVENTORY.md @@ -58,9 +58,9 @@ Follow-up automation should replace marker-only checks with generated or | `CLAUDE_INTEGRATION.md` | Keep | Design/user guide for Claude setup; paired with `SETUP.md`. | | `CLAUDE.md` | Revise | Kept as architecture orientation only; command/workflow duplication was reduced in favour of root `AGENTS.md` and `AGENT_INSTRUCTIONS.md`. | | `CLI_REFERENCE.md` | Keep/generated | Generated command reference from live help output. | -| `CI_CLEANUP_PLAN.md` | Keep with freshness | Accepted CI tier decisions, package gate policy, measurement plan, and implementation order. | +| `CI_CLEANUP_PLAN.md` | Keep as maintainer context | Dated CI policy, measurements, and implementation roadmap; current commands live in workflows and the `Makefile`. | | `CI_REQUIRED_CHECK_TOPOLOGY.md` | Keep | Design note for the required-check aggregate gate topology; explicitly flags pre-rollout status and gates branch-protection changes on the new checks passing first. | -| `CI_TEST_SURFACE_AUDIT.md` | Keep with freshness | Snapshot of local validation commands, GitHub Actions coverage, gaps, and CI cleanup roadmap. | +| `CI_TEST_SURFACE_AUDIT.md` | Keep as historical snapshot | Dated test-surface inventory; current commands live in workflows and the `Makefile`. | | `plugins/beads/skills/beads/resources/CLI_REFERENCE.md` | Keep pointer | Plugin skill resource intentionally points at live CLI sources to avoid duplicate generated command snapshots. | | `CODEX_INTEGRATION.md` | Keep | User-facing Codex integration guide. | | `COLLISION_MATH.md` | Keep | Mathematical background; low product drift. | @@ -101,7 +101,7 @@ Follow-up automation should replace marker-only checks with generated or | `PROJECT_CHARTER.md` | Keep | Canonical product scope and boundary policy. | | `PROTECTED_BRANCHES.md` | Keep | User-facing protected-branch workflow. | | `QUICKSTART.md` | Keep pointer | Short pointer to website quickstart; low drift. | -| `README_TESTING.md` | Consolidated pointer | Old duplicate staged; active path points to `TESTING.md` and `TESTING_PHILOSOPHY.md`. | +| `README_TESTING.md` | Compatibility pointer | Points to the sole canonical testing guide, `TESTING.md`. | | `RECOVERY.md` | Keep with freshness | Runtime recovery playbooks; marker tied to init safety constants/tests. | | `RELATED_PROJECTS.md` | Keep | Curated list of adjacent, non-integration projects; companion to `COMMUNITY_TOOLS.md`. | | `RELEASE-STABILITY-GATE.md` | Keep | Maintainer release gate policy. | @@ -116,8 +116,8 @@ Follow-up automation should replace marker-only checks with generated or | `superpowers/specs/2026-05-03-unclaim-command-design.md` | Removed | PR-passenger design-spec artifact paired with the plans file above; same removal rationale, content stays in git history. | | `SYNC_CONCEPTS.md` | Keep/canonical | Sanctioned website-mirror source for the sync conceptual model (Dolt source of truth, wire format, JSONL export, anti-patterns); listed in `deploy-docs.yml` and linked from `AGENTS.md`, `README.md`, `ARCHITECTURE.md`, `DOLT.md`, `GIT_INTEGRATION.md`, and `FAQ.md`. | | `SYNC_SETUP.md` | Revise | Links now point at canonical `DOLT.md`. | -| `TESTING.md` | Revise/canonical | Canonical test-running guide; stale line/test-count and stale skip entry removed. | -| `TESTING_PHILOSOPHY.md` | Keep | Canonical test-design guidance; duplicate/historical links removed. | +| `TESTING.md` | Keep/canonical | Sole authority for test commands, test design, and PR-readiness guidance. | +| `TESTING_PHILOSOPHY.md` | Compatibility pointer | Points to the sole canonical testing guide, `TESTING.md`. | | `TODO.md` | Keep | Behaviour/user guide for `bd todo`. | | `TROUBLESHOOTING.md` | Keep | User-facing recovery guide; debug/env tables need freshness review when edited. | | `UI_PHILOSOPHY.md` | Keep | Design philosophy. | diff --git a/engdocs/README_TESTING.md b/engdocs/README_TESTING.md index d1d92c6b9..22c49b02c 100644 --- a/engdocs/README_TESTING.md +++ b/engdocs/README_TESTING.md @@ -1,9 +1,3 @@ -# Testing Strategy +# Testing -The active testing workflow lives in [TESTING.md](TESTING.md). - -Testing rationale and test-design guidance live in -[TESTING_PHILOSOPHY.md](TESTING_PHILOSOPHY.md). - -The previous duplicate strategy note has been staged at -[staged-for-removal/README_TESTING.md](staged-for-removal/README_TESTING.md). +This compatibility page points to the canonical [Testing Guide](TESTING.md). diff --git a/engdocs/TESTING.md b/engdocs/TESTING.md index 84b67e417..a73949e03 100644 --- a/engdocs/TESTING.md +++ b/engdocs/TESTING.md @@ -1,286 +1,194 @@ # Testing Guide -## Overview +`TESTING.md` is the single authority for test commands, test selection, and +test design in this repository. Use the commands here for local work. For the +exact command behind a current CI check, inspect its workflow and corresponding +`Makefile` target; the CI audit and cleanup plan are dated maintainer context. -The beads project uses Go tests plus repository wrapper scripts. Prefer the -wrapper scripts for local validation because they apply the repository's normal -local build flags, skip policy, and timeout policy. The current GitHub Actions -PR contract still runs direct `go test` commands; the CI cleanup plan will move -that contract behind dedicated `scripts/ci/*` wrappers. +## Choose the Smallest Useful Test -## Test Performance +Test at the lowest seam that can fail for the user-visible reason. Add a +higher tier only when it covers a distinct risk that the lower tier cannot +show: integration wiring, a real persistence property, a process boundary, or +an external contract. -- Go compilation dominates full-suite runtime. -- Target package/test runs are usually the fastest way to validate focused changes. -- Docker-backed Dolt integration tests auto-detect prerequisites and skip when unavailable. +This keeps feedback fast and failures readable. It does not mean every test is +a unit test: use the real boundary when the defect could live there. -## Running Tests +| Need | Run | When | +|---|---|---| +| Docs-only validation | `git diff --check`, `go test -tags=gms_pure_go ./test/docsync`, and `./scripts/check-doc-freshness.sh` | For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full Go suite merely because a Markdown file changed. | +| Focused red/green loop | `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...` | While writing or fixing one behavior. | +| Affected-package confidence | `./scripts/test.sh ./path/to/package/...` | After the focused test passes; include directly affected neighbors when their contract changed. | +| Final Go baseline | `make test` | Once after focused work on Go code is green. It applies the normal local build flags, coverage, and local skip handling. | +| Named CI wrapper | `make ci-pr-core`, `make ci-pr-policy`, or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. | -### Quick Start +Do not replace the focused loop with repeated full-suite runs. Run the final +`make test` once the affected Go tests are green. For docs-only changes, use +the docs, link, and diff checks instead. + +## Commands and Local Environment + +`./scripts/test.sh` is the normal runner. It sources `.buildflags`, creates an +isolated test environment, applies `.test-skip`, and defaults to a **25m +per-package** Go-test timeout. The timeout is a hang backstop, not a target +runtime. Override it only when diagnosing a legitimate slow path: ```bash -# Run all tests (auto-skips known broken tests) -make test +TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/... +TEST_VERBOSE=1 ./scripts/test.sh ./cmd/bd/... +TEST_RUN='^TestExactName$' ./scripts/test.sh ./cmd/bd/... -# Or directly: -./scripts/test.sh +# Equivalent command-line options. +./scripts/test.sh -v -run '^TestExactName$' ./cmd/bd/... +./scripts/test.sh -timeout 30m ./cmd/bd/... +``` -# Run opt-in ICU regex path tests (maintainer-only, not normal validation) +Use the opt-in ICU regex path only when the change requires it: + +```bash make test-icu-path - -# Run specific package -./scripts/test.sh ./cmd/bd/... - -# Run specific test pattern -./scripts/test.sh -run TestCreate ./cmd/bd/... - -# Verbose output -./scripts/test.sh -v ``` -### Environment Variables +It is maintainer-only and not part of normal validation. `make test-full-cgo` +and `./scripts/test-cgo.sh` remain deprecated compatibility aliases. + +Use a named specialized target only when its risk is in scope: ```bash -# Set custom timeout (default: 3m) -TEST_TIMEOUT=5m ./scripts/test.sh - -# Enable verbose output -TEST_VERBOSE=1 ./scripts/test.sh - -# Run specific pattern -TEST_RUN=TestCreate ./scripts/test.sh +make test-regression +make test-upgrade +make test-cross-version +make test-migration ``` -### Docker (Dolt Integration Tests) +For a failing GitHub Actions check, follow the current workflow and its +`Makefile` target when exact reproduction matters. The local runner and CI +intentionally have different contracts in some cases. -Dolt integration tests require Docker with the exact Dolt image cached locally. -Tests auto-detect the environment and skip gracefully — no manual configuration -needed. +### Test Environment and Readiness -#### Readiness states +The runner isolates `HOME`, Git configuration, and Dolt state. By default its +test environment adds `dolt` to `BEADS_TEST_SKIP`; set +`BEADS_TEST_ENV_RUN_DOLT=1` only when deliberately exercising the Dolt path +and its prerequisites are available. Do not make ordinary tests depend on a +developer's database, daemon, global Git configuration, or filesystem state. -```csv -State,Condition,Behavior -doltSkipped,BEADS_TEST_SKIP contains "dolt",Silent skip (no warning) -doltNoDocker,Docker daemon not reachable,WARN + skip -doltNoImage,No Dolt image at all,WARN + skip with pull instruction -doltWrongVersion,Image repo cached but wrong tag,WARN + skip with pull instruction -doltReady,Exact image cached and Docker running,Run tests -``` - -States are checked once per test binary and cached. Order of evaluation: -`BEADS_TEST_SKIP` → Docker availability → exact image → any image version. - -#### Skipping Dolt tests explicitly - -Set `BEADS_TEST_SKIP` to opt out without Docker overhead (~1s `docker info`): +To skip an optional service explicitly, use the existing skip mechanism: ```bash -# Skip Dolt tests silently -BEADS_TEST_SKIP=dolt ./scripts/test.sh - -# Skip multiple services (comma-separated) -BEADS_TEST_SKIP=dolt,slow ./scripts/test.sh +BEADS_TEST_SKIP=dolt ./scripts/test.sh ./... ``` -### Short Mode and Test Boundaries +Tests that need a temporary repository or store should use `t.TempDir()` and +`t.Cleanup()`. Temporary repositories must set a repository-local hooks path; +do not inherit the developer's global hooks configuration. -`testing.Short()` is reserved for true runtime, stress, and large-fixture skips. -It must not be used as an implicit integration, e2e, API, Docker, or external -dependency boundary. - -Use these mechanisms instead: - -- `//go:build integration` or `//go:build e2e` for named suites. -- Environment readiness checks such as `BEADS_TEST_SKIP=dolt`, - `BEADS_TEST_EMBEDDED_DOLT=1`, or required API-key checks. -- Named wrappers such as `make ci-pr-core`, the main integration shards, and the - package gate wrappers. - -Run `make check-testing-short` to verify that new `testing.Short()` usage stays -within the approved runtime/stress/large-fixture allowlist. The PR policy wrapper -runs the same check. - -#### Enabling Dolt tests +For manual CLI experiments, run both initialization and subsequent commands +from a disposable working directory: ```bash -# Pull the exact Dolt image to enable integration tests -docker pull dolthub/dolt-sql-server:2.2.0 - -# Point tests at an existing Dolt server (skips container startup) -BEADS_DOLT_PORT=3308 ./scripts/test.sh +beads_manual_dir="$(mktemp -d)" +( + set -e + cd "$beads_manual_dir" + bd init --quiet --prefix test --skip-hooks --skip-agents + bd create "Test issue" -p 1 +) +rm -rf -- "$beads_manual_dir" ``` -`BEADS_DOLT_PORT` — when set, tests reuse the server at that port instead of -starting a container. Port 3307 is hardcoded as production and always rejected. +`BEADS_DB` selects a database for database-opening commands, but it does not by +itself redirect `bd init` workspace setup. Never run a manual `bd init` from a +production workspace merely because `BEADS_DB` points elsewhere. -### Advanced Usage +`testing.Short()` is for genuine runtime, stress, or large-fixture skips. It +is not a substitute for declaring an integration, end-to-end, API, Docker, or +external-dependency boundary. Keep new uses within the repository policy: ```bash -# Skip additional tests beyond .test-skip -./scripts/test.sh -skip SomeSlowTest - -# Run with custom timeout -./scripts/test.sh -timeout 5m - -# Combine flags -./scripts/test.sh -v -run TestCreate ./internal/beads/... +make check-testing-short ``` -## Known Broken Tests +## Test Design -Tests in `.test-skip` are automatically skipped by `scripts/test.sh`. +### Seams, Scenarios, and Doubles -At the time of this review, `.test-skip` contains only comments and no active -test-name patterns. Treat any new skip as a temporary exception: file the -upstream issue first, record it in `.test-skip`, and remove the skip when the -test is fixed. +Write one scenario at the smallest seam that demonstrates the behavior. Cover +the boundary or failure mode that changes the user result; use table-driven +subtests when examples share setup. Do not repeat the same scenario through a +helper, every caller, and the CLI merely because all are available. -## For Claude Code / AI Agents +A recording double or fake should be narrow: model only the calls, inputs, +outputs, and failures the test needs. It should not recreate a storage engine, +process manager, or another subsystem just to make a unit test look realistic. -When running tests during development: +A behavioral fake is different. If it stands in for a contract shared by +multiple production implementations, give it the same semantic-conformance +suite as those implementations. That shared suite defines observable behavior; +it prevents the fake from teaching callers a contract production code does not +honor. -### Best Practices +Semantic conformance asks whether an implementation produces the promised +results, errors, and state transitions for the same operation. Persistence +conformance asks whether the real persistence boundary preserves its required +durability, transaction, migration, and recovery properties. They answer +different questions. Do not claim backend parity unless a stated contract and +its conformance suite establish it. -1. **Use the test script:** Always use `./scripts/test.sh` instead of `go test` directly - - Automatically skips known broken tests - - Uses appropriate timeouts - - Matches local default validation; use future `scripts/ci/*` wrappers when - reproducing exact CI jobs - - Only if intentionally exercising the ICU regex path, use `./scripts/test-icu-path.sh` (or deprecated `make test-full-cgo`) +### Tier Admission -2. **Target specific tests when possible:** - ```bash - # Instead of running everything: - ./scripts/test.sh +An integration test belongs above the unit seam only when it covers a distinct +boundary that a narrow double cannot prove, such as configuration wiring, a +real filesystem or Git interaction, a subprocess protocol, or persistence +behavior. - # Run just what you changed: - ./scripts/test.sh -run TestSpecificFeature ./cmd/bd/... - ``` +An end-to-end test is admitted only when all of these are true: -3. **Compilation is the bottleneck:** - - The 180-second compilation time dominates - - Individual tests are fast - - Use `-run` to avoid recompiling unnecessarily +1. The failure would be user-visible. +2. A real process, setup, or wiring boundary owns a distinct failure risk that + no lower seam can prove. +3. Lower-tier tests cover the underlying behavior where practical, leaving the + end-to-end test focused on that boundary. +4. No existing end-to-end test already covers the same boundary risk. -4. **Check for new failures:** - ```bash - # If you see a new failure, check if it's known: - cat .test-skip - ``` +Lower-tier coverage of the same user journey does not disqualify the +end-to-end test; duplicate coverage of the same boundary risk does. State that +risk in the test name or nearby documentation. -### Adding Tests to Skip List +### Avoid Incidental Complexity -If you discover a broken test: +Avoid these patterns unless they are the behavior under test: -1. File a GitHub issue documenting the problem -2. Add to `.test-skip`: - ```bash - # Issue #NNN: Brief description - TestNameToSkip - ``` -3. Tests in `.test-skip` support regex patterns +- Duplicate scenarios at several layers. +- Global-state reset choreography. Prefer per-test state and cleanup. +- Subprocesses, listeners, sleeps, or real-store setup for a unit-level claim. +- Assertions on private implementation shape when observable behavior is the + contract. +- Performance claims without a repeatable measurement and stated workload. -## Test Organization +Sleeps, listeners, and real stores are appropriate when the test is specifically +about timing, lifecycle, protocol, or persistence. Keep the setup scoped and +make that reason apparent. -### Slowest Tests (>0.05s) +## Failures, Skips, and Review -The top slow tests in cmd/bd: -- `TestDoctorWithBeadsDir` (1.68s) - Only significantly slow test -- `TestFlushManagerDebouncing` (0.21s) -- `TestDebouncer_*` tests (0.06-0.12s each) - Intentional sleeps for concurrency testing -- `TestMultiWorkspaceDeletionSync` (0.12s) +`.test-skip` is a local, temporary exception list. If an unrelated failure is +already listed, report it rather than silently broadening the skip. Before +adding a new skip, record the issue it tracks and remove the skip when the +underlying failure is fixed. -Most tests are <0.01s and very fast. +Before opening a PR: -### Package Structure +1. For docs-only changes, run the applicable docs, link, freshness, and diff + checks; do not run the full Go suite by default. +2. For Go code, keep the focused and affected-package tests green, then run one + final `make test`. +3. Run only the named CI wrapper, specialized target, or risk gate required by + the changed surface, or the one needed to reproduce a CI result. -``` -cmd/bd/ - Main CLI tests (82 test files, most of the suite) -internal/beads/ - Core beads library tests -internal/storage/ - Storage backend tests (Dolt embedded/server, conformance) -internal/rpc/ - RPC protocol tests -internal/*/ - Various internal package tests -``` - -## Continuous Integration - -The current CI workflow does not yet call the `scripts/ci/*` wrappers for every -job. Until workflow migration is complete, use the command documented in the -failing workflow when reproducing an existing status check exactly. - -Use `scripts/test.sh` for local default validation and targeted development -runs. - -Use the CI wrappers for the accepted target PR contracts: - -```bash -make ci-pr-core -make ci-pr-policy -make ci-pr-lint -``` - -Use the package gate wrappers when touching package or docs surfaces: - -```bash -make ci-package-mcp -make ci-package-npm -``` - -### Coverage Signal Policy - -PR confidence is based on behavior checks, not raw coverage percentage. - -- Treat Codecov percentages as informational trend data. -- Prefer focused tests for risky paths (storage, sync/git, migrations, state - transitions, and corruption/integrity handling) over broad line-coverage - churn. -- Add or extend at least one targeted regression test when fixing risky logic. -- Do not block a change solely on overall coverage movement when behavioral - checks are strong. - -For the current CI/test-surface inventory and cleanup roadmap, see -[CI_TEST_SURFACE_AUDIT.md](CI_TEST_SURFACE_AUDIT.md). That audit documents -where local commands and GitHub Actions currently diverge before the CI cleanup -work starts changing workflow behavior. - -For accepted CI tier decisions and the implementation order, see -[CI_CLEANUP_PLAN.md](CI_CLEANUP_PLAN.md). - -## Debugging Test Failures - -### Get detailed output -```bash -./scripts/test.sh -v ./path/to/package/... -``` - -### Run a single test -```bash -./scripts/test.sh -run '^TestExactName$' ./cmd/bd/... -``` - -### Check which tests are being skipped -```bash -./scripts/test.sh 2>&1 | head -5 -``` - -Output shows: -``` -Running: go test -timeout 3m -skip TestFoo|TestBar ./... -Skipping: TestFoo|TestBar -``` - -## Contributing - -When adding new tests: - -1. Keep tests fast (<0.1s if possible) -2. Use `t.Parallel()` for independent tests -3. Clean up resources in `t.Cleanup()` or `defer` -4. Avoid sleeps unless testing concurrency - -When tests break: - -1. Fix them if possible -2. If unfixable right now, file an issue and add to `.test-skip` -3. Document the issue in `.test-skip` with issue number +For historical CI inventory and maintainer planning context, consult +[CI_TEST_SURFACE_AUDIT.md](CI_TEST_SURFACE_AUDIT.md) and +[CI_CLEANUP_PLAN.md](CI_CLEANUP_PLAN.md). For current commands, use the +workflow files and `Makefile`; these context documents are not a second testing +guide or a live CI inventory. diff --git a/engdocs/TESTING_PHILOSOPHY.md b/engdocs/TESTING_PHILOSOPHY.md index 2e5375183..a0386b48f 100644 --- a/engdocs/TESTING_PHILOSOPHY.md +++ b/engdocs/TESTING_PHILOSOPHY.md @@ -1,259 +1,4 @@ # Testing Philosophy -This document covers **what to test** and **what not to test**. For how to run tests, see [TESTING.md](TESTING.md). - -## The Test Pyramid - -``` - ┌─────────────────┐ - │ E2E Tests │ ← PR/Deploy only (slow, expensive) - │ ~5% of tests │ - └────────┬────────┘ - │ - ┌──────────────┴──────────────┐ - │ Integration Tests │ ← PR gate (moderate) - │ ~15% of tests │ - └──────────────┬──────────────┘ - │ - ┌────────────────────────┴────────────────────────┐ - │ Unit Tests (Fast) │ ← Every save/commit - │ ~80% of tests │ - └─────────────────────────────────────────────────┘ -``` - -### Tier 1: Fast Tests (< 5 seconds total) - -**When**: Every file save, pre-commit hooks, continuous during development - -- Pure function tests (no I/O) -- In-memory data structure tests -- Business logic validation -- Mock all external dependencies - -**In beads**: Core logic tests using `newTestStore()` with in-memory SQLite - -### Tier 2: Integration Tests (< 30 seconds) - -**When**: Pre-push, PR checks - -- Real file system operations -- Git operations with temp repos -- Config file parsing -- CLI argument handling - -**In beads**: Tests tagged with `//go:build integration`, server mode tests - -### Tier 3: E2E / Smoke Tests (1-5 minutes) - -**When**: PR merge, pre-deploy, nightly - -- Full `bd init` → `bd doctor` → `bd doctor --fix` workflow -- Real API calls (to staging) -- Cross-platform verification - ---- - -## What Makes a Test "Right" - -A good test: - -1. **Catches a bug you'd actually ship** - not theoretical edge cases -2. **Documents expected behavior** - serves as living documentation -3. **Runs fast enough to not skip** - slow tests get disabled -4. **Isn't duplicated elsewhere** - tests one thing, one way - ---- - -## What to Test (Priority Matrix) - -| Priority | What | Why | Examples in beads | -|----------|------|-----|-------------------| -| **High** | Core business logic | This is what users depend on | `sync`, `doctor`, `export`, `import` | -| **High** | Error paths that could corrupt data | Data loss is catastrophic | Config handling, git operations, database integrity | -| **Medium** | Edge cases from production bugs | Discovered through real issues | Orphan handling, ID collision detection | -| **Low** | Display/formatting | Visual output, can be manually verified | Table formatting, color output | - ---- - -## What NOT to Test Extensively - -### Simple utility functions -Trust the language. Don't test that `strings.TrimSpace` works. - -### Every permutation of inputs -Use table-driven tests with representative cases instead of exhaustive permutations. - -```go -// BAD: 10 separate test functions -func TestPriority0(t *testing.T) { ... } -func TestPriority1(t *testing.T) { ... } -func TestPriority2(t *testing.T) { ... } - -// GOOD: One table-driven test -func TestPriorityMapping(t *testing.T) { - cases := []struct{ in, want int }{ - {0, 4}, {1, 0}, {5, 3}, // includes boundary - } - for _, tc := range cases { - t.Run(fmt.Sprintf("priority_%d", tc.in), func(t *testing.T) { - got := mapPriority(tc.in) - if got != tc.want { t.Errorf(...) } - }) - } -} -``` - -### Obvious behavior -Don't test "if file exists, return true" - trust the implementation. - -### Same logic through different entry points -If you test a function directly, don't also test it through every caller. - ---- - -## Anti-Patterns to Avoid - -### 1. Trivial Assertions - -Testing obvious happy paths that would pass with trivial implementations. - -```go -// BAD: What bug would this catch? -func TestValidateBeadsWorkspace(t *testing.T) { - dir := setupTestWorkspace(t) - if err := validateBeadsWorkspace(dir); err != nil { - t.Errorf("expected no error, got: %v", err) - } -} - -// GOOD: Test the interesting error cases -func TestValidateBeadsWorkspace(t *testing.T) { - cases := []struct{ - name string - setup func(t *testing.T) string - wantErr string - }{ - {"missing .beads dir", setupNoBeadsDir, "not a beads workspace"}, - {"corrupted db", setupCorruptDB, "database is corrupted"}, - {"permission denied", setupNoReadAccess, "permission denied"}, - } - // ... -} -``` - -### 2. Duplicate Error Path Testing - -Testing the same logic multiple ways instead of once with table-driven tests. - -```go -// BAD: Repetitive individual assertions -if config.PriorityMap["0"] != 4 { t.Errorf(...) } -if config.PriorityMap["1"] != 0 { t.Errorf(...) } -if config.PriorityMap["2"] != 1 { t.Errorf(...) } - -// GOOD: Table-driven -for k, want := range expectedMap { - if got := config.PriorityMap[k]; got != want { - t.Errorf("PriorityMap[%q] = %d, want %d", k, got, want) - } -} -``` - -### 3. I/O Heavy Tests Without Mocking - -Unit tests that execute real commands or heavy I/O when they could mock. - -```go -// BAD: Actually executes external commands in unit test -func TestServerFix(t *testing.T) { - exec.Command("bd", "dolt", "stop").Run() - // ... -} - -// GOOD: Mock the execution or use integration test tag -func TestServerFix(t *testing.T) { - executor := &mockExecutor{} - fix := NewServerFix(executor) - // ... -} -``` - -### 4. Testing Implementation, Not Behavior - -Tests that break when you refactor, even though behavior is unchanged. - -```go -// BAD: Tests internal state -if len(server.connectionPool) != 3 { t.Error(...) } - -// GOOD: Tests observable behavior -if resp, err := server.HandleRequest(req); err != nil { t.Error(...) } -``` - -### 5. Missing Boundary Tests - -Testing known-good values but not boundaries and invalid inputs. - -```go -// BAD: Only tests middle values -TestPriority(1) // works -TestPriority(2) // works - -// GOOD: Tests boundaries and invalid -TestPriority(-1) // invalid - expect error -TestPriority(0) // boundary - min valid -TestPriority(4) // boundary - max valid -TestPriority(5) // boundary - first invalid -``` - ---- - -## Target Metrics - -| Metric | Target | Current (beads) | Status | -|--------|--------|-----------------|--------| -| Test-to-code ratio | 0.5:1 - 1.5:1 | 0.85:1 | Healthy | -| Fast test suite | < 5 seconds | 3.8 seconds | Good | -| Integration tests | < 30 seconds | ~15 seconds | Good | -| Compilation overhead | Minimize | 180 seconds | Bottleneck | - -### Interpretation - -- **0.5:1** - Light coverage, fast iteration (acceptable for utilities) -- **1:1** - Solid coverage for most projects (our target) -- **1.5:1** - Heavy coverage for critical systems -- **2:1+** - Over-engineered, maintenance burden - ---- - -## Beads-Specific Guidance - -### Well-Covered (Maintain) - -| Area | Why It's Well-Tested | -|------|---------------------| -| Sync/Export/Import | Data integrity critical - comprehensive edge cases | -| SQLite transactions | Rollback safety, atomicity guarantees | -| Merge operations | Dolt-native cell-level merge | -| Database locking | Prevents corruption from multiple instances | - -### Needs Attention - -| Area | Gap | Priority | -|------|-----|----------| -| Server lifecycle | Shutdown/signal handling | Medium | -| Concurrent operations | Stress testing under load | Medium | -| Boundary validation | Edge inputs in mapping functions | Low | - -### Skip These - -- Display formatting tests (manually verify) -- Simple getters/setters -- Tests that duplicate SQLite's guarantees - ---- - -## Related Docs - -- [TESTING.md](TESTING.md) - How to run tests -- [README_TESTING.md](README_TESTING.md) - Stable pointer to the active testing docs +This compatibility page points to the canonical [Testing Guide](TESTING.md), +which covers test design, test selection, and commands.