docs(test): make TESTING.md canonical (bd-mrs20) (#5226)
Consolidate test execution and design guidance around the smallest trustworthy seam, narrow doubles, semantic versus persistence conformance, strict E2E admission, and proportional verification. Convert the former philosophy documents to compatibility pointers. Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of CI Bot Co-authored-by: CI Bot <ci@beads.test>
This commit is contained in:
parent
47cae42efe
commit
e9a12ed3b1
8 changed files with 195 additions and 564 deletions
12
AGENTS.md
12
AGENTS.md
|
|
@ -88,16 +88,10 @@ echo 'Description with `backticks` and "quotes"' | bd create "Title" --descripti
|
|||
echo 'Updated text' | bd update <id> --description=-
|
||||
```
|
||||
|
||||
## Testing Commands (No Ambiguity)
|
||||
## Testing
|
||||
|
||||
- Default local test command: `make test` (or `./scripts/test.sh`).
|
||||
- Opt-in ICU regex path: `make test-icu-path` (or `./scripts/test-icu-path.sh ./...`).
|
||||
- This ICU path is maintainer-only and not part of normal validation; `make test-full-cgo` and `./scripts/test-cgo.sh` are deprecated aliases.
|
||||
- For package- or test-scoped shipped-config CGO runs, prefer:
|
||||
```bash
|
||||
CGO_ENABLED=1 go test -tags gms_pure_go ./cmd/bd/...
|
||||
CGO_ENABLED=1 go test -tags gms_pure_go -run '^TestName$' ./cmd/bd/...
|
||||
```
|
||||
Use [engdocs/TESTING.md](engdocs/TESTING.md) for the canonical commands,
|
||||
test-design guidance, and PR-readiness gates.
|
||||
|
||||
## Non-Interactive Shell Commands
|
||||
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ This document contains detailed operational instructions for AI agents working o
|
|||
|
||||
- **Go version**: see `go.mod` for the required version (currently 1.26+)
|
||||
- **Linting**: `golangci-lint run ./...` (baseline warnings documented in [engdocs/LINTING.md](engdocs/LINTING.md))
|
||||
- **Testing**: All new features need tests (`make test` for the normal local/CI path, `make test-icu-path` only when intentionally exercising the opt-in ICU regex path)
|
||||
- **Testing**: All new features need tests; use [engdocs/TESTING.md](engdocs/TESTING.md) for command selection, test design, and PR readiness.
|
||||
- **Documentation**: Update relevant .md files
|
||||
|
||||
### File Organization
|
||||
|
|
@ -30,17 +30,26 @@ beads/
|
|||
|
||||
**IMPORTANT:** Never pollute the production database with test issues!
|
||||
|
||||
**For manual testing**, use the `BEADS_DB` environment variable to point to a temporary database:
|
||||
For test commands, tier selection, and test-design guidance, use the canonical
|
||||
[engdocs/TESTING.md](engdocs/TESTING.md).
|
||||
|
||||
**For manual testing**, keep initialization and all experiments in a disposable
|
||||
working directory. This safety example supplements the canonical testing guide:
|
||||
|
||||
```bash
|
||||
# Create test issues in isolated database
|
||||
BEADS_DB=/tmp/test.db bd init --quiet --prefix test
|
||||
BEADS_DB=/tmp/test.db bd create "Test issue" -p 1
|
||||
|
||||
# Or for quick testing
|
||||
BEADS_DB=/tmp/test.db bd create "Test feature" -p 1
|
||||
beads_manual_dir="$(mktemp -d)"
|
||||
(
|
||||
set -e
|
||||
cd "$beads_manual_dir"
|
||||
bd init --quiet --prefix test --skip-hooks --skip-agents
|
||||
bd create "Test issue" -p 1
|
||||
)
|
||||
rm -rf -- "$beads_manual_dir"
|
||||
```
|
||||
|
||||
`BEADS_DB` alone does not redirect `bd init` workspace setup. Do not run manual
|
||||
initialization from a production workspace even when selecting another database.
|
||||
|
||||
**For automated tests**, use `t.TempDir()` in Go tests:
|
||||
|
||||
```go
|
||||
|
|
@ -61,7 +70,9 @@ git config core.hooksPath .git/hooks
|
|||
Do not rely on the developer's global git config. Global `core.hooksPath` can leak
|
||||
into temp repos and produce flaky test behavior.
|
||||
|
||||
**Warning:** bd will warn you when creating issues with "Test" prefix in the production database. Always use `BEADS_DB` for manual testing.
|
||||
**Warning:** bd will warn you when creating issues with a "Test" prefix in the
|
||||
production database. The disposable working directory is the isolation
|
||||
boundary for manual initialization and experiments.
|
||||
|
||||
**Tmpfs hosts:** the `cmd/bd` test suite creates an isolated `$HOME` and several
|
||||
test binaries under `$TMPDIR`. They are normally cleaned by the test process,
|
||||
|
|
@ -71,9 +82,9 @@ test runs if `du -sh /tmp/beads-* /tmp/bd-*` shows accumulation. See bd-3q2u.
|
|||
|
||||
### Before Committing
|
||||
|
||||
1. **Run tests**: `make test` (or `./scripts/test.sh`)
|
||||
- Only if intentionally exercising the ICU regex path: `make test-icu-path`
|
||||
2. **Run linter**: `golangci-lint run ./...` (ignore baseline warnings)
|
||||
1. **Run tests**: follow [engdocs/TESTING.md](engdocs/TESTING.md).
|
||||
2. **Run linter when its code surface changed**: `golangci-lint run ./...`
|
||||
(ignore baseline warnings)
|
||||
3. **Update docs**: If you changed behavior, update README.md or other docs
|
||||
4. **Commit**: With git hooks installed (`bd hooks install`), Dolt changes are auto-committed
|
||||
|
||||
|
|
@ -375,28 +386,21 @@ case types.StatusClosed:
|
|||
4. Link from `examples/README.md`
|
||||
5. Mention in main README.md
|
||||
|
||||
## Building and Testing
|
||||
## Building
|
||||
|
||||
```bash
|
||||
# Build and install bd to ~/.local/bin (the canonical location)
|
||||
make install
|
||||
|
||||
# Test (local baseline)
|
||||
make test
|
||||
|
||||
# Optional ICU regex path smoke (maintainer-only, not normal validation)
|
||||
make test-icu-path
|
||||
|
||||
# Coverage run
|
||||
go test -tags gms_pure_go -coverprofile=coverage.out ./...
|
||||
go tool cover -html=coverage.out
|
||||
|
||||
# Verify installed binary
|
||||
bd init --prefix test
|
||||
bd create "Test issue" -p 1
|
||||
bd ready
|
||||
```
|
||||
|
||||
For testing commands, test design, and PR-readiness gates, use
|
||||
[engdocs/TESTING.md](engdocs/TESTING.md).
|
||||
|
||||
> **WARNING**: Do NOT use `go build -o bd ./cmd/bd`, `go install ./cmd/bd`,
|
||||
> or raw `go run ./cmd/bd ...`.
|
||||
> These bypass the canonical build path, can create stale binaries in the
|
||||
|
|
|
|||
|
|
@ -44,20 +44,10 @@ beads/
|
|||
|
||||
## Running Tests
|
||||
|
||||
```bash
|
||||
# Run all tests (recommended — uses correct build tags)
|
||||
make test
|
||||
|
||||
# Run tests with coverage
|
||||
go test -tags gms_pure_go -v -coverprofile=coverage.out ./...
|
||||
go tool cover -html=coverage.out
|
||||
|
||||
# Run specific package tests
|
||||
go test -tags gms_pure_go ./internal/storage/dolt/ -v
|
||||
|
||||
# Run tests with race detection
|
||||
go test -tags gms_pure_go -race ./...
|
||||
```
|
||||
Use the canonical [testing guide](engdocs/TESTING.md) to choose focused tests,
|
||||
the proportional validation budget, and any applicable CI wrapper. The setup
|
||||
and safety notes in this file supplement that guide; they do not define a
|
||||
second test policy.
|
||||
|
||||
## Code Style
|
||||
|
||||
|
|
@ -139,22 +129,19 @@ If you are contributing code that involves AI decision-making or orchestration,
|
|||
|
||||
## Testing Guidelines
|
||||
|
||||
For how to run tests, see [engdocs/TESTING.md](engdocs/TESTING.md). For what to
|
||||
test and why (the test pyramid and tiering we follow), see
|
||||
[engdocs/TESTING_PHILOSOPHY.md](engdocs/TESTING_PHILOSOPHY.md).
|
||||
For test commands, test design, and PR-readiness gates, see the canonical
|
||||
[engdocs/TESTING.md](engdocs/TESTING.md).
|
||||
|
||||
### Before Opening a PR
|
||||
|
||||
- Run `make test` (or `./scripts/test.sh`) locally and make sure it passes.
|
||||
- Add tests for new functionality; extend existing tests when fixing bugs.
|
||||
- Write table-driven tests for multiple scenarios, use descriptive test
|
||||
names, use `t.Run()` for subtests, and clean up resources (database
|
||||
files, etc.) in test teardown.
|
||||
- Follow the proportional validation budget in
|
||||
[engdocs/TESTING.md](engdocs/TESTING.md): docs-only changes use docs checks;
|
||||
Go changes use focused and affected-package tests plus one final `make test`.
|
||||
- If you hit a test failure unrelated to your change, don't silently skip
|
||||
it -- check `.test-skip` and file an issue if it's not already tracked
|
||||
(see [engdocs/TESTING.md](engdocs/TESTING.md#known-broken-tests)).
|
||||
- Ensure CI passes (`make ci-pr-core`, `make ci-pr-policy`, `make
|
||||
ci-pr-lint`) before requesting review.
|
||||
(see [engdocs/TESTING.md](engdocs/TESTING.md#failures-skips-and-review)).
|
||||
- Run a named CI wrapper only when its risk or surface is affected, or when
|
||||
reproducing that CI check.
|
||||
- If your change touches ICU or build tags, see
|
||||
[engdocs/ICU-POLICY.md](engdocs/ICU-POLICY.md) for the policy and rationale.
|
||||
|
||||
|
|
|
|||
|
|
@ -61,19 +61,18 @@ See `internal/types/types.go`:
|
|||
|
||||
## Development Command Source
|
||||
|
||||
Use [AGENT_INSTRUCTIONS.md](../AGENT_INSTRUCTIONS.md#testing-commands-no-ambiguity)
|
||||
for the current command policy. This file should not duplicate command matrices
|
||||
or version-management workflows.
|
||||
Use the canonical [TESTING.md](TESTING.md) for test commands, test design, and
|
||||
PR-readiness gates. This file should not duplicate command matrices or
|
||||
version-management workflows.
|
||||
|
||||
> **Do NOT** use `go build -o bd` or `go install` directly — they create
|
||||
> stale binaries that shadow `~/.local/bin/bd`. Always use `make install`.
|
||||
|
||||
## Testing Philosophy
|
||||
## Testing
|
||||
|
||||
Testing guidance lives in [TESTING.md](TESTING.md) and
|
||||
[TESTING_PHILOSOPHY.md](TESTING_PHILOSOPHY.md). Architecture-specific notes for
|
||||
Claude are limited to where tests touch agent setup, hooks, or instruction-file
|
||||
generation.
|
||||
Testing guidance lives in [TESTING.md](TESTING.md). Architecture-specific notes
|
||||
for Claude are limited to where tests touch agent setup, hooks, or
|
||||
instruction-file generation.
|
||||
|
||||
## Important Notes
|
||||
|
||||
|
|
|
|||
|
|
@ -58,9 +58,9 @@ Follow-up automation should replace marker-only checks with generated or
|
|||
| `CLAUDE_INTEGRATION.md` | Keep | Design/user guide for Claude setup; paired with `SETUP.md`. |
|
||||
| `CLAUDE.md` | Revise | Kept as architecture orientation only; command/workflow duplication was reduced in favour of root `AGENTS.md` and `AGENT_INSTRUCTIONS.md`. |
|
||||
| `CLI_REFERENCE.md` | Keep/generated | Generated command reference from live help output. |
|
||||
| `CI_CLEANUP_PLAN.md` | Keep with freshness | Accepted CI tier decisions, package gate policy, measurement plan, and implementation order. |
|
||||
| `CI_CLEANUP_PLAN.md` | Keep as maintainer context | Dated CI policy, measurements, and implementation roadmap; current commands live in workflows and the `Makefile`. |
|
||||
| `CI_REQUIRED_CHECK_TOPOLOGY.md` | Keep | Design note for the required-check aggregate gate topology; explicitly flags pre-rollout status and gates branch-protection changes on the new checks passing first. |
|
||||
| `CI_TEST_SURFACE_AUDIT.md` | Keep with freshness | Snapshot of local validation commands, GitHub Actions coverage, gaps, and CI cleanup roadmap. |
|
||||
| `CI_TEST_SURFACE_AUDIT.md` | Keep as historical snapshot | Dated test-surface inventory; current commands live in workflows and the `Makefile`. |
|
||||
| `plugins/beads/skills/beads/resources/CLI_REFERENCE.md` | Keep pointer | Plugin skill resource intentionally points at live CLI sources to avoid duplicate generated command snapshots. |
|
||||
| `CODEX_INTEGRATION.md` | Keep | User-facing Codex integration guide. |
|
||||
| `COLLISION_MATH.md` | Keep | Mathematical background; low product drift. |
|
||||
|
|
@ -101,7 +101,7 @@ Follow-up automation should replace marker-only checks with generated or
|
|||
| `PROJECT_CHARTER.md` | Keep | Canonical product scope and boundary policy. |
|
||||
| `PROTECTED_BRANCHES.md` | Keep | User-facing protected-branch workflow. |
|
||||
| `QUICKSTART.md` | Keep pointer | Short pointer to website quickstart; low drift. |
|
||||
| `README_TESTING.md` | Consolidated pointer | Old duplicate staged; active path points to `TESTING.md` and `TESTING_PHILOSOPHY.md`. |
|
||||
| `README_TESTING.md` | Compatibility pointer | Points to the sole canonical testing guide, `TESTING.md`. |
|
||||
| `RECOVERY.md` | Keep with freshness | Runtime recovery playbooks; marker tied to init safety constants/tests. |
|
||||
| `RELATED_PROJECTS.md` | Keep | Curated list of adjacent, non-integration projects; companion to `COMMUNITY_TOOLS.md`. |
|
||||
| `RELEASE-STABILITY-GATE.md` | Keep | Maintainer release gate policy. |
|
||||
|
|
@ -116,8 +116,8 @@ Follow-up automation should replace marker-only checks with generated or
|
|||
| `superpowers/specs/2026-05-03-unclaim-command-design.md` | Removed | PR-passenger design-spec artifact paired with the plans file above; same removal rationale, content stays in git history. |
|
||||
| `SYNC_CONCEPTS.md` | Keep/canonical | Sanctioned website-mirror source for the sync conceptual model (Dolt source of truth, wire format, JSONL export, anti-patterns); listed in `deploy-docs.yml` and linked from `AGENTS.md`, `README.md`, `ARCHITECTURE.md`, `DOLT.md`, `GIT_INTEGRATION.md`, and `FAQ.md`. |
|
||||
| `SYNC_SETUP.md` | Revise | Links now point at canonical `DOLT.md`. |
|
||||
| `TESTING.md` | Revise/canonical | Canonical test-running guide; stale line/test-count and stale skip entry removed. |
|
||||
| `TESTING_PHILOSOPHY.md` | Keep | Canonical test-design guidance; duplicate/historical links removed. |
|
||||
| `TESTING.md` | Keep/canonical | Sole authority for test commands, test design, and PR-readiness guidance. |
|
||||
| `TESTING_PHILOSOPHY.md` | Compatibility pointer | Points to the sole canonical testing guide, `TESTING.md`. |
|
||||
| `TODO.md` | Keep | Behaviour/user guide for `bd todo`. |
|
||||
| `TROUBLESHOOTING.md` | Keep | User-facing recovery guide; debug/env tables need freshness review when edited. |
|
||||
| `UI_PHILOSOPHY.md` | Keep | Design philosophy. |
|
||||
|
|
|
|||
|
|
@ -1,9 +1,3 @@
|
|||
# Testing Strategy
|
||||
# Testing
|
||||
|
||||
The active testing workflow lives in [TESTING.md](TESTING.md).
|
||||
|
||||
Testing rationale and test-design guidance live in
|
||||
[TESTING_PHILOSOPHY.md](TESTING_PHILOSOPHY.md).
|
||||
|
||||
The previous duplicate strategy note has been staged at
|
||||
[staged-for-removal/README_TESTING.md](staged-for-removal/README_TESTING.md).
|
||||
This compatibility page points to the canonical [Testing Guide](TESTING.md).
|
||||
|
|
|
|||
|
|
@ -1,286 +1,194 @@
|
|||
# Testing Guide
|
||||
|
||||
## Overview
|
||||
`TESTING.md` is the single authority for test commands, test selection, and
|
||||
test design in this repository. Use the commands here for local work. For the
|
||||
exact command behind a current CI check, inspect its workflow and corresponding
|
||||
`Makefile` target; the CI audit and cleanup plan are dated maintainer context.
|
||||
|
||||
The beads project uses Go tests plus repository wrapper scripts. Prefer the
|
||||
wrapper scripts for local validation because they apply the repository's normal
|
||||
local build flags, skip policy, and timeout policy. The current GitHub Actions
|
||||
PR contract still runs direct `go test` commands; the CI cleanup plan will move
|
||||
that contract behind dedicated `scripts/ci/*` wrappers.
|
||||
## Choose the Smallest Useful Test
|
||||
|
||||
## Test Performance
|
||||
Test at the lowest seam that can fail for the user-visible reason. Add a
|
||||
higher tier only when it covers a distinct risk that the lower tier cannot
|
||||
show: integration wiring, a real persistence property, a process boundary, or
|
||||
an external contract.
|
||||
|
||||
- Go compilation dominates full-suite runtime.
|
||||
- Target package/test runs are usually the fastest way to validate focused changes.
|
||||
- Docker-backed Dolt integration tests auto-detect prerequisites and skip when unavailable.
|
||||
This keeps feedback fast and failures readable. It does not mean every test is
|
||||
a unit test: use the real boundary when the defect could live there.
|
||||
|
||||
## Running Tests
|
||||
| Need | Run | When |
|
||||
|---|---|---|
|
||||
| Docs-only validation | `git diff --check`, `go test -tags=gms_pure_go ./test/docsync`, and `./scripts/check-doc-freshness.sh` | For prose-only changes; add any generated-doc or surface-specific link check the changed paths require. Do not run the full Go suite merely because a Markdown file changed. |
|
||||
| Focused red/green loop | `./scripts/test.sh -run '^TestExactName$' ./path/to/package/...` | While writing or fixing one behavior. |
|
||||
| Affected-package confidence | `./scripts/test.sh ./path/to/package/...` | After the focused test passes; include directly affected neighbors when their contract changed. |
|
||||
| Final Go baseline | `make test` | Once after focused work on Go code is green. It applies the normal local build flags, coverage, and local skip handling. |
|
||||
| Named CI wrapper | `make ci-pr-core`, `make ci-pr-policy`, or `make ci-pr-lint` | Run the wrapper whose risk or surface is affected, or use it to reproduce that CI check. Do not run all three routinely for every edit. |
|
||||
|
||||
### Quick Start
|
||||
Do not replace the focused loop with repeated full-suite runs. Run the final
|
||||
`make test` once the affected Go tests are green. For docs-only changes, use
|
||||
the docs, link, and diff checks instead.
|
||||
|
||||
## Commands and Local Environment
|
||||
|
||||
`./scripts/test.sh` is the normal runner. It sources `.buildflags`, creates an
|
||||
isolated test environment, applies `.test-skip`, and defaults to a **25m
|
||||
per-package** Go-test timeout. The timeout is a hang backstop, not a target
|
||||
runtime. Override it only when diagnosing a legitimate slow path:
|
||||
|
||||
```bash
|
||||
# Run all tests (auto-skips known broken tests)
|
||||
make test
|
||||
TEST_TIMEOUT=30m ./scripts/test.sh ./cmd/bd/...
|
||||
TEST_VERBOSE=1 ./scripts/test.sh ./cmd/bd/...
|
||||
TEST_RUN='^TestExactName$' ./scripts/test.sh ./cmd/bd/...
|
||||
|
||||
# Or directly:
|
||||
./scripts/test.sh
|
||||
# Equivalent command-line options.
|
||||
./scripts/test.sh -v -run '^TestExactName$' ./cmd/bd/...
|
||||
./scripts/test.sh -timeout 30m ./cmd/bd/...
|
||||
```
|
||||
|
||||
# Run opt-in ICU regex path tests (maintainer-only, not normal validation)
|
||||
Use the opt-in ICU regex path only when the change requires it:
|
||||
|
||||
```bash
|
||||
make test-icu-path
|
||||
|
||||
# Run specific package
|
||||
./scripts/test.sh ./cmd/bd/...
|
||||
|
||||
# Run specific test pattern
|
||||
./scripts/test.sh -run TestCreate ./cmd/bd/...
|
||||
|
||||
# Verbose output
|
||||
./scripts/test.sh -v
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
It is maintainer-only and not part of normal validation. `make test-full-cgo`
|
||||
and `./scripts/test-cgo.sh` remain deprecated compatibility aliases.
|
||||
|
||||
Use a named specialized target only when its risk is in scope:
|
||||
|
||||
```bash
|
||||
# Set custom timeout (default: 3m)
|
||||
TEST_TIMEOUT=5m ./scripts/test.sh
|
||||
|
||||
# Enable verbose output
|
||||
TEST_VERBOSE=1 ./scripts/test.sh
|
||||
|
||||
# Run specific pattern
|
||||
TEST_RUN=TestCreate ./scripts/test.sh
|
||||
make test-regression
|
||||
make test-upgrade
|
||||
make test-cross-version
|
||||
make test-migration
|
||||
```
|
||||
|
||||
### Docker (Dolt Integration Tests)
|
||||
For a failing GitHub Actions check, follow the current workflow and its
|
||||
`Makefile` target when exact reproduction matters. The local runner and CI
|
||||
intentionally have different contracts in some cases.
|
||||
|
||||
Dolt integration tests require Docker with the exact Dolt image cached locally.
|
||||
Tests auto-detect the environment and skip gracefully — no manual configuration
|
||||
needed.
|
||||
### Test Environment and Readiness
|
||||
|
||||
#### Readiness states
|
||||
The runner isolates `HOME`, Git configuration, and Dolt state. By default its
|
||||
test environment adds `dolt` to `BEADS_TEST_SKIP`; set
|
||||
`BEADS_TEST_ENV_RUN_DOLT=1` only when deliberately exercising the Dolt path
|
||||
and its prerequisites are available. Do not make ordinary tests depend on a
|
||||
developer's database, daemon, global Git configuration, or filesystem state.
|
||||
|
||||
```csv
|
||||
State,Condition,Behavior
|
||||
doltSkipped,BEADS_TEST_SKIP contains "dolt",Silent skip (no warning)
|
||||
doltNoDocker,Docker daemon not reachable,WARN + skip
|
||||
doltNoImage,No Dolt image at all,WARN + skip with pull instruction
|
||||
doltWrongVersion,Image repo cached but wrong tag,WARN + skip with pull instruction
|
||||
doltReady,Exact image cached and Docker running,Run tests
|
||||
```
|
||||
|
||||
States are checked once per test binary and cached. Order of evaluation:
|
||||
`BEADS_TEST_SKIP` → Docker availability → exact image → any image version.
|
||||
|
||||
#### Skipping Dolt tests explicitly
|
||||
|
||||
Set `BEADS_TEST_SKIP` to opt out without Docker overhead (~1s `docker info`):
|
||||
To skip an optional service explicitly, use the existing skip mechanism:
|
||||
|
||||
```bash
|
||||
# Skip Dolt tests silently
|
||||
BEADS_TEST_SKIP=dolt ./scripts/test.sh
|
||||
|
||||
# Skip multiple services (comma-separated)
|
||||
BEADS_TEST_SKIP=dolt,slow ./scripts/test.sh
|
||||
BEADS_TEST_SKIP=dolt ./scripts/test.sh ./...
|
||||
```
|
||||
|
||||
### Short Mode and Test Boundaries
|
||||
Tests that need a temporary repository or store should use `t.TempDir()` and
|
||||
`t.Cleanup()`. Temporary repositories must set a repository-local hooks path;
|
||||
do not inherit the developer's global hooks configuration.
|
||||
|
||||
`testing.Short()` is reserved for true runtime, stress, and large-fixture skips.
|
||||
It must not be used as an implicit integration, e2e, API, Docker, or external
|
||||
dependency boundary.
|
||||
|
||||
Use these mechanisms instead:
|
||||
|
||||
- `//go:build integration` or `//go:build e2e` for named suites.
|
||||
- Environment readiness checks such as `BEADS_TEST_SKIP=dolt`,
|
||||
`BEADS_TEST_EMBEDDED_DOLT=1`, or required API-key checks.
|
||||
- Named wrappers such as `make ci-pr-core`, the main integration shards, and the
|
||||
package gate wrappers.
|
||||
|
||||
Run `make check-testing-short` to verify that new `testing.Short()` usage stays
|
||||
within the approved runtime/stress/large-fixture allowlist. The PR policy wrapper
|
||||
runs the same check.
|
||||
|
||||
#### Enabling Dolt tests
|
||||
For manual CLI experiments, run both initialization and subsequent commands
|
||||
from a disposable working directory:
|
||||
|
||||
```bash
|
||||
# Pull the exact Dolt image to enable integration tests
|
||||
docker pull dolthub/dolt-sql-server:2.2.0
|
||||
|
||||
# Point tests at an existing Dolt server (skips container startup)
|
||||
BEADS_DOLT_PORT=3308 ./scripts/test.sh
|
||||
beads_manual_dir="$(mktemp -d)"
|
||||
(
|
||||
set -e
|
||||
cd "$beads_manual_dir"
|
||||
bd init --quiet --prefix test --skip-hooks --skip-agents
|
||||
bd create "Test issue" -p 1
|
||||
)
|
||||
rm -rf -- "$beads_manual_dir"
|
||||
```
|
||||
|
||||
`BEADS_DOLT_PORT` — when set, tests reuse the server at that port instead of
|
||||
starting a container. Port 3307 is hardcoded as production and always rejected.
|
||||
`BEADS_DB` selects a database for database-opening commands, but it does not by
|
||||
itself redirect `bd init` workspace setup. Never run a manual `bd init` from a
|
||||
production workspace merely because `BEADS_DB` points elsewhere.
|
||||
|
||||
### Advanced Usage
|
||||
`testing.Short()` is for genuine runtime, stress, or large-fixture skips. It
|
||||
is not a substitute for declaring an integration, end-to-end, API, Docker, or
|
||||
external-dependency boundary. Keep new uses within the repository policy:
|
||||
|
||||
```bash
|
||||
# Skip additional tests beyond .test-skip
|
||||
./scripts/test.sh -skip SomeSlowTest
|
||||
|
||||
# Run with custom timeout
|
||||
./scripts/test.sh -timeout 5m
|
||||
|
||||
# Combine flags
|
||||
./scripts/test.sh -v -run TestCreate ./internal/beads/...
|
||||
make check-testing-short
|
||||
```
|
||||
|
||||
## Known Broken Tests
|
||||
## Test Design
|
||||
|
||||
Tests in `.test-skip` are automatically skipped by `scripts/test.sh`.
|
||||
### Seams, Scenarios, and Doubles
|
||||
|
||||
At the time of this review, `.test-skip` contains only comments and no active
|
||||
test-name patterns. Treat any new skip as a temporary exception: file the
|
||||
upstream issue first, record it in `.test-skip`, and remove the skip when the
|
||||
test is fixed.
|
||||
Write one scenario at the smallest seam that demonstrates the behavior. Cover
|
||||
the boundary or failure mode that changes the user result; use table-driven
|
||||
subtests when examples share setup. Do not repeat the same scenario through a
|
||||
helper, every caller, and the CLI merely because all are available.
|
||||
|
||||
## For Claude Code / AI Agents
|
||||
A recording double or fake should be narrow: model only the calls, inputs,
|
||||
outputs, and failures the test needs. It should not recreate a storage engine,
|
||||
process manager, or another subsystem just to make a unit test look realistic.
|
||||
|
||||
When running tests during development:
|
||||
A behavioral fake is different. If it stands in for a contract shared by
|
||||
multiple production implementations, give it the same semantic-conformance
|
||||
suite as those implementations. That shared suite defines observable behavior;
|
||||
it prevents the fake from teaching callers a contract production code does not
|
||||
honor.
|
||||
|
||||
### Best Practices
|
||||
Semantic conformance asks whether an implementation produces the promised
|
||||
results, errors, and state transitions for the same operation. Persistence
|
||||
conformance asks whether the real persistence boundary preserves its required
|
||||
durability, transaction, migration, and recovery properties. They answer
|
||||
different questions. Do not claim backend parity unless a stated contract and
|
||||
its conformance suite establish it.
|
||||
|
||||
1. **Use the test script:** Always use `./scripts/test.sh` instead of `go test` directly
|
||||
- Automatically skips known broken tests
|
||||
- Uses appropriate timeouts
|
||||
- Matches local default validation; use future `scripts/ci/*` wrappers when
|
||||
reproducing exact CI jobs
|
||||
- Only if intentionally exercising the ICU regex path, use `./scripts/test-icu-path.sh` (or deprecated `make test-full-cgo`)
|
||||
### Tier Admission
|
||||
|
||||
2. **Target specific tests when possible:**
|
||||
```bash
|
||||
# Instead of running everything:
|
||||
./scripts/test.sh
|
||||
An integration test belongs above the unit seam only when it covers a distinct
|
||||
boundary that a narrow double cannot prove, such as configuration wiring, a
|
||||
real filesystem or Git interaction, a subprocess protocol, or persistence
|
||||
behavior.
|
||||
|
||||
# Run just what you changed:
|
||||
./scripts/test.sh -run TestSpecificFeature ./cmd/bd/...
|
||||
```
|
||||
An end-to-end test is admitted only when all of these are true:
|
||||
|
||||
3. **Compilation is the bottleneck:**
|
||||
- The 180-second compilation time dominates
|
||||
- Individual tests are fast
|
||||
- Use `-run` to avoid recompiling unnecessarily
|
||||
1. The failure would be user-visible.
|
||||
2. A real process, setup, or wiring boundary owns a distinct failure risk that
|
||||
no lower seam can prove.
|
||||
3. Lower-tier tests cover the underlying behavior where practical, leaving the
|
||||
end-to-end test focused on that boundary.
|
||||
4. No existing end-to-end test already covers the same boundary risk.
|
||||
|
||||
4. **Check for new failures:**
|
||||
```bash
|
||||
# If you see a new failure, check if it's known:
|
||||
cat .test-skip
|
||||
```
|
||||
Lower-tier coverage of the same user journey does not disqualify the
|
||||
end-to-end test; duplicate coverage of the same boundary risk does. State that
|
||||
risk in the test name or nearby documentation.
|
||||
|
||||
### Adding Tests to Skip List
|
||||
### Avoid Incidental Complexity
|
||||
|
||||
If you discover a broken test:
|
||||
Avoid these patterns unless they are the behavior under test:
|
||||
|
||||
1. File a GitHub issue documenting the problem
|
||||
2. Add to `.test-skip`:
|
||||
```bash
|
||||
# Issue #NNN: Brief description
|
||||
TestNameToSkip
|
||||
```
|
||||
3. Tests in `.test-skip` support regex patterns
|
||||
- Duplicate scenarios at several layers.
|
||||
- Global-state reset choreography. Prefer per-test state and cleanup.
|
||||
- Subprocesses, listeners, sleeps, or real-store setup for a unit-level claim.
|
||||
- Assertions on private implementation shape when observable behavior is the
|
||||
contract.
|
||||
- Performance claims without a repeatable measurement and stated workload.
|
||||
|
||||
## Test Organization
|
||||
Sleeps, listeners, and real stores are appropriate when the test is specifically
|
||||
about timing, lifecycle, protocol, or persistence. Keep the setup scoped and
|
||||
make that reason apparent.
|
||||
|
||||
### Slowest Tests (>0.05s)
|
||||
## Failures, Skips, and Review
|
||||
|
||||
The top slow tests in cmd/bd:
|
||||
- `TestDoctorWithBeadsDir` (1.68s) - Only significantly slow test
|
||||
- `TestFlushManagerDebouncing` (0.21s)
|
||||
- `TestDebouncer_*` tests (0.06-0.12s each) - Intentional sleeps for concurrency testing
|
||||
- `TestMultiWorkspaceDeletionSync` (0.12s)
|
||||
`.test-skip` is a local, temporary exception list. If an unrelated failure is
|
||||
already listed, report it rather than silently broadening the skip. Before
|
||||
adding a new skip, record the issue it tracks and remove the skip when the
|
||||
underlying failure is fixed.
|
||||
|
||||
Most tests are <0.01s and very fast.
|
||||
Before opening a PR:
|
||||
|
||||
### Package Structure
|
||||
1. For docs-only changes, run the applicable docs, link, freshness, and diff
|
||||
checks; do not run the full Go suite by default.
|
||||
2. For Go code, keep the focused and affected-package tests green, then run one
|
||||
final `make test`.
|
||||
3. Run only the named CI wrapper, specialized target, or risk gate required by
|
||||
the changed surface, or the one needed to reproduce a CI result.
|
||||
|
||||
```
|
||||
cmd/bd/ - Main CLI tests (82 test files, most of the suite)
|
||||
internal/beads/ - Core beads library tests
|
||||
internal/storage/ - Storage backend tests (Dolt embedded/server, conformance)
|
||||
internal/rpc/ - RPC protocol tests
|
||||
internal/*/ - Various internal package tests
|
||||
```
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
The current CI workflow does not yet call the `scripts/ci/*` wrappers for every
|
||||
job. Until workflow migration is complete, use the command documented in the
|
||||
failing workflow when reproducing an existing status check exactly.
|
||||
|
||||
Use `scripts/test.sh` for local default validation and targeted development
|
||||
runs.
|
||||
|
||||
Use the CI wrappers for the accepted target PR contracts:
|
||||
|
||||
```bash
|
||||
make ci-pr-core
|
||||
make ci-pr-policy
|
||||
make ci-pr-lint
|
||||
```
|
||||
|
||||
Use the package gate wrappers when touching package or docs surfaces:
|
||||
|
||||
```bash
|
||||
make ci-package-mcp
|
||||
make ci-package-npm
|
||||
```
|
||||
|
||||
### Coverage Signal Policy
|
||||
|
||||
PR confidence is based on behavior checks, not raw coverage percentage.
|
||||
|
||||
- Treat Codecov percentages as informational trend data.
|
||||
- Prefer focused tests for risky paths (storage, sync/git, migrations, state
|
||||
transitions, and corruption/integrity handling) over broad line-coverage
|
||||
churn.
|
||||
- Add or extend at least one targeted regression test when fixing risky logic.
|
||||
- Do not block a change solely on overall coverage movement when behavioral
|
||||
checks are strong.
|
||||
|
||||
For the current CI/test-surface inventory and cleanup roadmap, see
|
||||
[CI_TEST_SURFACE_AUDIT.md](CI_TEST_SURFACE_AUDIT.md). That audit documents
|
||||
where local commands and GitHub Actions currently diverge before the CI cleanup
|
||||
work starts changing workflow behavior.
|
||||
|
||||
For accepted CI tier decisions and the implementation order, see
|
||||
[CI_CLEANUP_PLAN.md](CI_CLEANUP_PLAN.md).
|
||||
|
||||
## Debugging Test Failures
|
||||
|
||||
### Get detailed output
|
||||
```bash
|
||||
./scripts/test.sh -v ./path/to/package/...
|
||||
```
|
||||
|
||||
### Run a single test
|
||||
```bash
|
||||
./scripts/test.sh -run '^TestExactName$' ./cmd/bd/...
|
||||
```
|
||||
|
||||
### Check which tests are being skipped
|
||||
```bash
|
||||
./scripts/test.sh 2>&1 | head -5
|
||||
```
|
||||
|
||||
Output shows:
|
||||
```
|
||||
Running: go test -timeout 3m -skip TestFoo|TestBar ./...
|
||||
Skipping: TestFoo|TestBar
|
||||
```
|
||||
|
||||
## Contributing
|
||||
|
||||
When adding new tests:
|
||||
|
||||
1. Keep tests fast (<0.1s if possible)
|
||||
2. Use `t.Parallel()` for independent tests
|
||||
3. Clean up resources in `t.Cleanup()` or `defer`
|
||||
4. Avoid sleeps unless testing concurrency
|
||||
|
||||
When tests break:
|
||||
|
||||
1. Fix them if possible
|
||||
2. If unfixable right now, file an issue and add to `.test-skip`
|
||||
3. Document the issue in `.test-skip` with issue number
|
||||
For historical CI inventory and maintainer planning context, consult
|
||||
[CI_TEST_SURFACE_AUDIT.md](CI_TEST_SURFACE_AUDIT.md) and
|
||||
[CI_CLEANUP_PLAN.md](CI_CLEANUP_PLAN.md). For current commands, use the
|
||||
workflow files and `Makefile`; these context documents are not a second testing
|
||||
guide or a live CI inventory.
|
||||
|
|
|
|||
|
|
@ -1,259 +1,4 @@
|
|||
# Testing Philosophy
|
||||
|
||||
This document covers **what to test** and **what not to test**. For how to run tests, see [TESTING.md](TESTING.md).
|
||||
|
||||
## The Test Pyramid
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ E2E Tests │ ← PR/Deploy only (slow, expensive)
|
||||
│ ~5% of tests │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌──────────────┴──────────────┐
|
||||
│ Integration Tests │ ← PR gate (moderate)
|
||||
│ ~15% of tests │
|
||||
└──────────────┬──────────────┘
|
||||
│
|
||||
┌────────────────────────┴────────────────────────┐
|
||||
│ Unit Tests (Fast) │ ← Every save/commit
|
||||
│ ~80% of tests │
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Tier 1: Fast Tests (< 5 seconds total)
|
||||
|
||||
**When**: Every file save, pre-commit hooks, continuous during development
|
||||
|
||||
- Pure function tests (no I/O)
|
||||
- In-memory data structure tests
|
||||
- Business logic validation
|
||||
- Mock all external dependencies
|
||||
|
||||
**In beads**: Core logic tests using `newTestStore()` with in-memory SQLite
|
||||
|
||||
### Tier 2: Integration Tests (< 30 seconds)
|
||||
|
||||
**When**: Pre-push, PR checks
|
||||
|
||||
- Real file system operations
|
||||
- Git operations with temp repos
|
||||
- Config file parsing
|
||||
- CLI argument handling
|
||||
|
||||
**In beads**: Tests tagged with `//go:build integration`, server mode tests
|
||||
|
||||
### Tier 3: E2E / Smoke Tests (1-5 minutes)
|
||||
|
||||
**When**: PR merge, pre-deploy, nightly
|
||||
|
||||
- Full `bd init` → `bd doctor` → `bd doctor --fix` workflow
|
||||
- Real API calls (to staging)
|
||||
- Cross-platform verification
|
||||
|
||||
---
|
||||
|
||||
## What Makes a Test "Right"
|
||||
|
||||
A good test:
|
||||
|
||||
1. **Catches a bug you'd actually ship** - not theoretical edge cases
|
||||
2. **Documents expected behavior** - serves as living documentation
|
||||
3. **Runs fast enough to not skip** - slow tests get disabled
|
||||
4. **Isn't duplicated elsewhere** - tests one thing, one way
|
||||
|
||||
---
|
||||
|
||||
## What to Test (Priority Matrix)
|
||||
|
||||
| Priority | What | Why | Examples in beads |
|
||||
|----------|------|-----|-------------------|
|
||||
| **High** | Core business logic | This is what users depend on | `sync`, `doctor`, `export`, `import` |
|
||||
| **High** | Error paths that could corrupt data | Data loss is catastrophic | Config handling, git operations, database integrity |
|
||||
| **Medium** | Edge cases from production bugs | Discovered through real issues | Orphan handling, ID collision detection |
|
||||
| **Low** | Display/formatting | Visual output, can be manually verified | Table formatting, color output |
|
||||
|
||||
---
|
||||
|
||||
## What NOT to Test Extensively
|
||||
|
||||
### Simple utility functions
|
||||
Trust the language. Don't test that `strings.TrimSpace` works.
|
||||
|
||||
### Every permutation of inputs
|
||||
Use table-driven tests with representative cases instead of exhaustive permutations.
|
||||
|
||||
```go
|
||||
// BAD: 10 separate test functions
|
||||
func TestPriority0(t *testing.T) { ... }
|
||||
func TestPriority1(t *testing.T) { ... }
|
||||
func TestPriority2(t *testing.T) { ... }
|
||||
|
||||
// GOOD: One table-driven test
|
||||
func TestPriorityMapping(t *testing.T) {
|
||||
cases := []struct{ in, want int }{
|
||||
{0, 4}, {1, 0}, {5, 3}, // includes boundary
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(fmt.Sprintf("priority_%d", tc.in), func(t *testing.T) {
|
||||
got := mapPriority(tc.in)
|
||||
if got != tc.want { t.Errorf(...) }
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Obvious behavior
|
||||
Don't test "if file exists, return true" - trust the implementation.
|
||||
|
||||
### Same logic through different entry points
|
||||
If you test a function directly, don't also test it through every caller.
|
||||
|
||||
---
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
|
||||
### 1. Trivial Assertions
|
||||
|
||||
Testing obvious happy paths that would pass with trivial implementations.
|
||||
|
||||
```go
|
||||
// BAD: What bug would this catch?
|
||||
func TestValidateBeadsWorkspace(t *testing.T) {
|
||||
dir := setupTestWorkspace(t)
|
||||
if err := validateBeadsWorkspace(dir); err != nil {
|
||||
t.Errorf("expected no error, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// GOOD: Test the interesting error cases
|
||||
func TestValidateBeadsWorkspace(t *testing.T) {
|
||||
cases := []struct{
|
||||
name string
|
||||
setup func(t *testing.T) string
|
||||
wantErr string
|
||||
}{
|
||||
{"missing .beads dir", setupNoBeadsDir, "not a beads workspace"},
|
||||
{"corrupted db", setupCorruptDB, "database is corrupted"},
|
||||
{"permission denied", setupNoReadAccess, "permission denied"},
|
||||
}
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Duplicate Error Path Testing
|
||||
|
||||
Testing the same logic multiple ways instead of once with table-driven tests.
|
||||
|
||||
```go
|
||||
// BAD: Repetitive individual assertions
|
||||
if config.PriorityMap["0"] != 4 { t.Errorf(...) }
|
||||
if config.PriorityMap["1"] != 0 { t.Errorf(...) }
|
||||
if config.PriorityMap["2"] != 1 { t.Errorf(...) }
|
||||
|
||||
// GOOD: Table-driven
|
||||
for k, want := range expectedMap {
|
||||
if got := config.PriorityMap[k]; got != want {
|
||||
t.Errorf("PriorityMap[%q] = %d, want %d", k, got, want)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 3. I/O Heavy Tests Without Mocking
|
||||
|
||||
Unit tests that execute real commands or heavy I/O when they could mock.
|
||||
|
||||
```go
|
||||
// BAD: Actually executes external commands in unit test
|
||||
func TestServerFix(t *testing.T) {
|
||||
exec.Command("bd", "dolt", "stop").Run()
|
||||
// ...
|
||||
}
|
||||
|
||||
// GOOD: Mock the execution or use integration test tag
|
||||
func TestServerFix(t *testing.T) {
|
||||
executor := &mockExecutor{}
|
||||
fix := NewServerFix(executor)
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
### 4. Testing Implementation, Not Behavior
|
||||
|
||||
Tests that break when you refactor, even though behavior is unchanged.
|
||||
|
||||
```go
|
||||
// BAD: Tests internal state
|
||||
if len(server.connectionPool) != 3 { t.Error(...) }
|
||||
|
||||
// GOOD: Tests observable behavior
|
||||
if resp, err := server.HandleRequest(req); err != nil { t.Error(...) }
|
||||
```
|
||||
|
||||
### 5. Missing Boundary Tests
|
||||
|
||||
Testing known-good values but not boundaries and invalid inputs.
|
||||
|
||||
```go
|
||||
// BAD: Only tests middle values
|
||||
TestPriority(1) // works
|
||||
TestPriority(2) // works
|
||||
|
||||
// GOOD: Tests boundaries and invalid
|
||||
TestPriority(-1) // invalid - expect error
|
||||
TestPriority(0) // boundary - min valid
|
||||
TestPriority(4) // boundary - max valid
|
||||
TestPriority(5) // boundary - first invalid
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Target Metrics
|
||||
|
||||
| Metric | Target | Current (beads) | Status |
|
||||
|--------|--------|-----------------|--------|
|
||||
| Test-to-code ratio | 0.5:1 - 1.5:1 | 0.85:1 | Healthy |
|
||||
| Fast test suite | < 5 seconds | 3.8 seconds | Good |
|
||||
| Integration tests | < 30 seconds | ~15 seconds | Good |
|
||||
| Compilation overhead | Minimize | 180 seconds | Bottleneck |
|
||||
|
||||
### Interpretation
|
||||
|
||||
- **0.5:1** - Light coverage, fast iteration (acceptable for utilities)
|
||||
- **1:1** - Solid coverage for most projects (our target)
|
||||
- **1.5:1** - Heavy coverage for critical systems
|
||||
- **2:1+** - Over-engineered, maintenance burden
|
||||
|
||||
---
|
||||
|
||||
## Beads-Specific Guidance
|
||||
|
||||
### Well-Covered (Maintain)
|
||||
|
||||
| Area | Why It's Well-Tested |
|
||||
|------|---------------------|
|
||||
| Sync/Export/Import | Data integrity critical - comprehensive edge cases |
|
||||
| SQLite transactions | Rollback safety, atomicity guarantees |
|
||||
| Merge operations | Dolt-native cell-level merge |
|
||||
| Database locking | Prevents corruption from multiple instances |
|
||||
|
||||
### Needs Attention
|
||||
|
||||
| Area | Gap | Priority |
|
||||
|------|-----|----------|
|
||||
| Server lifecycle | Shutdown/signal handling | Medium |
|
||||
| Concurrent operations | Stress testing under load | Medium |
|
||||
| Boundary validation | Edge inputs in mapping functions | Low |
|
||||
|
||||
### Skip These
|
||||
|
||||
- Display formatting tests (manually verify)
|
||||
- Simple getters/setters
|
||||
- Tests that duplicate SQLite's guarantees
|
||||
|
||||
---
|
||||
|
||||
## Related Docs
|
||||
|
||||
- [TESTING.md](TESTING.md) - How to run tests
|
||||
- [README_TESTING.md](README_TESTING.md) - Stable pointer to the active testing docs
|
||||
This compatibility page points to the canonical [Testing Guide](TESTING.md),
|
||||
which covers test design, test selection, and commands.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue