beads/tests/oracle-a/README.md
Julian Knutsen c86a87bb20
test(oracle-a): pin BEADS_TIP_SEED so human-output scenarios stop flaking (#7707)
After a successful human-output create/show/list/ready, bd may append a
"💡 Tip:" line chosen by a probability roll whose RNG is clock-seeded
unless BEADS_TIP_SEED is set. run_scenario env_clear()s and passes only
PATH/HOME/TMPDIR + BEADS_TEST_MODE=1, so on a host with ~/.claude (claude_setup
tip eligible, p=0.6) every bd process rolled independently and out-of-scope
human-output scenarios (create_human_readable_output, show_short_mode_text,
show_multiple_short, ready_empty_no_issues_plain, ...) diverged main-vs-main.

Pass BEADS_TIP_SEED=1214 to every step for both reference and candidate.
Go math/rand's first Float64() for that seed is 0.0000159, so the
highest-priority eligible tip always wins its roll. Where the host makes a
tip eligible, it stays visible and lands on a deterministic step, so a change
to which tip prints, on which step, or to its text (including a tip leaking
into --json) still diverges instead of being normalized away. Where no tip is
eligible (no ~/.claude, beads plugin enabled, or a `bd prime` hook) none
prints and tip changes go unobserved; the README now says so.

The new unit test spawns printenv through run_scenario. That races the
existing stub-bd test, which writes its stub with std::fs::write and then
execs it: a fork while the write fd is open makes the exec fail with ETXTBSY
(rust-lang/rust#114554). Serialize both spawning tests on a SPAWN mutex.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-19 22:02:44 +00:00

9.9 KiB
Raw Blame History

Oracle A — refactor-safety differential conformance for bd

Oracle A answers one question mechanically: did this change alter any user-visible bd behavior?

It runs the same curated CLI scenarios against two separately-built bd binaries and diffs every step:

role built from meaning
REFERENCE the merge-base with origin/main (overridable via REF_REF) the "before"
CANDIDATE the current working tree (HEAD + uncommitted) the "after"

Each scenario is an ordered list of bd CLI argv steps run as real processes in a throwaway workspace (bd init + steps; a scenario whose first step is its own bd init starts in an empty workspace). For every step the harness compares exit code, stderr, and JSON-aware stdout (object key order ignored; array order compared as a multiset by default, as an ordered sequence for scenarios flagged ordered). Volatile values — timestamps, UUIDs, the host actor identity, the bd show --json "revision" token (matched by member name, see below), the run's own throwaway workspace path, and the repository and clone IDs bd init hashes from it — are normalized to <TS>/<UUID>/<ACTOR>/<EMAIL>/<REV>/<WORKSPACE>/<REPO_ID>/<CLONE_ID> before comparison. Tolerated in-scope divergences: zero.

Run it

tests/oracle-a/run-oracle-a.sh

Exit status: 0 = 100% in-scope pass, 1 = at least one in-scope divergence, 2 = setup/build error. On failure, each divergence is printed with the reference vs candidate value for the offending step.

Overrides:

  • REF_REF=<gitref> — compare against something other than the merge-base (e.g. a release tag, or origin/main's tip).
  • ORACLE_CATALOG=1 — deep tier: in addition to the curated scenarios, run the full enumerated catalog (harness/scenarios/enumerated.json, ~500 deterministic scenarios covering the wider bd CLI surface). Much broader coverage, ~10–15 min. Off by default so the everyday gate stays fast.
  • KEEP_ARTIFACTS=1 — keep the scratch build dir (binaries, goldens, scoreboard output) for inspection instead of deleting it on exit. The per-divergence detail is written to that run's own oracle-a-failures.txt there (via ORACLE_FAILURES_FILE), never to a path shared with other runs.

Two tiers

  • Curated (default) — the hand-maintained scenarios in harness/src/scenarios.rs: a small, always-green set targeting the highest-value contract behaviors. Fast enough to run on every change.
  • Enumerated catalog (ORACLE_CATALOG=1) — ~500 deterministic scenarios in harness/scenarios/enumerated.json, generated to sweep the wider CLI surface as data. Non-deterministic entries (ID minting, non-reproducible init output) and entries not runnable embedded-only are excluded automatically; entries already covered by the curated set are de-duped. The in-scope predicate (below) applies identically to both tiers.

Prerequisites

  • Rust / cargo — builds the conformance harness (harness/).
  • A CGO toolchain (gcc/cc) and go — bd embeds Dolt, which is cgo; both binaries build with CGO_ENABLED=1 -tags gms_pure_go (the gms_pure_go tag is mandatory per engdocs/ICU-POLICY.md).
  • git with origin/main fetched (the script resolves REF_REF locally; run git fetch first if it is stale).
  • jq — used for the golden floor assertion (the reference's create steps must exit 0 with an id, so a broken environment can't score a false green).

Runtime

Dominated by the cold reference build (a full bd compile, ~1 min on a warm module cache, several minutes cold). The candidate build reuses the local build cache and is fast. Harness build is ~10 s. Golden capture + scoring run the curated scenarios as real bd processes (~30–60 s). End-to-end: ~2–7 minutes depending on Go build-cache warmth (the deep tier, ORACLE_CATALOG=1, adds the ~500-scenario catalog for ~10–15 minutes total). There is no Dolt server in the loop — every scenario uses embedded Dolt in its own tempdir.

What green PROVES

  • For the curated contract scenarios (see harness/src/scenarios.rs), the candidate bd produces byte-identical (post-normalization) exit codes, stderr, and JSON-structural stdout to the reference bd, step for step.
  • This covers the semantics most likely to break under a storage or serialization refactor: the close→ready is_blocked propagation, transitive/parent-child blocking, cycle rejection, claim lifecycle + idempotent self-reclaim, storage tiers (ephemeral/no-history) set algebra, delete and a real (non-dry-run) purge with re-seed (both must recompute surviving neighbours' is_blocked), comment add/list, config set/get success and reject paths, metadata storage, label-based query filtering (list --label), and the error contracts (bd sql embedded-unsupported, not-found, config-key rejection).

What green does NOT prove

  • Only the in-scope surface. The predicate is the contract commands + flags in harness/src/bin/scoreboard.rs (IN_SCOPE_CMDS / IN_SCOPE_FLAGS, with --json required for the JSON-output commands). Behavior outside that set — most of the wider bd command surface, human/plain output modes, and any flag not listed — is reported as out-of-scope and is not a gate. A change that touches only out-of-scope behavior passes Oracle A.
  • Ordering tiebreaks are UNGATED. ready/list order among equal-priority issues is empirically nondeterministic here: every issue is created seconds apart, bd orders equal priorities by created_at DESC (with an id ASC final tiebreak), and created_at is both sub-second-precision-dependent AND normalized to <TS> — so the same equal-priority creates land in a different order run to run. Array output is therefore compared as a multiset by default; only scenarios with all-DISTINCT priorities are marked ordered and compared as a sequence. What this leaves unpinned: the id ASC / created_at DESC tiebreak chain, and the >48h hybrid-priority arm (unreachable without an injectable clock). A change that breaks tiebreaks or the hybrid arm passes Oracle A.
  • Minted-ID paths are UNGATED. Every create scenario passes an explicit --id. A --id-less create mints a hash-derived ID that hashes the wall-clock timestamp, so the reference and candidate mint different IDs for "the same" create and the ID is NOT normalized (it is not a UUID) — an --id-less scenario would flake. ID minting is a property-test concern, not a byte-diff one.
  • Anything invisible to stdout/stderr/exit diffing. Post-commit hook firing, rollback-suppressed side effects, and N-commits-instead-of-one are deliberately NOT asserted here. Timestamp values are erased by <TS> normalization, so a local-vs-UTC divergence in a timestamp value is also out of reach. Likewise a non-zero "revision" token is erased to <REV> (every create, update, close, reopen, claim and unclaim remints it from crypto/rand), so whether a write remints it is not gated; the member's presence, its string type, and a never-stamped "0" still are. The Repository ID/Clone ID values a scenario's own bd init prints are erased to <REPO_ID>/<CLONE_ID> too (they hash the random workspace path), so how they are derived is not gated; the lines and their widths still are.
  • <REV> matches a member name, not a position. The "revision" collapse runs over the raw stdout/stderr text before any JSON parse, so it erases EVERY "revision" member holding a non-zero decimal string — at any depth, and in plain (non-JSON) text too — not only the bd show --json issue token. That includes a user metadata key named revision, which the in-scope --set-metadata/--metadata flags can set: a change to its value passes the gate. No scenario sets one today; use another key name in any that would.
  • The scenario environment is scrubbed, and BEADS_TEST_MODE=1 is ungated. Scenarios run with an explicit env whitelist (PATH/HOME/TMPDIR + BEADS_TEST_MODE=1 + BEADS_TIP_SEED) rather than the inherited host env, so a green certifies the same configuration every run. But BEADS_TEST_MODE=1 itself changes store construction (auto-server decisions off, port rewiring), so this rig does NOT cover the mode/topology construction paths — do not cite Oracle A green for those.
  • Tips are pinned, not erased, and depend on the host. Human output may end with a 💡 Tip: line picked by a probability roll. BEADS_TIP_SEED (TIP_SEED in harness/src/differential.rs) pins that roll so the highest-priority eligible tip always wins: its text, the step it prints on, and its frequency gate are compared like any other output, but a tip's probability value is not gated. Whether the claude_setup tip is eligible depends on the host HOME (~/.claude present, no beads plugin enabled, no bd prime hook). Both binaries run on the same host, so this is symmetric, but on a host where no tip is eligible (a CI runner, or a beads developer with the plugin installed) none prints and tip changes go unobserved, including a tip leaking into --json.
  • This is a same-backend gate. Oracle A is same-backend (embedded Dolt) before-vs-after. It is a refactor-safety net, not a cross-backend conformance run.

The harness

The Rust crate under harness/ is a self-contained conformance rig (MIT OR Apache-2.0). It builds two binaries — capture_golden (records the reference bd's per-step traces) and scoreboard (replays every scenario against the candidate bd and diffs it against the goldens). Goldens are re-captured from the reference build into harness/testdata/golden/ (git-ignored) on every run, so the gate always reflects the current "before" — nothing is pinned to a stale snapshot. See harness/PROVENANCE.md for how goldens, the environment contract, and the standalone Cargo manifest are handled.