After a successful human-output create/show/list/ready, bd may append a "💡 Tip:" line chosen by a probability roll whose RNG is clock-seeded unless BEADS_TIP_SEED is set. run_scenario env_clear()s and passes only PATH/HOME/TMPDIR + BEADS_TEST_MODE=1, so on a host with ~/.claude (claude_setup tip eligible, p=0.6) every bd process rolled independently and out-of-scope human-output scenarios (create_human_readable_output, show_short_mode_text, show_multiple_short, ready_empty_no_issues_plain, ...) diverged main-vs-main. Pass BEADS_TIP_SEED=1214 to every step for both reference and candidate. Go math/rand's first Float64() for that seed is 0.0000159, so the highest-priority eligible tip always wins its roll. Where the host makes a tip eligible, it stays visible and lands on a deterministic step, so a change to which tip prints, on which step, or to its text (including a tip leaking into --json) still diverges instead of being normalized away. Where no tip is eligible (no ~/.claude, beads plugin enabled, or a `bd prime` hook) none prints and tip changes go unobserved; the README now says so. The new unit test spawns printenv through run_scenario. That races the existing stub-bd test, which writes its stub with std::fs::write and then execs it: a fork while the write fd is open makes the exec fail with ETXTBSY (rust-lang/rust#114554). Serialize both spawning tests on a SPAWN mutex. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
9.9 KiB
Oracle A — refactor-safety differential conformance for bd
Oracle A answers one question mechanically: did this change alter any
user-visible bd behavior?
It runs the same curated CLI scenarios against two separately-built bd
binaries and diffs every step:
| role | built from | meaning |
|---|---|---|
| REFERENCE | the merge-base with origin/main (overridable via REF_REF) |
the "before" |
| CANDIDATE | the current working tree (HEAD + uncommitted) | the "after" |
Each scenario is an ordered list of bd CLI argv steps run as real processes in
a throwaway workspace (bd init + steps; a scenario whose first step is its own
bd init starts in an empty workspace). For every step the harness compares
exit code, stderr, and JSON-aware stdout (object key order ignored;
array order compared as a multiset by default, as an ordered sequence for
scenarios flagged ordered). Volatile values — timestamps, UUIDs, the host
actor identity, the bd show --json "revision" token (matched by member name,
see below), the run's own throwaway workspace path, and the repository and clone
IDs bd init hashes from it — are normalized to
<TS>/<UUID>/<ACTOR>/<EMAIL>/<REV>/<WORKSPACE>/<REPO_ID>/<CLONE_ID>
before comparison.
Tolerated in-scope divergences: zero.
Run it
tests/oracle-a/run-oracle-a.sh
Exit status: 0 = 100% in-scope pass, 1 = at least one in-scope divergence,
2 = setup/build error. On failure, each divergence is printed with the
reference vs candidate value for the offending step.
Overrides:
REF_REF=<gitref>— compare against something other than the merge-base (e.g. a release tag, ororigin/main's tip).ORACLE_CATALOG=1— deep tier: in addition to the curated scenarios, run the full enumerated catalog (harness/scenarios/enumerated.json, ~500 deterministic scenarios covering the widerbdCLI surface). Much broader coverage, ~10–15 min. Off by default so the everyday gate stays fast.KEEP_ARTIFACTS=1— keep the scratch build dir (binaries, goldens, scoreboard output) for inspection instead of deleting it on exit. The per-divergence detail is written to that run's ownoracle-a-failures.txtthere (viaORACLE_FAILURES_FILE), never to a path shared with other runs.
Two tiers
- Curated (default) — the hand-maintained scenarios in
harness/src/scenarios.rs: a small, always-green set targeting the highest-value contract behaviors. Fast enough to run on every change. - Enumerated catalog (
ORACLE_CATALOG=1) — ~500 deterministic scenarios inharness/scenarios/enumerated.json, generated to sweep the wider CLI surface as data. Non-deterministic entries (ID minting, non-reproducible init output) and entries not runnable embedded-only are excluded automatically; entries already covered by the curated set are de-duped. The in-scope predicate (below) applies identically to both tiers.
Prerequisites
- Rust /
cargo— builds the conformance harness (harness/). - A CGO toolchain (
gcc/cc) andgo—bdembeds Dolt, which is cgo; both binaries build withCGO_ENABLED=1 -tags gms_pure_go(thegms_pure_gotag is mandatory perengdocs/ICU-POLICY.md). gitwithorigin/mainfetched (the script resolvesREF_REFlocally; rungit fetchfirst if it is stale).jq— used for the golden floor assertion (the reference'screatesteps must exit 0 with an id, so a broken environment can't score a false green).
Runtime
Dominated by the cold reference build (a full bd compile, ~1 min on a warm
module cache, several minutes cold). The candidate build reuses the local build
cache and is fast. Harness build is ~10 s. Golden capture + scoring run the
curated scenarios as real bd processes (~30–60 s). End-to-end: ~2–7 minutes
depending on Go build-cache warmth (the deep tier, ORACLE_CATALOG=1, adds the
~500-scenario catalog for ~10–15 minutes total). There is no Dolt server in
the loop — every scenario uses embedded Dolt in its own tempdir.
What green PROVES
- For the curated contract scenarios (see
harness/src/scenarios.rs), the candidatebdproduces byte-identical (post-normalization) exit codes, stderr, and JSON-structural stdout to the referencebd, step for step. - This covers the semantics most likely to break under a storage or serialization
refactor: the close→ready
is_blockedpropagation, transitive/parent-child blocking, cycle rejection, claim lifecycle + idempotent self-reclaim, storage tiers (ephemeral/no-history) set algebra,deleteand a real (non-dry-run)purgewith re-seed (both must recompute surviving neighbours'is_blocked),commentadd/list,config set/getsuccess and reject paths, metadata storage, label-based query filtering (list --label), and the error contracts (bd sqlembedded-unsupported, not-found, config-key rejection).
What green does NOT prove
- Only the in-scope surface. The predicate is the contract commands + flags in
harness/src/bin/scoreboard.rs(IN_SCOPE_CMDS/IN_SCOPE_FLAGS, with--jsonrequired for the JSON-output commands). Behavior outside that set — most of the widerbdcommand surface, human/plain output modes, and any flag not listed — is reported as out-of-scope and is not a gate. A change that touches only out-of-scope behavior passes Oracle A. - Ordering tiebreaks are UNGATED.
ready/listorder among equal-priority issues is empirically nondeterministic here: every issue is created seconds apart, bd orders equal priorities bycreated_at DESC(with anid ASCfinal tiebreak), andcreated_atis both sub-second-precision-dependent AND normalized to<TS>— so the same equal-priority creates land in a different order run to run. Array output is therefore compared as a multiset by default; only scenarios with all-DISTINCT priorities are markedorderedand compared as a sequence. What this leaves unpinned: theid ASC/created_at DESCtiebreak chain, and the >48h hybrid-priority arm (unreachable without an injectable clock). A change that breaks tiebreaks or the hybrid arm passes Oracle A. - Minted-ID paths are UNGATED. Every create scenario passes an explicit
--id. A--id-less create mints a hash-derived ID that hashes the wall-clock timestamp, so the reference and candidate mint different IDs for "the same" create and the ID is NOT normalized (it is not a UUID) — an--id-less scenario would flake. ID minting is a property-test concern, not a byte-diff one. - Anything invisible to stdout/stderr/exit diffing. Post-commit hook firing,
rollback-suppressed side effects, and N-commits-instead-of-one are deliberately
NOT asserted here. Timestamp values are erased by
<TS>normalization, so a local-vs-UTC divergence in a timestamp value is also out of reach. Likewise a non-zero"revision"token is erased to<REV>(every create, update, close, reopen, claim and unclaim remints it from crypto/rand), so whether a write remints it is not gated; the member's presence, its string type, and a never-stamped"0"still are. TheRepository ID/Clone IDvalues a scenario's ownbd initprints are erased to<REPO_ID>/<CLONE_ID>too (they hash the random workspace path), so how they are derived is not gated; the lines and their widths still are. <REV>matches a member name, not a position. The"revision"collapse runs over the raw stdout/stderr text before any JSON parse, so it erases EVERY"revision"member holding a non-zero decimal string — at any depth, and in plain (non-JSON) text too — not only thebd show --jsonissue token. That includes a user metadata key namedrevision, which the in-scope--set-metadata/--metadataflags can set: a change to its value passes the gate. No scenario sets one today; use another key name in any that would.- The scenario environment is scrubbed, and
BEADS_TEST_MODE=1is ungated. Scenarios run with an explicit env whitelist (PATH/HOME/TMPDIR+BEADS_TEST_MODE=1+BEADS_TIP_SEED) rather than the inherited host env, so a green certifies the same configuration every run. ButBEADS_TEST_MODE=1itself changes store construction (auto-server decisions off, port rewiring), so this rig does NOT cover the mode/topology construction paths — do not cite Oracle A green for those. - Tips are pinned, not erased, and depend on the host. Human output may end
with a
💡 Tip:line picked by a probability roll.BEADS_TIP_SEED(TIP_SEEDinharness/src/differential.rs) pins that roll so the highest-priority eligible tip always wins: its text, the step it prints on, and its frequency gate are compared like any other output, but a tip's probability value is not gated. Whether theclaude_setuptip is eligible depends on the hostHOME(~/.claudepresent, no beads plugin enabled, nobd primehook). Both binaries run on the same host, so this is symmetric, but on a host where no tip is eligible (a CI runner, or a beads developer with the plugin installed) none prints and tip changes go unobserved, including a tip leaking into--json. - This is a same-backend gate. Oracle A is same-backend (embedded Dolt) before-vs-after. It is a refactor-safety net, not a cross-backend conformance run.
The harness
The Rust crate under harness/ is a self-contained conformance rig
(MIT OR Apache-2.0). It builds two binaries — capture_golden (records the
reference bd's per-step traces) and scoreboard (replays every scenario against
the candidate bd and diffs it against the goldens). Goldens are re-captured from
the reference build into harness/testdata/golden/ (git-ignored) on every run,
so the gate always reflects the current "before" — nothing is pinned to a stale
snapshot. See harness/PROVENANCE.md for how goldens, the environment contract,
and the standalone Cargo manifest are handled.