test(oracle-a): normalize per-run revision token and workspace path so main-vs-main is clean (#7680)

* test(oracle-a): normalize the per-write revision token so main-vs-main is clean

Since #5535, `bd show --json` publishes `"revision"`, the decimal spelling
of the row's row_lock. Every create and status/ownership write remints that
token from crypto/rand (issueops.freshRowLock), so it differs between any
two runs of the same scenario. The harness did not normalize it, so Oracle A
failed against itself: on the curated tier, origin/main vs origin/main scored
31 pass / 13 fail in scope, and each failure was a `show --json` step that
differed only in `"revision"`.

normalize() now collapses a non-zero decimal `"revision"` string to `<REV>`.
It keeps everything else literal: the member name, its presence and its
string type, "0" (a row that was never stamped), number-typed values, other
`*_revision` members, and escaped text inside a JSON string. A new unit test
covers each of these cases.

Curated tier, REF_REF=origin/main, run against itself: before 31 pass /
13 fail in scope (0/1 out of scope); after 44 pass / 0 fail (1/0).

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(oracle-a): normalize the per-run workspace path

Each scenario runs in a fresh tempfile::tempdir(), so any bd output that
echoes the workspace path differs on every run. When origin/main's bd was
captured twice with the deep tier (ORACLE_CATALOG=1, 546 scenarios), 40
catalog scenarios differed only in that path. All 40 start with their own
`init --prefix X --non-interactive` step, which runs in a workspace the
harness already initialized and fails with "Found existing Dolt database:
/tmp/.tmpXXXXXX/.beads/embeddeddolt/X".

run_scenario now hands normalize_in_workspace() the exact workspace path
and its symlink-resolved spelling, and that function replaces only those
strings with <WORKSPACE>, longest first. Other paths stay literal, so no
stable value is collapsed. All 40 scenarios are out of scope, so in-scope
counts do not change.

Deep tier, origin/main's bd against itself, scored by the scoreboard:
- base harness: 256 pass / 67 fail in scope, 154 / 69 out of scope.
- with both normalizers: 323 / 0 in scope, 219 / 4 out of scope.

The out-of-scope remainder changes from run to run and none of it is a
field value. It comes from randomized "Tip:" lines in human output and one
unordered `list --format dot`.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(oracle-a): document and pin the <REV> match scope and the workspace wiring (mc-ro9kg)

Review fixes for #7680, iteration 1.

revision_regex runs over the raw stdout/stderr text before any JSON
parse, so it collapses every "revision" member holding a non-zero
decimal string, at any depth and in plain text, not only the
bd show --json issue token. That includes a user metadata key named
revision, which the in-scope --set-metadata/--metadata flags can set,
so a change to its value passes the gate. The regex comment and a new
README "What green does NOT prove" bullet now say so, and
normalize_collapses_revision_by_member_name_at_any_depth pins the
scope: a nested metadata member and plain text are collapsed, while a
metadata key with any other name still gates. The old test name said
nothing else was collapsed; it is now
normalize_collapses_non_zero_revision_strings.

run_scenario_collapses_the_workspace_in_both_step_streams runs
run_scenario against a stub bd that prints its cwd on stdout and its
resolved path on stderr, and expects <WORKSPACE> in both. Before, the
two normalize_in_workspace calls in run_scenario could be reverted with
every unit test and the curated tier still green.

The remint wording in the comment and the README now names every
writer (create, update, close, reopen, claim and unclaim) instead of
only status/ownership writes.

Tests: cargo test --locked --offline (7 passed), clippy clean, no new
rustfmt hunks. Mutants, each in its own copy and target dir: reverting
either or both normalize_in_workspace calls fails the run_scenario
test; skipping a nested metadata member, or matching only after { or
",", fails the scope test.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com>
This commit is contained in:
Julian Knutsen 2026-10-19 04:27:28 +00:00 • committed by GitHub
commit 8fac20a7f1
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 195 additions and 13 deletions

View file

@ -15,9 +15,11 @@ Each scenario is an ordered list of `bd` CLI argv steps run as real processes in
a throwaway workspace (`bd init` + steps). For every step the harness compares
**exit code**, **stderr**, and **JSON-aware stdout** (object key order ignored;
array order compared as a multiset by default, as an ordered sequence for
scenarios flagged `ordered`). Volatile values — timestamps, UUIDs, and the host
actor identity — are normalized to `<TS>`/`<UUID>`/`<ACTOR>`/`<EMAIL>` before
comparison. **Tolerated in-scope divergences: zero.**
scenarios flagged `ordered`). Volatile values — timestamps, UUIDs, the host
actor identity, the `bd show --json` `"revision"` token (matched by member name,
see below), and the run's own throwaway workspace path — are normalized to
`<TS>`/`<UUID>`/`<ACTOR>`/`<EMAIL>`/`<REV>`/`<WORKSPACE>` before comparison.
**Tolerated in-scope divergences: zero.**
## Run it
@ -112,7 +114,18 @@ the loop — every scenario uses embedded Dolt in its own tempdir.
- **Anything invisible to stdout/stderr/exit diffing.** Post-commit hook firing,
rollback-suppressed side effects, and N-commits-instead-of-one are deliberately
NOT asserted here. Timestamp *values* are erased by `<TS>` normalization, so a
local-vs-UTC divergence in a timestamp value is also out of reach.
local-vs-UTC divergence in a timestamp value is also out of reach. Likewise a
non-zero `"revision"` token is erased to `<REV>` (every create, update, close,
reopen, claim and unclaim remints it from crypto/rand), so whether a write
remints it is not gated; the member's presence, its string type, and a
never-stamped `"0"` still are.
- **`<REV>` matches a member name, not a position.** The `"revision"` collapse runs
over the raw stdout/stderr text before any JSON parse, so it erases EVERY
`"revision"` member holding a non-zero decimal string — at any depth, and in
plain (non-JSON) text too — not only the `bd show --json` issue token. That
includes a user metadata key named `revision`, which the in-scope
`--set-metadata`/`--metadata` flags can set: a change to its value passes the
gate. No scenario sets one today; use another key name in any that would.
- **The scenario environment is scrubbed, and `BEADS_TEST_MODE=1` is ungated.**
Scenarios run with an explicit env whitelist (`PATH`/`HOME`/`TMPDIR` +
`BEADS_TEST_MODE=1`) rather than the inherited host env, so a green certifies the

View file

@ -89,20 +89,72 @@ fn uuid_regex() -> &'static regex::Regex {
})
}
/// Replace volatile values (timestamps, host identity, random UUIDs) with stable
/// tokens. UUIDs (e.g. comment ids) are random per run, so byte-diffing them is
/// meaningless — collapse them like timestamps.
fn revision_regex() -> &'static regex::Regex {
static RE: OnceLock<regex::Regex> = OnceLock::new();
RE.get_or_init(|| {
// The `"revision": "<token>"` JSON member (types.IssueDetails.Revision, on
// `bd show --json` since #5535). The token is the row's row_lock, which
// every create, update, close, reopen, claim and unclaim remints from
// crypto/rand (issueops.freshRowLock), so its value differs on every run.
// Only a NON-ZERO decimal token is collapsed: "0" (a never-stamped row)
// stays literal, and the member's name, presence and string type are
// untouched.
//
// The match is by member NAME over the raw text, before any JSON parse,
// so it collapses EVERY `"revision"` member holding such a string, at any
// depth and in plain (non-JSON) text too, not only the issue token. That
// includes a user metadata key named `revision`, which the in-scope
// --set-metadata/--metadata flags can set: a change to its value would
// pass the gate. No scenario sets one; use another key name in any that
// would. Pinned by normalize_collapses_revision_by_member_name_at_any_depth.
regex::Regex::new(r#"("revision"\s*:\s*")-?[1-9][0-9]*(")"#).expect("valid revision regex")
})
}
/// Replace volatile values (timestamps, host identity, random UUIDs, revision
/// tokens) with stable tokens. UUIDs (e.g. comment ids) and revision tokens are
/// random per run, so byte-diffing them is meaningless — collapse them like
/// timestamps.
pub fn normalize(s: &str) -> String {
let s = ts_regex().replace_all(s, "<TS>").into_owned();
let s = revision_regex()
.replace_all(s, "${1}<REV>${2}")
.into_owned();
let s = ts_regex().replace_all(&s, "<TS>").into_owned();
let s = uuid_regex().replace_all(&s, "<UUID>").into_owned();
s.replace(ACTOR_EMAIL, "<EMAIL>").replace(ACTOR_NAME, "<ACTOR>")
}
/// `normalize`, after first collapsing this run's own workspace directory to
/// `<WORKSPACE>`. Each run gets a fresh random tempdir, so any output that
/// echoes it (e.g. the "Found existing Dolt database: <dir>/.beads/..." error a
/// scenario's own `init` step hits) differs on every run. Only these exact path
/// strings are replaced, longest first, so a spelling that contains another
/// (`/private/var/...` around `/var/...`) is replaced whole.
pub fn normalize_in_workspace(s: &str, workspace: &[String]) -> String {
let mut paths: Vec<&str> = workspace
.iter()
.map(String::as_str)
.filter(|p| !p.is_empty())
.collect();
paths.sort_by_key(|p| std::cmp::Reverse(p.len()));
let mut s = s.to_string();
for p in paths {
s = s.replace(p, "<WORKSPACE>");
}
normalize(&s)
}
/// Run a scenario against `bin` in a throwaway workspace, returning the
/// normalized trace. Requires the binary to support `init -p <prefix> --quiet`
/// and to honor the workspace cwd (real bd uses embedded Dolt under `.beads/`).
pub fn run_scenario(bin: &Path, sc: &Scenario) -> std::io::Result<Trace> {
let work = tempfile::tempdir()?;
// Every spelling of the workspace bd may print: the tempdir as created and
// its symlink-resolved form (macOS tempdirs sit under /var -> /private/var).
let mut workspace = vec![work.path().to_string_lossy().into_owned()];
if let Ok(real) = work.path().canonicalize() {
workspace.push(real.to_string_lossy().into_owned());
}
let run = |args: &[String]| -> std::io::Result<std::process::Output> {
// Scrub the environment to an explicit whitelist. The upstream harness
// inherited the FULL host env, so any host BEADS_*/BD_* var
@ -142,8 +194,8 @@ pub fn run_scenario(bin: &Path, sc: &Scenario) -> std::io::Result<Trace> {
steps.push(StepResult {
args: args.clone(),
exit: out.status.code().unwrap_or(-1),
stdout: normalize(&String::from_utf8_lossy(&out.stdout)),
stderr: normalize(&String::from_utf8_lossy(&out.stderr)),
stdout: normalize_in_workspace(&String::from_utf8_lossy(&out.stdout), &workspace),
stderr: normalize_in_workspace(&String::from_utf8_lossy(&out.stderr), &workspace),
});
}
Ok(Trace {
@ -325,6 +377,123 @@ mod tests {
assert!(!n.contains("2026"));
}
#[test]
fn normalize_collapses_non_zero_revision_strings() {
// Two runs of the same `show --json`: only the minted token differs.
let run1 =
"[\n {\n \"id\": \"g-root\",\n \"revision\": \"-1228122709301203494\"\n }\n]\n";
let run2 =
"[\n {\n \"id\": \"g-root\",\n \"revision\": \"5267907219149755461\"\n }\n]\n";
assert_eq!(normalize(run1), normalize(run2));
assert!(normalize(run1).contains(r#""revision": "<REV>""#));
assert!(stdout_equiv(&normalize(run1), &normalize(run2), true));
assert_eq!(normalize(r#"{"revision":"42"}"#), r#"{"revision":"<REV>"}"#);
// Everything that is NOT a minted token keeps its literal value:
// "0" (a never-stamped row) must still diverge from a minted token.
assert_eq!(normalize(r#"{"revision":"0"}"#), r#"{"revision":"0"}"#);
assert!(!stdout_equiv(
&normalize(r#"{"revision":"0"}"#),
&normalize(r#"{"revision":"7"}"#),
false
));
// a number-typed member (a wire-type change) is not collapsed,
assert_eq!(normalize(r#"{"revision":42}"#), r#"{"revision":42}"#);
// nor are other members whose names merely contain "revision",
assert_eq!(
normalize(r#"{"local_revision":"3"}"#),
r#"{"local_revision":"3"}"#
);
// nor the same text quoted inside a JSON string value.
let quoted = r#"{"description":"see \"revision\": \"12\""}"#;
assert_eq!(normalize(quoted), quoted);
// Dropping the member is still a divergence.
assert!(!stdout_equiv(
&normalize(r#"{"id":"a","revision":"9"}"#),
&normalize(r#"{"id":"a"}"#),
false
));
}
#[test]
fn normalize_collapses_revision_by_member_name_at_any_depth() {
// The scope documented on revision_regex: the match is by member name over
// the raw text, not by position, so a user metadata key named "revision"
// is collapsed too and a change to its value is NOT a divergence.
let meta3 = r#"[{"id":"g-1","revision":"11","metadata":{"revision":"3"}}]"#;
let meta4 = r#"[{"id":"g-1","revision":"22","metadata":{"revision":"4"}}]"#;
assert_eq!(
normalize(meta3),
r#"[{"id":"g-1","revision":"<REV>","metadata":{"revision":"<REV>"}}]"#
);
assert!(stdout_equiv(&normalize(meta3), &normalize(meta4), false));
// Plain (non-JSON) text is matched as well.
assert_eq!(
normalize(r#"config "revision": "12" pinned"#),
r#"config "revision": "<REV>" pinned"#
);
// A metadata key with any other name keeps its value, so it still gates.
assert!(!stdout_equiv(
&normalize(r#"{"metadata":{"rev":"3"}}"#),
&normalize(r#"{"metadata":{"rev":"4"}}"#),
false
));
}
#[test]
fn normalize_in_workspace_collapses_only_this_runs_workspace() {
// Listed short-first: the longer spelling must still be replaced whole.
let ws = vec![
"/var/t/.tmpAbC123".to_string(),
"/private/var/t/.tmpAbC123".to_string(),
];
let err = "Found existing Dolt database: /private/var/t/.tmpAbC123/.beads/embeddeddolt/c\n";
assert_eq!(
normalize_in_workspace(err, &ws),
"Found existing Dolt database: <WORKSPACE>/.beads/embeddeddolt/c\n"
);
// Two runs that differ only in their random workspace normalize equal.
let other = vec!["/var/t/.tmpXyZ789".to_string()];
assert_eq!(
normalize_in_workspace("db at /var/t/.tmpAbC123/.beads", &ws),
normalize_in_workspace("db at /var/t/.tmpXyZ789/.beads", &other)
);
// A path that is not this run's workspace stays literal.
let foreign = "db at /var/t/.tmpXyZ789/.beads";
assert_eq!(normalize_in_workspace(foreign, &ws), foreign);
// The volatile-value normalizers still apply.
assert_eq!(
normalize_in_workspace(r#"{"revision":"5"}"#, &ws),
r#"{"revision":"<REV>"}"#
);
}
#[cfg(unix)]
#[test]
fn run_scenario_collapses_the_workspace_in_both_step_streams() {
// A stub bd that echoes its cwd on stdout and its symlink-resolved form
// on stderr, like the "Found existing Dolt database" error: both must
// reach the trace as <WORKSPACE>, so the run's random tempdir never does.
use std::os::unix::fs::PermissionsExt;
let dir = tempfile::tempdir().unwrap();
let bin = dir.path().join("bd");
std::fs::write(
&bin,
"#!/bin/sh\npwd\necho \"Found existing Dolt database: $(pwd -P)/.beads/x\" >&2\n",
)
.unwrap();
std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).unwrap();
let sc = Scenario::new("wiring", "w", &[&["show", "--json"]]);
let trace = run_scenario(&bin, &sc).unwrap();
assert_eq!(trace.steps.len(), 1);
assert_eq!(trace.steps[0].stdout, "<WORKSPACE>\n");
assert_eq!(
trace.steps[0].stderr,
"Found existing Dolt database: <WORKSPACE>/.beads/x\n"
);
}
#[test]
fn stdout_equiv_ignores_key_and_array_order_but_catches_set_diffs() {
// key order ignored

View file

@ -5,9 +5,9 @@
# REFERENCE bd — built from the merge base of HEAD and origin/main (the "before")
# CANDIDATE bd — built from the current working tree (the "after")
# and diffs each step (exit code, stderr, JSON-aware stdout) with volatile
# normalization (<TS>/<UUID>/<ACTOR>/<EMAIL>). ZERO in-scope divergences is the
# gate: any in-scope FAIL means the change altered a user-visible bd behavior on
# the covered contract surface.
# normalization (<TS>/<UUID>/<ACTOR>/<EMAIL>/<REV>/<WORKSPACE>). ZERO in-scope
# divergences is the gate: any in-scope FAIL means the change altered a
# user-visible bd behavior on the covered contract surface.
#
# The differential is driven by the Rust conformance harness under harness/
# (see harness/PROVENANCE.md); the harness is pointed at two Go bd binaries and