beads/tests
Test User e7484b32f7 feat(oracle): add the opt-in enumerated deep tier (~500 scenarios)
Adds a second, opt-in tier to Oracle A: the enumerated catalog
(`tests/oracle-a/harness/scenarios/enumerated.json` — 523 entries, 501
deterministic) that sweeps the wider bd CLI surface as data. The curated set
stays the fast default (~2-7 min); `ORACLE_CATALOG=1` pulls in the catalog for a
deep run (~10-15 min).

- `catalog()` now loads `scenarios/enumerated.json` (harness-relative), and
  `scoreboard` gates it on `ORACLE_CATALOG` — symmetric with `capture_golden` —
  so a default run scores exactly the curated set with no no-golden noise.
- `run-oracle-a.sh` exports `ORACLE_CATALOG` through to both the capture and score
  steps (only when non-empty; the tools treat any set value as "on").
- Floor assertion reworked. The old per-scenario "every create must exit 0 with an
  id" check false-positived on the catalog's intentional error-path scenarios
  (invalid priority/type, missing title, unknown flag, ...), whose creates SHOULD
  exit non-zero, and aborted before scoring. It now aggregates: the reference must
  produce >= 1 successful create (proving bd can init + create a workspace);
  per-scenario error behavior is validated by the ref-vs-candidate diff, not the
  floor.

Verified (bd-vs-bd on origin/main d23ba38f8): deep-tier IN-SCOPE 323/323 PASS, 0
divergences; fast tier remains 44/44. The 45 out-of-scope failures are
informational only (build-stamp / human-output deltas that always vary between two
builds — the in-scope predicate excludes them, e.g. the non-deterministic
`version --json` scenarios).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 23:44:45 +00:00
..
integration Document supported go install modes (bd-o5a) (#3480) 2026-04-25 11:30:19 -07:00
oracle-a feat(oracle): add the opt-in enumerated deep tier (~500 scenarios) 2026-07-10 23:44:45 +00:00
regression : refactor to use RunE, emit usage metrics, skip metrics in CI 2026-06-22 20:44:43 +00:00