From 9c6f6e15cdfdc4478e634520e8320795914e68e5 Mon Sep 17 00:00:00 2001 From: Jim Wordelman Date: Sun, 18 Oct 2026 17:28:16 +0000 Subject: [PATCH 01/41] =?UTF-8?q?bd=20versions=20+=20a=20config=20switch?= =?UTF-8?q?=20for=20versioned=20history=20=E2=80=94=20the=20minimum=20cut?= =?UTF-8?q?=20that=20makes=20versioned=20beads=20usable=20(#6138)=20(#6661?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(conformance): author R7.1 as-of-read conformance contract Adds a new Phase 0 conformance contract for R7.1 (as-of reads, gastownhall/beads#6136, design doc #5898 rev 9) -- no fixture existed for this anywhere in backend/conformance/, so this is authored from scratch, not un-skipped. AsOfReadFixture supplies three hooks: Resolve (the read under test), MintAt (seeds a version at a controlled accept-time), and MakeUnanswerable (arranges a Gone/Unknown refusal independently of RetentionFixture/EpochFixture's own removal machinery, since R7.1's refusal is a sibling contract to R17/R20, not a dependent of them). Five Run* functions cover the bead's five required-coverage points verbatim: - selector accepts either an instant or a version address, and both name the same answer when they resolve to the same version; - resolution returns the latest version at-or-before T, with a version accepted after T strictly excluded even when close by; - a served result reports the exact address it served; - an unanswerable read refuses as a typed Gone*/Unknown outcome with a reason, and never substitutes a neighboring version, the nearest surviving version, or current state -- asserted directly by requiring Address == "" on every refusal; - local completeness only (TC3): a lineage minted into one of two stores resolves there and refuses Unknown (FR-06) on the store that never received it, asserting nothing about agreement between the two. The file's doc comment states the Memory-plane-consumer-policy non-goal explicitly (closed-before-T boundary, sibling exclusion, supersession exclusion are consumer policy per #5877, not part of this primitive) so the gap reads as deliberate scope, not an oversight. Wired onto all three storage legs (dolt, embeddeddolt, uow) with unconditional "not implemented" stubs rather than partial-real hooks, so every case fails via MintAt on its first fixture call -- deterministic RED with no risk of an accidental pass, per the precedent set by be-x5jqd.1. Sentinel error messages reference be-x5jqd.5, the implementation child this suite is written ahead of. TestEveryLegWiresEveryRoleContract requires zero waivers: all 5 new entrypoints are fully wired (not nil) on all three legs. Branch cut from the rebased be-2h167 head (5fdfb92fe5). Verified via go build ./..., go vet ./..., gofmt -l, and go test -tags cgo across all three legs: all 5 new subtests (8 counting the 4-way Gone/Unknown table) FAIL for the documented reason, TestEveryLegWiresEveryRoleContract passes cleanly on all three legs, and TestVersionedHistoryPhase0RunsInFullSkipMode is unaffected. Bead: be-x5jqd.2 * fix(history): refuse metadata integers binary64 cannot hold exactly (refs be-wdlod, #6358 follow-up) canonicalDurableState canonicalizes numbers per RFC 8785 (JCS), which renders every number as an IEEE-754 binary64 double regardless of magnitude. An integer-valued metadata literal past 2^53-1 therefore rounds to a DIFFERENT VALUE on the way into durable_state -- not a formatting change, a value change -- which can collapse two distinct issue states onto one durable_state and therefore one content token. refuseUnrepresentableIntegers is the fix: an admission gate, one layer above the JCS step, that parses each JSON number literal exactly with math/big (never float64 -- float64 rounding is the exact thing being guarded against) and refuses whenever the literal's VALUE, not its spelling, is an integer whose magnitude exceeds 2^53-1. Value-not-form matters because the first cut of this gate skipped any literal containing '.', 'e', or 'E' outright, reasoning that non-integer forms round-trip losslessly -- true for genuinely fractional literals, but not for an integer VALUE spelled in decimal or exponential form: QA found that 9007199254740993.0 and 9.007199254740993e15 walked straight past that first gate and canonicalized to the same colliding target as the bare literal. The predicate now inspects the value via big.Rat.SetString/.IsInt, so all three spellings are refused alike. 1e300 is refused too, no notation carve-out: it denotes the integer 10^300, and binary64 spacing near that magnitude is roughly 2^248 -- vastly wider than the collision window at the 2^53 boundary this gate exists to close. Admitting that spelling while refusing 9007199254740993 would be the same defect with a bigger window (mayor's ruling, citing this collision-window argument as decisive). Non-integer values stay admissible past the bound (1.5e300 is fine): ES6 shortest-round-trip formatting is a bijection on doubles, so two non-integer literals collide only if they already denoted the same double before this gate ever ran, and a fractional literal never carried a promise of exact decimal storage the way an integer -- an id, a count, a timestamp -- carries a promise of exactness that silent rounding would corrupt. jcsCanonicalize is canonicalDurableState's RFC 8785 step pulled out on its own, below the gate rather than behind it, because refusing 1e300 puts the admission policy at odds with one of this store's own spec- conformance fixtures: testdata/jcs/input/values.json, one of the 10 official RFC 8785 vectors shipped verbatim from the spec's reference suite (not authored here), itself contains 1E30. RFC 8785 conformance is a claim about the canonicalizer, not about this store's admission policy, so TestCanonicalDurableStateIsJCS now runs all 10 shipped vectors -- values.json included, still asserting 1E30 folds to 1e+30 per spec -- through the ungated jcsCanonicalize, while its two local ad hoc cases keep exercising the full gated canonicalDurableState, since neither conflicts and both are more faithful to production traffic that way. The gate's own refusal of that magnitude is pinned separately, on its own terms, in version_history_integer_admission_test.go. Severity is MINOR, not MAJOR: Dolt's JSON column type already renormalizes metadata to float64 on every read regardless of this gate (measured on dolt 2.2.3: 9007199254740993 reads back as 9007199254740992, 1e300 as 1e+300), which converges two distinct in-memory states to one stored state before any version row is minted. The collision this gate closes is therefore unreachable through normal bd update traffic on this backend; it stays real for off-path writes -- bd sql, a hand-written JSONL import, or a TEXT-column backend -- which is why this landed as a follow-up rather than gating Phase 2 itself. Verified clean: a full module compile; static analysis over the storage tree; the complete issueops package suite (260 subtests); the schema package suite; and the broader storage tree, apart from one dolt-package test that fails identically against the pre-this-diff baseline (a Dolt server port collision with a concurrent session's process on this host) -- confirmed pre-existing and environmental, not caused by this change. * feat(issueops): implement R7.1 as-of read's store-layer primitive (be-x5jqd.5) Turns backend/conformance/versioned_read_contract.go (authored RED in 6aefdce60, gastownhall/beads#5898 rev 9, #6136) towards GREEN. This commit lands the shared production seam; wiring the three per-leg fixtures (dolt, embeddeddolt, uow) onto it is a follow-up commit. asof_read.go (new): AsOfReadInTx resolves an AsOfSelector (an instant or a previously-served version address) to one issue_versions row -- the latest version accepted at or before T, everything later strictly excluded -- and reports either its state and address, or a typed refusal (gone-retention/gone-erasure/gone-reorganization/unknown) with a reason. storeID is accepted only for signature parity with the fixture hook shape (Design D: the conformance suite's own ad hoc dispatch string selects among per-leg isolated test stores; issue_versions carries no store_id column, matching CurrentEpochInTx's precedent). AsOfRestriction is string-backed, unlike R20's int-backed EpochRestriction, because it round-trips directly through the new removed_restriction column with a plain cast. Migration 0069 adds issue_versions.removed_restriction VARCHAR(30), nullable (NULL = live, never a stored value). Sits alongside removed_at/ removed_reason (migration 0067, virgin until this slice); AsOfReadInTx only READS all three -- a future R17 is the real writer via hold/remove/ erase primitives, no collision. Guarded PREPARE + CLI direct-DDL override + schema_test.go hygiene entries, all modeled on 0068's pattern; TestLatestVersionIncludesMigration0068 is superseded per the one-pin-at-a-time convention. version_history.go: splits RecordVersionInTx into a versionedHistoryEnabled- gated wrapper and a new exported RecordVersionAtInTx, both delegating to a shared recordVersionAtInTx(..., at time.Time) body. RecordVersionAtInTx is test-support only -- ungated, caller-chosen timestamp -- so an as-of- read fixture's MintAt hook can mint version rows dated in the past without flipping a store's history-enabled flag. No behavior change for existing RecordVersionInTx callers. The new/moved code shifted which function each completeness guard sees as the enclosing write site, so all three needed updating, not bypassing: - row_lock_guard_test.go's funcNameExemptions key moves from RecordVersionInTx to recordVersionAtInTx (the write's new actual enclosing function), reasoning extended to cover both callers. - journal_completeness_test.go's beadDMLExemptions and version_completeness_test.go's versionExemptions each gain a RecordVersionAtInTx entry mirroring RecordVersionInTx's own (test- support twin, no production caller, no mutation of its own to journal/mint beyond the seam itself). Also fixed a latent test-helper gotcha found while writing the 0069 Dolt-CLI test: queryDoltCSV trims the whole `dolt sql -r csv` output before parsing, so a row whose ONLY selected column is NULL renders as a bare blank line that strings.TrimSpace silently swallows when it's the trailing line, collapsing "one row, NULL value" into "zero rows". Fixed by selecting a non-NULL anchor column (issue_id) alongside the column under test. Verification performed: full module compile and static analysis, gofmt clean; the complete internal/storage/schema suite with the real-Dolt-CLI lane enabled; and the complete internal/storage/issueops suite, including every completeness/row-lock structural guard -- all clean. * feat(issueops): wire dolt, embeddeddolt, and uow onto R7.1 as-of read (be-x5jqd.5) Turns backend/conformance/versioned_read_contract.go fully GREEN (authored RED in 6aefdce60, gastownhall/beads#5898 rev 9, #6136): this follows up on 4da98a9a8's shared production seam by wiring all three storage legs' AsOfReadFixture (Resolve/MintAt/MakeUnanswerable) onto issueops.AsOfReadInTx, per that commit's own stated plan. Each leg's asOfReadHarness is Design D's dispatch table: the conformance suite invents storeID strings to select among N isolated stores, and isolating them is entirely the harness's job -- AsOfReadInTx itself stays storeID-oblivious. The three legs differ only in how they get that isolation cheaply: - dolt: one shared testcontainer, N randomly-named databases - embeddeddolt: N fully independent on-disk databases - uow: one shared dolt sql-server subprocess (NewDoltServerUOWProvider's adoption semantics collapse repeat calls against the same root dir onto the same process), N randomly-named databases All three harnesses are Fatalf/Skip-free below their own TestAsOfReadContract body: AsOfReadFixture's hooks carry no *testing.T of their own and run from inside t.Run subtest closures (sometimes nested two deep), where FailNow/SkipNow would panic with "subtest may have called FailNow on a parent test". Failures instead flow back as a plain error into conformance.RunAsOfReadXXX, which Fatalfs correctly on the subtest's own goroutine. Two small companion fixes surfaced while wiring the legs on: - issueops.asOfVersionAddress -> AsOfVersionAddress (exported): each leg's MintAt needs to build the exact token its own RecordVersionAtInTx write will resolve against, from a different package than issueops. - recordVersionAtInTx now truncates `at` to whole seconds before writing it to issue_versions.change_at (DATETIME, migration 0067). Measured on dolt 2.2.3: the column ROUNDS a sub-second value rather than truncating it, so a change_at of 13:32:04.77 lands at 13:32:05 -- strictly later than the instant it was meant to record. That breaks resolveAsOfRevisionInTx's "at or before T" comparison (asof_read.go) for any T between the true instant and the rounded- up one. This is a production correctness fix, not test-only: RecordVersionInTx hits the identical hazard on every real time.Now().UTC() call whose nanosecond component rounds up. Verification performed: - uow: TestAsOfReadContract green standalone (5 subtests, 4 nested restriction variants) and the full uow package green in isolation (673s, includes pre-existing TestReleaserContract and everything else). A `go test ./internal/storage/...` full-tree sweep flagged uow FAIL via a 600s hang inside TestReleaserContract's migration lock -- traced to cross-package parallel resource contention from running the whole storage tree's dolt-sql-server-spinning suites concurrently, not a regression: the same test is clean when the package runs alone. - embeddeddolt: TestAsOfReadContract green (22.89s, all subtests and all 4 restriction variants). - dolt: skipped on this host -- no Docker for the leg's testcontainer, the same pre-existing skipIfNoDolt gate the whole dolt package already runs under, unrelated to this change. - TestEveryLegWiresEveryRoleContract and TestEveryContractImporterInTheStorageTreeIsRegistered both pass for all three legs. - gofmt clean; whole-module build and vet clean. Bead: be-x5jqd.5 * feat(versions): bd versions + a config switch for versioned history (refs be-hs42e.6.2) Versioned history has been buildable but not usable: migrations 0067+ create issue_versions, and the Phase 2 writer fills it, but SetVersionedHistoryEnabled was a Go method nothing in cmd/bd ever called. There was no supported way for a person to turn the feature on, and no way at all to read it back without raw Dolt SQL. This adds the switch and a reader. - versioned-history.enabled (env BD_VERSIONED_HISTORY_ENABLED), default off, applied at the top of wireStorageDecorators. It MUST run on the raw store: the capability is a type assertion and no decorator forwards it, so applying it after a wrap would silently leave history off while the config said it was on. TestVersionedHistoryConfigAppliesToTheRawStore pins the ordering, and fails rather than skips if a decorator ever starts forwarding. - storage.VersionLister + issueops.ListVersionsInTx over issue_versions, on the dolt and embeddeddolt legs. A narrow capability rather than a Storage method, following ExternalRefHistoryQuerier: a backend that cannot know must be able to say so instead of answering "none". - bd versions , with three distinguishable outcomes rather than two. Off, unsupported, and empty are different answers with different fixes; an empty list is a claim about the bead, and returning it for the other two would be a wrong answer rather than an empty one. The four removal restrictions render as four sentences, and unknown (no lineage knowledge here) is not gone. - REV is printed as a local ordinal with a footer saying so. It is deliberately NOT presented as a citable address: AsOfVersionAddress mints over the ordinal, two clones can both hold revision 8 of the same bead, and the durable version_id primary key has not landed. An address printed now would break every citation once it does. - bd version now points at bd versions instead of printing bd's own build and looking like the bead has no history. Bare bd version is unchanged. No migration: 0067/0068/0069 already carry every column this reads. Verified on a scratch store: create + 2 updates + close with the flag on lists 4 attributed versions; with the flag off, 0 rows in issue_versions and the command refuses naming the key. Co-Authored-By: Claude Opus 5 (1M context) * fix(versions): emit local_revision, not revision, in --json (bee's #5898 consumer read) types.Issue already ships `json:"revision"` (types.go:1206) and it is the row-lock CAS token — an unrelated value, and a string where this one is an int64. Emitting "revision" from the versions payload would put two meanings and two types on one key in the place machines read: the `bd version` collision again, with worse consequences, since a script that learned one shape gets the other and cannot tell. local_revision also carries the non-citability with the value rather than only in a footer no script parses, and needs no rename when version_id lands beside it. Raised by @bee-ghosttrack's consumer read on #5898, who reported a couple of hundred script files parsing `bd … --json` — this is the part of the surface they would actually touch. Pinned by TestVersionsJSONDoesNotCollideWithIssueRevision. Co-Authored-By: Claude Opus 5 (1M context) * fix(versions): make the advertised switch actually switch it on (#6661 review) @bee-ghosttrack's blocking finding: the help text and the off-refusal both told the user to run `bd config set versioned-history.enabled true`, and that command did not turn the feature on. The key is not yaml-only, so `bd config set` routed it to the workspace DATABASE; both readers called config.GetBool, which is viper (yaml + env) only. The user ran exactly what they were told, was told it was set, and recording stayed off — configured on, actually off, one layer up from the decorator-ordering hazard this command already guards. Takes bee's option (b): read the switch from the store's own config, where `bd config set` already writes it. That also makes recording a property of the STORE rather than of whoever is calling, which is what closes finding 2 — two clients of one sql-server can no longer disagree about whether a write is recorded. The two planes are OR'd rather than ranked. The hazard is a write that fails to record, never one that records when it needn't, so either saying yes is enough and neither can silently switch the other off. BD_VERSIONED_HISTORY_ENABLED still works for a one-off run. A store with no settings plane (proxied, no-db) falls back to viper instead of failing the command. Also from that review: - finding 3: the read no longer refuses just because recording is off. A store that recorded for a month and was switched off HAS versions; refusing made a claim about the store from a fact about this invocation's config, which VersionLister's own doc says are independent. Rows are listed either way, with a note when recording is currently off; refuse-with-remedy only when off AND empty. - finding 4: an unresolved id with no versions is "no such bead", not "none yet". Falling through on an unresolved id stays right — a deleted bead can still have versions — but only until the answer is empty. - finding 5: the ordering test's escape hatch is t.Fatal, not t.Skip; a skip returned before the assertion that actually guards the ordering. The storage_chain.go doc cited a test name that never existed. version_lister.go promised a list of compile-time assertions that was not there; the assertions now exist in each leg's own package, where they can compile. - `versioned-history.` is a recognized config prefix, so the set no longer prints "not a recognized config key" immediately before confirming the set. Test: TestVersionedHistorySwitchRoundTrip_Embedded drives the real `bd config set` through to a real `bd versions` against a real store, with BD_VERSIONED_HISTORY_ENABLED cleared so a pass proves the store setting carried it alone. This is the round trip bee asked for, and it is the only shape that could have caught the original defect — a fake-backed unit test cannot see a disagreement between two real planes. Co-Authored-By: Claude Opus 5 (1M context) * docs(versions): name what the OR'd switch planes cost (#6661 review) @bee-ghosttrack, round 2: with the two planes OR'd rather than ranked, the consequence worth naming is that once the store setting says on, BD_VERSIONED_HISTORY_ENABLED=0 cannot countermand it for a single run. The help now says so, and says where to turn it off instead. Co-Authored-By: Claude Opus 5 (1M context) * fix(versions): gate the wiring store read on the capability, not a recover (#6661 review) @bee-ghosttrack's round-3 blocking finding. wireStorageDecorators reads the versioned-history switch off the raw store on every bd invocation, and did so before anything established that the store could use the answer. cmd/bd's own fakeProbingDoltStore embeds a nil storage.DoltStorage, so the promoted WorkspaceConfig SIGSEGV'd at wiring time -- and a panic takes the whole test binary down, which is why one unguarded read showed up as two red CI steps (Test (macos-latest), PR Core (wrapper timing)) rather than one failed test. Takes the fix bee prescribed: assert storage.VersionedHistoryConfigurer first and read the store plane only when the store could act on the answer. The previous round fixed this with a recover plus a WorkspaceConfig on the fake, and left a comment arguing that no capability assertion could work. That argument was wrong, and wrong in a specific way worth recording: it reasoned about asserting WorkspaceConfig, which IS declared on storage.DoltStorage and so matches every store including the nil-embedding fake. bee proposed asserting VersionedHistoryConfigurer, which is a different interface -- SetVersionedHistoryEnabled is NOT part of DoltStorage, so the fake does not promote it and does not satisfy it. The comment rebutted a fix that had not been proposed. It now states what is actually true. So the fake goes back to byte-identical with the pushed head: the wiring fix fixes that test without touching it, as bee said it would, and the fake keeps the "only the methods below may be called" constraint its own doc asserts. The recover stays, as a backstop rather than the guard. It covers what the assertion does not: versionedHistoryEnabled is also called from `bd versions`, where the read is the point and no capability gate precedes it, and a store that DOES implement the capability can still have a half-built settings plane. The viper plane is still read first and unconditionally, so a set-but-unsupported store still reaches applyVersionedHistoryConfig and still gets its warning. Only the store-plane read is gated. The one case that loses a warning is a store whose own settings plane says true but which lacks the capability -- and `bd config set` could not have written that plane on such a store to begin with. Tests, split so each layer fails for its own reason: - TestWireStorageDecoratorsSkipsTheStoreReadForANonConfigurer asserts calls == 0, not merely that it survived. "It survived" is what the recover delivers too, which is how this regressed the first time. - TestWireStorageDecoratorsSurvivesAStoreThatCannotAnswer keeps the recover's case, now on a store that IS a configurer so the guard lets it through. - TestVersionedHistoryEnabledForWiringReadsTheStoreWhenItCanRecord pins that the guard did not stop reading the plane finding 1 existed to start reading. Mutation-verified both ways: removing the guard fails the first test on calls=1/want=0 while everything still "passes" by surviving; removing the recover leaves the first test green and fails only the second. Co-Authored-By: Claude Opus 5 (1M context) * test(versions): name the embedded round-trip so CI selects it (#6661 review) @bee-ghosttrack's round-3 finding 2. The round-trip test that proves the two config planes agree ran only on the author's machine. .github/scripts/embedded-test-shard.sh discovers what to run with `grep -rh '^func TestEmbedded' cmd/bd/*_embedded_test.go` and validates manifest entries against ^TestEmbedded[A-Za-z0-9_]+$. TestVersionedHistorySwitchRoundTrip_Embedded matched neither, so the only lane that sets BEADS_TEST_EMBEDDED_DOLT=1 for cmd/bd never selected it; the generic Test jobs run without that env, where the test's own guard skipped it. Skipped and passed look alike from outside, which is what made it invisible. Renaming is the whole fix -- a test absent from the shard manifest still gets a shard from the fallback distribution. Verified by running the script's own selector over the tree and confirming the new name is discovered. The rationale is now a comment on the test, because the old name reads more naturally and nothing else in the file explains why the prefix is load-bearing. Co-Authored-By: Claude Opus 5 (1M context) * test(versions): drop the dead fixture, name the empty arm (#6661 review) @bee-ghosttrack's round-3 finding 3. TestVersionsRefusesWhenFlagOff built a backend holding one recorded version, handed it to nothing, and ended in `_ = backend`. The fixture was load-bearing for the assertion the test was named after -- that the read was refused BEFORE the backend was consulted. Finding 3 removed that behaviour on purpose, so the assertion went with it; the fixture stayed behind, unused, under a name still promising the old contract. Renamed to TestVersionsOffAndEmptyRefusesWithRemedy for what it actually pins, moved beside TestVersionsOffButRecordedStillLists so the two read as the pair they are -- recording off decides the FOOTER, never whether to read; off with rows lists them, off and empty is the one case that refuses -- and the dead fixture is gone. Co-Authored-By: Claude Opus 5 (1M context) * fix(proxy): give bd versions a capability row, a typed refusal (#6661) main's proxied-server capability registry (#6582) requires every command path to carry a row, and TestProxyCapabilityRegistryCoversCommandTree fails for `bd versions`, which this PR adds. It is refused, reason unimplemented, rather than permitted. It is a pure read (issueops.ListVersionsInTx in one read transaction), so nothing about it is unsafe on a shared backend and "design" would be wrong. But it reaches the backend only through the optional storage.VersionLister, which the dolt and embedded stores implement and the uow provider does not. Permitting it would open the proxied provider in the root pre-run (starting the proxy and dolt) and then fail untyped at errVersionsUnsupported. The row refuses with proxy.versions.unsupported before any side effect, and its tracking names the missing piece: a VersionLister over the proxied uow provider (tracker #6132). Co-Authored-By: Claude Opus 5.5 (1M context) * test(uow): register verified shutdown cleanup in TestAsOfReadContract (#6661) main's TestEveryServerFixtureRegistersVerifiedCleanup (#6441) requires every server fixture to use verifiedShutdownCleanup(t, storeRootDir) instead of calling proxy.Shutdown directly, so a shutdown that refuses cannot leave a live dolt sql-server behind a green run. TestAsOfReadContract still had its own proxy.Shutdown cleanup that only logged the error. The cleanup is registered at the same point, so the LIFO ordering the harness relies on (every provider closed before the shared server stops) is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) * style(schema): gofmt cli_migrations.go after the 0070 renumber The rebase onto upstream's 0069_widen_issue_versions_datetime_precision (#6675) joined its const and the renumbered 0070 comment without the separating blank line, which make fmt-check rejects. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(history): stop flooring change_at now that 0069 widened it (refs be-hs42e.10) recordVersionAtInTx carried `at = at.Truncate(time.Second)`. That was correct while 0067's plain DATETIME was in force: Dolt's datetime(0) does not truncate sub-second input, it ROUNDS half-up, so a change_at could be stored strictly LATER than the instant it recorded and break AsOfReadInTx's "latest version accepted at or before T". Flooring first left nothing to round. Migration 0069 widened change_at and removed_at to DATETIME(6), which removes the rounding entirely and makes the floor actively harmful — it threw away the microseconds the widening exists to keep. Two versions minted in the same second collapsed onto one stored value and stopped being distinguishable by change_at, which defeats --at, the only PORTABLE selector the read surface offers while revision stays a per-store ordinal and no durable version address has shipped. This was measured, not inferred. On a store carrying both the widening and the floor, four versions minted inside one second all read back 15:40:38.000000: the column could hold microseconds and the writer refused to supply them. Two correct fixes cancelling each other, and only the second one was visible. removed_at needs no matching change: MarkLatestVersionRemovedInTx already passes time.Now().UTC() straight through and was never floored. Audited rather than assumed. Pinned by TestRecordVersionKeepsSubSecondChangeAt (embeddeddolt), which mints two versions 500ms apart inside one second and asserts both that the fractional part survives and that the two differ. Verified it FAILS with the floor restored — reporting exactly the 11:04:07.000000 collapse — and passes without it, so it is a real guard rather than a test that passes either way. The column's own half of the contract stays pinned by TestMigration0069ChangeAtSurvivesSubSecondPrecisionThroughDoltCLI. Also drops this branch's own tx-minting guard commit, which upstream landed independently as #6833. Co-Authored-By: Claude Opus 5 (1M context) * test(feat): red — Activate versioned history in every store and provider factory, each store reading its own #6661 switch (stacked on #6661) (refs be-qxq70) Tests first, all failing on the right reasons at fb6e219d6a (#6661's head): - D1: on real embedded stores a routed update, a routed close and `bd create --repo` into a rig whose OWN row is on each gain 0 issue_versions rows, want 1; the control (a write inside the rig) gains 1. - D3: BD_VERSIONED_HISTORY_ENABLED=1 records into no routed store, and =0 is shown not to switch off a store whose row is on. - D4: TestEveryStoreConstructionActivatesVersionedHistory flags all 9 store and provider factories, and its mutation probe reports "probed 0 of 9". - wiring: wireStorageDecorators still reads the store's row (1 read) and switches it on; activation belongs to the factories. - internal/versionedhistory (does not exist yet) and the doctor repair opener: the resolver, the provider read (K1), the error-vs-absent diagnostics and the shared-activation requirement. * feat: green — Activate versioned history in every store and provider factory, each store reading its own #6661 switch (stacked on #6661) (refs be-qxq70) Stacked on #6661 (fb6e219d6a). #6661's plane and approved commits are unchanged: the switch stays a row of the store's own config table, OR'd with viper, and `bd config set versioned-history.enabled true` still writes that row and not config.yaml. The defect: applyVersionedHistoryConfig ran only from wireStorageDecorators, so the one store a command opens for its own workspace was activated and every store built by newDoltStoreFromConfig (prefix-routed update/close, `bd create --repo`, the direct-mode open) was not. With the switch on they recorded no issue_versions row while the command reported success. - Activation moves to where a store or unit-of-work provider is constructed (the store_factory.go/_nocgo.go twins, uow_factory.go's two provider constructors, the personal-migration planning store, and bd doctor's repair opener), on the raw store, before anything can wrap it. wireStorageDecorators no longer activates, so each store's row is read once per open. - One resolver, in internal/versionedhistory, reachable from doctor/fix which cannot import package main. #6661's semantics are preserved: one key spelling, viper first, then the capability check, then the row read, with the recover backstop. The behavior-preserving refactor of #6661's versioned_history_switch.go / storage_chain.go is the only edit to files #6661 introduced; the resolver tests moved with the code, assertions unchanged. - K1: a provider has no DoltStorage, so its row is read through a unit of work (NewUOW + ConfigUseCase().GetConfig) at construction: a pooled connection, one SELECT, rolled back. The constructors have already run initSchema, so this starts no server and creates nothing. It does not fall back to viper alone. - A row read that errors resolves to off but is reported under BD_DEBUG; an absent row stays silent. - Docs (configuration.md), a CHANGELOG entry, and a hint at `bd config set ... true` time that the setting replicates and is single-writer only. The bazel-sync run also wanted unrelated BUILD entries for files that already exist at #6661's head (versions.go, versioned_read_contract.go, ...); they are not part of this change and are left out. * test(feat): green — a served mutation records from the provider's own row, on a real server (refs be-qxq70) K1 is the one place the resolver cannot ask a DoltStorage: `bd serve` against a server-mode workspace writes through a unit-of-work provider, so its flag is read through NewUOW + ConfigUseCase().GetConfig at construction. The fakes in internal/versionedhistory prove the read; this proves it WORKS against a real Dolt server at that moment (pool open, database selected, schema migrated). Two server-mode workspaces on the shared test container: the one whose own row says on records a served HTTP mutation (+1 issue_versions row), the one with no row records none. Measured both ways: passes with the provider activation in place, and fails ("added 0 issue_versions row(s), want 1") with it deleted from both provider constructors in uow_factory.go. Proxied lane only (BEADS_TEST_PROXIED_SERVER=1 BEADS_TEST_ENV_RUN_DOLT=1 TESTCONTAINERS_RYUK_DISABLED=true), like TestServeActivatesTheEventsJournal, so it skips in a plain make test. * test(feat): red — #6661 review round, small items: findings 2, 4, 5, 6 and 8 Tests for the small findings in the second review round on #6661. - 2: `bd versions` must decide "recording" from the store's own setting. BD_VERSIONED_HISTORY_ENABLED=1 with the store's row unset and nothing recorded must refuse, not answer "none yet"; the help text must not offer the variable as a per-run switch for a read. - 4: `bd versions --json` is one object, {"versions": [...], "recording": bool}, with [] rather than null when nothing is recorded. - 5: AsOfReadInTx reports any stored removed_restriction outside its four values, "live" included, as unknown. - 6: the 0070 header spells the restriction vocabulary with the underscore spellings the code stores and compares. - 8: `bd version ` exits 1 and points at `bd versions `. This one pins behaviour the branch already has, so it passes at red; it was shown to fail with the guard removed. Item 7 is a comment-only fix and has no test. * feat: green — #6661 review round, small items: findings 2, 4, 5, 6 and 7 - 2: `bd versions` reports the store's own setting. It read the environment and config.yaml as well, so BD_VERSIONED_HISTORY_ENABLED=1 made a store that never recorded answer "No versions recorded yet" instead of refusing with the remedy. It only reads, so those planes cannot make the store record. The "for one run" hint is dropped from the help and the refusal, since it could only ever produce the wrong answer. - 4: `bd versions --json` is always {"versions": [...], "recording": bool}, with [] rather than null when nothing is recorded. Previously a bare array, or null, and no recording flag. - 5: AsOfReadInTx reports a stored removed_restriction outside its four values, "live" included, as unknown instead of passing it through. - 6: the 0070 header and its test use the underscore spellings the code stores and compares. 0070 is unreleased, and this changes its content hash. - 7: comment only. DATETIME(6) still rounds to the microsecond; the comment said 0069 removed the rounding entirely. * test(fix): red — #6661 number admission: fractions past the bound, exponent cost, pre-enable scan Tests for the review round on the number-admission gate, written before the change: - a fraction whose nearest double is at or past 2^53 is refused (9007199254740993.5, 9007199254740994.5, 9007199254740991.5), and the gate is closed under its own canonicalization, checked as a property over a corpus - classification depends on magnitude, not on the size of an exponent: 1e100000000 and -1e400 are refusals, 1e-100000000 and 1e-400 are admitted, and allocation does not grow with the exponent - the sentinel's text names the I-JSON exact-integer range - CheckMetadataVersionable agrees with the mint, and FindUnversionableMetadata finds what the mint would refuse and skips what it never versions - bd config set versioned-history.enabled true refuses, writing nothing, over rows the mint would refuse, and false never scans Guards that pass before and after by design: the verdict on every literal as the store returns it, the mint's refusal writing nothing, and, on the server and embedded legs, the store-level property that no two literals reading back as different numbers share a canonical form. Those store tests also log what each Dolt holds for a number outside the range when history is off, and pin that a later write to such a row is refused once history is on. * fix: green — #6661 number admission: the nearest-double rule, ParseFloat classification, a pre-enable scan at the switch The admission gate now refuses a JSON number literal if and only if the binary64 nearest to it has magnitude above 2^53-1, or it overflows, whatever the literal's spelling: - a fraction that rounds to or past 2^53 (9007199254740993.5, 9007199254740994.5, 9007199254740991.5) is refused, so the gate is closed under its own canonicalization - literals are classified with strconv.ParseFloat, linear in their length, so 1e1000000 no longer builds a million-digit integer inside the writer's transaction; an overflow is the sentinel and a magnitude too small for binary64 is admitted; math/big is gone - the sentinel reads "number is outside the I-JSON exact-integer range (magnitude above 2^53-1)"; ErrIntegerNotRepresentable keeps its name, with a comment saying why - ordinary fractions such as 0.1 stay admitted; no verdict on a value the store hands back changes The header now says what the mint's refusal is: an integrity backstop inside the writer's transaction, not an admission policy for metadata. bd config set versioned-history.enabled true now checks the store first. It reads every issue the mint would version, runs the mint's own function over each one's metadata (issueops.CheckMetadataVersionable, through issueops.FindUnversionableMetadata), and refuses, writing nothing, while any is refused, printing the count, the ids and the fix. There is no override; setting it false never scans. The environment and config.yaml planes do not pass through the command and rely on the refusal at write time. Also: docs and a CHANGELOG entry, and comments in the existing admission tests that described the old rule. * test(fix): keep the exponent-cost allocation test stable under the race detector TestNumberAdmissionAllocationsDoNotGrowWithTheExponent compared two independently averaged allocation counts with a strict >. Under -race sync.Pool drops one Put in four at random, and the refusal's fmt.Errorf takes its printer from a pool, so a call now and then allocates one more than it would. The test failed on most runs of `go test -race -short`, the flag set of the required lane, and never without -race. That noise only ever adds allocations, so compare the fewest allocations over many single-call samples instead. The comparison stays strict, and it still fails on the exact big.Rat gate this test exists to keep out (49 allocations for 1e1000000 against 36 for 1e400). The doc comment no longer claims the count is deterministic under -race. * test(fix): red — set-many skips the pre-enable scan; pin what the scan reads bd config set-many versioned-history.enabled=true switches versioned history on with no pre-enable scan: only bd config set makes it. With a row holding a number outside the I-JSON exact-integer range the setting is turned on and the next write to that row fails. The new embedded test drives the real verb over such a row and expects the whole batch to be refused, writing nothing. The scan's own embedded test only used open issues, so deleting AllFlag or IncludeAllTypes, or replacing the unlimited Limit, left it green. It now adds a closed, a pinned and a gate offender, and a second test imports more rows than one default page and expects every one of them counted. * fix: green — bd config set-many runs the pre-enable scan for versioned-history.enabled bd config set refuses to turn versioned history on while an issue holds metadata a version could not record. bd config set-many writes the same setting through a validation loop of its own and made no such check, so it switched history on over a row whose next write then failed. It now runs the same check for that one key inside its validation loop, before any pair of the batch is written, so a refusal stores nothing. The check reads through the accessor it already used on both routes: the store the command has opened on the direct route, the provider on the proxied one. * chore: strip internal tracker ids from the comments and tests across #6661's range Comment and test-text changes only; no behaviour changes. Across every commit from the merge-base with upstream main to here, the comments and tests named internal tracker ids (one bead id in eleven comments and in one test failure message, an example id in the `bd version` guard's comment, and an internal decision's attribution in the huge-exponent admission test). They now cite gastownhall/beads#5898 and #6136 only, which is what a reader of the public branch can follow. The 0070 migration's header comment is one of the edits. That migration is new on this branch (it does not exist on upstream main), so the change is not an edit to a shipped migration, but it does change the file's content hash; the schema tests that compute hashes are self-consistent. Text that already exists on upstream main is left as it is, even where it mentions an id: this commit does not rewrite lines the range did not add. * chore(bazel): bazel-sync (gazelle) for the #6661 range * fix(bazel): give fix_test the store.go its activation guard parses TestVersionedHistoryDoctorRepairOpenerUsesTheSharedActivation parses store.go from its working directory. `go test` runs in the package directory, so the file is there. A Bazel test only sees its declared runfiles, so the test failed with parse store.go: open store.go: no such file or directory in //cmd/bd/doctor/fix:fix_test (--config=ci) and in shard 1 of //cmd/bd/doctor/fix:fix_dolt_test (--config=doltserver), which runs the same binary through data = [":fix_test"]. Declare store.go as data on fix_test. * test(fix): red — the pre-enable scan must check the whole issue, as the mint does bd config set versioned-history.enabled true runs a check over every issue before it turns history on. The check looked at each issue's metadata alone, while the mint canonicalizes the whole issue. A gate's timeout is a time.Duration that a version holds as nanoseconds, so a gate whose timeout is past 2^53-1 ns (about 104.25 days) holds a number outside the I-JSON exact-integer range: the mint refuses every write to it, the scan never looked, and the switch could be turned on over it. - The agreement test's domain widens from metadata documents to whole issues: the metadata corpus, plus timeouts at 2500h (admitted), 2600h, 8760h and -2600h (refused), exactly 2^53-1 ns (admitted) and 2^53 ns (refused). - The scan test covers a gate over 105 days and expects the verdict to name the field, "timeout". - The mint-level refusal test covers the same gate and expects the refusal to name the field as well as the literal, for a number in metadata too. - An end-to-end test on the direct route and one on the proxied route create gates with bd gate create --timeout, expect the enable to refuse and name them, and expect the issues the scan names to be exactly the issues whose next write the mint refuses. - Three guards keep the scan's lite read from hiding a number: every column it leaves out is a string or a list of strings, nothing the mint adds and the scan does not load can hold a number, and no exported check over part of an issue exists beside the one the mint runs. Each guard is also run against shapes it must refuse, so it cannot pass by finding nothing. The new symbols (CheckIssueVersionable, FindUnversionable and the Field of UnversionableIssue) do not exist yet, so the issueops tests do not build. * fix: green — the pre-enable scan runs the mint's own check over the whole issue The check behind bd config set versioned-history.enabled true looked at each issue's metadata alone. The mint canonicalizes the whole issue, and a gate's timeout, a time.Duration held as nanoseconds, is a number outside the I-JSON exact-integer range once it is past 2^53-1 ns (about 104.25 days). The mint refuses every write to such a gate, the scan cleared the switch over it, and the first write after that failed. CheckIssueVersionable is now exactly canonicalDurableState's refusal over the whole issue, and FindUnversionable runs it over every issue the mint would version. It replaces FindUnversionableMetadata, and CheckMetadataVersionable is gone, so no check over part of an issue is exported beside the one the mint runs. A refusal names the top-level field it found the value in ("metadata", "timeout") as well as the literal, on the mint and on the scan; the field is located only after a refusal, so a write that is admitted still reads the document once. A gate timeout above 2^53-1 ns is unrecordable by policy: it is refused at the write, named by field and documented, not re-encoded. The refusal at the switch gives the fix for each field. For a number in metadata that is bd update --metadata or --unset-metadata, which works with history on or off because a version is recorded from the state after the write. For a gate timeout, which bd update cannot change, it is removing the gate with bd delete, or setting the timeout_ns column with bd sql while history is off; bd sql needs a server-backed store, and the text says so. It also says the check runs once, now: a writer that does not record (an older bd, bd sql, a pull from a clone that had history off) can add such a row afterwards, and the first write to it then fails with the same refusal. Making the check and the write of the setting one transaction would narrow that window, not close it. Corrected in the comments, the docs and the changelog as well: an import is not a path that skips the mint, since it goes through the same batch create; and Dolt's JSON column collapses duplicate keys when it stores a document, so on a Dolt store neither the scan nor the mint can see one, and the duplicate-key refusal is a defence for a backend that keeps them. * test(fix): red — the listing must answer as the as-of read does; the SDK opens must be accounted for Three review findings on the versioned-history change and one wording fix, tests first. - A removed version's restriction. AsOfReadInTx answers a stored value outside the four-value vocabulary (a NULL or empty column, "live", a hyphenated or arbitrary string) as unknown, but ListVersionsInTx hands the column back as stored, for every row, and the label bd versions prints passes such a string through. A table test runs the listing and the as-of read on the same stored value, for a version that was removed and one that was not, and expects the same answer, and none for a version that was never removed. The label gets the same out-of-vocabulary values and is expected to render what unknown renders. - The SDK's store opens. The shared construction guard now scans the repository root as well, so the Go SDK's five opens (Open, OpenFromConfig and OpenGated, and OpenBestAvailable in each build-tag twin) are noticed. Nothing accounts for them yet, so both guards fail on exactly those five sites. - A bytes bound beside the allocation-count bound on the number admission gate. A classifier that builds the number it is asked about makes one allocation whatever the exponent, so the count bound passes it, and the bytes grow. The fewest bytes one call allocates for a literal with a huge exponent must be no more than for a small one plus a margin. It passes today, because the gate never builds the number; it fails for a gate that does. - The refusal at the switch names the gate remedy as bd delete --force, says that without --force bd only previews, and says that removing the gate unblocks anything it was blocking. The embedded and proxied end-to-end tests expect all three. * fix: green — the listing answers as the as-of read does; the SDK opens are accounted for Three review findings on the versioned-history change and one wording fix. A removed version's restriction. A stored removed_restriction is a free string, and only the four values of the vocabulary are answers. The as-of read already reported anything else (a NULL or empty column, "live", a hyphenated or arbitrary string) as unknown, but the listing handed the column back as stored, for every row, and the label bd versions prints passed it through. NormalizeRemovedRestriction is now the one place that decision is made. AsOfReadInTx calls it; ListVersionsInTx calls it for a version that was removed and gives a version that was not removed no restriction; and restrictionLabel renders through it. The listing, its JSON and the label now say what the as-of read says for the same row. The SDK's store opens. The Go SDK's five opens (Open, OpenFromConfig and OpenGated, and OpenBestAvailable in each build-tag twin) apply neither per-store switch, the events journal's or versioned history's, and they still do not. Applying them would change the library's behavior for every embedder, so that is a separate change; this one declares the boundary. The shared construction guard now accounts for the five with one exemption reason that is true of both switches, so a new SDK open is noticed rather than missed, and its staleness check keeps the exemptions honest. The docs gain one bullet saying that writers built on the SDK do not apply the switch. The bytes bound on the number admission gate needs no production change: the gate never builds the number, which is what the bound pins. The refusal at the switch now names the gate remedy as bd delete --force, says that without --force bd only previews, and says that removing the gate also unblocks anything it was blocking. The docs line says the same. For a gate that blocks an issue, bd delete without --force prints a preview and exits 1, and with --force the gate is removed and what it blocked is ready again. * test(bazel): declare the root package's Go source as data for //cmd/bd:bd_test The shared store-construction scan now reads the repository root, where the Go SDK's opens live, but //cmd/bd:bd_test declared none of the root package's source. Under bazel test the scan therefore saw an empty directory, reported the five SDK exemptions as no longer matching a store construction site, and failed the events-journal guard, while go test, which sees the whole checkout, passed. List the root in tools/bazel/go_srcs.py so the generator emits its managed go_srcs filegroup there, and add that filegroup to bd_test's data. The new block is the generator's own output and nothing else changes. * ci: list the switch scan's proxied gate-timeout test in the 30-shard manifest Upstream main (#7129) moved the proxied-server tier to 30 duration-balanced shards, and TestProxiedShardManifestGeneratorNotStale requires every TestProxiedServer*/TestServerMode* test exactly once in that block. This branch adds TestProxiedServerVersionedHistorySwitchRefusesAnUnrecordableGateTimeout (cmd/bd/versioned_history_switch_scan_proxied_integration_test.go), so the block needs its line. Written by the generator itself: python3 scripts/ci/gen_proxied_shard_manifest.py 30 --weights=duration --write (one line, shard 8); --check exits 0 afterwards. Rebase note (maintainer): regenerated on current main, whose proxied-server tier uses 34 shards; the generator's incremental output (gen_proxied_shard_manifest.py 34 --weights=duration --write) placed this test at shard 6 of the 34-shard block. * ci: list the PR's embedded versioned-history tests in the embedded shard manifests #7156 made the embedded-Dolt tier's shard manifests exact-once (50 cmd shards, 15 storage shards), so every embedded test must be listed. This PR adds five embedded cmd tests and three embedded storage tests; this commit is exactly what the generators write for them: python3 scripts/ci/gen_embedded_cmd_shard_manifest.py 50 --weights=duration --write python3 scripts/ci/gen_embedded_storage_shard_manifest.py 15 --weights=duration --write Incremental: 8 lines added, none moved. Rebase note (maintainer): regenerated on current main, whose embedded manifests use 100 cmd shards and 40 storage shards; the generator's incremental output placed these tests at cmd shards 55 (TestEmbeddedVersionedHistorySwitchScansEveryRowNotOnePage), 56 (...SwitchRoundTrip), 57 (...SwitchRefusesUnversionableMetadata), 58 (...SwitchRefusesAnUnrecordableGateTimeout) and 59 (...SetManyRefusesUnversionableMetadata) of the 100-shard block, and at storage shards 5 (TestRecordVersionKeepsSubSecondChangeAt), 6 (TestVersionedHistoryKeepsNumbersTheStoreDistinguishesApart) and 23 (TestAsOfReadContract) of the 40-shard block. * test: exempt OpenBestAvailableWith, where #7188 moved the SDK's store construction Upstream #7188 split beads.OpenBestAvailable: it now delegates to the new OpenBestAvailableWith, which builds the store. This PR's two construction guards (TestEveryStoreConstructionActivatesTheEventsJournal and TestEveryStoreConstructionActivatesVersionedHistory) share one exemption list keyed by function, so after merging main they flagged OpenBestAvailableWith as an unexempted construction site and reported the OpenBestAvailable entries as stale. The two build-tag twins' keys now name OpenBestAvailableWith. The reason is unchanged: it is the SDK's library surface for embedders, and no code in the bd binary calls it. gofmt realigned the three neighbouring entries. * chore(schema): renumber removed_restriction to 0071 after #6943's 0070 Migrations are numbered by landing order. #6943 (participation_generation, the versioned-history write fence) lands first and owns main-plane 0070 and ignored/0028, so this PR's removed_restriction migration moves to the next free slot, 0071. The rename is done here, as one maintainer commit, rather than inside the replayed commits, so the history shows exactly what changed relative to the PR as reviewed; the intermediate commits of the rebase carry a duplicate 0070 and are not expected to build. - migrations/0070_add_removed_restriction.{up,down}.sql become 0071_*, with their header comments updated (the SQL itself is unchanged). Neither file exists on main or on #6943, so this is not an edit to a shipped migration. - cli_migrations.go: the CLI-bundle override is keyed to the new file name and renamed cliMigration0071AddRemovedRestriction, and its const now sits after 0070's. - migration_0070_add_removed_restriction_test.go becomes migration_0071_add_removed_restriction_test.go; its tests and messages say 0071, and it now owns the one LatestVersion pin (71). TestLatestVersionIncludesMigration0070 in #6943's participation_generation test is superseded by it and replaced by the usual pointer comment, and 0069's pointer comment now names the 0070 file that superseded it. - schema_test.go hygiene comments, BUILD.bazel embedsrcs/srcs, and the asof_read.go comment that cited the migration number. Checked: go test ./internal/storage/schema/... (with the dolt CLI lane, including TestAllMigrationsSQLAppliesThroughDoltCLIAndRecordsLatestVersion) and scripts/check-migration-hygiene.sh against origin/main and against #6943's tip. Co-Authored-By: Claude Opus 5.5 (1M context) * test(versions): reconcile #6661's tests and docs with #6943's write fence #6943's write fence (design §16.2b) makes an update-shaped write to a legacy record -- one whose participation_generation is NULL because it was created while versioned history was off -- a skip, not a refusal: the fence returns before the snapshot is canonicalized, so nothing is versioned and the mint's admission gate never runs. Several of #6661's tests seed rows with history off and then expect a later write to them to be versioned or refused, and several comments say every later write to an offending row fails. On a tree that carries both PRs those tests fail and those comments are wrong. This ports, without changing their substance, the fence-integration commits Jim Wordelman wrote for the donnabox integration fork, where both PRs were already folded together: - eff898db80 / 5d2e5a2ac4 (issueops, dolt, embeddeddolt): the mocked mint sequence scripts the fence's participation_generation read; the number fidelity test splits phase 2 into 2a (a legacy row is written, not versioned, not refused, stays NULL) and 2b (a participating row is refused and left untouched); TestRecordVersionInTxSkipsALegacyRowBefore- TheAdmissionGate pins the fence's order against the gate. - e0ad9d1e3c / 1528022dfa (cmd/bd): the switch round trip and the routing/env end-to-end tests build the rows they expect a version for after the setting is on, and assert that the pre-setting seeds gain nothing. - 18b9015bf5 (cmd/bd): the embedded and proxied gate-timeout scan tests assert the whole scan-vs-mint truth table (legacy offenders are named by the scan but not refused by the mint; participating offenders are both). - 5929ebd50c / 8ce872be03 (comments, CHANGELOG, configuration reference): the mint's refusal binds rows that participate in history; the pre-enable scan cannot see participation, so it over-approximates in the safe direction. Left out on purpose: the fork's mintSkipsRow helper and defer-wake sweep change, and its bd doctor "Versionable Issues" check and the docs that point at it; none of those are part of #6661 or #6943. Co-Authored-By: Claude Opus 5.5 (1M context) Co-authored-by: Jim Wordelman * test(versions): adapt #6661's guards to what main changed under it Three test-side adjustments for code that landed on main after #6661's last merge of it; no production code changes. - internal/storage/leg_contract_wiring_test.go: main gained an HTTP storage leg, and TestEveryLegWiresEveryRoleContract requires every leg to name or waive every role contract. The five R7.1 as-of read contracts this PR adds are a storage capability the HTTP client does not serve (internal/httpclient answers AsOf with an explicit unsupported error, and the wire has no instant/address selector or gone/unknown refusal), so they are waived on the http leg with that reason, as the donnabox integration fork does. - cmd/bd/versioned_history_construction_test.go: main split newDoltStoreFromConfig into two thin wrappers over openDoltStoreFromConfig, which is where the activation defer now lives (this rebase's resolution of the activation commit), so the factory guard keys on that body. - cmd/bd/versioned_history_serve_e2e_test.go: main's TestProxiedGatedTestsRunInTheProxiedLane requires a test that skips unless BEADS_TEST_PROXIED_SERVER=1 to be one the proxied lane selects, so TestVersionedHistoryServeActivatesTheProviderFromItsOwnRow becomes TestProxiedServerVersionedHistoryServeActivatesTheProviderFromItsOwnRow, and the 34-shard proxied manifest gains its line (written by gen_proxied_shard_manifest.py 34 --weights=duration --write, incremental; --check passes). Co-Authored-By: Claude Opus 5.5 (1M context) * chore(bazel): emit go_srcs.py's managed blocks for internal/versionedhistory Main's tools/bazel/go_srcs.py now manages a repo_files filegroup block in every package BUILD file, lists those filegroups in the root BUILD.bazel's aggregates, and lists every go_library in tools/bazel's release_cross_build target. internal/versionedhistory is a package this PR adds, so after the rebase those managed blocks were stale and TestBazelGoSrcsBlocksCurrent (go test ./scripts/...) failed. This is the generator's own output (python3 tools/bazel/go_srcs.py) and nothing else; --check passes afterwards. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(versions): don't publish change_agent in bd versions --json `bd versions --json` prints storage.IssueVersion as-is, so every JSON tag on that struct is part of a public contract. Provenance V2 (gastownhall/beads#7510, as ruled by the maintainer) removes the ad hoc agent field in favour of an actor chain; shipping a "change_agent" key now would make a public contract out of a field that is on its way out. IssueVersion.ChangeAgent is now tagged json:"-". Nothing else changes: the issue_versions.change_agent column (migration 0067) stays, ListVersionsInTx still reads it, change_actor and change_message are still published, and the human listing never showed the agent. No doc or golden named the key. TestVersionsJSONDoesNotPublishChangeAgent pins the absence with a non-empty value, so an omitempty-style regression would still fail, and pins that change_actor and change_message are still there. Co-Authored-By: Claude Opus 5.5 (1M context) * test(issueops): pin what the unfenced test-support mint does on #6943's new paths #6943's two fixes to the shared mint body -- (a) no issues row means no mint and no error, and (b) a create-shaped call on a new issues-table row flagged no-history stamps participation_generation without minting -- were written for its two kinds. This PR's shared body has a third, mintUnfenced, behind RecordVersionAtInTx (the R7.1 as-of fixtures' controlled-timestamp mint). The rebase resolution decided what (a) and (b) mean for it; this pins that decision: - (a) applies. It is the plane rule (FR-8: the table that holds the row decides its plane), not the fence, and the unfenced kind already honoured the flag-based form of the same rule (IsWisp). Without it, an unfenced mint for an unflagged wisps-table row would insert a version row that no issues row points at. The shared body's issues-table read now runs for all three kinds; only the update kind acts on the value it reads. - (b) does not apply. The unfenced mint never declares participation, so a flagged row returns right after the snapshot, as before: no read, no stamp. - The fence still does not apply: a legacy row is minted, and the closing UPDATE advances current_revision alone. TestRecordVersionAtInTxKeepsThePlaneRuleButNotTheFence scripts each of those three sequences exactly with sqlmock. TestTestSupportMintsStayOutsideTheShapeSplit places RecordVersionAtInTx outside #6943's create / dual-shape / update classification explicitly (it is the seam's exempt twin, not a mutation entry point or mint site): it must be in none of the shape lists, stay in versionExemptions, and reach neither RecordVersionForCreateInTx (it never stamps) nor RecordVersionInTx (whose gate and fence would stop the as-of fixtures minting history-off rows). The row-lock guard's exemption comment for recordVersionAtInTx already names the stamp-only UPDATE: #6943's own wording of it (all three UPDATE issues statements) was carried into that exemption when the as-of read commit was replayed. Co-Authored-By: Claude Opus 5.5 (1M context) * test(storage): say exactly why the as-of read contracts stay waived on http #6943 now runs most of its participation-generation contracts over the served wire (internal/httpclient/served_participation_generation_test.go): their subject is a create or an update, which the client can send, and the fixture reads the reference store's raw columns. This checked whether the five R7.1 as-of read contracts can be wired the same way. They cannot, cheaply or otherwise, without a new wire operation: - their subject is the read itself (AsOfReadFixture.Resolve), and R7.1's read, issueops.AsOfReadInTx, is reached by no publicops role and served by no wire operation; - the wire carries neither selector (an instant or a served version address) nor the gone/unknown refusal the read answers with; - the client's only point-in-time read, DoltStorage.AsOf, is a Dolt commit-ref read, not R7.1, and is on the client's D8 unsupported allowlist. Binding Resolve to the reference store behind the served env would pass, but as a property of that store, which the store legs already pin, never of the client. So the waiver stays; its comment and reason string now say the above instead of the earlier shorthand ("HTTP AsOf is explicitly unsupported"), which conflated the client's commit-ref AsOf with R7.1. Co-Authored-By: Claude Opus 5.5 (1M context) * docs(versions): describe what the write fence does to beads that predate recording With #6943's write fence (design §16.2b) a bead that existed before versioned history was turned on (participation_generation NULL) is legacy: an update-shaped write to it is skipped, so it is never versioned however often it is edited. Several user-facing strings from #6661 still described the behaviour before the fence, and an independent review reproduced each with a built bd (a legacy bead edited twice after enabling still lists no versions; a legacy bead the switch refuses over is written without refusal under BD_VERSIONED_HISTORY_ENABLED=1). Text only; no behaviour changes. - bd versions help: only beads created while recording is on are versioned; a bead that already existed is legacy and editing it records nothing; no backfill. (Was: "a bead edited yesterday shows nothing until it is edited again".) - bd versions empty listing: drops "yet" and says why the list is empty -- a bead created while recording is on always has its create version, so an empty list means the bead predates recording (or was made by a writer that was not recording), or is no-history or a wisp. The line still starts "No versions recorded for ", which the switch round-trip test pins. - bd versions with recording off: only beads created after it is turned on are versioned. - bd config set versioned-history.enabled true refusal: Jim's scan stays as conservative as he made it -- it cannot see participation, and a legacy row can still be minted create-shaped if it re-enters the issues plane -- but the text now says a listed issue's writes are refused only if it participates (created while history was on, or entered the issues plane while it was on), and that legacy issues are listed because the check cannot tell. The closing line says the same of a value a non-recording writer adds later. - docs/reference/configuration.md: the section opener says every accepted change to an issue created while history is on is snapshotted, and that a pre-existing issue is legacy; the "participates" parenthetical drops "or written" (a write never stamps participation; only a create-shaped mint does) and names entering the issues plane, matching the refusal text. CHANGELOG already says "if it participates in history" and is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(versions): keep updated_at across update-shaped version mints The update-shaped mint ended with UPDATE issues SET current_revision = ? WHERE id = ?. current_revision always changes there, so the column's ON UPDATE CURRENT_TIMESTAMP clause fired: with versioned history on, every label, dependency or import write moved issues.updated_at to the session's wall clock (local time, read back as UTC). An import with RejectStaleUpserts then stored the wall clock instead of the source's updated_at, and the source's next genuine edit was rejected as stale. The statement now assigns updated_at to itself, the rule #4298, blocked_state.go's mark/unmark templates and the create-shaped mint statements already follow, so history on leaves updated_at exactly where history off does. It is the one statement every production update-shaped mint ends with. Tests: - TestEmbeddedMintLeavesUpdatedAtWhereHistoryOffDoes (embeddeddolt) ages a bead, adds and removes a label and a dependency, and runs two RejectStaleUpserts imports, with history off first as the control. With the fix reverted, history on fails all six arms (updated_at reads the wall clock, and the later source edit is rejected as stale); history off passes either way. - RunParticipationGenerationFlagOffSkipsUpdateOnParticipatingRecord, wired on the dolt, embeddeddolt, uow and http legs: a record created with the flag on and updated with it off mints nothing and keeps its current_revision. Every other flag-off case uses a legacy record, which the fence stops first, so with RecordVersionInTx's gate removed this is the only case that fails, on all four legs. That gate is the single entry for every production update-shaped mint, so the one update arm covers label and close writes too. - The sqlmock pin of the statement (version_history_preenable_scan_test.go) and the journal-completeness exemption comment follow the new SQL. - The createbatchequiv issues_updated_at goldens of small, depadd, import458 and import458-reject-stale are re-recorded. They pinned the wall-clock bump, so each row minted a second time (eq-n2; eq-db1 and eq-dl; eq-b3 and eq-b250) had dropped out of the compared set; CI's embedded and server legs both printed the new digests. Recorded on embedded Dolt, the fix-reverted tree reproduces every committed golden byte for byte, HEAD moves only these four digests, and its rows equal the history-off rows in every scenario. GoldenDirEnv's comment lists it as the goldens' third deliberate change. - The new embedded test has its line in the storage shard manifest (gen_embedded_storage_shard_manifest.py 40 --weights=duration --write), so TestStorageEmbeddedShardManifestGeneratorNotStale stays green. It sits after TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion rather than at the end of the file, where #7646 appends its own test, so the two PRs still merge without a conflict in that file. Docs: - CHANGELOG: the feature's two entries under ### Fixed described fixes to a switch no released bd has. One ### Added entry replaces them, covering bd versions, the versioned-history.enabled switch and its per-store semantics, the pre-enable check, the single-writer caveat, migration 0071 and the bd version refusal. - The bd versions help, its recording-off refusal and empty-listing note, and the opening paragraph of the configuration doc's Versioned Issue History section now state the rule the pre-enable refusal prints: a bead participates if it was created, or entered the issues plane (a wisp promoted or made persistent), while recording was on. bd version: the "did you mean bd versions" refusal now runs after the command's metrics event is created, like the other in-RunE argument refusals, so a mistyped bd version is counted as a version invocation instead of going uncounted. Co-Authored-By: Claude Opus 5.5 Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen --------- Co-authored-by: Jim Wordelman Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Julian Knutsen Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com> --- .github/scripts/embedded-cmd-test-shards.txt | 5 + .../scripts/embedded-storage-test-shards.txt | 4 + .github/scripts/proxied-cmd-test-shards.txt | 2 + BUILD.bazel | 17 + CHANGELOG.md | 35 + backend/conformance/BUILD.bazel | 1 + ...write_participation_generation_contract.go | 59 ++ .../conformance/versioned_read_contract.go | 356 ++++++++ cmd/bd/BUILD.bazel | 13 + cmd/bd/capability_registry.go | 13 + cmd/bd/config.go | 16 +- cmd/bd/config_side_effects.go | 7 + cmd/bd/config_side_effects_test.go | 29 + cmd/bd/doctor/fix/BUILD.bazel | 5 + cmd/bd/doctor/fix/store.go | 28 +- .../fix/versioned_history_activation_test.go | 87 ++ cmd/bd/events_journal_construction_test.go | 30 +- cmd/bd/migrate_personal.go | 4 +- cmd/bd/storage_chain.go | 7 + cmd/bd/store_factory.go | 3 + cmd/bd/store_factory_nocgo.go | 3 + cmd/bd/uow_factory.go | 2 + cmd/bd/version.go | 12 + cmd/bd/versioned_history.go | 63 ++ cmd/bd/versioned_history_construction_test.go | 322 ++++++++ cmd/bd/versioned_history_e2e_test.go | 383 +++++++++ cmd/bd/versioned_history_serve_e2e_test.go | 88 ++ cmd/bd/versioned_history_switch.go | 170 ++++ ...ioned_history_switch_scan_embedded_test.go | 436 ++++++++++ ...ry_switch_scan_proxied_integration_test.go | 158 ++++ cmd/bd/versioned_history_switch_test.go | 97 +++ cmd/bd/versions.go | 305 +++++++ cmd/bd/versions_switch_embedded_test.go | 122 +++ cmd/bd/versions_test.go | 419 ++++++++++ docs/reference/configuration.md | 23 +- internal/config/config.go | 12 + .../served_participation_generation_test.go | 3 + internal/storage/BUILD.bazel | 1 + .../createbatchequiv/createbatchequiv.go | 13 +- .../createbatchequiv/golden/depadd.json | 4 +- .../golden/import458-reject-stale.json | 4 +- .../createbatchequiv/golden/import458.json | 4 +- .../createbatchequiv/golden/small.json | 4 +- internal/storage/dolt/BUILD.bazel | 2 + ..._participation_generation_contract_test.go | 3 + .../dolt/metadata_number_fidelity_test.go | 272 ++++++ internal/storage/dolt/versioned.go | 24 + .../dolt/versioned_read_contract_test.go | 337 ++++++++ internal/storage/embeddeddolt/BUILD.bazel | 3 + ..._participation_generation_contract_test.go | 3 + .../metadata_number_fidelity_test.go | 286 +++++++ internal/storage/embeddeddolt/store.go | 16 + .../version_completeness_behaviour_test.go | 113 +++ .../embeddeddolt/version_subsecond_test.go | 99 +++ .../versioned_read_contract_test.go | 309 +++++++ internal/storage/issueops/BUILD.bazel | 7 + internal/storage/issueops/asof_read.go | 286 +++++++ internal/storage/issueops/asof_read_test.go | 173 ++++ .../issueops/journal_completeness_test.go | 25 +- .../storage/issueops/row_lock_guard_test.go | 40 +- .../issueops/version_completeness_test.go | 65 ++ internal/storage/issueops/version_history.go | 442 +++++++++- .../version_history_durable_state_test.go | 80 +- .../version_history_integer_admission_test.go | 186 +++++ .../version_history_number_admission_test.go | 298 +++++++ .../version_history_preenable_scan_test.go | 680 +++++++++++++++ .../storage/issueops/version_history_scan.go | 53 ++ internal/storage/issueops/version_list.go | 77 ++ internal/storage/leg_contract_wiring_test.go | 30 + internal/storage/schema/BUILD.bazel | 3 + internal/storage/schema/cli_migrations.go | 15 + .../migration_0069_datetime_precision_test.go | 6 + ..._0070_add_participation_generation_test.go | 19 +- ...ation_0071_add_removed_restriction_test.go | 171 ++++ .../0071_add_removed_restriction.down.sql | 16 + .../0071_add_removed_restriction.up.sql | 49 ++ internal/storage/schema/schema_test.go | 6 + internal/storage/uow/BUILD.bazel | 1 + ..._participation_generation_contract_test.go | 3 + .../uow/versioned_read_contract_test.go | 404 +++++++++ internal/storage/version_lister.go | 76 ++ internal/versionedhistory/BUILD.bazel | 92 +++ internal/versionedhistory/activation.go | 350 ++++++++ internal/versionedhistory/activation_test.go | 776 ++++++++++++++++++ tools/bazel/BUILD.bazel | 1 + tools/bazel/go_srcs.py | 1 + 86 files changed, 9156 insertions(+), 111 deletions(-) create mode 100644 backend/conformance/versioned_read_contract.go create mode 100644 cmd/bd/doctor/fix/versioned_history_activation_test.go create mode 100644 cmd/bd/versioned_history.go create mode 100644 cmd/bd/versioned_history_construction_test.go create mode 100644 cmd/bd/versioned_history_e2e_test.go create mode 100644 cmd/bd/versioned_history_serve_e2e_test.go create mode 100644 cmd/bd/versioned_history_switch.go create mode 100644 cmd/bd/versioned_history_switch_scan_embedded_test.go create mode 100644 cmd/bd/versioned_history_switch_scan_proxied_integration_test.go create mode 100644 cmd/bd/versioned_history_switch_test.go create mode 100644 cmd/bd/versions.go create mode 100644 cmd/bd/versions_switch_embedded_test.go create mode 100644 cmd/bd/versions_test.go create mode 100644 internal/storage/dolt/metadata_number_fidelity_test.go create mode 100644 internal/storage/dolt/versioned_read_contract_test.go create mode 100644 internal/storage/embeddeddolt/metadata_number_fidelity_test.go create mode 100644 internal/storage/embeddeddolt/version_subsecond_test.go create mode 100644 internal/storage/embeddeddolt/versioned_read_contract_test.go create mode 100644 internal/storage/issueops/asof_read.go create mode 100644 internal/storage/issueops/asof_read_test.go create mode 100644 internal/storage/issueops/version_history_integer_admission_test.go create mode 100644 internal/storage/issueops/version_history_number_admission_test.go create mode 100644 internal/storage/issueops/version_history_preenable_scan_test.go create mode 100644 internal/storage/issueops/version_history_scan.go create mode 100644 internal/storage/issueops/version_list.go create mode 100644 internal/storage/schema/migration_0071_add_removed_restriction_test.go create mode 100644 internal/storage/schema/migrations/0071_add_removed_restriction.down.sql create mode 100644 internal/storage/schema/migrations/0071_add_removed_restriction.up.sql create mode 100644 internal/storage/uow/versioned_read_contract_test.go create mode 100644 internal/storage/version_lister.go create mode 100644 internal/versionedhistory/BUILD.bazel create mode 100644 internal/versionedhistory/activation.go create mode 100644 internal/versionedhistory/activation_test.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index fe342055e..983c57438 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -133,14 +133,19 @@ 100 54 TestEmbeddedSearchConcurrent 100 55 TestEmbeddedUndefer +100 55 TestEmbeddedVersionedHistorySwitchScansEveryRowNotOnePage 100 56 TestEmbeddedDepNoCycleCheck +100 56 TestEmbeddedVersionedHistorySwitchRoundTrip 100 57 TestEmbeddedQueryConcurrent +100 57 TestEmbeddedVersionedHistorySwitchRefusesUnversionableMetadata 100 58 TestEmbeddedDepAAddRejectionsAndBulk +100 58 TestEmbeddedVersionedHistorySwitchRefusesAnUnrecordableGateTimeout 100 59 TestEmbeddedDeferAutoWake +100 59 TestEmbeddedVersionedHistorySetManyRefusesUnversionableMetadata 100 60 TestEmbeddedCommentsCLI 100 60 TestEmbeddedRoutedWriteMissDoesNotCreateTargetDatabase diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index 089738a24..d127ad9a6 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -186,6 +186,7 @@ 40 5 TestEmbeddedClaimerRefusesIncompleteRequestsAndAnAbsentIDWithoutTouchingState 40 5 TestEmbeddedIssueOperationsCloseAndReopenAWisp 40 5 TestEmbeddedReaderListKeysetPositionResumesTheCreatedDescIDAscOrder +40 5 TestRecordVersionKeepsSubSecondChangeAt 40 5 TestWithPeerAuth 40 6 TestCreateBatchFastPathsMatchPerRowApplyWaitsFor_Embedded @@ -195,6 +196,7 @@ 40 6 TestEmbeddedRelationsAnswersAWispTargetInTheOutDirection 40 6 TestPromoteStillWorksWithCollisionGuard 40 6 TestSearchWithMissingWispPlaneTables +40 6 TestVersionedHistoryKeepsNumbersTheStoreDistinguishesApart 40 7 TestConcurrencyMultiProcess 40 7 TestEmbeddedBatchCloserRecordsOneHistoryEntryForWhatLanded @@ -208,6 +210,7 @@ 40 8 TestDeleterContract 40 8 TestEmbeddedDependencyEditorRoutesWispSourcedRemovalToTheWispPlane 40 8 TestEmbeddedMergeAndSettleDirtyWorkingSetSurvivesRefusedMerge +40 8 TestEmbeddedMintLeavesUpdatedAtWhereHistoryOffDoes 40 8 TestEmbeddedReaderGetBriefDepsProjectsTheDependencyRows 40 8 TestEmbeddedRelationsOrdersNeighborsFromBothPlanesTogether 40 8 TestEmbeddedReopenIssueCategorySemantics @@ -369,6 +372,7 @@ 40 22 TestLeaseLifecycleEmbedded 40 22 TestPolicyBlockingAnnotatorForeignOpensAreSublinear +40 23 TestAsOfReadContract 40 23 TestEmbeddedBatchCloserBackendFailureReturnsNoOutcomes 40 23 TestEmbeddedBatchCloserClaimNextHydratesWhenSomethingClosed 40 23 TestEmbeddedDependencyEditorAcceptsBlockingAcrossIssueTypes diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index 7c60e35ad..7c2e916b9 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -69,6 +69,7 @@ 34 6 TestProxiedServerServeReadsTheJournalItJustWrote 34 6 TestProxiedServerSwarm 34 6 TestProxiedServerUpdate2B +34 6 TestProxiedServerVersionedHistorySwitchRefusesAnUnrecordableGateTimeout 34 6 TestProxiedServerWispList 34 7 TestProxiedServerCloseConcurrent @@ -254,6 +255,7 @@ 34 30 TestProxiedServerStatus 34 30 TestProxiedServerTodo 34 30 TestProxiedServerUpdate2 +34 30 TestProxiedServerVersionedHistoryServeActivatesTheProviderFromItsOwnRow 34 31 TestProxiedServerCloseBeadGate 34 31 TestProxiedServerComment diff --git a/BUILD.bazel b/BUILD.bazel index 7b10becbc..fed381d93 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -246,6 +246,19 @@ exports_files( visibility = ["//internal/httpclient/encode:__pkg__"], ) +# --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + ), + visibility = ["//:__subpackages__"], +) + +# --- end go_srcs --- + # --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- filegroup( @@ -413,6 +426,7 @@ filegroup( "//internal/utils:repo_doc_files", "//internal/validation:repo_doc_files", "//internal/versioncheck:repo_doc_files", + "//internal/versionedhistory:repo_doc_files", "//internal/workapi:repo_doc_files", "//internal/workapi/storecounter:repo_doc_files", "//internal/workapi/storequerier:repo_doc_files", @@ -647,6 +661,7 @@ filegroup( "//internal/utils:repo_go_srcs", "//internal/validation:repo_go_srcs", "//internal/versioncheck:repo_go_srcs", + "//internal/versionedhistory:repo_go_srcs", "//internal/workapi:repo_go_srcs", "//internal/workapi/storecounter:repo_go_srcs", "//internal/workapi/storequerier:repo_go_srcs", @@ -866,6 +881,7 @@ filegroup( "//internal/utils:repo_go_test_srcs", "//internal/validation:repo_go_test_srcs", "//internal/versioncheck:repo_go_test_srcs", + "//internal/versionedhistory:repo_go_test_srcs", "//internal/workapi:repo_go_test_srcs", "//internal/workapi/storecounter:repo_go_test_srcs", "//internal/workapi/storequerier:repo_go_test_srcs", @@ -1094,6 +1110,7 @@ filegroup( "//internal/utils:repo_other_files", "//internal/validation:repo_other_files", "//internal/versioncheck:repo_other_files", + "//internal/versionedhistory:repo_other_files", "//internal/workapi:repo_other_files", "//internal/workapi/storecounter:repo_other_files", "//internal/workapi/storequerier:repo_other_files", diff --git a/CHANGELOG.md b/CHANGELOG.md index cf1021419..60d4d31e9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `bd versions ` lists the versions recorded for a bead, and + `versioned-history.enabled` turns recording on. While it is on, every + accepted change to a bead created while it is on, or that enters the issues + plane while it is on (a wisp promoted or made persistent), stores a + snapshot of it as a row of `issue_versions` in the same transaction. It is + off by default and nothing is backfilled: a bead already on the issues + plane when it is turned on is legacy, and editing it records nothing. A + recorded write leaves `updated_at` where an unrecorded one would, so + `bd import`'s stale check sees the same timestamps either way. + `bd versions --json` answers `{"versions": [...], "recording": bool}`; the + command still lists what was recorded after recording is turned off, and + proxied-server and HTTP workspaces refuse it. The setting is a row of each + store's `config` table (`bd config set versioned-history.enabled true`), + and every store reads its own when it is opened: a write routed into + another workspace (a prefix-routed `bd update` or `bd close`, + `bd create --repo`) follows the target store's row, a proxied-server + provider reads it when it is constructed, and a long-lived `bd serve` + keeps the value it read until it restarts. + `BD_VERSIONED_HISTORY_ENABLED=1` (or a `config.yaml` value) turns + recording on for every store the process opens, and never turns it off. + The row replicates on `bd dolt push`/`pull`, and recording is single-writer + only: two clones recording at once can mint the same revision and fail the + pull, and a write from a writer that is not recording is folded into the + next recorded version under that version's actor, so keep recording on for + every writer of a store or off for all of them. Turning it on with + `bd config set` first checks every issue the store would version and + refuses, writing nothing, while any holds a number outside the I-JSON + exact-integer range (magnitude above 2^53-1, however it is spelled) in its + metadata or in a gate's timeout (held as nanoseconds, so about 104.25 + days), and prints the count, the ids, the field and the fix. There is no + override, and the check runs once; after that, a recorded write that would + hold such a number fails atomically. Migration 0071 adds a nullable + `removed_restriction` column to `issue_versions`. `bd version` now refuses + an argument and suggests `bd versions `, instead of printing bd's + build for a mistyped `bd version bd-123`. - `bd blocked --flags` prints the blocked flag for each id read from stdin (one per line; blank lines and repeats ignored), with `--json` as `[{"id","is_blocked"}]` in the order the ids were given, omitting an id that diff --git a/backend/conformance/BUILD.bazel b/backend/conformance/BUILD.bazel index 545efa1aa..2ba90b30b 100644 --- a/backend/conformance/BUILD.bazel +++ b/backend/conformance/BUILD.bazel @@ -65,6 +65,7 @@ go_library( "transaction_parity.go", "tree_walker_contract.go", "version_reconciler_contract.go", + "versioned_read_contract.go", "workspaceconfig_contract.go", ], importpath = "github.com/steveyegge/beads/backend/conformance", diff --git a/backend/conformance/dualwrite_participation_generation_contract.go b/backend/conformance/dualwrite_participation_generation_contract.go index e1e283380..7943e7e4a 100644 --- a/backend/conformance/dualwrite_participation_generation_contract.go +++ b/backend/conformance/dualwrite_participation_generation_contract.go @@ -267,6 +267,65 @@ func RunParticipationGenerationProceedsForPromotedRecord(t *testing.T, ctx conte } } +// RunParticipationGenerationFlagOffSkipsUpdateOnParticipatingRecord pins the +// switch's off half for a record the fence would let through: created with +// the flag on (so participation_generation is stamped), then updated after +// the flag is turned back off, it must mint nothing. ProceedsForPromotedRecord +// shows the fence passes such a record, so only the flag stands between this +// update and a mint. No other case can see a writer that ignores the flag on +// updates: every other flag-off record is legacy, and the fence stops those +// first. +func RunParticipationGenerationFlagOffSkipsUpdateOnParticipatingRecord(t *testing.T, ctx context.Context, fixture ParticipationGenerationFixture) { + t.Helper() + if err := fixture.SetFlag(ctx, true); err != nil { + t.Fatalf("enabling versioned history: %v", err) + } + id := fixture.IssuePrefix + "-flag-off-skip" + if err := fixture.Mutate(ctx, id); err != nil { + t.Fatalf("mutating %s: %v", id, err) + } + if gen, err := fixture.ParticipationGeneration(ctx, id); err != nil { + t.Fatalf("reading participation_generation for %s: %v", id, err) + } else if gen == nil { + t.Fatalf("participation_generation for %s is NULL immediately after a create with the flag on — "+ + "this case cannot test the flag against a record the fence would have stopped anyway", id) + } + + if err := fixture.SetFlag(ctx, false); err != nil { + t.Fatalf("disabling versioned history: %v", err) + } + beforeRevision, err := fixture.CurrentRevision(ctx, id) + if err != nil { + t.Fatalf("reading current_revision for %s: %v", id, err) + } + beforeCount, err := fixture.VersionRowCount(ctx, id) + if err != nil { + t.Fatalf("counting version rows for %s: %v", id, err) + } + + if err := fixture.MutateExisting(ctx, id); err != nil { + t.Fatalf("mutating existing %s: %v", id, err) + } + + afterRevision, err := fixture.CurrentRevision(ctx, id) + if err != nil { + t.Fatalf("reading current_revision for %s: %v", id, err) + } + afterCount, err := fixture.VersionRowCount(ctx, id) + if err != nil { + t.Fatalf("counting version rows for %s: %v", id, err) + } + if afterCount != beforeCount { + t.Errorf("version row count for %s went from %d to %d across an update-shaped mutation with the "+ + "flag off, want unchanged: turning versioned history off stops minting for every record, "+ + "including one that already participates", id, beforeCount, afterCount) + } + if afterRevision != beforeRevision { + t.Errorf("current_revision for %s went from %d to %d across the same update with the flag off, "+ + "want unchanged", id, beforeRevision, afterRevision) + } +} + // RunParticipationGenerationWispPromotionBeginsHistory pins that promoting a // wisp onto the issues plane is CREATE-shaped in design §16.2b's sense. (This // is wisp promotion, PromoteFromEphemeral: not R2.3's promotion of a legacy diff --git a/backend/conformance/versioned_read_contract.go b/backend/conformance/versioned_read_contract.go new file mode 100644 index 000000000..0e187117c --- /dev/null +++ b/backend/conformance/versioned_read_contract.go @@ -0,0 +1,356 @@ +package conformance + +import ( + "context" + "testing" + "time" +) + +// This file holds the contract for R7.1 (gastownhall/beads#6136, design doc +// gastownhall/beads#5898 revision 9): the as-of read — the "versioned +// reads" half of Phase 3's title, authored here from scratch, since no +// fixture for it exists anywhere in this package to un-skip. +// +// R7.1, VERBATIM (design doc #5898 rev 9): "An as-of read is a named read +// whose selector is an instant. Given an instant T or a version address, a +// store returns one Bead's durable state as of that point — the latest +// version it holds that was accepted at or before T, with everything later +// excluded — under R7's record shape, marking the address it served, or +// refuses as gone or unknown with the reason (R8, R20). Like every history +// answer it is complete for the answering store's holdings and may change +// after a sync (TC3)." +// +// ADDRESS, RESTRICTION, AND CHANGEATTRIBUTION ARE DECLARED IN +// expected_revision_contract.go — see that file's package-level note on +// where this suite's shared vocabulary lives and why. This file reuses +// Restriction's five values as-is for the gone-or-unknown refusal outcome +// (R8): an as-of read never invents a sixth state, and never distinguishes +// "gone" from "unknown" any differently than R20 already does. +// +// EVERY HOOK ON THE FIXTURE BELOW IS INDEPENDENTLY NILABLE — see the +// package-level note in expected_revision_contract.go. Every case +// nil-checks every hook it uses and SKIPS BY NAME when one is missing. No +// backend implements as-of reads yet — the implementation is +// the child this suite is written ahead of. +// +// EXPLICIT NON-GOAL: MEMORY-PLANE CONSUMER POLICY IS NOT PART OF THIS +// PRIMITIVE. R7.1's own text is explicit that a closed-before-T boundary, +// sibling exclusion, and supersession exclusion "are Memory-plane policies +// a consumer layers on this primitive (#5877), not part of it." This suite +// does not implement or assert any of the three — a future reader who finds +// them absent should read that as this file honoring R7.1's own scope, not +// as an oversight. This is the read-side analog of the R16/R16.1 boundary +// risk already recorded in expected_revision_contract.go: do not conflate +// the primitive with policy layered on top of it. +// +// THIS SUITE ASSERTS LOCAL COMPLETENESS ONLY (TC3). Nothing below requires, +// or even checks, that two stores asked about the same lineage agree — R7.1 +// itself says an as-of answer "may change after a sync," so two stores at +// different sync points are expected to answer differently, and neither is +// wrong. RunAsOfReadIsCompleteOnlyForTheAnsweringStoresHoldings is the one +// case that touches more than one store, and what it asserts is exactly +// this: each store answers strictly from its own holdings, never a +// sibling's — not that the two agree. + +// AsOfSelector names the point R7.1 resolves against: either a wall-clock +// instant, or a specific version Address treated as the selector itself — +// "the read that names its own answer" (R7.1: "an instant T or a version +// address"). Exactly one of At/Version is populated in every case this file +// exercises; R7.1 does not speak to the shape where both or neither are +// set, so this suite does not either. +type AsOfSelector struct { + At *time.Time + Version *Address +} + +// AsOfReadRefusal is R8's typed outcome for an as-of read that cannot be +// answered. Restriction is always one of the four Gone*/Unknown values, +// never RestrictionLive — a live answer is a served AsOfReadResult, not a +// refusal. Reason explains why (R8). +type AsOfReadRefusal struct { + Restriction Restriction + Reason string +} + +// AsOfReadResult is the outcome of an as-of read (R7.1). Address/State and +// Refusal are mutually exclusive: read Address/State only when Refusal is +// nil. +type AsOfReadResult struct { + // Address is the version this read served, marking "the address it + // served" (R7.1). Populated only when Refusal is nil. + Address Address + // State is the served version's durable state under R7's record shape. + // Opaque to this suite: the fixture adapter translates a backend's real + // record shape into this map, and no case here asserts its field-level + // content — only that a served result carries an Address. + State map[string]any + // Refusal is populated instead of Address/State when the read cannot be + // answered (R8). Nil means the read was served. + Refusal *AsOfReadRefusal +} + +// AsOfRead resolves storeID's answer to an as-of read against id's lineage, +// at selector (R7.1). +type AsOfRead func(ctx context.Context, storeID, id string, selector AsOfSelector) (AsOfReadResult, error) + +// AsOfReadFixture supplies the capability under test for R7.1: a read whose +// selector is an instant or a version address, resolving to the latest +// version accepted at or before that point, or a typed refusal. +type AsOfReadFixture struct { + IssuePrefix string + + // Resolve performs the as-of read under test (R7.1). A nil hook means + // this backend does not implement as-of reads yet, and every case in + // this file skips with that reason. + Resolve AsOfRead + + // MintAt mints a fresh version for id under storeID, recorded as + // accepted at the given instant, and returns its Address. Cases use + // this to build a lineage with known accept-times to resolve an as-of + // read against. A nil hook means this backend has no way to mint a + // version at a controlled accept-time, and cases that need one skip + // with that reason. + MintAt func(ctx context.Context, storeID, id string, acceptedAt time.Time) (Address, error) + + // MakeUnanswerable arranges for id to be unanswerable at storeID for + // the given Restriction (one of RestrictionGoneRetention/GoneErasure/ + // GoneReorganization/Unknown) — independently of + // RetentionFixture/EpochFixture's own removal machinery, since R7.1's + // refusal is a sibling contract to R17/R20, not a dependent of them. A + // nil hook means this backend has no way to arrange this, and the case + // that needs it skips with that reason. + MakeUnanswerable func(ctx context.Context, storeID, id string, restriction Restriction) error +} + +// RunAsOfReadSelectorAcceptsEitherAnInstantOrAVersionAddress pins R7.1's +// selector shape: an instant T and the version address that instant +// resolves to are two names for the same answer, and a store must accept +// either. +func RunAsOfReadSelectorAcceptsEitherAnInstantOrAVersionAddress(t *testing.T, ctx context.Context, fixture AsOfReadFixture) { + t.Helper() + if fixture.Resolve == nil { + t.Skip("this backend does not implement as-of reads (Resolve is nil)") + } + if fixture.MintAt == nil { + t.Skip("this backend has no way to mint a version at a controlled accept-time (MintAt is nil)") + } + store := asOfReadStore(fixture, "selector-kinds") + id := fixture.IssuePrefix + "-asof-selector" + t0 := time.Now().UTC().Add(-time.Hour) + asOfReadMint(t, ctx, fixture, store, id, t0) + t1 := t0.Add(time.Minute) + second := asOfReadMint(t, ctx, fixture, store, id, t1) + + byInstant, err := fixture.Resolve(ctx, store, id, AsOfSelector{At: &t1}) + if err != nil { + t.Fatalf("Resolve(selector=instant): %v", err) + } + if byInstant.Refusal != nil { + t.Fatalf("Resolve(selector=instant) = %+v, want a served result", byInstant) + } + + byVersion, err := fixture.Resolve(ctx, store, id, AsOfSelector{Version: &second}) + if err != nil { + t.Fatalf("Resolve(selector=version address): %v", err) + } + if byVersion.Refusal != nil { + t.Fatalf("Resolve(selector=version address) = %+v, want a served result", byVersion) + } + + if byInstant.Address != second { + t.Errorf("Resolve(selector=instant matching a version's own accept-time).Address = %s, want %s", byInstant.Address, second) + } + if byVersion.Address != second { + t.Errorf("Resolve(selector=version address).Address = %s, want the same address named %s", byVersion.Address, second) + } + if byInstant.Address != byVersion.Address { + t.Errorf("an instant selector (served %s) and a version-address selector naming the same version (served %s) resolved to different addresses; R7.1 accepts either as a name for the same answer", byInstant.Address, byVersion.Address) + } +} + +// RunAsOfReadReturnsTheLatestVersionAtOrBeforeTStrictlyExcludingLater pins +// R7.1's ordering promise: resolving at T returns the latest version +// accepted at or before T, and a version accepted after T is excluded even +// when its own instant is close by. +func RunAsOfReadReturnsTheLatestVersionAtOrBeforeTStrictlyExcludingLater(t *testing.T, ctx context.Context, fixture AsOfReadFixture) { + t.Helper() + if fixture.Resolve == nil { + t.Skip("this backend does not implement as-of reads (Resolve is nil)") + } + if fixture.MintAt == nil { + t.Skip("this backend has no way to mint a version at a controlled accept-time (MintAt is nil)") + } + store := asOfReadStore(fixture, "ordering") + id := fixture.IssuePrefix + "-asof-ordering" + t0 := time.Now().UTC().Add(-time.Hour) + asOfReadMint(t, ctx, fixture, store, id, t0) + t1 := t0.Add(time.Minute) + middle := asOfReadMint(t, ctx, fixture, store, id, t1) + t2 := t0.Add(2 * time.Minute) + asOfReadMint(t, ctx, fixture, store, id, t2) + + atExactly, err := fixture.Resolve(ctx, store, id, AsOfSelector{At: &t1}) + if err != nil { + t.Fatalf("Resolve(T=middle's own instant): %v", err) + } + if atExactly.Refusal != nil || atExactly.Address != middle { + t.Errorf("Resolve(T=middle's own instant) = %+v, want the version accepted exactly at T (%s), not a later one", atExactly, middle) + } + + between := t1.Add(30 * time.Second) + betweenResult, err := fixture.Resolve(ctx, store, id, AsOfSelector{At: &between}) + if err != nil { + t.Fatalf("Resolve(T strictly between two versions): %v", err) + } + if betweenResult.Refusal != nil || betweenResult.Address != middle { + t.Errorf("Resolve(T strictly between two versions) = %+v, want the latest version at-or-before T (%s); the version accepted after T must be strictly excluded", betweenResult, middle) + } +} + +// RunAsOfReadReportsTheAddressItServed pins R7.1's "marking the address it +// served": a served result names the exact Address of the version it +// returned, not merely a truthy success. +func RunAsOfReadReportsTheAddressItServed(t *testing.T, ctx context.Context, fixture AsOfReadFixture) { + t.Helper() + if fixture.Resolve == nil { + t.Skip("this backend does not implement as-of reads (Resolve is nil)") + } + if fixture.MintAt == nil { + t.Skip("this backend has no way to mint a version at a controlled accept-time (MintAt is nil)") + } + store := asOfReadStore(fixture, "reports-address") + id := fixture.IssuePrefix + "-asof-reports-address" + at := time.Now().UTC() + address := asOfReadMint(t, ctx, fixture, store, id, at) + + result, err := fixture.Resolve(ctx, store, id, AsOfSelector{At: &at}) + if err != nil { + t.Fatalf("Resolve: %v", err) + } + if result.Refusal != nil { + t.Fatalf("Resolve = %+v, want a served result", result) + } + if result.Address == "" { + t.Fatal("Resolve returned an empty Address on a served result, want the address of the version it served (R7.1)") + } + if result.Address != address { + t.Errorf("Resolve(...).Address = %s, want the served version's own address %s", result.Address, address) + } +} + +// RunAsOfReadRefusesAsGoneOrUnknownNeverSubstitutingAnotherVersion pins R8: +// a read that cannot be answered refuses as a typed Gone or Unknown outcome +// with a reason, and never silently returns a neighboring version, the +// nearest surviving version, or current state as a substitute — R8's own +// text names all three as non-answers. +func RunAsOfReadRefusesAsGoneOrUnknownNeverSubstitutingAnotherVersion(t *testing.T, ctx context.Context, fixture AsOfReadFixture) { + t.Helper() + if fixture.Resolve == nil { + t.Skip("this backend does not implement as-of reads (Resolve is nil)") + } + if fixture.MintAt == nil { + t.Skip("this backend has no way to mint a version at a controlled accept-time (MintAt is nil)") + } + if fixture.MakeUnanswerable == nil { + t.Skip("this backend has no way to make a lineage unanswerable (MakeUnanswerable is nil)") + } + + for _, restriction := range []Restriction{ + RestrictionGoneRetention, + RestrictionGoneErasure, + RestrictionGoneReorganization, + RestrictionUnknown, + } { + t.Run(restriction.String(), func(t *testing.T) { + store := asOfReadStore(fixture, "unanswerable-"+restriction.String()) + id := fixture.IssuePrefix + "-asof-unanswerable-" + restriction.String() + at := time.Now().UTC() + address := asOfReadMint(t, ctx, fixture, store, id, at) + + if err := fixture.MakeUnanswerable(ctx, store, id, restriction); err != nil { + t.Fatalf("MakeUnanswerable(%s, %s): %v", id, restriction, err) + } + + result, err := fixture.Resolve(ctx, store, id, AsOfSelector{At: &at}) + if err != nil { + t.Fatalf("Resolve after MakeUnanswerable(%s): %v", restriction, err) + } + if result.Refusal == nil { + t.Fatalf("Resolve after MakeUnanswerable(%s) = %+v, want a Refusal, not a served result (R8)", restriction, result) + } + if result.Refusal.Restriction != restriction { + t.Errorf("Refusal.Restriction = %s, want %s", result.Refusal.Restriction, restriction) + } + if result.Refusal.Reason == "" { + t.Error("Refusal.Reason is empty, want a reason (R8)") + } + if result.Address != "" { + t.Errorf("Resolve on a refusal returned Address = %s, want empty: R8 forbids silently substituting a neighboring version, the nearest surviving version, or current state (the pre-refusal address was %s)", result.Address, address) + } + }) + } +} + +// RunAsOfReadIsCompleteOnlyForTheAnsweringStoresHoldings pins R7.1's closing +// clause and TC3: an as-of answer "is complete for the answering store's +// holdings and may change after a sync." This case asserts NOTHING about +// agreement between two stores asked about the same lineage — proving that +// absence is the point, not a gap. It mints a lineage into exactly one of +// two stores and confirms each store answers strictly from its own +// holdings: the store that holds the lineage serves it, and the store that +// never received it answers Unknown (FR-06, reused from R20) rather than +// reaching across to a sibling store's knowledge, erroring, or answering +// Gone — which would claim knowledge of a lineage it never held. +func RunAsOfReadIsCompleteOnlyForTheAnsweringStoresHoldings(t *testing.T, ctx context.Context, fixture AsOfReadFixture) { + t.Helper() + if fixture.Resolve == nil { + t.Skip("this backend does not implement as-of reads (Resolve is nil)") + } + if fixture.MintAt == nil { + t.Skip("this backend has no way to mint a version at a controlled accept-time (MintAt is nil)") + } + holds := asOfReadStore(fixture, "tc3-holds-the-lineage") + neverSynced := asOfReadStore(fixture, "tc3-never-synced") + id := fixture.IssuePrefix + "-asof-tc3" + at := time.Now().UTC() + address := asOfReadMint(t, ctx, fixture, holds, id, at) + + local, err := fixture.Resolve(ctx, holds, id, AsOfSelector{At: &at}) + if err != nil { + t.Fatalf("Resolve on the store that holds the lineage: %v", err) + } + if local.Refusal != nil || local.Address != address { + t.Fatalf("Resolve on the store that holds the lineage = %+v, want it served from that store's own holdings (%s)", local, address) + } + + remote, err := fixture.Resolve(ctx, neverSynced, id, AsOfSelector{At: &at}) + if err != nil { + t.Fatalf("Resolve on a store that never received this lineage: %v", err) + } + if remote.Refusal == nil { + t.Fatalf("Resolve on a store that never received this lineage = %+v, want a Refusal: an as-of answer is complete for the answering store's own holdings, not a sibling's (TC3)", remote) + } + if remote.Refusal.Restriction != RestrictionUnknown { + t.Errorf("Refusal.Restriction on the never-synced store = %s, want RestrictionUnknown: no lineage knowledge is Unknown, not Gone (FR-06)", remote.Refusal.Restriction) + } +} + +// --- fixture helpers ------------------------------------------------------- + +// asOfReadStore names a storeID namespaced by the fixture's IssuePrefix and +// tag, so cases that need more than one store (e.g. the TC3 local- +// completeness case) get independent ones. +func asOfReadStore(fixture AsOfReadFixture, tag string) string { + return fixture.IssuePrefix + "-asof-store-" + tag +} + +// asOfReadMint mints a version for id on store at the given instant via +// fixture.MintAt, fataling the case if minting fails. +func asOfReadMint(t *testing.T, ctx context.Context, fixture AsOfReadFixture, store, id string, at time.Time) Address { + t.Helper() + address, err := fixture.MintAt(ctx, store, id, at) + if err != nil { + t.Fatalf("MintAt(%s, %s, %s): %v", store, id, at, err) + } + return address +} diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 409654492..6eb4de6e7 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -375,6 +375,9 @@ go_library( "version.go", "version_proxied_server.go", "version_tracking.go", + "versioned_history.go", + "versioned_history_switch.go", + "versions.go", "where.go", "wisp.go", "wisp_proxied_server.go", @@ -463,6 +466,7 @@ go_library( "//internal/utils", "//internal/validation", "//internal/versioncheck", + "//internal/versionedhistory", "//internal/workapi", "//internal/workspacegate", "//internal/worktreeremove", @@ -1290,6 +1294,14 @@ go_test( "version_embedded_test.go", "version_test.go", "version_tracking_test.go", + "versioned_history_construction_test.go", + "versioned_history_e2e_test.go", + "versioned_history_serve_e2e_test.go", + "versioned_history_switch_scan_embedded_test.go", + "versioned_history_switch_scan_proxied_integration_test.go", + "versioned_history_switch_test.go", + "versions_switch_embedded_test.go", + "versions_test.go", "where_cgo_test.go", "where_embedded_test.go", "where_test.go", @@ -1319,6 +1331,7 @@ go_test( ":go_srcs", ":preflight_go_fixture", "//:bd_test_files", + "//:go_srcs", "//cmd/bd/doctor:go_srcs", "//cmd/bd/doctor/fix:go_srcs", "//internal/storage/embeddeddolt/cmd:go_srcs", diff --git a/cmd/bd/capability_registry.go b/cmd/bd/capability_registry.go index 1ac36eca9..b52d5df13 100644 --- a/cmd/bd/capability_registry.go +++ b/cmd/bd/capability_registry.go @@ -79,6 +79,9 @@ const ( trackSync = "proxied-parity S5 (bd sync)" trackDegradation = "proxied-parity S6 (doctor, config show, degradations)" trackLongTail = "proxied-parity S7 (long-tail policy pass)" + // Versioned history (gastownhall/beads#6132) is not a proxied-parity slice: + // its reads need a storage.VersionLister over the proxied uow provider. + trackVersionedHistory = "versioned history: VersionLister over the proxied uow provider (gastownhall/beads#6132)" ) // capabilityRow is one command-path policy row. ArgSet is the sorted set of @@ -451,6 +454,16 @@ var proxyCapabilityRegistry = []capabilityRow{ // at the front door would leave those modes with no way to flush. permitted("dolt commit").withHistory(HistoryProxySupported), + // --- versioned history ---------------------------------------------------- + // `bd versions` is a pure read (issueops.ListVersionsInTx in one read + // transaction), so nothing about it is unsafe on a shared backend. It is + // refused only because no proxied route exists yet: it reaches the backend + // through the optional storage.VersionLister, which the dolt and embedded + // stores implement and the uow provider does not. Permitting it would open + // the proxied provider in the root pre-run and then fail untyped at + // errVersionsUnsupported, so it is refused here, before any side effect. + refusedPath("versions", "proxy.versions.unsupported", ProxyReasonUnimplemented, trackVersionedHistory), + // --- untyped refusals inside RunE --------------------------------------- // The gate permits these paths and the command refuses itself with a bare // string, so a JSON consumer gets no code and no mutates flag. They are diff --git a/cmd/bd/config.go b/cmd/bd/config.go index 6ea9df1fb..6756f3e49 100644 --- a/cmd/bd/config.go +++ b/cmd/bd/config.go @@ -280,6 +280,12 @@ var configSetCmd = &cobra.Command{ if err != nil { return HandleError("%v", err) } + // Per-key validation that needs the store rather than the key alone: turning + // versioned history on refuses, writing nothing, while any issue holds a value a + // version could not record. The store is open here, on whichever route this is. + if err := checkVersionedHistoryCanBeEnabled(rootCtx, key, value); err != nil { + return HandleError("%v", err) + } result, err := settings.SetSetting(rootCtx, issueops.SetSettingRequest{Key: key, Value: value}) if err != nil { return HandleError("setting config: %v", err) @@ -1222,6 +1228,14 @@ Examples: return HandleError("%v", err) } } + // The pre-enable check `bd config set` makes: turning versioned history + // on refuses while any issue holds a value a version could not record. + // This verb writes the same setting through a loop of its own, so it + // makes the check itself, here and before any pair is written: a refusal + // stores nothing from the batch. + if err := checkVersionedHistoryCanBeEnabled(rootCtx, p.key, p.value); err != nil { + return HandleError("%v", err) + } } var yamlPairs, gitPairs, dbPairs []kvPair @@ -1352,7 +1366,7 @@ var recognizedConfigPrefixes = []string{ "status.", "types.", "doctor.suppress.", "routing.", "sync.", "git.", "directory.", "repos.", "external_projects.", "validation.", "lint.", "hierarchy.", "ai.", "backup.", "federation.", "metrics.", - "agent.", "claim.", "storage-class.", + "agent.", "claim.", "storage-class.", "versioned-history.", } // validateStorageClassConfig validates a storage-class. per-type diff --git a/cmd/bd/config_side_effects.go b/cmd/bd/config_side_effects.go index 9876732dd..5f70d26c2 100644 --- a/cmd/bd/config_side_effects.go +++ b/cmd/bd/config_side_effects.go @@ -4,6 +4,8 @@ import ( "fmt" "os" "strings" + + "github.com/steveyegge/beads/internal/versionedhistory" ) // configSideEffect describes a hint or warning to show after a config change. @@ -65,6 +67,11 @@ func checkConfigSetSideEffects(key, value string) []configSideEffect { Message: fmt.Sprintf("Git sync remote set to %q. Ensure this git remote exists.", value), Command: fmt.Sprintf("git remote -v | grep %s", value), }) + + case key == versionedHistorySettingKey && versionedhistory.ValueEnables(value): + effects = append(effects, configSideEffect{ + Message: "Versioned history now records for this store. The setting replicates on 'bd dolt push' and 'bd dolt pull', so every clone that pulls it records too, and it is safe with a single writer only: keep one writer at a time per store (docs/reference/configuration.md#versioned-issue-history).", + }) } return effects diff --git a/cmd/bd/config_side_effects_test.go b/cmd/bd/config_side_effects_test.go index faa53db88..d8b9f329e 100644 --- a/cmd/bd/config_side_effects_test.go +++ b/cmd/bd/config_side_effects_test.go @@ -2,6 +2,7 @@ package main import ( "os" + "strings" "testing" ) @@ -105,6 +106,34 @@ func TestCheckConfigSetSideEffects_UnknownKey(t *testing.T) { } } +// versioned-history.enabled is a row of the store's config table, which +// replicates on `bd dolt push` and `bd dolt pull`, and the feature is single +// writer only. `bd config set ... true` is the moment someone turns it on, so it +// is where that caveat belongs -- said once, at the command that has the effect. +func TestVersionedHistoryConfigSetHintsAboutReplicationAndSingleWriter(t *testing.T) { + for _, on := range []string{"true", "TRUE", " true ", "1", "t"} { + effects := checkConfigSetSideEffects(versionedHistorySettingKey, on) + if len(effects) != 1 { + t.Errorf("value %q turns recording on: expected 1 hint, got %d", on, len(effects)) + continue + } + for _, want := range []string{"replicates", "single writer"} { + if !strings.Contains(strings.ToLower(effects[0].Message), want) { + t.Errorf("value %q: the hint does not mention %q: %s", on, want, effects[0].Message) + } + } + } + // Turning it off, or anything that does not turn it on, needs no caveat. + for _, off := range []string{"false", "0", "", "off", "nonsense"} { + if effects := checkConfigSetSideEffects(versionedHistorySettingKey, off); len(effects) != 0 { + t.Errorf("value %q does not turn recording on: expected no hint, got %d", off, len(effects)) + } + } + if effects := checkConfigUnsetSideEffects(versionedHistorySettingKey); len(effects) != 0 { + t.Errorf("unsetting the key needs no hint, got %d", len(effects)) + } +} + func TestCheckConfigUnsetSideEffects_FederationRemote(t *testing.T) { effects := checkConfigUnsetSideEffects("federation.remote") if len(effects) != 1 { diff --git a/cmd/bd/doctor/fix/BUILD.bazel b/cmd/bd/doctor/fix/BUILD.bazel index fd7245c04..7255c6111 100644 --- a/cmd/bd/doctor/fix/BUILD.bazel +++ b/cmd/bd/doctor/fix/BUILD.bazel @@ -43,6 +43,7 @@ go_library( "//internal/storage/issueops", "//internal/types", "//internal/utils", + "//internal/versionedhistory", "@com_github_burntsushi_toml//:toml", "@com_github_go_sql_driver_mysql//:mysql", "@in_gopkg_yaml_v3//:yaml_v3", @@ -81,7 +82,11 @@ go_test( "testmain_cgo_test.go", "testmain_nocgo_test.go", "validation_test.go", + "versioned_history_activation_test.go", ], + # versioned_history_activation_test.go parses store.go from its working + # directory, and a Bazel test only sees its declared runfiles. + data = ["store.go"], embed = [":fix"], deps = [ "//internal/configfile", diff --git a/cmd/bd/doctor/fix/store.go b/cmd/bd/doctor/fix/store.go index cadd5ea31..793d975fe 100644 --- a/cmd/bd/doctor/fix/store.go +++ b/cmd/bd/doctor/fix/store.go @@ -5,10 +5,11 @@ import ( "github.com/steveyegge/beads/internal/eventsjournal" "github.com/steveyegge/beads/internal/storage/dolt" + "github.com/steveyegge/beads/internal/versionedhistory" ) // openBeadMutatingStore opens a repair handler's store WITH the workspace's -// events-journal activation applied. +// events-journal and versioned-history activation applied. // // Most of bd doctor inspects and repairs workspace state — schema cursors, // metadata, remotes, fingerprints, git — and none of that belongs in a journal @@ -17,7 +18,10 @@ import ( // fresh-clone import paths CREATE them. Those are ordinary bead mutations, and // a consumer whose mirror silently diverges because someone ran // `bd doctor --fix` is exactly the failure the journal exists to prevent — the -// more so because a repair is unattended and nobody is watching the diff. +// more so because a repair is unattended and nobody is watching the diff. The +// same repairs are also ordinary issue mutations for version history, and an +// issue_versions trail with a silent gap where `bd doctor --fix` ran is the same +// failure one table over. // // The read-only and workspace-state handlers deliberately keep opening the // store directly; they carry their exemption in the construction guard, with a @@ -27,7 +31,7 @@ import ( // is what a repair path legitimately does. func openBeadMutatingStore(ctx context.Context, beadsDir string) (*dolt.DoltStore, error) { store, err := dolt.NewFromConfig(ctx, beadsDir) - return activated(beadsDir, store, err) + return activated(ctx, beadsDir, store, err) } // openBeadMutatingStoreCreating is openBeadMutatingStore for the fresh-clone @@ -36,15 +40,23 @@ func openBeadMutatingStore(ctx context.Context, beadsDir string) (*dolt.DoltStor // guard rather than failing on an absent database. func openBeadMutatingStoreCreating(ctx context.Context, beadsDir string) (*dolt.DoltStore, error) { store, err := dolt.NewFromConfigWithOptions(ctx, beadsDir, &dolt.Config{CreateIfMissing: true}) - return activated(beadsDir, store, err) + return activated(ctx, beadsDir, store, err) } -// activated applies the workspace's journal setting and closes the store if it -// cannot be honored. A failed open passes straight through, so the typed-nil -// store never reaches the activation. -func activated(beadsDir string, store *dolt.DoltStore, err error) (*dolt.DoltStore, error) { +// activated applies the workspace's journal setting and the store's own +// versioned-history setting, and closes the store if the journal cannot be +// honored. A failed open passes straight through, so the typed-nil store never +// reaches the activation. +// +// Versioned history cannot fail an open (a plumbing that cannot version simply +// records nothing), so it needs no error handling of its own; it runs after the +// journal so a store the journal refused is never activated. It is the shared +// internal/versionedhistory rule, not a copy: package main cannot be imported +// from here. +func activated(ctx context.Context, beadsDir string, store *dolt.DoltStore, err error) (*dolt.DoltStore, error) { if _, err = eventsjournal.ActivateStore(beadsDir, store, err); err != nil { return nil, err } + _, _ = versionedhistory.ActivateStore(ctx, store, nil) return store, nil } diff --git a/cmd/bd/doctor/fix/versioned_history_activation_test.go b/cmd/bd/doctor/fix/versioned_history_activation_test.go new file mode 100644 index 000000000..22e6fa0e8 --- /dev/null +++ b/cmd/bd/doctor/fix/versioned_history_activation_test.go @@ -0,0 +1,87 @@ +package fix + +import ( + "go/ast" + "go/parser" + "go/token" + "strconv" + "strings" + "testing" +) + +const ( + versionedHistoryPkg = "github.com/steveyegge/beads/internal/versionedhistory" + // package main cannot be imported at all; the guard below is that the + // activation this package applies lives in a package it CAN import, so the + // repair handlers apply the identical rule as the command surface rather than + // a copy of it. + beadMutatingStoreFile = "store.go" +) + +// TestVersionedHistoryDoctorRepairOpenerUsesTheSharedActivation holds the one +// construction site the cmd/bd construction guard cannot see. bd doctor is exempted +// from that guard as a package (its CHECKS are read-only), so the repair opener -- +// which opens the store the bead-mutating repairs (stale-closed cleanup, pollution +// cleanup, the fresh-clone imports) write through -- is held to the property here. +// +// A repair that deletes or creates issues is an ordinary mutation for version +// history: an issue_versions trail with a silent gap where `bd doctor --fix` ran is +// the same failure the routed-write bug produced, one command over, and the more +// dangerous for being unattended. +// +// The check is structural for the same reason the cmd/bd guard is: activation +// failing is invisible from inside the process. It requires the activation that +// `activated` applies to come from internal/versionedhistory -- the ONE +// implementation of the rule, which this package can import and package main cannot +// be. +func TestVersionedHistoryDoctorRepairOpenerUsesTheSharedActivation(t *testing.T) { + fset := token.NewFileSet() + file, err := parser.ParseFile(fset, beadMutatingStoreFile, nil, 0) + if err != nil { + t.Fatalf("parse %s: %v", beadMutatingStoreFile, err) + } + + imports := map[string]string{} + for _, spec := range file.Imports { + path, err := strconv.Unquote(spec.Path.Value) + if err != nil { + t.Fatalf("unquote import %s: %v", spec.Path.Value, err) + } + name := path[strings.LastIndex(path, "/")+1:] + if spec.Name != nil { + name = spec.Name.Name + } + imports[name] = path + } + + var activated *ast.FuncDecl + for _, decl := range file.Decls { + if fn, ok := decl.(*ast.FuncDecl); ok && fn.Name.Name == "activated" { + activated = fn + } + } + if activated == nil { + t.Fatalf("%s has no `activated` function: the repair opener's activation moved and this guard no longer looks at it", beadMutatingStoreFile) + } + + called := false + ast.Inspect(activated, func(n ast.Node) bool { + call, ok := n.(*ast.CallExpr) + if !ok { + return true + } + sel, ok := call.Fun.(*ast.SelectorExpr) + if !ok { + return true + } + if ident, ok := sel.X.(*ast.Ident); ok && imports[ident.Name] == versionedHistoryPkg && sel.Sel.Name == "ActivateStore" { + called = true + } + return true + }) + if !called { + t.Errorf("`activated` in %s never calls versionedhistory.ActivateStore: every repair that mutates issues through it "+ + "mints NO issue_versions row while `bd doctor --fix` reports success. Apply the shared activation there; "+ + "it is the same rule as the command surface and package main cannot be imported to reuse it.", beadMutatingStoreFile) + } +} diff --git a/cmd/bd/events_journal_construction_test.go b/cmd/bd/events_journal_construction_test.go index 0a2f5c345..fbf8682d9 100644 --- a/cmd/bd/events_journal_construction_test.go +++ b/cmd/bd/events_journal_construction_test.go @@ -89,15 +89,21 @@ var qualifiedActivationCalls = map[string]map[string]bool{ // scannedPackages are the directories searched for construction sites: every // package that builds a store or provider for the bd binary, plus the standalone -// embedded-Dolt utility so it is accounted for rather than merely unnoticed. -// Each maps to the prefix its sites are keyed under. +// embedded-Dolt utility and the Go SDK's own opens (the repository root), so they +// are accounted for rather than merely unnoticed. Each maps to the prefix its +// sites are keyed under. var scannedPackages = map[string]string{ ".": "", "doctor": "doctor/", "doctor/fix": "doctor/fix/", "../../internal/storage/embeddeddolt/cmd": "embeddeddolt-cmd/", + "../..": "sdk/", } +// sdkOpenExemptionReason is the one reason the Go SDK's five opens share. It has +// to be true of both per-store switches, because the exemption list is shared. +const sdkOpenExemptionReason = "SDK entry point (library surface, not the bd binary): applies neither per-store switch, the events journal's or versioned history's; the bd binary's factories do" + // constructionExemptions are construction sites that legitimately do NOT // activate the events journal, each with a reason. Keyed by // "/:" so the two build-tag twins of a factory are @@ -172,6 +178,17 @@ var constructionExemptions = map[string]string{ // A standalone developer utility binary, not bd. It has no workspace config // to read and never runs as part of a bd command. "embeddeddolt-cmd/main.go:main": "standalone embeddeddolt debug utility, not the bd binary; no workspace config and no bd command context", + + // The Go SDK's own opens (the repository root's beads package). They are a + // library surface that embedders call directly, and they build stores without + // applying either per-store switch. They are declared here so that a new SDK + // open is noticed rather than missed. Making them apply the switches would + // change the library's behavior for every embedder, so it is a separate change. + "sdk/beads.go:Open": sdkOpenExemptionReason, + "sdk/beads.go:OpenFromConfig": sdkOpenExemptionReason, + "sdk/beads.go:OpenGated": sdkOpenExemptionReason, + "sdk/beads_cgo.go:OpenBestAvailableWith": sdkOpenExemptionReason, + "sdk/beads_nocgo.go:OpenBestAvailableWith": sdkOpenExemptionReason, } func TestEveryStoreConstructionActivatesTheEventsJournal(t *testing.T) { @@ -235,6 +252,9 @@ func exemptionFor(key string) (exemption, reason string, ok bool) { type constructionSite struct { constructors []string activates bool + // activatesVersionedHistory is the same question for the versioned-history + // guard, which reuses this scan (see versioned_history_construction_test.go). + activatesVersionedHistory bool } // scanStoreConstructionSites parses each scanned package and returns one entry @@ -364,10 +384,16 @@ func inspectConstructionSite(node ast.Node, imports map[string]string) *construc if qualifiedActivationCalls[path][fun.Sel.Name] { site.activates = true } + if qualifiedVersionedHistoryActivationCalls[path][fun.Sel.Name] { + site.activatesVersionedHistory = true + } case *ast.Ident: if activationCalls[fun.Name] { site.activates = true } + if versionedHistoryActivationCalls[fun.Name] { + site.activatesVersionedHistory = true + } } return true }) diff --git a/cmd/bd/migrate_personal.go b/cmd/bd/migrate_personal.go index bb79b51bc..619cbf9be 100644 --- a/cmd/bd/migrate_personal.go +++ b/cmd/bd/migrate_personal.go @@ -25,8 +25,10 @@ import ( // workspace and then deletes them from the project. Those are ordinary bead // mutations in another workspace, so they belong in that workspace's events // journal if it has one enabled — and the setting is read from there, not from -// the workspace bd was launched in. +// the workspace bd was launched in. The same goes for its issue-version history, +// which the store's own settings row decides. func openMigrationPlanningStore(ctx context.Context, planningBeadsDir string) (s storage.DoltStorage, err error) { + defer func() { s, err = activateVersionedHistoryStore(ctx, s, err) }() defer func() { s, err = activateEventsJournalStore(planningBeadsDir, s, err) }() return dolt.New(ctx, &dolt.Config{ Path: doltserver.ResolveDoltDir(planningBeadsDir), diff --git a/cmd/bd/storage_chain.go b/cmd/bd/storage_chain.go index 0c420d5c2..85e554b53 100644 --- a/cmd/bd/storage_chain.go +++ b/cmd/bd/storage_chain.go @@ -23,6 +23,13 @@ import ( // storage spans measure pure DB time without hook-firing overhead. The policy // sits directly beneath hooks so serve's one hook-layer peel retains it. // +// Versioned-history activation is NOT applied here. It used to be, and this runs +// on the one store a command opens for its own workspace, so every store a +// routed write opens for another workspace missed it. It now happens in the +// factories that construct a store, on the raw store before anything can wrap it +// (see versioned_history.go) -- which is also why the decorators above, none of +// which forwards SetVersionedHistoryEnabled, cannot get in its way. +// // Extracted from main.go's PersistentPreRunE so the chain composition is // unit-testable — the bug this PR fixes was a missing WrapStorage call, // and the regression class deserves test coverage. diff --git a/cmd/bd/store_factory.go b/cmd/bd/store_factory.go index d9a720e69..22ff0edb2 100644 --- a/cmd/bd/store_factory.go +++ b/cmd/bd/store_factory.go @@ -56,6 +56,7 @@ func usesProxiedServer() bool { // registered backend decides for itself what read-only means, and refusing to // offer it the setting would be this factory guessing on its behalf. func newRegisteredBackendStore(ctx context.Context, name, beadsDir string, readOnly bool) (s storage.DoltStorage, err error) { + defer func() { s, err = activateVersionedHistoryStore(ctx, s, err) }() defer func() { s, err = activateEventsJournalStore(beadsDir, s, err) }() backend, ok := backends.Lookup(name) if !ok { @@ -68,6 +69,7 @@ func newRegisteredBackendStore(ctx context.Context, name, beadsDir string, readO } func newDoltStore(ctx context.Context, cfg *dolt.Config) (s storage.DoltStorage, err error) { + defer func() { s, err = activateVersionedHistoryStore(ctx, s, err) }() defer func() { s, err = activateEventsJournalStore(cfg.BeadsDir, s, err) }() if cfg.ProxiedServer { return nil, errProxiedStoreUnrouted() @@ -176,6 +178,7 @@ func newExistingDoltStoreFromConfig(ctx context.Context, beadsDir string) (stora // openDoltStoreFromConfig is the body of newDoltStoreFromConfig and // newExistingDoltStoreFromConfig; requireExisting selects the latter. func openDoltStoreFromConfig(ctx context.Context, beadsDir string, requireExisting bool) (s storage.DoltStorage, err error) { + defer func() { s, err = activateVersionedHistoryStore(ctx, s, err) }() defer func() { s, err = activateEventsJournalStore(beadsDir, s, err) }() cfg, err := configfile.Load(beadsDir) if err != nil { diff --git a/cmd/bd/store_factory_nocgo.go b/cmd/bd/store_factory_nocgo.go index 566953d21..48dc24981 100644 --- a/cmd/bd/store_factory_nocgo.go +++ b/cmd/bd/store_factory_nocgo.go @@ -34,6 +34,7 @@ func usesProxiedServer() bool { // exists here too so the root pre-run's registry arm has one activating // construction path in both builds. func newRegisteredBackendStore(ctx context.Context, name, beadsDir string, readOnly bool) (s storage.DoltStorage, err error) { + defer func() { s, err = activateVersionedHistoryStore(ctx, s, err) }() defer func() { s, err = activateEventsJournalStore(beadsDir, s, err) }() backend, ok := backends.Lookup(name) if !ok { @@ -48,6 +49,7 @@ func newRegisteredBackendStore(ctx context.Context, name, beadsDir string, readO // newDoltStore applies events-journal activation for the same reason its CGO // twin does — see the note at the top of events_journal.go. func newDoltStore(ctx context.Context, cfg *dolt.Config) (s storage.DoltStorage, err error) { + defer func() { s, err = activateVersionedHistoryStore(ctx, s, err) }() defer func() { s, err = activateEventsJournalStore(cfg.BeadsDir, s, err) }() if cfg.ProxiedServer { return nil, errProxiedStoreUnrouted() @@ -65,6 +67,7 @@ func acquireEmbeddedLock(_ string, _ bool) (util.Unlocker, error) { // newDoltStoreFromConfig creates a SQL-server-backed storage backend from config. func newDoltStoreFromConfig(ctx context.Context, beadsDir string) (s storage.DoltStorage, err error) { + defer func() { s, err = activateVersionedHistoryStore(ctx, s, err) }() defer func() { s, err = activateEventsJournalStore(beadsDir, s, err) }() cfg, err := configfile.Load(beadsDir) if err != nil { diff --git a/cmd/bd/uow_factory.go b/cmd/bd/uow_factory.go index 576cb5d99..802e91f75 100644 --- a/cmd/bd/uow_factory.go +++ b/cmd/bd/uow_factory.go @@ -463,6 +463,7 @@ func resolveServerModeUOWTopologyWithTransportResolver(ctx context.Context, bead // the paths it resolves (root, log) are the same ones proxied mode uses because // both modes root their server at the same directory. func newExternalProxiedServerUOWProvider(ctx context.Context, beadsDir string, topology sqlServerUOWTopology, opts ...uow.ProviderOption) (p uow.UnitOfWorkProvider, err error) { + defer func() { p, err = activateVersionedHistoryProvider(ctx, p, err) }() defer func() { p, err = activateEventsJournalProvider(ctx, beadsDir, p, err) }() rootPath, err := resolveProxiedServerRootPath(beadsDir) if err != nil { @@ -503,6 +504,7 @@ func newExternalProxiedServerUOWProvider(ctx context.Context, beadsDir string, t } func newManagedProxiedServerUOWProvider(ctx context.Context, beadsDir string, topology sqlServerUOWTopology, opts ...uow.ProviderOption) (p uow.UnitOfWorkProvider, err error) { + defer func() { p, err = activateVersionedHistoryProvider(ctx, p, err) }() defer func() { p, err = activateEventsJournalProvider(ctx, beadsDir, p, err) }() // Resolve and hardened-probe the external dolt binary before spawning // it: an env/sidecar override that is explicitly named but broken diff --git a/cmd/bd/version.go b/cmd/bd/version.go index 3bf32c13f..cc2ca37a6 100644 --- a/cmd/bd/version.go +++ b/cmd/bd/version.go @@ -29,6 +29,12 @@ var versionCmd = &cobra.Command{ Short: "Print version information", SilenceUsage: true, SilenceErrors: true, + // Accept args only so the "did you mean bd versions" guard below can see + // them. `bd version` names bd's own build; `bd versions ` lists a + // bead's recorded versions. One character apart, unrelated meanings -- + // without this a mistyped `bd version bd-123` silently prints the build + // and looks like the bead simply has no versions. + Args: cobra.ArbitraryArgs, RunE: func(cmd *cobra.Command, args []string) error { evt := metrics.NewCommandEvent("version") defer func() { @@ -37,6 +43,12 @@ var versionCmd = &cobra.Command{ } }() + if len(args) > 0 { + return HandleErrorRespectJSON( + "bd version prints bd's own version information and takes no arguments.\n"+ + "Did you mean: bd versions %s", args[0]) + } + commit := resolveCommitHash() branch := resolveBranch() diff --git a/cmd/bd/versioned_history.go b/cmd/bd/versioned_history.go new file mode 100644 index 000000000..d39028c5e --- /dev/null +++ b/cmd/bd/versioned_history.go @@ -0,0 +1,63 @@ +package main + +import ( + "context" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/uow" + "github.com/steveyegge/beads/internal/versionedhistory" +) + +// Versioned-history activation, applied in the SAME place events-journal +// activation is (see the note at the top of events_journal.go): the factories +// that construct a store or a unit-of-work provider -- store_factory.go, +// store_factory_nocgo.go, uow_factory.go, the personal-migration planning store, +// and bd doctor's repair handlers through the same shared package. Every +// plumbing bd opens goes through one of those, so a routed write cannot acquire a +// store that silently records no history. +// +// It did not start there. It was first applied in wireStorageDecorators, which +// the root pre-run calls on the ONE store a command opens for its own workspace. +// A routed update or close (routed.go), a routed create (`bd create --repo`) and +// the direct-mode open each build their OWN write-capable store through +// newDoltStoreFromConfig, so with the switch on they recorded nothing while the +// command reported success -- the "configured on, actually off" class this +// feature exists to prevent. TestEveryStoreConstructionActivatesVersionedHistory +// keeps it in the factories. +// +// Each store answers for itself. The switch is a row of THAT store's own config +// table (what `bd config set versioned-history.enabled true` writes), OR'd with +// the process-wide environment, which can only turn recording on -- so a write +// routed into rig B reads rig B's row, never the launching workspace's. Nothing +// here takes a beadsDir for that reason: the store already knows where it lives. +// +// The two defers in a factory are ordered on purpose. Deferred calls run +// last-in-first-out, and this one is registered ABOVE the events-journal +// activation, so the journal runs first. A journal activation that fails closes +// the store and returns (nil, err); this one then passes that through instead of +// reading a row out of a closed store. +// +// The policy itself lives in internal/versionedhistory so bd doctor's fix +// package, which cannot import package main, applies the identical rule. + +// activateVersionedHistoryStore is versionedhistory.ActivateStore under the name +// the construction guard matches. Kept as a wrapper rather than called directly so +// cmd/bd has one spelling of the idiom and the guard has one name to look for, +// exactly as activateEventsJournalStore does for its feature. +func activateVersionedHistoryStore(ctx context.Context, s storage.DoltStorage, err error) (storage.DoltStorage, error) { + return versionedhistory.ActivateStore(ctx, s, err) +} + +// activateVersionedHistoryProvider is the same for a unit-of-work provider -- bd's +// second write plumbing, with its own transactions and its own activation switch, +// which proxied-server mode and `bd serve` against a server-mode workspace write +// through instead of a DoltStorage. +// +// A provider has no DoltStorage to ask for a settings plane, so its row is read +// through a unit of work at construction (versionedhistory.ProviderSetting): one +// pooled connection, one SELECT, rolled back. The provider constructors have +// already opened the pool and run their schema setup by then, so the read starts +// no server and creates nothing. +func activateVersionedHistoryProvider(ctx context.Context, p uow.UnitOfWorkProvider, err error) (uow.UnitOfWorkProvider, error) { + return versionedhistory.ActivateProvider(ctx, p, err) +} diff --git a/cmd/bd/versioned_history_construction_test.go b/cmd/bd/versioned_history_construction_test.go new file mode 100644 index 000000000..c225bfbd0 --- /dev/null +++ b/cmd/bd/versioned_history_construction_test.go @@ -0,0 +1,322 @@ +package main + +import ( + "fmt" + "go/ast" + "go/parser" + "go/token" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/journalscan" +) + +// The twin of TestEveryStoreConstructionActivatesTheEventsJournal, for the other +// per-store switch bd resolves when it opens a store: dual-write issue-version +// history. +// +// It exists for the same reason and has the same failure shape. Activation that +// is applied by whichever command happens to use a store, rather than by the +// factory that builds it, reaches only the stores those commands open. The first +// cut of this feature was wired in wireStorageDecorators, which the root pre-run +// calls on the ONE store a command opens for its own workspace. A routed update +// and close (routed.go), a routed create (`bd create --repo`) and the direct-mode +// open each build their OWN write-capable store through newDoltStoreFromConfig, +// ran with history off, and reported success -- leaving issue_versions with +// silent gaps in a workspace that had asked for it. Nothing in the process can +// tell: a missing version row looks exactly like a quiet one. +// +// The scan, the constructor set, the scanned packages and the exemption list are +// the events-journal guard's, shared rather than copied. A store that refuses +// writes, a store used only for a config read and a store that writes only +// workspace state are exempt from both for the same reason: there is no bead +// mutation for a journal row OR a version row to accompany, and one list means +// an exemption cannot be true of one feature and quietly forgotten for the other. +// +// Limits, stated rather than papered over. The check is syntactic and local, like +// its twin: a function that calls a store or provider constructor must call an +// activation helper in its OWN body. bd doctor is exempted as a package by the +// shared list, so the repair opener in doctor/fix/store.go is held to the +// property by its own test in that package +// (TestVersionedHistoryDoctorRepairOpenerUsesTheSharedActivation). + +// versionedHistoryActivationCalls are the cmd/bd helpers that apply a store's own +// configured switch to the thing just constructed. +var versionedHistoryActivationCalls = map[string]bool{ + "activateVersionedHistoryStore": true, + "activateVersionedHistoryProvider": true, +} + +// qualifiedVersionedHistoryActivationCalls are the same, keyed by import path and +// function, for callers that cannot import package main. +// +// Apply is deliberately NOT here, unlike its events-journal twin. Apply binds a +// value somebody else resolved; it is ActivateStore/ActivateProvider that resolve +// the switch, so a caller that only Applies has decided the answer by hand. +var qualifiedVersionedHistoryActivationCalls = map[string]map[string]bool{ + "github.com/steveyegge/beads/internal/versionedhistory": { + "ActivateStore": true, + "ActivateProvider": true, + }, +} + +// versionedHistoryConstructionViolations returns one message per non-exempt +// construction site that never calls a versioned-history activation helper. It +// is the guard's whole verdict, factored out so the mutation probe below asks the +// SAME question of a factory whose activation has been deleted. +func versionedHistoryConstructionViolations(sites map[string]*constructionSite) []string { + var out []string + for _, key := range sortedKeys(sites) { + site := sites[key] + if _, reason, ok := exemptionFor(key); ok { + if strings.TrimSpace(reason) == "" { + out = append(out, fmt.Sprintf("%s has an empty exemption reason", key)) + } + continue + } + if !site.activatesVersionedHistory { + out = append(out, fmt.Sprintf("%s constructs a store or unit-of-work provider (%s) but never calls %v on it -- "+ + "every mutation through it mints NO issue_versions row while the command reports success. "+ + "Apply activation in this function (see newDoltStore for the idiom), or add it to "+ + "constructionExemptions with a reason that is true of issue versions as well as the events journal.", + key, strings.Join(site.constructors, ", "), sortedKeys(versionedHistoryActivationCalls))) + } + } + return out +} + +func TestEveryStoreConstructionActivatesVersionedHistory(t *testing.T) { + sites := scanStoreConstructionSites(t) + if len(sites) == 0 { + t.Fatal("found no store construction sites -- the constructor set or the scan changed and this guard is not actually running") + } + checked := 0 + for key := range sites { + if _, _, exempt := exemptionFor(key); !exempt { + checked++ + } + } + if checked == 0 { + t.Fatal("every construction site is exempt -- the guard is not actually checking anything") + } + for _, v := range versionedHistoryConstructionViolations(sites) { + t.Error(v) + } +} + +// TestVersionedHistoryGuardSeesTheFactoriesEveryWriteUses pins that the scan +// really reaches the sites this feature was wrong about, in BOTH build-tag +// twins. A guard that quietly stops seeing a factory reports success while +// checking nothing, which is the worst way for one to fail. +// +// routed.go, create.go and direct_mode.go do not construct a store themselves; +// they call newDoltStoreFromConfig, so it is that factory the guard has to hold. +func TestVersionedHistoryGuardSeesTheFactoriesEveryWriteUses(t *testing.T) { + sites := scanStoreConstructionSites(t) + for _, key := range versionedHistoryFactoryKeys { + site, ok := sites[key] + if !ok { + t.Errorf("the construction scan no longer sees %s", key) + continue + } + if _, _, exempt := exemptionFor(key); exempt { + t.Errorf("%s is exempt from the guard, but it opens a write-capable plumbing", key) + continue + } + if !site.activatesVersionedHistory { + t.Errorf("%s does not activate versioned history", key) + } + } +} + +// versionedHistoryFactoryKeys are the construction sites every routed, direct and +// served write goes through. Each is named so that a factory disappearing from +// the scan fails a test instead of shrinking the guard. +var versionedHistoryFactoryKeys = []string{ + "store_factory.go:newDoltStore", + // Both routed-open wrappers (newDoltStoreFromConfig and + // newExistingDoltStoreFromConfig) delegate to this activating body. + "store_factory.go:openDoltStoreFromConfig", + "store_factory.go:newRegisteredBackendStore", + "store_factory_nocgo.go:newDoltStore", + "store_factory_nocgo.go:newDoltStoreFromConfig", + "store_factory_nocgo.go:newRegisteredBackendStore", + "migrate_personal.go:openMigrationPlanningStore", + "uow_factory.go:newExternalProxiedServerUOWProvider", + "uow_factory.go:newManagedProxiedServerUOWProvider", +} + +// TestVersionedHistoryConstructionGuardFailsWhenAFactoryLosesItsActivation is the +// mutation probe. For each guarded factory it takes the factory's real source, +// deletes the versioned-history activation from THAT function alone, re-runs the +// scanner's own inspection on the result, and asks the guard's own verdict +// function whether it now objects. +// +// It answers a different question from the guard above. The guard says "every +// factory activates today". This says "the guard would notice if one stopped": +// a detector that matched too loosely (any function that merely mentions the +// helper's name, a substring in a comment, the wrong build-tag twin) would pass +// the guard forever and catch nothing. The probe also checks that the deletion +// removed the versioned-history activation ONLY, leaving the events-journal +// activation standing, so a probe that over-deleted cannot pass for the right +// reason. +func TestVersionedHistoryConstructionGuardFailsWhenAFactoryLosesItsActivation(t *testing.T) { + sites := scanStoreConstructionSites(t) + + probed := 0 + for _, key := range versionedHistoryFactoryKeys { + site, ok := sites[key] + if !ok { + t.Errorf("the construction scan no longer sees %s", key) + continue + } + fn, imports := parseConstructionSiteFunc(t, key) + before := inspectConstructionSite(fn, imports) + if before == nil || !before.activatesVersionedHistory { + t.Errorf("%s has no versioned-history activation to delete, so this probe cannot show that removing it is noticed", key) + continue + } + + dropped := dropVersionedHistoryActivation(fn, imports) + if dropped == 0 { + t.Errorf("%s: the scanner sees an activation but the probe found no statement to delete -- the probe and the scanner disagree about what activation looks like", key) + continue + } + after := inspectConstructionSite(fn, imports) + if after == nil { + t.Errorf("%s: deleting the activation also deleted the construction itself, so the probe proved nothing; the factory must keep the open and its activation as separate statements", key) + continue + } + if after.activates != site.activates { + t.Errorf("%s: the probe changed the events-journal activation (%v -> %v); it must remove the versioned-history activation only", key, site.activates, after.activates) + } + if after.activatesVersionedHistory { + t.Errorf("%s: still reads as activating versioned history after its activation was deleted -- the detector is too loose", key) + continue + } + violations := versionedHistoryConstructionViolations(map[string]*constructionSite{key: after}) + if len(violations) == 0 { + t.Errorf("%s: the guard raised no violation for a factory whose versioned-history activation was deleted", key) + continue + } + if !strings.Contains(violations[0], key) { + t.Errorf("%s: the violation does not name the factory: %q", key, violations[0]) + } + probed++ + } + if probed != len(versionedHistoryFactoryKeys) { + t.Errorf("probed %d of %d guarded factories", probed, len(versionedHistoryFactoryKeys)) + } +} + +// parseConstructionSiteFunc re-parses the source file a construction-site key +// names and returns the function declaration together with that file's import +// table. It parses afresh on every call because the probe mutates the tree. +func parseConstructionSiteFunc(t *testing.T, key string) (*ast.FuncDecl, map[string]string) { + t.Helper() + dir, file, name := "", "", "" + for candidatePrefix, candidateDir := range prefixToScannedDir() { + if strings.HasPrefix(key, candidatePrefix) && len(candidatePrefix) >= len(dir) { + rest := strings.TrimPrefix(key, candidatePrefix) + if before, after, ok := strings.Cut(rest, ":"); ok && !strings.Contains(before, "/") { + dir, file, name = candidateDir, before, after + } + } + } + if file == "" { + t.Fatalf("cannot locate the source of construction site %q", key) + } + fset := token.NewFileSet() + parsed, err := parser.ParseFile(fset, filepath.Join(dir, file), nil, 0) + if err != nil { + t.Fatalf("parse %s/%s: %v", dir, file, err) + } + imports := fileImports(t, parsed) + for _, decl := range parsed.Decls { + fn, ok := decl.(*ast.FuncDecl) + if !ok { + continue + } + got := fn.Name.Name + if fn.Recv != nil && len(fn.Recv.List) > 0 { + if recv := journalscan.ReceiverTypeName(fn.Recv.List[0].Type); recv != "" { + got = recv + "." + got + } + } + if got == name { + return fn, imports + } + } + t.Fatalf("construction site %q: no function %q in %s/%s", key, name, dir, file) + return nil, nil +} + +// prefixToScannedDir inverts scannedPackages: the key prefix each package's sites +// carry, mapped back to the directory the scanner read them from. +func prefixToScannedDir() map[string]string { + out := make(map[string]string, len(scannedPackages)) + for dir, prefix := range scannedPackages { + out[prefix] = dir + } + return out +} + +// dropVersionedHistoryActivation deletes, in place and at any depth, every simple +// statement in fn that calls a versioned-history activation helper, and returns +// how many it removed. Compound statements are recursed into rather than removed, +// so the surrounding control flow (and any events-journal activation beside it) +// survives. +func dropVersionedHistoryActivation(fn *ast.FuncDecl, imports map[string]string) int { + dropped := 0 + filter := func(list []ast.Stmt) []ast.Stmt { + out := make([]ast.Stmt, 0, len(list)) + for _, st := range list { + switch st.(type) { + case *ast.DeferStmt, *ast.ExprStmt, *ast.AssignStmt, *ast.GoStmt: + if callsVersionedHistoryActivation(st, imports) { + dropped++ + continue + } + } + out = append(out, st) + } + return out + } + ast.Inspect(fn, func(n ast.Node) bool { + switch b := n.(type) { + case *ast.BlockStmt: + b.List = filter(b.List) + case *ast.CaseClause: + b.Body = filter(b.Body) + case *ast.CommClause: + b.Body = filter(b.Body) + } + return true + }) + return dropped +} + +func callsVersionedHistoryActivation(node ast.Node, imports map[string]string) bool { + found := false + ast.Inspect(node, func(n ast.Node) bool { + call, ok := n.(*ast.CallExpr) + if !ok { + return true + } + switch fun := call.Fun.(type) { + case *ast.Ident: + if versionedHistoryActivationCalls[fun.Name] { + found = true + } + case *ast.SelectorExpr: + if ident, ok := fun.X.(*ast.Ident); ok { + if path, imported := imports[ident.Name]; imported && qualifiedVersionedHistoryActivationCalls[path][fun.Sel.Name] { + found = true + } + } + } + return !found + }) + return found +} diff --git a/cmd/bd/versioned_history_e2e_test.go b/cmd/bd/versioned_history_e2e_test.go new file mode 100644 index 000000000..7ad763d25 --- /dev/null +++ b/cmd/bd/versioned_history_e2e_test.go @@ -0,0 +1,383 @@ +//go:build cgo + +package main + +import ( + "bytes" + "database/sql" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/embeddeddolt" +) + +// The end-to-end claim of versioned-history activation, against real embedded +// stores and the table it exists to fill: issue_versions. +// +// It is end-to-end on purpose, for the reason the events-journal E2Es are. +// Activation failing is INVISIBLE from inside the process -- the command +// succeeds, the write lands, every exit code is 0 -- and only a later count of +// version rows shows that nothing was recorded. This feature was wrong that way +// once already: the switch was read and applied where the root pre-run wraps the +// command's own store, which is the one store a routed update, a routed close and +// `bd create --repo` do NOT use. Each of those builds its own write-capable store +// for the target workspace, ran with history off, and reported success. A green +// unit suite was consistent with the feature doing nothing. +// +// The switch is a property of the STORE. `bd config set versioned-history.enabled +// true` writes a row of the workspace's own config table, and a store answers +// for itself: a write routed into rig B reads rig B's row, never the launching +// workspace's. The environment (and a hand-edited yaml value) is process-wide and +// can only turn recording ON; nothing but the row can turn a store off. +// +// Runs in the embedded-Dolt lane only (BEADS_TEST_EMBEDDED_DOLT=1), like every +// other test that builds and drives a real bd binary. The by-construction +// coverage that runs everywhere is the construction guard and the resolver's own +// tests. + +// vhEnvVar is viper's spelling of versioned-history.enabled: the BD_ prefix, with +// dots and hyphens mapped to underscores. +const vhEnvVar = "BD_VERSIONED_HISTORY_ENABLED" + +func vhRequireEmbeddedLane(t *testing.T) { + t.Helper() + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } +} + +// vhEnv is a subprocess environment in which nothing from the developer's own +// shell can turn the feature on or off: only what a step passes explicitly. +func vhEnv(dir string, extra ...string) []string { + env := envWithout(bdEnv(dir), vhEnvVar) + env = envWithout(env, "BD_EVENTS_JOURNAL") + return append(env, extra...) +} + +// vhBD runs bd and returns stdout and stderr separately. It fails the test on a +// non-zero exit. +func vhBD(t *testing.T, bd, dir string, extraEnv []string, args ...string) (stdout, stderr string) { + t.Helper() + cmd := exec.Command(bd, args...) + cmd.Dir = dir + cmd.Env = vhEnv(dir, extraEnv...) + var out, errOut bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &errOut + if err := cmd.Run(); err != nil { + t.Fatalf("bd %s in %s: %v\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), dir, err, out.String(), errOut.String()) + } + return out.String(), errOut.String() +} + +type vhWorkspace struct { + name string + dir string + beadsDir string + database string +} + +func vhNewWorkspace(t *testing.T, bd, dir, prefix string) vhWorkspace { + t.Helper() + if err := os.MkdirAll(dir, 0o750); err != nil { + t.Fatal(err) + } + initGitRepoAt(t, dir) + runBDInit(t, bd, dir, "--prefix", prefix, "--skip-hooks", "--skip-agents") + return vhWorkspace{name: prefix, dir: dir, beadsDir: filepath.Join(dir, ".beads"), database: prefix} +} + +// vhQuery runs one query against a workspace's embedded database. Callers must +// not have a bd process running against it: the embedded store is single-writer. +func vhQuery[T any](t *testing.T, w vhWorkspace, scan func(*sql.Row) (T, error), query string, args ...any) T { + t.Helper() + db, cleanup, err := embeddeddolt.OpenSQL(t.Context(), filepath.Join(w.beadsDir, "embeddeddolt"), w.database, "main") + if err != nil { + t.Fatalf("OpenSQL(%s): %v", w.database, err) + } + defer func() { _ = cleanup() }() + got, err := scan(db.QueryRowContext(t.Context(), query, args...)) + if err != nil { + t.Fatalf("%s: %v", query, err) + } + return got +} + +// vhVersions counts the issue_versions rows in one workspace's database. +func vhVersions(t *testing.T, w vhWorkspace) int { + t.Helper() + return vhQuery(t, w, func(r *sql.Row) (int, error) { + var n int + return n, r.Scan(&n) + }, "SELECT COUNT(*) FROM issue_versions") +} + +// vhRow reads the switch's row out of one workspace's config table. present is +// false when there is no such row, which is the state of a workspace that never +// ran `bd config set`. +func vhRow(t *testing.T, w vhWorkspace) (value string, present bool) { + t.Helper() + type result struct { + value string + present bool + } + got := vhQuery(t, w, func(r *sql.Row) (result, error) { + var v string + err := r.Scan(&v) + if errors.Is(err, sql.ErrNoRows) { + return result{}, nil + } + return result{v, true}, err + }, "SELECT value FROM config WHERE `key` = ?", versionedHistorySettingKey) + return got.value, got.present +} + +// TestVersionedHistoryConfigSetWritesTheStoreRow is the front door, and D5's +// regression pin for the plane #6661 chose. `bd config set` has to write the +// database row the stores read, and must not write config.yaml: the key is not a +// yaml-only key, and making it one would move the switch off the store and back +// to the per-process plane this feature was moved away from. +func TestVersionedHistoryConfigSetWritesTheStoreRow(t *testing.T) { + vhRequireEmbeddedLane(t) + bd := buildEmbeddedBD(t) + w := vhNewWorkspace(t, bd, filepath.Join(t.TempDir(), "ws"), "vhc") + + if value, present := vhRow(t, w); present { + t.Fatalf("a fresh workspace already has a %s row (%q); the test's premise is wrong", versionedHistorySettingKey, value) + } + + vhBD(t, bd, w.dir, nil, "config", "set", versionedHistorySettingKey, "true") + + if value, present := vhRow(t, w); !present || value != "true" { + t.Errorf("config table holds (%q, present=%v) for %s after `bd config set ... true`, want (\"true\", true)", value, present, versionedHistorySettingKey) + } + for _, name := range []string{"config.yaml", "config.local.yaml"} { + raw, err := os.ReadFile(filepath.Join(w.beadsDir, name)) + if err != nil { + continue + } + if strings.Contains(string(raw), "versioned-history") { + t.Errorf("`bd config set %s` wrote to %s; the switch is a store row, not a yaml key:\n%s", versionedHistorySettingKey, name, raw) + } + } + stdout, _ := vhBD(t, bd, w.dir, nil, "config", "get", versionedHistorySettingKey) + if !strings.Contains(stdout, "true") { + t.Errorf("`bd config get` does not read back the value `bd config set` wrote: %q", stdout) + } + + vhBD(t, bd, w.dir, nil, "config", "set", versionedHistorySettingKey, "false") + if value, present := vhRow(t, w); !present || value != "false" { + t.Errorf("config table holds (%q, present=%v) after `bd config set ... false`, want (\"false\", true): the row is how a store is turned off", value, present) + } +} + +// vhMatrix is the fixture for the routed-write matrix: a launching workspace +// (the "town") and two rigs it routes into by issue-ID prefix. +type vhMatrix struct { + bd string + town, rigA, rigB vhWorkspace +} + +func (m vhMatrix) workspaces() []vhWorkspace { return []vhWorkspace{m.town, m.rigA, m.rigB} } + +func (m vhMatrix) setRow(t *testing.T, w vhWorkspace, value string) { + t.Helper() + vhBD(t, m.bd, w.dir, nil, "config", "set", versionedHistorySettingKey, value) +} + +// routedUpdate writes into a rig FROM THE TOWN, through the prefix route +// (routed.go: newDoltStoreFromConfig on the target's .beads). +func (m vhMatrix) routedUpdate(t *testing.T, id, title string, env ...string) { + t.Helper() + vhBD(t, m.bd, m.town.dir, env, "update", id, "--title", title) +} + +// routedClose is the other prefix-routed write. +func (m vhMatrix) routedClose(t *testing.T, id string, env ...string) { + t.Helper() + vhBD(t, m.bd, m.town.dir, env, "close", id) +} + +// routedCreate writes a new issue into a rig FROM THE TOWN, through --repo +// (create.go: newDoltStoreFromConfig on the target's .beads). +func (m vhMatrix) routedCreate(t *testing.T, target vhWorkspace, title string, env ...string) { + t.Helper() + vhBD(t, m.bd, m.town.dir, env, "create", "--silent", "--repo", target.dir, title) +} + +func (m vhMatrix) create(t *testing.T, w vhWorkspace, title string) string { + t.Helper() + out, _ := vhBD(t, m.bd, w.dir, nil, "create", "--silent", title) + return strings.TrimSpace(out) +} + +// expectDelta runs a write and asserts how many version rows EVERY workspace +// gained, so a write that lands in the wrong place fails as loudly as one that +// records nothing. want is keyed by workspace name; anything unlisted must gain +// none -- including the launching workspace, which never records a write that +// was routed into another one. +func (m vhMatrix) expectDelta(t *testing.T, step string, want map[string]int, write func()) { + t.Helper() + before := map[string]int{} + for _, w := range m.workspaces() { + before[w.name] = vhVersions(t, w) + } + write() + var counts []string + for _, w := range m.workspaces() { + after := vhVersions(t, w) + counts = append(counts, fmt.Sprintf("%s %d->%d", w.name, before[w.name], after)) + if got := after - before[w.name]; got != want[w.name] { + t.Errorf("%s: %s gained %d issue_versions row(s), want %d (before %d, after %d)", + step, w.name, got, want[w.name], before[w.name], after) + } + } + // The measured counts, not just the verdict: with -v this is the + // before/after evidence for every routed write. + t.Logf("%s: issue_versions rows: %s", step, strings.Join(counts, ", ")) +} + +// newVHMatrix builds the town and its two rigs, with history off everywhere and +// a routes file mapping each rig's prefix to it. +func newVHMatrix(t *testing.T) vhMatrix { + t.Helper() + bd := buildEmbeddedBD(t) + root := t.TempDir() + m := vhMatrix{bd: bd} + m.town = vhNewWorkspace(t, bd, filepath.Join(root, "town"), "vht") + m.rigA = vhNewWorkspace(t, bd, filepath.Join(m.town.dir, "rig-a"), "vha") + m.rigB = vhNewWorkspace(t, bd, filepath.Join(m.town.dir, "rig-b"), "vhb") + routes := `{"prefix":"vha-","path":"rig-a"}` + "\n" + `{"prefix":"vhb-","path":"rig-b"}` + "\n" + if err := os.WriteFile(filepath.Join(m.town.beadsDir, "routes.jsonl"), []byte(routes), 0o644); err != nil { + t.Fatalf("write routes.jsonl: %v", err) + } + return m +} + +// TestVersionedHistoryFollowsTheStoreItWritesInto is the routed-write matrix, with +// a count of issue_versions rows in every workspace after every write. +// +// D1: with the TARGET's row on and the launching workspace's row absent, a routed +// update, a routed close and `bd create --repo` each add exactly one version row +// to the target. At the commit this was built on, they add none: the store they +// open is never activated. +// +// D2: the reverse. The launching workspace's row on and the target's off (or +// absent), env unset: the routed write adds none. A store answers for itself, so +// the town's answer cannot leak into a store it opened for someone else. +func TestVersionedHistoryFollowsTheStoreItWritesInto(t *testing.T) { + vhRequireEmbeddedLane(t) + m := newVHMatrix(t) + + // One issue in each rig, plus one to close, all created while history is off. + idA, idAClose, idB := m.create(t, m.rigA, "seed a"), m.create(t, m.rigA, "seed a close"), m.create(t, m.rigB, "seed b") + + m.expectDelta(t, "control: history off everywhere, routed update", nil, func() { m.routedUpdate(t, idA, "control") }) + + t.Run("D1 target row on, launching workspace row absent", func(t *testing.T) { + m.setRow(t, m.rigA, "true") + if _, present := vhRow(t, m.town); present { + t.Fatal("the launching workspace has a row; this subtest's premise is that it has none") + } + + // The rows the writes below expect a version for are created AFTER the setting + // is on, so they participate in history. The seeds above were created with it + // off, so they are legacy and design §16.2b's write fence skips an update-shaped + // write to them whole: that is pinned here, at the command level. + idAOn, idACloseOn := m.create(t, m.rigA, "seed a, created with history on"), m.create(t, m.rigA, "seed a close, created with history on") + m.expectDelta(t, "routed update of a seed created before history was enabled", nil, func() { m.routedUpdate(t, idA, "routed update a, legacy seed") }) + m.expectDelta(t, "routed close of a seed created before history was enabled", nil, func() { m.routedClose(t, idAClose) }) + + m.expectDelta(t, "routed update into the enabling rig", map[string]int{"vha": 1}, func() { m.routedUpdate(t, idAOn, "routed update a") }) + m.expectDelta(t, "routed close into the enabling rig", map[string]int{"vha": 1}, func() { m.routedClose(t, idACloseOn) }) + m.expectDelta(t, "bd create --repo into the enabling rig", map[string]int{"vha": 1}, func() { m.routedCreate(t, m.rigA, "routed create a") }) + // A second rig that never asked stays at zero while the first records. + m.expectDelta(t, "routed update into a rig that never enabled history", nil, func() { m.routedUpdate(t, idB, "routed update b") }) + m.expectDelta(t, "bd create --repo into a rig that never enabled history", nil, func() { m.routedCreate(t, m.rigB, "routed create b") }) + m.expectDelta(t, "direct write inside the enabling rig", map[string]int{"vha": 1}, func() { + vhBD(t, m.bd, m.rigA.dir, nil, "update", idAOn, "--title", "direct a") + }) + }) + + t.Run("flipping it back is a plain config change", func(t *testing.T) { + m.setRow(t, m.rigA, "false") + m.expectDelta(t, "routed update after setting the rig's row back to false", nil, func() { m.routedUpdate(t, idA, "after flip back") }) + m.setRow(t, m.rigA, "true") + // Created after the setting is on again, so it participates; idA is a legacy seed. + idAFlip := m.create(t, m.rigA, "seed a, created after enabling again") + m.expectDelta(t, "routed update after enabling again, with no restart of anything", map[string]int{"vha": 1}, func() { + m.routedUpdate(t, idAFlip, "after flip forward") + }) + m.setRow(t, m.rigA, "false") + }) + + t.Run("D2 launching workspace row on, target row off", func(t *testing.T) { + m.setRow(t, m.town, "true") + // The routed stores the town opens belong to workspaces that did not ask + // (rig A says false outright, rig B says nothing) and must not inherit the + // town's answer. + m.expectDelta(t, "routed update into a rig whose own row says false", nil, func() { m.routedUpdate(t, idA, "leak check update") }) + m.expectDelta(t, "bd create --repo into a rig whose own row says false", nil, func() { m.routedCreate(t, m.rigA, "leak check create") }) + m.expectDelta(t, "routed update into a rig with no row at all", nil, func() { m.routedUpdate(t, idB, "leak check b") }) + // The town's own store, in the town's own workspace, does record. + m.expectDelta(t, "the launching workspace's own write", map[string]int{"vht": 1}, func() { + m.create(t, m.town, "town's own issue") + }) + m.setRow(t, m.town, "false") + }) +} + +// TestVersionedHistoryEnvIsProcessWideAndOnlyTurnsRecordingOn is D3. The +// environment applies to every store the process opens, routed targets included; +// it can turn recording on for a store that never asked, and it can never turn it +// off for a store whose row says on. Both planes are OR'd, never ranked: the +// hazard is a write that fails to record, never one that records when it needn't. +func TestVersionedHistoryEnvIsProcessWideAndOnlyTurnsRecordingOn(t *testing.T) { + vhRequireEmbeddedLane(t) + m := newVHMatrix(t) + idA, idB := m.create(t, m.rigA, "seed a"), m.create(t, m.rigB, "seed b") + + on := []string{vhEnvVar + "=1"} + // The rows the writes below expect a version for are created AFTER the setting is on + // (here, the environment), so they participate in history. The seeds above were created + // with it off, so they are legacy and design §16.2b's write fence skips an update-shaped + // write to them whole: that is pinned here, at the command level. + createOn := func(w vhWorkspace, title string) string { + out, _ := vhBD(t, m.bd, w.dir, on, "create", "--silent", title) + return strings.TrimSpace(out) + } + idAOn, idBOn := createOn(m.rigA, "seed a, created under the env override"), createOn(m.rigB, "seed b, created under the env override") + m.expectDelta(t, "routed writes to seeds created before the env override", nil, func() { + m.routedUpdate(t, idA, "env legacy a", on...) + m.routedUpdate(t, idB, "env legacy b", on...) + }) + // Every store this process opens records, whatever its own row says. + m.expectDelta(t, "routed writes into both rigs under the env override", map[string]int{"vha": 1, "vhb": 1}, func() { + m.routedUpdate(t, idAOn, "env a", on...) + m.routedUpdate(t, idBOn, "env b", on...) + }) + m.expectDelta(t, "bd create --repo under the env override", map[string]int{"vhb": 1}, func() { + m.routedCreate(t, m.rigB, "env create b", on...) + }) + + // =0 is not "off". A store whose row says on keeps recording, and a store + // with no row stays quiet. + m.setRow(t, m.rigA, "true") + off := []string{vhEnvVar + "=0"} + m.expectDelta(t, "env=0 over a rig whose row enables history", map[string]int{"vha": 1}, func() { + m.routedUpdate(t, idAOn, "env off a", off...) + }) + m.expectDelta(t, "env=0 over a rig with no row", nil, func() { + m.routedUpdate(t, idB, "env off b", off...) + }) + + // The only way off is the row. + m.setRow(t, m.rigA, "false") + m.expectDelta(t, "routed update after the row is set back to false", nil, func() { + m.routedUpdate(t, idA, "row off a") + }) +} diff --git a/cmd/bd/versioned_history_serve_e2e_test.go b/cmd/bd/versioned_history_serve_e2e_test.go new file mode 100644 index 000000000..23b1d4d9e --- /dev/null +++ b/cmd/bd/versioned_history_serve_e2e_test.go @@ -0,0 +1,88 @@ +//go:build cgo && unix + +package main + +import ( + "context" + "database/sql" + "fmt" + "net/http" + "testing" + "time" +) + +// K1, against a real server: the unit-of-work provider `bd serve` builds for a +// server-mode workspace. +// +// A provider has no storage.DoltStorage to ask for a settings plane, so its flag +// comes from viper OR its own database's row, read through a unit of work when +// the provider is constructed. The unit tests prove the read with fakes; what only +// a real server can prove is that the read WORKS at that moment -- a pooled +// connection, a database already selected, a config table already migrated -- and +// that a provider which silently fell back to the environment alone would have +// looked exactly the same from outside: the served mutation succeeds, the response +// is normal, and issue_versions is simply empty. +// +// Runs in the proxied-server lane only (BEADS_TEST_PROXIED_SERVER=1), like the +// events-journal twin, TestServeActivatesTheEventsJournal. + +// serverVersionRows counts issue_versions rows in one database on the shared test +// Dolt container. +func serverVersionRows(t *testing.T, port int, database string) int { + t.Helper() + dsn := fmt.Sprintf("root:@tcp(127.0.0.1:%d)/%s?parseTime=true", port, database) + db, err := sql.Open("mysql", dsn) + if err != nil { + t.Fatalf("sql.Open %s: %v", dsn, err) + } + defer func() { _ = db.Close() }() + ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second) + defer cancel() + var n int + if err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM issue_versions").Scan(&n); err != nil { + t.Fatalf("count issue_versions in %s: %v", database, err) + } + return n +} + +// serveOneMutation starts `bd serve` on a server-mode workspace, makes one HTTP +// mutation through the provider serve built, shuts serve down, and returns how many +// issue_versions rows that mutation added. +func serveOneMutation(t *testing.T, bd string, port int, p serverModeProject) int { + t.Helper() + before := serverVersionRows(t, port, p.database) + + // Nothing from the developer's shell may switch the feature on for serve. + env := envWithout(p.env, vhEnvVar) + sp := startServe(t, bd, p.dir, env) + status, body := sp.postJSON(t, "/v0/beads/issues:batchCreate", + `{"actor":"tester","items":[{"title":"served mutation","issue_type":"task"}]}`) + if status != http.StatusOK { + t.Fatalf("POST /v0/beads/issues:batchCreate = %d: %v\nstderr:\n%s", status, body, sp.stderr.String()) + } + sp.shutdown(t) + + return serverVersionRows(t, port, p.database) - before +} + +// TestProxiedServerVersionedHistoryServeActivatesTheProviderFromItsOwnRow: two server-mode +// workspaces on one server. The one whose OWN row says on records a served +// mutation; the one with no row does not, and does not inherit the other's answer. +// Neither is helped by the environment, which is unset. +func TestProxiedServerVersionedHistoryServeActivatesTheProviderFromItsOwnRow(t *testing.T) { + port := requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + + enabled := newServerModeProject(t, bd, "vhse") + quiet := newServerModeProject(t, bd, "vhsq") + enabled.run(t, bd, "config", "set", versionedHistorySettingKey, "true") + + if got := serveOneMutation(t, bd, port, enabled); got != 1 { + t.Errorf("a served mutation into a workspace whose row says on added %d issue_versions row(s), want 1: "+ + "the provider serve builds is not reading its own database's row", got) + } + if got := serveOneMutation(t, bd, port, quiet); got != 0 { + t.Errorf("a served mutation into a workspace with no row added %d issue_versions row(s), want 0", got) + } +} diff --git a/cmd/bd/versioned_history_switch.go b/cmd/bd/versioned_history_switch.go new file mode 100644 index 000000000..cd6dd56cf --- /dev/null +++ b/cmd/bd/versioned_history_switch.go @@ -0,0 +1,170 @@ +package main + +import ( + "context" + "errors" + "fmt" + "strings" + + "github.com/steveyegge/beads/internal/storage" + storageissueops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/versionedhistory" + "github.com/steveyegge/beads/issueops" +) + +// The resolver for the versioned-history switch -- which planes it reads, why they +// are OR'd, how a store that cannot answer is handled -- lives in +// internal/versionedhistory, so that bd doctor's fix package (which cannot import +// package main) applies the same rule. This file keeps the two names the rest of +// cmd/bd uses. + +// versionedHistorySettingKey is the one spelling of the switch, used for both the +// store-config read and the `bd config set` line the help text and the off-refusal +// tell people to run. They must not drift, so it is defined once, in the package +// that reads it. +const versionedHistorySettingKey = versionedhistory.ConfigKey + +// versionedHistoryEnabled reports whether THIS STORE is recording versions: its +// own settings row, and nothing else. It is what `bd versions` calls, where the +// read is the point and no capability gate precedes it. +// +// It deliberately does not fold in the environment or config.yaml the way the +// rule for writers does. Those are process-wide and can only turn recording on +// for the writes this process makes; `bd versions` only reads, so they cannot +// make the store record anything. Letting them answer here made a store that +// never recorded read as one that was, and "No versions recorded" is the +// empty answer this command's three outcomes exist to prevent. What other +// clients of the store will do is what the store's row says. +func versionedHistoryEnabled(ctx context.Context, st storage.DoltStorage) bool { + return versionedhistory.StoreSetting(ctx, st) +} + +// versionedHistoryRefusalListLimit is how many offending issues the refusal spells +// out. The count it leads with is always the whole number. +const versionedHistoryRefusalListLimit = 20 + +// checkVersionedHistoryCanBeEnabled is the check `bd config set +// versioned-history.enabled true` and `bd config set-many +// versioned-history.enabled=true` make before they write the setting, and it +// makes it for that value alone: any other key, and every value that does not +// turn recording on (`false` included), passes without reading a row. +// +// With history on, recording a version runs in the same transaction as the write, +// so a write that introduces a number outside the I-JSON exact-integer range, in +// metadata or in a gate's timeout, already fails atomically. A row that holds one +// BEFORE the switch is turned on -- written while history was off, or by a path +// that does not record -- would, if it participates in history, fail every later +// write to it. So the switch reads every issue the store would version, runs the +// check recording runs over each one (issueops.FindUnversionable: the whole +// issue, not its metadata alone), and refuses, writing nothing, while any is +// refused. It cannot see participation, so it checks legacy rows too: an +// over-approximation that can only refuse more. +// +// That check runs once, now. A writer that does not record (an older bd, bd sql, a +// pull from a clone that had history off) can add such a row at any time +// afterwards, even right after the switch is on. If the row participates in +// history, the first write to it then fails with the same refusal, and the same fix +// applies; a legacy row is skipped by the write fence and stays writable. Making +// the check and the write of the setting one transaction would narrow that window, +// not close it. +// +// It reads through the same role accessor `bd list` does, so the direct and the +// proxied route both answer, and with the request that means "everything": every +// status, pinned rows, every type, and both planes (the scan itself skips the rows +// recording never versions). Limit 0 with no MaxRows is an unbounded read: a store +// larger than a page must not be checked on a page. A read that fails refuses the +// switch too, because a check that could not run has not passed. The read is a +// brief one, without the large text columns or the labels, and loads no +// dependencies; that is sound because none of those can hold a number, which the +// tests in internal/storage/issueops pin. +// +// On the direct route it reads through the store the command has already opened: +// `config set` opens it through openWorkspaceConfig before it asks, and the +// command's pre-run has opened it for a `config set-many` that names a database +// key, before the batch is validated. The proxied route reads through its provider. +// +// The environment and config.yaml planes do not pass through these commands, so +// they are not checked here; for those, recording's own refusal at write time is +// the control. +func checkVersionedHistoryCanBeEnabled(ctx context.Context, key, value string) error { + if key != versionedHistorySettingKey || !versionedhistory.ValueEnables(value) { + return nil + } + reader, err := openIssueReader() + if err != nil { + return fmt.Errorf("cannot check the store's issues before turning versioned history on: %w", err) + } + unlimited := 0 + page, err := reader.List(ctx, issueops.ListRequest{ + AllFlag: true, + IncludeAllTypes: true, + Limit: &unlimited, + Brief: true, + SkipLabels: true, + SkipCounts: true, + }) + if err != nil { + return fmt.Errorf("cannot check the store's issues before turning versioned history on: %w", err) + } + rows := make([]*types.Issue, 0, len(page.Items)) + for _, item := range page.Items { + rows = append(rows, item.Issue) + } + if found := storageissueops.FindUnversionable(rows); len(found) > 0 { + return errors.New(unversionableIssueRefusal(found)) + } + return nil +} + +// unversionableIssueRefusal is the text of the refusal: how many issues, which +// (the first versionedHistoryRefusalListLimit), what is wrong with each and the +// field it is in, and the fix for each kind of field that is wrong. It says there +// is no override because there is none. +// +// The fix depends on the field. A number in metadata is replaced or removed with +// bd update, which works with history on or off, because a version is recorded +// from the state after the write. bd update cannot change a gate's timeout, so the +// gate is removed or its column is set with bd sql, while history is off; with +// history on, every write to that row, a close included, is refused if the row +// participates in history. bd sql needs a server-backed store, which the text says. +func unversionableIssueRefusal(found []storageissueops.UnversionableIssue) string { + noun, verb := "issues", "hold" + if len(found) == 1 { + noun, verb = "issue", "holds" + } + var b strings.Builder + fmt.Fprintf(&b, "cannot turn versioned history on: %d %s %s a value a version could not record.\n", len(found), noun, verb) + shown := found + if len(shown) > versionedHistoryRefusalListLimit { + shown = shown[:versionedHistoryRefusalListLimit] + } + for _, f := range shown { + fmt.Fprintf(&b, " %s %v\n", f.ID, f.Err) + } + if len(found) > len(shown) { + fmt.Fprintf(&b, " ... and %d more\n", len(found)-len(shown)) + } + var inMetadata, inTimeout bool + for _, f := range found { + switch f.Field { + case "metadata", "": + inMetadata = true + case "timeout": + inTimeout = true + } + } + b.WriteString("With history on, a write to one of these issues that leaves its value in place would be refused if the issue participates in history: it was created while history was on, or it enters the issues plane while history is on (a wisp promoted or made persistent). This check cannot tell which issues participate, so it also lists legacy issues, created before history was on, whose writes would not be refused. Fix each one, then run this command again.\n") + if inMetadata { + b.WriteString("A number in metadata, one command per issue (this works with history on or off):\n") + b.WriteString(" bd update --metadata '{\"\": \"\"}' replace the value; a string keeps it exact\n") + b.WriteString(" bd update --unset-metadata or remove the key\n") + } + if inTimeout { + b.WriteString("A gate timeout is held as nanoseconds, and 9007199254740991 (about 104.25 days) is the most a version can record. bd update cannot change a gate's timeout. While history is off, remove the gate or set its timeout within that limit:\n") + b.WriteString(" bd delete --force remove the gate; without --force bd only previews, and removing the gate also unblocks anything it was blocking\n") + b.WriteString(" bd sql \"UPDATE issues SET timeout_ns = WHERE id = ''\" set the timeout; bd sql needs a server-backed store, not an embedded one\n") + } + b.WriteString("There is no override. This check runs once, now: if a writer that does not record (an older bd, bd sql, a pull from a clone that had history off) later puts such a value into an issue that participates in history, the next write to it fails with the same refusal, and the same fix applies.") + return b.String() +} diff --git a/cmd/bd/versioned_history_switch_scan_embedded_test.go b/cmd/bd/versioned_history_switch_scan_embedded_test.go new file mode 100644 index 000000000..6c4c1a286 --- /dev/null +++ b/cmd/bd/versioned_history_switch_scan_embedded_test.go @@ -0,0 +1,436 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/workapi" +) + +// THE NAME IS LOAD-BEARING for the same reason TestEmbeddedVersionedHistorySwitchRoundTrip's is: +// .github/scripts/embedded-test-shard.sh selects the tests the embedded job runs by +// `grep -rh '^func TestEmbedded' cmd/bd/*_embedded_test.go`, so a test that keeps this prefix +// and this file suffix runs in the one job that sets BEADS_TEST_EMBEDDED_DOLT=1. +// +// TestEmbeddedVersionedHistorySwitchRefusesUnversionableMetadata drives the REAL +// `bd config set versioned-history.enabled true` against a real store that already holds rows +// a version could not be recorded for. +// +// With history on, recording a version runs in the same transaction as the write, so a write +// that introduces a number outside the I-JSON exact-integer range already fails atomically. +// What that cannot help is a row that holds such a number BEFORE the switch is turned on: +// written while history was off, or through a path that does not mint (bd sql, a pull from a +// clone that had history off). Once history is on, every later write to that row would fail +// if the row participates in history. The switch therefore checks first, with the same +// function the mint runs, and refuses to turn history on while any issue the mint would +// version holds a value it would refuse. +// +// What this pins, end to end and in a database of its own: +// - the refusal exits non-zero and prints the count, the ids and the remedy, +// - it names every issue the mint would version, whatever its status or type: a closed issue, +// a pinned one and a gate are all hidden from a default `bd list`, so a scan that read only +// what `bd list` shows would clear the switch over them, +// - it names only those: an ephemeral row and a no-history row holding the same value are +// never versioned, so they are not listed and do not block, +// - it writes NOTHING: the setting is still off afterwards, +// - turning history OFF never scans, +// - once the rows are fixed with the command the refusal names, the switch turns on. +func TestEmbeddedVersionedHistorySwitchRefusesUnversionableMetadata(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + // The environment can only turn recording on and would make a passing run prove nothing + // about the store's own setting, so clear it. No t.Parallel: t.Setenv is incompatible. + t.Setenv("BD_VERSIONED_HISTORY_ENABLED", "") + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "vsc") + + const timestampInNanoseconds = `{"ts":1727000000000000000}` + const largeInteger = `{"n":9007199254740993}` + + clean := bdCreateSilent(t, bd, dir, "clean", "--metadata", `{"ok":1,"fraction":0.1}`) + first := bdCreateSilent(t, bd, dir, "holds a nanosecond timestamp", "--metadata", timestampInNanoseconds) + second := bdCreateSilent(t, bd, dir, "holds an integer past 2^53", "--metadata", largeInteger) + wisp := bdCreateSilent(t, bd, dir, "ephemeral, never versioned", "--ephemeral", "--metadata", timestampInNanoseconds) + noHistory := bdCreateSilent(t, bd, dir, "no-history, never versioned", "--no-history", "--metadata", timestampInNanoseconds) + + // Three more the mint versions that a default `bd list` does not show. The scan has to ask for + // every status and every type to see them. + closed := bdCreateSilent(t, bd, dir, "closed, still versioned", "--metadata", timestampInNanoseconds) + bdRunOK(t, bd, dir, "close", closed) + pinned := bdCreateSilent(t, bd, dir, "pinned, still versioned", "--metadata", largeInteger) + bdRunOK(t, bd, dir, "update", pinned, "--status", "pinned") + gate := bdCreateSilent(t, bd, dir, "a gate, still versioned", "--type", "gate", "--metadata", timestampInNanoseconds) + + out, code := bdRunFailCode(t, bd, dir, "config", "set", "versioned-history.enabled", "true") + if code == 0 { + t.Fatalf("enabling versioned history over unrecordable rows exited 0; out=%s", out) + } + offenders := map[string]string{ + "the open issue holding a timestamp": first, + "the open issue holding a large integer": second, + "the closed issue": closed, + "the pinned issue": pinned, + "the gate": gate, + } + for name, id := range offenders { + if !strings.Contains(out, id) { + t.Errorf("the refusal does not name %s (%s), which holds a value a version could not record:\n%s", name, id, out) + } + } + for name, id := range map[string]string{"the clean issue": clean, "the ephemeral issue": wisp, "the no-history issue": noHistory} { + if strings.Contains(out, id) { + t.Errorf("the refusal names %s (%s), which the mint would never version:\n%s", name, id, out) + } + } + if want := fmt.Sprintf("%d issues", len(offenders)); !strings.Contains(out, want) { + t.Errorf("the refusal does not give the count of offending issues (%d):\n%s", len(offenders), out) + } + for _, want := range []string{"bd update", "--metadata", "I-JSON"} { + if !strings.Contains(out, want) { + t.Errorf("the refusal does not contain %q, so it does not say what is wrong or how to fix it:\n%s", want, out) + } + } + + // It refuses WITHOUT writing: the setting is still off. + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", "versioned-history.enabled")); strings.HasPrefix(got, "true") { + t.Fatalf("the refused `config set ... true` wrote the setting anyway: config get says %q", got) + } + + // Turning it off never scans, so it works with offenders present and is how a + // store that is somehow already on gets out from under a poisoned row. + bdConfig(t, bd, dir, "set", "versioned-history.enabled", "false") + + // The remedy the refusal names: one update per row, while history is off. + bdRunOK(t, bd, dir, "update", first, "--metadata", `{"ts":"1727000000000000000"}`) + bdRunOK(t, bd, dir, "update", second, "--unset-metadata", "n") + bdRunOK(t, bd, dir, "update", closed, "--metadata", `{"ts":"1727000000000000000"}`) + bdRunOK(t, bd, dir, "update", pinned, "--unset-metadata", "n") + bdRunOK(t, bd, dir, "update", gate, "--metadata", `{"ts":"1727000000000000000"}`) + + bdConfig(t, bd, dir, "set", "versioned-history.enabled", "true") + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", "versioned-history.enabled")); !strings.HasPrefix(got, "true") { + t.Fatalf("after fixing every row the switch did not turn on: config get says %q", got) + } +} + +// TestEmbeddedVersionedHistorySwitchScansEveryRowNotOnePage pins that the scan reads the whole +// store. A list read that sets no limit returns one page (workapi.DefaultListLimit rows), so a +// scan that stopped asking for everything would clear the switch over any offender past the first +// page and count only that page's worth in its refusal. +// +// The rows are imported in one process: creating a page and a half of issues one command at a +// time costs the better part of a minute in this job. +func TestEmbeddedVersionedHistorySwitchScansEveryRowNotOnePage(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Setenv("BD_VERSIONED_HISTORY_ENABLED", "") + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "vsp") + + // More rows than one page, and more than the refusal lists, so both counts are exercised. + n := workapi.DefaultListLimit + 10 + var rows strings.Builder + for i := 1; i <= n; i++ { + fmt.Fprintf(&rows, `{"title":"offender %03d","metadata":{"ts":1727000000000000000}}`+"\n", i) + } + file := filepath.Join(t.TempDir(), "offenders.jsonl") + if err := os.WriteFile(file, []byte(rows.String()), 0o600); err != nil { + t.Fatalf("writing the rows to import: %v", err) + } + bdRunOK(t, bd, dir, "import", file) + + out, _ := bdRunFailCode(t, bd, dir, "config", "set", "versioned-history.enabled", "true") + if want := fmt.Sprintf("%d issues", n); !strings.Contains(out, want) { + t.Errorf("the refusal does not count all %d offending issues (a scan that read one page would say %d):\n%s", n, workapi.DefaultListLimit, out) + } + if want := fmt.Sprintf("... and %d more", n-versionedHistoryRefusalListLimit); !strings.Contains(out, want) { + t.Errorf("the refusal does not say %q after the first %d ids:\n%s", want, versionedHistoryRefusalListLimit, out) + } +} + +// TestEmbeddedVersionedHistorySetManyRefusesUnversionableMetadata drives the REAL +// `bd config set-many versioned-history.enabled=true` against a store that holds a row a version +// could not be recorded for. set-many writes the same setting as `bd config set` through a +// validation loop of its own, and it once turned history on over such a row with no check at +// all: the next write to that row then failed. +// +// What this pins, end to end and in a database of its own: +// - the batch is refused: it exits non-zero and the refusal names the issue and the remedy, +// - it writes NOTHING, neither the setting nor the other pair in the same batch, because the +// check runs before any pair is written, +// - a batch that turns history OFF never scans, +// - once the row is fixed, the same batch turns history on and writes its other pair. +func TestEmbeddedVersionedHistorySetManyRefusesUnversionableMetadata(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Setenv("BD_VERSIONED_HISTORY_ENABLED", "") + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "vsm") + + offender := bdCreateSilent(t, bd, dir, "holds a nanosecond timestamp", "--metadata", `{"ts":1727000000000000000}`) + + // The other pair comes first, so a check made after the writes would already have stored it. + out, _ := bdRunFailCode(t, bd, dir, "config", "set-many", "vsm.marker=1", "versioned-history.enabled=true") + if !strings.Contains(out, offender) { + t.Errorf("the refusal does not name %s, which holds a value a version could not record:\n%s", offender, out) + } + for _, want := range []string{"1 issue", "bd update", "--metadata", "I-JSON"} { + if !strings.Contains(out, want) { + t.Errorf("the refusal does not contain %q, so it does not say what is wrong or how to fix it:\n%s", want, out) + } + } + + // It refuses WITHOUT writing: the setting is still off and the other pair was not stored. + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", "versioned-history.enabled")); strings.HasPrefix(got, "true") { + t.Fatalf("the refused `config set-many ... versioned-history.enabled=true` wrote the setting anyway: config get says %q", got) + } + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", "vsm.marker")); got == "1" { + t.Fatalf("the refused batch stored its other pair anyway: config get vsm.marker says %q", got) + } + + // Turning it off never scans, so a batch that does so works with the offender present. + bdConfig(t, bd, dir, "set-many", "versioned-history.enabled=false") + + // The remedy the refusal names, then the same batch that was refused goes through. + bdRunOK(t, bd, dir, "update", offender, "--unset-metadata", "ts") + bdConfig(t, bd, dir, "set-many", "vsm.marker=1", "versioned-history.enabled=true") + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", "versioned-history.enabled")); !strings.HasPrefix(got, "true") { + t.Fatalf("after fixing the row the batch did not turn the switch on: config get says %q", got) + } + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", "vsm.marker")); got != "1" { + t.Fatalf("after fixing the row the batch did not store its other pair: config get vsm.marker says %q", got) + } +} + +// gateIDFromCreateJSON reads the id out of `bd gate create --json` output, which may carry +// warnings ahead of the object. +func gateIDFromCreateJSON(t *testing.T, out string) string { + t.Helper() + start := strings.Index(out, "{") + if start < 0 { + t.Fatalf("no JSON object in `bd gate create --json` output:\n%s", out) + } + var created struct { + ID string `json:"id"` + } + if err := json.NewDecoder(strings.NewReader(out[start:])).Decode(&created); err != nil || created.ID == "" { + t.Fatalf("parsing `bd gate create --json` output: %v\n%s", err, out) + } + return created.ID +} + +// createTimerGate creates a timer gate that blocks one issue, through the real command, so what +// reaches the store is what a person can write. +func createTimerGate(t *testing.T, bd, dir, blocks, timeout string) string { + t.Helper() + return gateIDFromCreateJSON(t, bdRunOK(t, bd, dir, "gate", "create", "--type=timer", "--blocks", blocks, "--timeout", timeout, "--json")) +} + +// idsNamedIn returns the distinct issue ids with the given prefix that a command's output names. +func idsNamedIn(out, prefix string) map[string]bool { + ids := map[string]bool{} + for _, id := range regexp.MustCompile(regexp.QuoteMeta(prefix)+`-[0-9a-z]+`).FindAllString(out, -1) { + ids[id] = true + } + return ids +} + +func sortedIDSet(set map[string]bool) []string { + ids := make([]string, 0, len(set)) + for id := range set { + ids = append(ids, id) + } + sort.Strings(ids) + return ids +} + +// writeRefusedWithHistoryOn reports whether an ordinary write to the issue is refused when the +// process records versions because its ENVIRONMENT says so. That plane does not pass through the +// pre-enable scan, which makes it the way to ask the mint itself what it would refuse. +func writeRefusedWithHistoryOn(t *testing.T, bd, dir, id string) bool { + t.Helper() + cmd := exec.Command(bd, "update", id, "--notes", "touched with history on") + cmd.Dir = dir + cmd.Env = append(bdEnv(dir), vhEnvVar+"=1") + out, err := cmd.CombinedOutput() + if err == nil { + return false + } + var exitErr *exec.ExitError + if !errors.As(err, &exitErr) { + t.Fatalf("bd update %s failed without an exit code: %v\n%s", id, err, out) + } + if !strings.Contains(string(out), "I-JSON") { + t.Fatalf("bd update %s failed, but not with the mint's refusal of a number outside the I-JSON range:\n%s", id, out) + } + return true +} + +// createWithHistoryOn creates an issue in a process that records versions because its ENVIRONMENT +// says so, which makes the row participate in history from its first write. A row created while +// history is off does not participate: design §16.2b's write fence skips an update-shaped write to it. +func createWithHistoryOn(t *testing.T, bd, dir string, args ...string) string { + t.Helper() + cmd := exec.Command(bd, append([]string{"create", "--silent"}, args...)...) + cmd.Dir = dir + cmd.Env = append(bdEnv(dir), vhEnvVar+"=1") + out, err := cmd.Output() + if err != nil { + var exitErr *exec.ExitError + if errors.As(err, &exitErr) { + t.Fatalf("bd create --silent %s with history on failed: %v\n%s", strings.Join(args, " "), err, exitErr.Stderr) + } + t.Fatalf("bd create --silent %s with history on failed: %v", strings.Join(args, " "), err) + } + return strings.TrimSpace(string(out)) +} + +// TestEmbeddedVersionedHistorySwitchRefusesAnUnrecordableGateTimeout drives the REAL +// `bd config set versioned-history.enabled true` against a store that holds gates whose timeout a +// version could not record. +// +// A gate's timeout is a time.Duration, which a version holds as a count of nanoseconds, so one past +// 2^53-1 ns (about 104.25 days) is a number outside the I-JSON exact-integer range. Nothing stops +// `bd gate create --timeout 2600h` while history is off, and once history is on every write to a +// participating gate, a close included, is refused. A scan that read only metadata cleared the +// switch over it. +// +// What this pins, end to end and in a database of its own: +// - the refusal names each gate past the range, open or closed, and the field the number was +// found in ("timeout"), and does not name a gate that is in range, +// - it offers the remedy for the field that is wrong: removing the gate, not a metadata edit, until +// an issue with a number in its metadata is there too, +// - it writes NOTHING: the setting is still off afterwards, +// - the scan names every offender, because it cannot see participation: a row written while +// history was off is a legacy row, which design §16.2b's write fence makes the mint skip, so +// with history on through the environment, a plane the scan does not guard, no write to one is +// refused. The scan can only refuse more than the mint does, +// - for a row that participates in history, the scan and the mint AGREE: the issue whose next +// write is refused is exactly the one the scan named, +// - once the remedies are applied, the switch turns on. +func TestEmbeddedVersionedHistorySwitchRefusesAnUnrecordableGateTimeout(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("embedded dolt integration tests are not enabled") + } + // The environment can only turn recording on and would make a passing run prove nothing + // about the store's own setting, so clear it. No t.Parallel: t.Setenv is incompatible. + t.Setenv(vhEnvVar, "") + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "vsg") + + blocked := bdCreateSilent(t, bd, dir, "blocked by three gates") + clean := bdCreateSilent(t, bd, dir, "clean") + // History is off, so nothing refuses a gate whose timeout a version could not record. + overlong := createTimerGate(t, bd, dir, blocked, "2600h") + inRange := createTimerGate(t, bd, dir, blocked, "2500h") + closedOverlong := createTimerGate(t, bd, dir, blocked, "8760h") + bdRunOK(t, bd, dir, "gate", "resolve", closedOverlong) + + out, code := bdRunFailCode(t, bd, dir, "config", "set", "versioned-history.enabled", "true") + if code == 0 { + t.Fatalf("enabling versioned history over a gate whose timeout is past the range exited 0; out=%s", out) + } + for name, id := range map[string]string{"the open gate past the range": overlong, "the closed gate past the range": closedOverlong} { + if !strings.Contains(out, id) { + t.Errorf("the refusal does not name %s (%s), whose timeout a version could not record:\n%s", name, id, out) + } + } + for name, id := range map[string]string{"the gate in range": inRange, "the issue the gates block": blocked, "the clean issue": clean} { + if strings.Contains(out, id) { + t.Errorf("the refusal names %s (%s), which the mint would record:\n%s", name, id, out) + } + } + for _, want := range []string{"2 issues", `"timeout"`, "I-JSON", "bd delete --force", "only previews", "unblocks anything it was blocking"} { + if !strings.Contains(out, want) { + t.Errorf("the refusal does not contain %q, so it does not say what is wrong or how to fix it:\n%s", want, out) + } + } + if strings.Contains(out, "--unset-metadata") { + t.Errorf("the refusal offers a metadata edit although nothing holds a number in its metadata:\n%s", out) + } + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", "versioned-history.enabled")); strings.HasPrefix(got, "true") { + t.Fatalf("the refused `config set ... true` wrote the setting anyway: config get says %q", got) + } + + // Add the other kind of offender. The refusal now has two fields to fix and offers a remedy for each. + holdsNumber := bdCreateSilent(t, bd, dir, "holds a nanosecond timestamp", "--metadata", `{"ts":1727000000000000000}`) + out, _ = bdRunFailCode(t, bd, dir, "config", "set", "versioned-history.enabled", "true") + for _, want := range []string{"3 issues", `"timeout"`, `"metadata"`, "bd delete --force", "only previews", "unblocks anything it was blocking", "--unset-metadata"} { + if !strings.Contains(out, want) { + t.Errorf("the refusal over a gate and a metadata number does not contain %q:\n%s", want, out) + } + } + + // The scan reads rows, not participation, so it names every row that holds a value the mint would + // refuse. Ask the mint, through a process that records because its environment says so (a plane the + // scan does not guard), what a write to each row does. Everything so far was written while history + // was off, so every row is a legacy row, and design §16.2b's write fence skips an update-shaped write + // to one before the refusal could run: none of these writes is refused, offenders included. That is + // the scan's deliberate over-approximation, in the safe direction: it can only refuse more than the + // mint does. + legacy := []string{blocked, clean, overlong, inRange, closedOverlong, holdsNumber} + for _, id := range legacy { + if writeRefusedWithHistoryOn(t, bd, dir, id) { + t.Errorf("the mint refused a write to %s, a row written while history was off: the write fence should have skipped it", id) + } + } + if got, want := strings.Join(sortedIDSet(idsNamedIn(out, "vsg")), ","), strings.Join(sortedIDSet(map[string]bool{overlong: true, closedOverlong: true, holdsNumber: true}), ","); got != want { + t.Errorf("the scan named %s, want exactly the offenders %s", got, want) + } + + // A row participates in history when it was created while history was on. Create two that way, in + // range, then take one of them out of range while history is off, which nothing refuses. For a row + // that participates the scan and the mint agree: the offender is named and every write to it is + // refused, and the clean one is neither. + participatingClean := createWithHistoryOn(t, bd, dir, "participates, clean", "--metadata", `{"ts":1}`) + participating := createWithHistoryOn(t, bd, dir, "participates, then holds a nanosecond timestamp", "--metadata", `{"ts":1}`) + bdRunOK(t, bd, dir, "update", participating, "--metadata", `{"ts":1727000000000000000}`) + + out, _ = bdRunFailCode(t, bd, dir, "config", "set", "versioned-history.enabled", "true") + named := idsNamedIn(out, "vsg") + namedParticipating := map[string]bool{} + refused := map[string]bool{} + for _, id := range []string{participatingClean, participating} { + if named[id] { + namedParticipating[id] = true + } + if writeRefusedWithHistoryOn(t, bd, dir, id) { + refused[id] = true + } + } + if got, want := strings.Join(sortedIDSet(namedParticipating), ","), strings.Join(sortedIDSet(refused), ","); got != want { + t.Errorf("of the rows that participate, the scan named %s but the mint refused a write to %s: they must be the same issues", got, want) + } + if got := strings.Join(sortedIDSet(refused), ","); got != participating { + t.Errorf("the mint refused a write to %v, want exactly %s", sortedIDSet(refused), participating) + } + if got, want := strings.Join(sortedIDSet(named), ","), strings.Join(sortedIDSet(map[string]bool{overlong: true, closedOverlong: true, holdsNumber: true, participating: true}), ","); got != want { + t.Errorf("the scan named %s, want the three offenders written while history was off and the participating one, %s", got, want) + } + + // The remedies the refusal names, then the switch turns on. + bdRunOK(t, bd, dir, "delete", overlong, closedOverlong, "--force") + bdRunOK(t, bd, dir, "update", holdsNumber, "--unset-metadata", "ts") + bdRunOK(t, bd, dir, "update", participating, "--unset-metadata", "ts") + bdConfig(t, bd, dir, "set", "versioned-history.enabled", "true") + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", "versioned-history.enabled")); !strings.HasPrefix(got, "true") { + t.Fatalf("after applying the remedies the switch did not turn on: config get says %q", got) + } +} diff --git a/cmd/bd/versioned_history_switch_scan_proxied_integration_test.go b/cmd/bd/versioned_history_switch_scan_proxied_integration_test.go new file mode 100644 index 000000000..c1d7343ab --- /dev/null +++ b/cmd/bd/versioned_history_switch_scan_proxied_integration_test.go @@ -0,0 +1,158 @@ +//go:build cgo + +package main + +import ( + "fmt" + "strings" + "testing" +) + +// THE NAME IS LOAD-BEARING: .github/scripts/proxied-test-shard.sh finds the tests the proxied-server +// lane runs by the name prefix (TestProxiedServer, TestServerMode) across cmd/bd/*_test.go, and that +// lane is the only one that enables the proxied-server tests. A test that dropped the prefix would +// skip everywhere else and never run. + +// proxiedWriteRefusedWithHistoryOn is writeRefusedWithHistoryOn on the proxied route: it asks +// whether an ordinary write to the issue is refused when the process records versions because its +// environment says so, a plane the pre-enable scan does not guard. +func proxiedWriteRefusedWithHistoryOn(t *testing.T, bd, dir, id string) bool { + t.Helper() + _, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, dir, []string{vhEnvVar + "=1"}, "update", id, "--notes", "touched with history on") + if err == nil { + return false + } + if !strings.Contains(stderr, "I-JSON") { + t.Fatalf("bd update %s failed, but not with the mint's refusal of a number outside the I-JSON range:\n%s", id, stderr) + } + return true +} + +// proxiedCreateWithHistoryOn is createWithHistoryOn on the proxied route: the row participates in +// history from its first write because the process records versions as its environment says so. +func proxiedCreateWithHistoryOn(t *testing.T, bd, dir string, args ...string) string { + t.Helper() + stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, dir, []string{vhEnvVar + "=1"}, append([]string{"create", "--silent"}, args...)...) + if err != nil { + t.Fatalf("bd create --silent %s with history on failed: %v\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), err, stdout, stderr) + } + return strings.TrimSpace(stdout) +} + +// TestProxiedServerVersionedHistorySwitchRefusesAnUnrecordableGateTimeout is the proxied-server twin +// of TestEmbeddedVersionedHistorySwitchRefusesAnUnrecordableGateTimeout: the same scan, read through +// the proxied route's own reader instead of the store the command opened. +// +// That is the point of running it. The scan reads a lite projection, and each route builds its own, +// so a route whose projection left out a gate's timeout would clear the switch over a participating +// gate the mint then refuses every write to. A timeout past 2^53-1 ns (about 104.25 days) must be +// refused here exactly as on the direct route. +// +// What this pins, against a real server: +// - the refusal names the gate past the range and the field ("timeout"), and not the gate in range, +// - it writes NOTHING: the setting is still off afterwards, +// - the scan names every offender, because it cannot see participation: a row written while +// history was off is a legacy row, which design §16.2b's write fence makes the mint skip, so +// with history on through the environment no write to one is refused. The scan can only refuse +// more than the mint does, +// - for a row that participates in history, the scan and the mint AGREE: the issue whose next +// write is refused with history on through the environment is exactly the one the scan named, +// - the remedies it names work on this route, `bd sql` for the timeout included (it is not +// available on an embedded store), and the switch then turns on. +func TestProxiedServerVersionedHistorySwitchRefusesAnUnrecordableGateTimeout(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + p := newSharedProxiedProject(t, bd, "vsx") + + blocked := bdProxiedCreate(t, bd, p.dir, "blocked by two gates") + clean := bdProxiedCreate(t, bd, p.dir, "clean") + createGate := func(timeout string) string { + t.Helper() + stdout, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "gate", "create", "--type=timer", "--blocks", blocked.ID, "--timeout", timeout, "--json") + if err != nil { + t.Fatalf("bd gate create --timeout %s failed: %v\nstdout:\n%s\nstderr:\n%s", timeout, err, stdout, stderr) + } + return gateIDFromCreateJSON(t, stdout) + } + // History is off, so nothing refuses a gate whose timeout a version could not record. + overlong := createGate("2600h") + inRange := createGate("2500h") + holdsNumber := bdProxiedCreate(t, bd, p.dir, "holds a nanosecond timestamp", "--metadata", `{"ts":1727000000000000000}`) + + out := bdProxiedConfigFail(t, bd, p.dir, "set", versionedHistorySettingKey, "true") + for name, id := range map[string]string{"the gate past the range": overlong, "the issue holding a number in metadata": holdsNumber.ID} { + if !strings.Contains(out, id) { + t.Errorf("the refusal does not name %s (%s), which holds a value a version could not record:\n%s", name, id, out) + } + } + for name, id := range map[string]string{"the gate in range": inRange, "the issue the gates block": blocked.ID, "the clean issue": clean.ID} { + if strings.Contains(out, id) { + t.Errorf("the refusal names %s (%s), which the mint would record:\n%s", name, id, out) + } + } + for _, want := range []string{"2 issues", `"timeout"`, `"metadata"`, "I-JSON", "bd delete --force", "only previews", "unblocks anything it was blocking", "bd sql", "--unset-metadata"} { + if !strings.Contains(out, want) { + t.Errorf("the refusal does not contain %q, so it does not say what is wrong or how to fix it:\n%s", want, out) + } + } + if got := strings.TrimSpace(bdProxiedConfig(t, bd, p.dir, "get", versionedHistorySettingKey)); strings.HasPrefix(got, "true") { + t.Fatalf("the refused `config set ... true` wrote the setting anyway: config get says %q", got) + } + + // The scan reads rows, not participation, so it names every row that holds a value the mint would + // refuse. Ask the mint, through a process that records because its environment says so (a plane the + // scan does not guard), what a write to each row does. Everything so far was written while history + // was off, so every row is a legacy row, and design §16.2b's write fence skips an update-shaped write + // to one before the refusal could run: none of these writes is refused, offenders included. That is + // the scan's deliberate over-approximation, in the safe direction: it can only refuse more than the + // mint does. + legacy := []string{blocked.ID, clean.ID, overlong, inRange, holdsNumber.ID} + for _, id := range legacy { + if proxiedWriteRefusedWithHistoryOn(t, bd, p.dir, id) { + t.Errorf("the mint refused a write to %s, a row written while history was off: the write fence should have skipped it", id) + } + } + if got, want := strings.Join(sortedIDSet(idsNamedIn(out, "vsx")), ","), strings.Join(sortedIDSet(map[string]bool{overlong: true, holdsNumber.ID: true}), ","); got != want { + t.Errorf("the scan named %s, want exactly the offenders %s", got, want) + } + + // A row participates in history when it was created while history was on. Create two that way, in + // range, then take one of them out of range while history is off, which nothing refuses. For a row + // that participates the scan and the mint agree: the offender is named and every write to it is + // refused, and the clean one is neither. + participatingClean := proxiedCreateWithHistoryOn(t, bd, p.dir, "participates, clean", "--metadata", `{"ts":1}`) + participating := proxiedCreateWithHistoryOn(t, bd, p.dir, "participates, then holds a nanosecond timestamp", "--metadata", `{"ts":1}`) + bdProxiedUpdate(t, bd, p.dir, participating, "--metadata", `{"ts":1727000000000000000}`) + + out = bdProxiedConfigFail(t, bd, p.dir, "set", versionedHistorySettingKey, "true") + named := idsNamedIn(out, "vsx") + namedParticipating := map[string]bool{} + refused := map[string]bool{} + for _, id := range []string{participatingClean, participating} { + if named[id] { + namedParticipating[id] = true + } + if proxiedWriteRefusedWithHistoryOn(t, bd, p.dir, id) { + refused[id] = true + } + } + if got, want := strings.Join(sortedIDSet(namedParticipating), ","), strings.Join(sortedIDSet(refused), ","); got != want { + t.Errorf("of the rows that participate, the scan named %s but the mint refused a write to %s: they must be the same issues", got, want) + } + if got := strings.Join(sortedIDSet(refused), ","); got != participating { + t.Errorf("the mint refused a write to %v, want exactly %s", sortedIDSet(refused), participating) + } + if got, want := strings.Join(sortedIDSet(named), ","), strings.Join(sortedIDSet(map[string]bool{overlong: true, holdsNumber.ID: true, participating: true}), ","); got != want { + t.Errorf("the scan named %s, want the two offenders written while history was off and the participating one, %s", got, want) + } + + // The remedies the refusal names, then the switch turns on. + bdProxiedSQL(t, bd, p.dir, fmt.Sprintf("UPDATE issues SET timeout_ns = 7200000000000 WHERE id = '%s'", overlong)) + bdProxiedUpdate(t, bd, p.dir, holdsNumber.ID, "--unset-metadata", "ts") + bdProxiedUpdate(t, bd, p.dir, participating, "--unset-metadata", "ts") + bdProxiedConfig(t, bd, p.dir, "set", versionedHistorySettingKey, "true") + if got := strings.TrimSpace(bdProxiedConfig(t, bd, p.dir, "get", versionedHistorySettingKey)); !strings.HasPrefix(got, "true") { + t.Fatalf("after applying the remedies the switch did not turn on: config get says %q", got) + } +} diff --git a/cmd/bd/versioned_history_switch_test.go b/cmd/bd/versioned_history_switch_test.go new file mode 100644 index 000000000..47fecc685 --- /dev/null +++ b/cmd/bd/versioned_history_switch_test.go @@ -0,0 +1,97 @@ +package main + +import ( + "context" + "testing" + + "github.com/steveyegge/beads/internal/config" + "github.com/steveyegge/beads/internal/hooks" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/issueops" +) + +// The resolver's own tests -- the table of ways a store can fail to answer, how the +// stored string is read back, the capability guard, the recover, and the raw-store +// ordering -- moved to internal/versionedhistory with the resolver, assertions +// unchanged, which is what shows the move preserved #6661's behavior. What stays +// here is the one thing that is cmd/bd's: how the decorator chain is wired. + +// configProbeStore is a raw store at the bottom of a decorator chain whose +// settings plane answers however a case needs. It counts the reads so a test can +// prove it was actually asked -- or, here, that it was NOT. +// +// Every other method promotes from the embedded nil interface and panics if +// called, which is deliberate: nothing here should be calling them. +type configProbeStore struct { + storage.DoltStorage + cfg issueops.WorkspaceConfig + calls int + enabled bool +} + +func (s *configProbeStore) WorkspaceConfig() (issueops.WorkspaceConfig, error) { + s.calls++ + return s.cfg, nil +} + +// SetVersionedHistoryEnabled makes this store a storage.VersionedHistoryConfigurer, +// which is what would earn it a store-plane read if wiring still did one. +func (s *configProbeStore) SetVersionedHistoryEnabled(enabled bool) { s.enabled = enabled } + +// settingsPlane is a workspace settings plane with one canned answer. +type settingsPlane struct { + issueops.WorkspaceConfig + value string +} + +func (p *settingsPlane) GetSetting(context.Context, issueops.GetSettingRequest) (issueops.SettingResult, error) { + return issueops.SettingResult{Key: versionedHistorySettingKey, Value: p.value}, nil +} + +// TestWireStorageDecoratorsLeavesActivationToTheFactories pins the other half of +// moving activation to where a store is constructed. +// +// wireStorageDecorators used to be the one place the switch was read and applied, +// and it runs on the ONE store a command opens for its own workspace. Every store +// a routed update, a routed close or `bd create --repo` opens for ANOTHER +// workspace is built by newDoltStoreFromConfig and never passed through here, so +// it recorded nothing while the command reported success. Activation now happens +// in the factories, for every store they hand back; if wiring ALSO read the row it +// would read it twice per open, and would read as the place that activates -- the +// misreading that produced the defect. +// +// The count is the assertion that matters, not "it survived": the store here would +// answer true, and would be switched on, if wiring still consulted it. +func TestWireStorageDecoratorsLeavesActivationToTheFactories(t *testing.T) { + for _, tt := range []struct { + name string + env string + }{ + {"the store's row says on", ""}, + {"the environment says on", "1"}, + } { + t.Run(tt.name, func(t *testing.T) { + clearTelemetryEnv(t) + t.Setenv("BD_VERSIONED_HISTORY_ENABLED", tt.env) + // config.GetBool answers false until viper is initialised, so without + // this the env subtest would pass whatever wiring did with the env. + if err := config.Initialize(); err != nil { + t.Fatalf("config.Initialize(): %v", err) + } + t.Cleanup(func() { _ = config.Initialize() }) + + raw := &configProbeStore{cfg: &settingsPlane{value: "true"}} + chain := wireStorageDecorators(raw, hooks.NewRunner("/nonexistent"), false) + + if chain == nil { + t.Fatal("wireStorageDecorators returned nil for a non-nil store") + } + if raw.calls != 0 { + t.Errorf("wireStorageDecorators read the store's settings plane %d time(s), want 0: activation belongs to the factory that constructed the store", raw.calls) + } + if raw.enabled { + t.Error("wireStorageDecorators switched versioned history on; only the factory that constructs a store may, at construction") + } + }) + } +} diff --git a/cmd/bd/versions.go b/cmd/bd/versions.go new file mode 100644 index 000000000..d24e96416 --- /dev/null +++ b/cmd/bd/versions.go @@ -0,0 +1,305 @@ +package main + +import ( + "context" + "errors" + "fmt" + "strings" + + "github.com/spf13/cobra" + "github.com/steveyegge/beads/internal/metrics" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/ui" + "github.com/steveyegge/beads/internal/utils" +) + +// errVersionedHistoryOff and errVersionsUnsupported are distinct on purpose. +// +// #5898's third goal is that a history answer has three shapes, not two: +// here it is, there is no such thing, and something prevented a complete +// answer. An empty list is a WRONG answer for both of these cases -- it reads +// as "this bead has no versions", which is a claim about the bead rather than +// about the store. Same principle get_issue.go:38-41 applies to a missing +// leases table: a wrong answer, not an empty one. +var ( + errVersionedHistoryOff = errors.New("versioned history is not enabled on this store") + errVersionsUnsupported = errors.New("this storage backend cannot serve version history") + errNoSuchBead = errors.New("no such bead") +) + +// versionsOutcome is what runVersions resolved, kept apart from the rendering +// so the four ways of having nothing to show stay four different answers. +// +// It is also the --json payload: one object for every answer that is not an +// error, so a script parses one shape whether or not anything was recorded. +// Versions is never nil on a success, so an empty list is [] and never null. +type versionsOutcome struct { + Versions []storage.IssueVersion `json:"versions"` + // Recording reports whether the store is recording versions RIGHT NOW. + // It is independent of whether Versions is empty: a store that recorded + // for a month and was then switched off still has versions, and saying + // otherwise would be a claim about the store made from a fact about this + // invocation's config. + Recording bool `json:"recording"` +} + +var versionsCmd = &cobra.Command{ + Use: "versions ", + GroupID: "views", + Short: "List the recorded versions of a bead", + Long: `List the versions recorded for a bead by versioned history. + +Recording is off by default. Turn it on with: + + bd config set versioned-history.enabled true + +That writes the setting into the store, so every client of that store agrees +about whether a write is recorded, and it is the setting this command reports. +BD_VERSIONED_HISTORY_ENABLED=1 (or a value in config.yaml) also turns recording +on for the writes a process makes, but it says nothing about the store, and +this command only reads, so it does not consult them. + +Either source turning it on is enough for a writer; neither can switch the +other off. The consequence worth knowing: once the store setting says on, +BD_VERSIONED_HISTORY_ENABLED=0 will not turn recording off for one run. Turn +it off where it was turned on: + + bd config set versioned-history.enabled false + +Only beads created while recording is on, or that enter the issues plane +while it is on (a wisp promoted or made persistent), are versioned. A bead +already on the issues plane when recording was turned on is legacy: editing +it records nothing (the write fence, design §16.2b), and nothing is +backfilled. + +Turning recording off does not hide what was already recorded — this command +still lists it, and says that recording is currently off. + +A caution while the feature is young: keep recording ON for every writer of a +shared store, or leave it off for all of them. A write from a client with +recording off advances nothing, and the next recorded write mints a version +that absorbs that unrecorded change under its own actor — the revisions stay +contiguous and the listing looks gapless when it is not. + +Examples: + bd versions bd-123 + bd versions bd-123 --json`, + Args: cobra.ExactArgs(1), + SilenceUsage: true, + SilenceErrors: true, + RunE: func(cmd *cobra.Command, args []string) error { + evt := metrics.NewCommandEvent("versions") + defer func() { + if c := metrics.Global(); c != nil { + c.CloseEventAndAdd(evt) + } + }() + + issueID := args[0] + resolved := false + if full, err := utils.ResolvePartialID(rootCtx, store, issueID); err == nil { + issueID, resolved = full, true + } else if errors.Is(err, utils.ErrAmbiguousID) { + return HandleErrorRespectJSON("%v", err) + } + + recording := versionedHistoryEnabled(rootCtx, store) + + outcome, err := runVersions(rootCtx, store, issueID, recording, resolved) + switch { + case errors.Is(err, errNoSuchBead): + return HandleErrorRespectJSON( + "no bead %s in this store, and no versions recorded under that id.", issueID) + case errors.Is(err, errVersionedHistoryOff): + return HandleErrorRespectJSON( + "versioned history is not being recorded on this store, and nothing was recorded earlier.\n"+ + "Turn it on with: bd config set %s true\n"+ + "Only beads created, or brought onto the issues plane (a wisp promoted or made persistent), after that are versioned: beads already on it stay unversioned, and nothing is backfilled.", + versionedHistorySettingKey) + case errors.Is(err, errVersionsUnsupported): + return HandleErrorRespectJSON( + "this storage backend cannot serve version history (proxied, no-db and non-Dolt backends cannot).\n" + + "Run this against a Dolt-backed workspace.") + case err != nil: + return HandleErrorRespectJSON("failed to list versions: %v", err) + } + + if jsonOutput { + return outputJSON(outcome) + } + printVersions(issueID, outcome) + return nil + }, +} + +// runVersions is the testable core. It separates four situations that all +// look like "nothing here" and each need a different response. +// +// It deliberately READS EVEN WHEN RECORDING IS OFF. An earlier version +// refused up front, which meant a store that recorded for a month and was +// then switched off reported "no versions are recorded" -- a claim about the +// store, derived from a fact about this invocation's config. storage +// .VersionLister's own doc already says the two are independent: holding the +// capability says the backend can READ versions and says nothing about +// whether recording is on. (bee-ghosttrack, #6661 review, finding 3.) +// +// resolved reports whether issueID named a bead the store knows. An +// unresolved id with no versions is "no such bead", not "none yet" -- falling +// through on an unresolved id is right, because a deleted bead can still have +// versions, but only until the answer turns out to be empty. (Same review, +// finding 4.) +func runVersions(ctx context.Context, backend any, issueID string, recording, resolved bool) (versionsOutcome, error) { + lister, ok := versionListerFor(backend) + if !ok { + return versionsOutcome{}, errVersionsUnsupported + } + + versions, err := lister.ListVersions(ctx, issueID) + if err != nil { + return versionsOutcome{}, err + } + if len(versions) > 0 { + // Something was recorded. Whether recording is on right now changes + // the footnote, never whether these rows are shown. + return versionsOutcome{Versions: versions, Recording: recording}, nil + } + + // Nothing to show. Which of the three reasons it is decides the message. + if !resolved { + return versionsOutcome{}, errNoSuchBead + } + if !recording { + return versionsOutcome{}, errVersionedHistoryOff + } + // An empty non-nil list: this is also the --json payload, where nil would + // print null. + return versionsOutcome{Versions: []storage.IssueVersion{}, Recording: true}, nil +} + +// versionListerFor finds the VersionLister behind whatever cmd/bd is holding. +// +// The store in cmd/bd is the DECORATED chain (telemetry, externaldeps, +// hooks), and VersionLister is an optional capability rather than a method on +// DoltStorage, so none of those decorators forwards it -- asserting against +// the wrapper reports "unsupported" for a store that can serve versions +// perfectly well. storage.UnwrapStore's own doc names this case: peel before +// asserting optional interfaces. +// +// The direct assertion is tried FIRST so a test double, or any future +// decorator that genuinely implements the capability itself, wins over the +// concrete store underneath -- the same precedence cmd/bd/dolt.go uses. +func versionListerFor(backend any) (storage.VersionLister, bool) { + if lister, ok := backend.(storage.VersionLister); ok { + return lister, true + } + if ds, ok := backend.(storage.DoltStorage); ok { + if lister, ok := storage.UnwrapStore(ds).(storage.VersionLister); ok { + return lister, true + } + } + return nil, false +} + +func printVersions(issueID string, outcome versionsOutcome) { + versions := outcome.Versions + if len(versions) == 0 { + // Recording is on (runVersions refuses otherwise) and the bead + // resolved. A bead created, or brought onto the issues plane, while + // recording is on always has its create-shaped version, so an empty + // list means this bead does not participate: it is legacy (on the + // issues plane before recording was on, or put there by a writer that + // was not recording), or it is no-history or a wisp. + // Under design §16.2b's write fence it will not gain versions by + // being edited, so this says why rather than "not yet". + fmt.Printf("\nNo versions recorded for %s.\n", issueID) + fmt.Println(ui.RenderMuted("Only beads created while recording is on, or that enter the issues plane while it is on (a wisp promoted or made persistent), are versioned. This one reached the issues plane before recording was turned on (or through a writer that was not recording), or it is a no-history bead or a wisp, so editing it records nothing. Nothing is backfilled.")) + return + } + + fmt.Printf("\nVersions of %s (%d)\n\n", issueID, len(versions)) + fmt.Printf(" %-5s %-19s %-22s %s\n", + ui.RenderMuted("REV"), ui.RenderMuted("WHEN"), ui.RenderMuted("WHO"), ui.RenderMuted("ATTRIB")) + + for _, v := range versions { + if v.Removed() { + // A removed version is a restriction, not a blank row: name which + // restriction applied and when, so it is never mistaken for an + // absence. #5898's four values are distinct answers. + fmt.Printf(" %-5d %s\n", v.Revision, + ui.RenderMuted(fmt.Sprintf("✗ gone · %s%s · %s", + restrictionLabel(v.RemovedRestriction), + reasonSuffix(v.RemovedReason), + v.RemovedAt.Format("2006-01-02 15:04")))) + continue + } + who := v.ChangeActor + if who == "" { + who = ui.RenderMuted("—") + } + fmt.Printf(" %-5d %-19s %-22s %s\n", + v.Revision, + v.ChangeAt.Format("2006-01-02 15:04"), + who, + attributionLabel(v.AttributionStatus)) + if v.ChangeMessage != "" { + fmt.Printf(" %s\n", ui.RenderMuted(v.ChangeMessage)) + } + } + + fmt.Println() + if !outcome.Recording { + // The rows are real; recording is simply off now. Saying nothing here + // would let a stale listing read as current. + fmt.Println(ui.RenderWarn("Recording is currently OFF — this listing ends where it was switched off.")) + fmt.Println(ui.RenderMuted(fmt.Sprintf("Turn it back on with: bd config set %s true", versionedHistorySettingKey))) + fmt.Println() + } + fmt.Println(ui.RenderMuted("REV is local to this store and is not a citable address: two clones can")) + fmt.Println(ui.RenderMuted("both hold revision 8 of the same bead. Portable version addresses arrive")) + fmt.Println(ui.RenderMuted("with the version_id change.")) +} + +// restrictionLabel renders #5898's removal vocabulary. "unknown" is NOT +// "gone": it means this store has no lineage knowledge, which the conformance +// suite pins separately (a never-synced store answers Unknown, not Gone). +// +// A string outside the vocabulary renders as unknown, because that is what the +// as-of read answers for it: the label never prints a stored value the read +// would not give. +func restrictionLabel(r string) string { + switch issueops.NormalizeRemovedRestriction(r) { + case issueops.AsOfRestrictionGoneRetention: + return "outside the retention window" + case issueops.AsOfRestrictionGoneErasure: + return "erased" + case issueops.AsOfRestrictionGoneReorganization: + return "voided by an epoch change" + default: + return "no lineage knowledge in this store" + } +} + +func attributionLabel(s string) string { + switch s { + case "claimed": + return "claimed" + case "unknown": + return ui.RenderMuted("unknown") + case "": + return ui.RenderMuted("—") + default: + return s + } +} + +func reasonSuffix(reason string) string { + if strings.TrimSpace(reason) == "" { + return "" + } + return " (" + reason + ")" +} + +func init() { + rootCmd.AddCommand(versionsCmd) +} diff --git a/cmd/bd/versions_switch_embedded_test.go b/cmd/bd/versions_switch_embedded_test.go new file mode 100644 index 000000000..797d145de --- /dev/null +++ b/cmd/bd/versions_switch_embedded_test.go @@ -0,0 +1,122 @@ +//go:build cgo + +package main + +import ( + "os" + "strings" + "testing" +) + +// THE NAME IS LOAD-BEARING: it must keep the TestEmbedded prefix, because +// .github/scripts/embedded-test-shard.sh selects what runs with +// `grep -rh '^func TestEmbedded' cmd/bd/*_embedded_test.go` and validates each +// manifest entry against `^TestEmbedded[A-Za-z0-9_]+$`. As +// TestVersionedHistorySwitchRoundTrip_Embedded this test matched neither, so +// the only job that sets BEADS_TEST_EMBEDDED_DOLT=1 for cmd/bd never selected +// it, and the generic Test jobs run without that env so its own guard skipped +// it there. The one test that can see the two planes disagree therefore ran +// only on the author's machine. Renaming it is the whole fix -- a test absent +// from the manifest still gets a shard from the fallback distribution. +// (bee-ghosttrack, #6661 third review, blocking finding 2.) +// +// TestEmbeddedVersionedHistorySwitchRoundTrip drives the REAL +// `bd config set versioned-history.enabled true` through to a real +// `bd versions`, against a real store. +// +// This exists because the two ends disagreed and no unit test could see it. +// `bd config set` routes a key that is not yaml-only to the workspace +// DATABASE; the readers called config.GetBool, which is viper (yaml + env) +// only. So the command the help text and the off-refusal both named wrote one +// plane while the reader read the other: the user ran exactly what they were +// told, was told it worked, and recording stayed off. A fake-backed unit test +// cannot catch that, because the bug IS the difference between the two real +// planes. (bee-ghosttrack, #6661 review, finding 1: "a test that drives the +// real bd config set -> reader round trip would have caught this and would +// keep it caught.") +// +// The env var is explicitly cleared for the write and read below, so a +// passing run proves the STORE setting alone carried it. +func TestEmbeddedVersionedHistorySwitchRoundTrip(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + // No t.Parallel: t.Setenv below is incompatible with it, and clearing the + // env var is the whole point — a pass has to prove the STORE setting + // carried the switch on its own. + t.Setenv("BD_VERSIONED_HISTORY_ENABLED", "") + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "vsw") + + created := bdRunOK(t, bd, dir, "create", "switch round trip") + id := extractFirstID(created, "vsw-") + if id == "" { + t.Fatalf("could not find a vsw- id in create output: %s", created) + } + + // Off by default: a bead with no recorded versions must refuse, not + // report an empty history. + if out, code := bdRunFailCode(t, bd, dir, "versions", id); code == 0 { + t.Errorf("bd versions succeeded with recording off and nothing recorded; want a refusal. out=%s", out) + } else if !strings.Contains(out, "bd config set versioned-history.enabled true") { + t.Errorf("the refusal must name the command that actually turns it on; got: %s", out) + } + + // The advertised command. If this writes a plane the reader does not + // consult, everything below fails. + bdConfig(t, bd, dir, "set", "versioned-history.enabled", "true") + + // A bead that existed before the switch was turned on never participated in + // history (design §16.2b: the write fence skips a legacy record), so writing + // to it versions nothing. That is the fence, pinned at the command level. + legacyID := id + bdRunOK(t, bd, dir, "update", legacyID, "--title", "switch round trip, edited while legacy") + if legacy := bdRunOK(t, bd, dir, "versions", legacyID); !strings.Contains(legacy, "No versions recorded for "+legacyID) { + t.Errorf("a bead created before the switch was turned on gained a version from an ordinary write; the fence must skip it. out=%s", legacy) + } + + // The round trip itself is asserted on a bead created AFTER the switch, which + // participates, so its writes are versioned. + created = bdRunOK(t, bd, dir, "create", "switch round trip, created with history on") + id = extractFirstID(created, "vsw-") + if id == "" { + t.Fatalf("could not find a vsw- id in create output: %s", created) + } + + bdRunOK(t, bd, dir, "update", id, "--title", "switch round trip, edited") + + out := bdRunOK(t, bd, dir, "versions", id) + if !strings.Contains(out, "Versions of "+id) { + t.Fatalf("no versions listed after enabling via bd config set — the write and read planes have diverged again. out=%s", out) + } + if strings.Contains(out, "Recording is currently OFF") { + t.Errorf("recording reported OFF immediately after bd config set enabled it: %s", out) + } + + // Turning it back off must not hide what was already recorded: that is a + // claim about the store made from a fact about this invocation's config + // (same review, finding 3). + bdConfig(t, bd, dir, "set", "versioned-history.enabled", "false") + + off := bdRunOK(t, bd, dir, "versions", id) + if !strings.Contains(off, "Versions of "+id) { + t.Errorf("recorded versions vanished when recording was switched off; they are still real. out=%s", off) + } + if !strings.Contains(off, "Recording is currently OFF") { + t.Errorf("a listing taken while recording is off must say so, or it reads as current: %s", off) + } +} + +// extractFirstID pulls the first whitespace-delimited token carrying prefix +// out of command output. +func extractFirstID(out, prefix string) string { + for _, f := range strings.FieldsFunc(out, func(r rune) bool { + return r == ' ' || r == '\n' || r == '\t' || r == ':' || r == '\r' + }) { + if strings.HasPrefix(f, prefix) { + return strings.TrimRight(f, ".,") + } + } + return "" +} diff --git a/cmd/bd/versions_test.go b/cmd/bd/versions_test.go new file mode 100644 index 000000000..8e4b2ab98 --- /dev/null +++ b/cmd/bd/versions_test.go @@ -0,0 +1,419 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/issueops" +) + +type fakeVersionLister struct { + storage.DoltStorage + called bool + versions []storage.IssueVersion +} + +func (f *fakeVersionLister) ListVersions(_ context.Context, _ string) ([]storage.IssueVersion, error) { + f.called = true + return f.versions, nil +} + +type notAVersionLister struct{ storage.DoltStorage } + +// settingsVersionLister is a store with both halves `bd versions` reads: a +// settings plane whose row is canned, and a version history that is canned. +// The nil storage.DoltStorage underneath makes anything else the command +// touches panic, so a case cannot pass through a path it did not mean to +// exercise. +type settingsVersionLister struct { + fakeVersionLister + cfg issueops.WorkspaceConfig +} + +func (s *settingsVersionLister) WorkspaceConfig() (issueops.WorkspaceConfig, error) { + return s.cfg, nil +} + +// TestVersionsOffButRecordedStillLists is finding 3 from bee's #6661 review: +// a store that recorded for a month and was then switched off HAS versions, +// and refusing with "nothing is recorded" states a fact about this +// invocation's config as if it were a fact about the store. +func TestVersionsOffButRecordedStillLists(t *testing.T) { + backend := &fakeVersionLister{versions: []storage.IssueVersion{{Revision: 3}, {Revision: 2}}} + + got, err := runVersions(context.Background(), backend, "bd-1", false, true) + + if err != nil { + t.Fatalf("runVersions(recording=false, 2 rows) = %v, want the rows", err) + } + if len(got.Versions) != 2 { + t.Fatalf("got %d versions, want 2 — recorded history must survive the switch going off", len(got.Versions)) + } + if got.Recording { + t.Error("Recording should be false so the caller can say the listing ends where recording stopped") + } +} + +// TestVersionsOffAndEmptyRefusesWithRemedy is the empty arm of the pair above, +// and together they are the whole of finding 3: recording off decides the +// FOOTER, never whether to read. Off with rows recorded lists them; off with +// nothing recorded is the one case that refuses, and it refuses with the +// remedy rather than returning an empty list. +// +// That distinction is what #5898 exists to make. An empty list is a claim +// about the bead ("it has no versions"); the truth here is a claim about the +// store ("it never recorded any"). Returning the former for the latter is the +// wrong-answer-not-an-empty-one failure. +// +// This was TestVersionsRefusesWhenFlagOff, which asserted that the read was +// refused BEFORE the backend was consulted. Finding 3 removed that behaviour +// deliberately, so the assertion went with it -- but the fixture it had been +// asserting against stayed behind, built, unused and ending in `_ = backend`, +// under a name that still promised the old contract. Renamed to what it +// actually pins, and the dead fixture is gone. +// (bee-ghosttrack, #6661 third review, finding 3.) +func TestVersionsOffAndEmptyRefusesWithRemedy(t *testing.T) { + got, err := runVersions(context.Background(), &fakeVersionLister{}, "bd-1", false, true) + + if !errors.Is(err, errVersionedHistoryOff) { + t.Fatalf("runVersions(recording=false, empty) error = %v, want errVersionedHistoryOff", err) + } + if len(got.Versions) != 0 { + t.Errorf("returned %d versions, want none", len(got.Versions)) + } +} + +// TestVersionsUnresolvedIDIsNotAnEmptyBead is finding 4: falling through on an +// unresolved id is right (a deleted bead can still have versions), but only +// until the answer is empty. Then it is "no such bead", not "none yet". +func TestVersionsUnresolvedIDIsNotAnEmptyBead(t *testing.T) { + _, err := runVersions(context.Background(), &fakeVersionLister{}, "nosuch-id", true, false) + + if !errors.Is(err, errNoSuchBead) { + t.Fatalf("runVersions(resolved=false, empty) error = %v, want errNoSuchBead", err) + } +} + +// TestVersionsRefusesUnsupportedBackend pins that "this backend cannot answer" +// is distinguishable from "there are no versions" -- the third shape of an +// answer, not a smaller success. +func TestVersionsRefusesUnsupportedBackend(t *testing.T) { + _, err := runVersions(context.Background(), ¬AVersionLister{}, "bd-1", true, true) + + if !errors.Is(err, errVersionsUnsupported) { + t.Fatalf("runVersions(unsupported backend) error = %v, want errVersionsUnsupported", err) + } + if errors.Is(err, errVersionedHistoryOff) { + t.Error("an unsupported backend must not be reported as the feature being off: different fixes") + } +} + +// TestVersionsEmptyIsNotAnError pins the other side: a supported, enabled +// store with nothing recorded returns an empty list and no error. That is a +// truthful answer, and the command renders the no-backfill explanation. +func TestVersionsEmptyIsNotAnError(t *testing.T) { + got, err := runVersions(context.Background(), &fakeVersionLister{}, "bd-1", true, true) + if err != nil { + t.Fatalf("runVersions on an empty store = %v, want nil", err) + } + if len(got.Versions) != 0 { + t.Errorf("got %d versions, want 0", len(got.Versions)) + } +} + +// TestVersionsDecidesRecordingFromTheStoreNotTheProcessEnvironment is the +// reviewer's probe on #6661 (finding 2): BD_VERSIONED_HISTORY_ENABLED=1, the +// store's own setting unset, nothing recorded. +// +// `bd versions` only reads, so the variable cannot make the store record +// anything; it can only make THIS invocation claim that the store does. With +// the environment ORed in, a store that never recorded answered "No versions +// recorded for yet" -- the empty list this command's three outcomes exist +// to prevent, because an empty list is a claim about the bead and the truth is +// a claim about the store. The variable still turns recording on for the writes +// of a process that has it; what this command reports is the store's own +// setting. +// +// The rows that already pass stay in the table on purpose. They are the other +// half of the claim: a store whose setting says on keeps answering "none yet", +// and the variable never switches that off. +func TestVersionsDecidesRecordingFromTheStoreNotTheProcessEnvironment(t *testing.T) { + for _, tt := range []struct { + name string + env string + row string + wantRefusal bool + }{ + {"env on, row unset (the reviewer's probe)", "1", "", true}, + {"env on, row false", "1", "false", true}, + {"env off, row unset", "", "", true}, + {"env on, row true", "1", "true", false}, + {"env off, row true", "", "true", false}, + {"env 0 does not switch off a row that says true", "0", "true", false}, + } { + t.Run(tt.name, func(t *testing.T) { + // The variable has to be in place before viper is initialised, and + // viper has to be initialised at all: config.GetBool answers false + // until it is, which would make every env-on row pass whatever the + // code did with the variable. + t.Setenv("BD_VERSIONED_HISTORY_ENABLED", tt.env) + initConfigForTest(t) + st := &settingsVersionLister{cfg: &settingsPlane{value: tt.row}} + + recording := versionedHistoryEnabled(context.Background(), st) + got, err := runVersions(context.Background(), st, "bd-1", recording, true) + + if tt.wantRefusal { + if !errors.Is(err, errVersionedHistoryOff) { + t.Fatalf("env=%q store row=%q, nothing recorded: runVersions = (%+v, %v), want errVersionedHistoryOff: the process environment must not make a store that never recorded look like one that is recording", tt.env, tt.row, got, err) + } + return + } + if err != nil { + t.Fatalf("env=%q store row=%q, nothing recorded: runVersions error = %v, want an empty listing", tt.env, tt.row, err) + } + if !got.Recording { + t.Errorf("env=%q store row=%q: Recording = false, want true for a store whose own setting says on", tt.env, tt.row) + } + }) + } +} + +// TestVersionsHelpDoesNotOfferTheEnvironmentAsAReadSwitch is the other half of +// finding 2. The help text told people to run +// `BD_VERSIONED_HISTORY_ENABLED=1 bd versions ` "for one run". A read +// records nothing, so that line could only ever make the answer wrong; and +// once the command reports the store's own setting it does not work at all. +func TestVersionsHelpDoesNotOfferTheEnvironmentAsAReadSwitch(t *testing.T) { + if strings.Contains(versionsCmd.Long, "BD_VERSIONED_HISTORY_ENABLED=1 bd versions") { + t.Errorf("bd versions help still suggests BD_VERSIONED_HISTORY_ENABLED=1 bd versions ; a read cannot record anything, so the variable cannot make it answer for a store that never recorded.\nhelp:\n%s", versionsCmd.Long) + } +} + +// TestVersionListerPeelsDecorators is the regression guard for the bug this +// command shipped with first: cmd/bd holds the DECORATED store, VersionLister +// is an optional capability none of the decorators forwards, and asserting +// against the wrapper reported "unsupported" for a store that could serve +// versions. The capability must be found through the decorator chain. +func TestVersionListerPeelsDecorators(t *testing.T) { + inner := &fakeVersionLister{versions: []storage.IssueVersion{{Revision: 7}}} + decorated := storage.NewHookFiringStore(inner, nil) + + lister, ok := versionListerFor(decorated) + if !ok { + t.Fatal("versionListerFor could not see through the decorator chain") + } + got, err := lister.ListVersions(context.Background(), "bd-1") + if err != nil { + t.Fatalf("ListVersions through decorator = %v", err) + } + if len(got) != 1 || got[0].Revision != 7 { + t.Errorf("got %+v, want the inner store's revision 7", got) + } +} + +// TestRestrictionLabelsAreDistinct pins that the four removal reasons render +// as four different sentences. Collapsing them into one "gone" would erase +// the distinction R10 requires, and in particular "unknown" (this store has +// no lineage knowledge) is NOT "gone" -- the conformance suite pins that a +// never-synced store answers Unknown rather than Gone. +func TestRestrictionLabelsAreDistinct(t *testing.T) { + seen := map[string]string{} + for _, r := range []string{"gone_retention", "gone_erasure", "gone_reorganization", "unknown"} { + label := restrictionLabel(r) + if label == "" { + t.Errorf("restrictionLabel(%q) is empty", r) + } + if prev, dup := seen[label]; dup { + t.Errorf("restrictionLabel(%q) and (%q) both render %q; they are different answers", r, prev, label) + } + seen[label] = r + } + if restrictionLabel("unknown") == restrictionLabel("gone_erasure") { + t.Error("unknown must not read as gone: no lineage knowledge is not erasure") + } +} + +// TestRestrictionLabelAnswersAnythingOutsideTheVocabularyAsUnknown is finding 3 of the +// #6661 review, on the label's side. The as-of read answers a stored restriction outside +// the four-value vocabulary as unknown, and so is "live", which is never stored; the +// label printed such a string raw, so a listing could show "quarantined" for a version +// the as-of read calls unknown. It renders as unknown does. +func TestRestrictionLabelAnswersAnythingOutsideTheVocabularyAsUnknown(t *testing.T) { + want := restrictionLabel("unknown") + for _, r := range []string{"", "live", "gone-retention", "quarantined"} { + if got := restrictionLabel(r); got != want { + t.Errorf("restrictionLabel(%q) = %q, want %q (what unknown renders as): a value outside the vocabulary is not an answer", r, got, want) + } + } +} + +// TestVersionsJSONDoesNotCollideWithIssueRevision pins that the versions +// payload does not emit a "revision" key. +// +// types.Issue already ships `json:"revision"` and it is the row-lock CAS +// token -- an unrelated value, and a STRING where this one is a number. One +// key carrying two meanings, in the place machines read, is the `bd version` +// collision again with worse consequences: a script that learned one shape +// gets the other and cannot tell. Raised by bee-ghosttrack's consumer read on +// #5898, who reported a couple of hundred script files parsing bd --json. +func TestVersionsJSONDoesNotCollideWithIssueRevision(t *testing.T) { + blob, err := json.Marshal(storage.IssueVersion{Revision: 7, IssueID: "bd-1"}) + if err != nil { + t.Fatalf("marshal: %v", err) + } + + var generic map[string]any + if err := json.Unmarshal(blob, &generic); err != nil { + t.Fatalf("unmarshal: %v", err) + } + + if _, collides := generic["revision"]; collides { + t.Errorf(`IssueVersion emits a "revision" key; types.Issue already uses that key for the row-lock token. Got: %s`, blob) + } + got, ok := generic["local_revision"] + if !ok { + t.Fatalf(`IssueVersion has no "local_revision" key. Got: %s`, blob) + } + if n, isNum := got.(float64); !isNum || n != 7 { + t.Errorf("local_revision = %v, want 7", got) + } +} + +// TestVersionsJSONDoesNotPublishChangeAgent pins that the versions payload +// carries no "change_agent" key, while change_actor and change_message stay. +// +// Provenance V2 (gastownhall/beads#7510) replaces the ad hoc agent field with +// an actor chain, so a "change_agent" key in `bd versions --json` would become +// a public contract for a field that is on its way out. The column (migration +// 0067) and IssueVersion.ChangeAgent are kept; only the JSON key is withheld. +// The value is set here so an omitempty-style regression, which would hide it +// only when empty, still fails. +func TestVersionsJSONDoesNotPublishChangeAgent(t *testing.T) { + blob, err := json.Marshal(storage.IssueVersion{ + IssueID: "bd-1", + Revision: 3, + ChangeActor: "alice", + ChangeAgent: "some-agent", + ChangeMessage: "why it changed", + }) + if err != nil { + t.Fatalf("marshal: %v", err) + } + + var generic map[string]any + if err := json.Unmarshal(blob, &generic); err != nil { + t.Fatalf("unmarshal: %v", err) + } + if _, published := generic["change_agent"]; published { + t.Errorf(`IssueVersion emits a "change_agent" key; Provenance V2 replaces that field, so it must not become part of bd versions --json. Got: %s`, blob) + } + if got := generic["change_actor"]; got != "alice" { + t.Errorf(`change_actor = %v, want "alice". Got: %s`, got, blob) + } + if got := generic["change_message"]; got != "why it changed" { + t.Errorf(`change_message = %v, want "why it changed". Got: %s`, got, blob) + } +} + +// TestVersionsJSONIsOneObjectWhetherOrNotAnythingWasRecorded is finding 4: +// `bd versions --json` printed `null` for a store with nothing recorded +// yet (a nil slice through outputJSON) and never said whether recording was on, +// so a JSON consumer could tell neither "empty" from "no such list" nor a +// listing that ends where recording was switched off from one that is current. +// +// The shape is one object for every answer that is not an error: +// {"versions": [...], "recording": bool}, with [] for the empty list, never +// null. It goes through outputJSON, the way the command does, so the check sees +// what a script sees. +func TestVersionsJSONIsOneObjectWhetherOrNotAnythingWasRecorded(t *testing.T) { + t.Setenv("BD_JSON_ENVELOPE", "") + rows := []storage.IssueVersion{{Revision: 2, IssueID: "bd-1"}, {Revision: 1, IssueID: "bd-1"}} + + for _, tt := range []struct { + name string + backend *fakeVersionLister + recording bool + wantVersions int + }{ + {"nothing recorded yet", &fakeVersionLister{}, true, 0}, + {"recorded, recording on", &fakeVersionLister{versions: rows}, true, 2}, + {"recorded, recording since switched off", &fakeVersionLister{versions: rows}, false, 2}, + } { + t.Run(tt.name, func(t *testing.T) { + got, err := runVersions(context.Background(), tt.backend, "bd-1", tt.recording, true) + if err != nil { + t.Fatalf("runVersions: %v", err) + } + + out := captureStdout(t, func() error { return outputJSON(got) }) + + var payload map[string]json.RawMessage + if err := json.Unmarshal([]byte(out), &payload); err != nil { + t.Fatalf("--json output is not one JSON object: %v\n%s", err, out) + } + + rawVersions, ok := payload["versions"] + if !ok { + t.Fatalf(`--json output has no "versions" key. Got: %s`, out) + } + var list []json.RawMessage + if err := json.Unmarshal(rawVersions, &list); err != nil { + t.Fatalf(`"versions" is not an array: %v. Got: %s`, err, out) + } + if list == nil { + t.Errorf(`"versions" is null, want [] for an empty list. Got: %s`, out) + } + if len(list) != tt.wantVersions { + t.Errorf(`"versions" has %d entries, want %d. Got: %s`, len(list), tt.wantVersions, out) + } + + rawRecording, ok := payload["recording"] + if !ok { + t.Fatalf(`--json output has no "recording" key. Got: %s`, out) + } + var recording bool + if err := json.Unmarshal(rawRecording, &recording); err != nil { + t.Fatalf(`"recording" is not a boolean: %v. Got: %s`, err, out) + } + if recording != tt.recording { + t.Errorf(`"recording" = %v, want %v. Got: %s`, recording, tt.recording, out) + } + }) + } +} + +// TestVersionWithAnArgumentPointsAtVersions pins the guard on `bd version`. +// +// `bd version` prints bd's own build and `bd versions ` lists a bead's +// recorded versions: one character apart, unrelated meanings. Before the guard, +// a mistyped `bd version ` silently printed the build and exited 0, which +// reads as "this bead has no versions". It now exits 1 and names the command +// that was probably meant (#6661 review, finding 8). The guard is deliberate, +// and it is a change from the exit 0 it replaced, so both halves are pinned: +// the exit code and the hint, for the argument the user typed. +// +// The hint is matched as two fragments rather than one string so that spacing +// inside the message can change without losing the pin. +func TestVersionWithAnArgumentPointsAtVersions(t *testing.T) { + pinJSONOutput(t, false) + + var err error + stderr := captureStderr(t, func() { + err = versionCmd.RunE(versionCmd, []string{"bd-123"}) + }) + + code, isExitError := exitCodeFromError(err) + if !isExitError || code != 1 { + t.Fatalf("bd version bd-123 returned %v (exit error: %v, code: %d), want an exit error with code 1", err, isExitError, code) + } + for _, want := range []string{"Did you mean", "bd versions bd-123"} { + if !strings.Contains(stderr, want) { + t.Errorf("bd version bd-123 stderr is missing %q. Got: %q", want, stderr) + } + } +} diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 5dc748b16..cf3b5d4c9 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -292,7 +292,7 @@ export BEADS_ACTOR="my-github-handle" ## Project-Level Settings (Database) -These are written to the Dolt database by `bd config set` and have no env var override. Common namespaces: +These are written to the Dolt database by `bd config set` and have no env var override, with one exception: `versioned-history.enabled` also honors `BD_VERSIONED_HISTORY_ENABLED` (see [Versioned Issue History](#versioned-issue-history)). Common namespaces: | Namespace | Purpose | |---|---| @@ -312,6 +312,7 @@ These are written to the Dolt database by `bd config set` and have no env var ov | `min_hash_length`, `max_hash_length` | Adaptive ID bounds (defaults `3` and `8`) | | `max_collision_prob` | Hash ID collision tolerance (default `0.25`) | | `claim.pools` | Comma-separated pool aliases: placeholder assignees that any actor can take with `bd update --claim` (see [below](#claim-pools)). Unset by default, which turns pool claiming off | +| `versioned-history.enabled` | `true` records a version row for every issue write to this store; off by default. Unlike the rest of this table it also honors an environment variable, which can only turn it on (see [below](#versioned-issue-history)) | | `doctor.suppress.*` | Suppress specific `bd doctor` warnings by check slug (warnings only; errors always show) | Issue prefix (`issue_prefix`) is **not** settable via `bd config set` — use `bd init --prefix`, `bd bootstrap`, or `bd rename-prefix`. @@ -402,6 +403,25 @@ Claiming reads this key from the database only. A `claim.pools` value in `config - **`bd ready --claim`** takes only unassigned issues, so it skips pool-assigned ones even though `bd ready` lists them. Claim those by ID. - **Lease expiry.** If the claimer's lease expires, `bd reclaim` sets the issue back to `open` with no assignee. It does not return the issue to the pool alias, so a dispatcher that wants it back in the pool has to reassign it. +### Versioned Issue History + +While `versioned-history.enabled` is on, every accepted change to an issue that participates in history (one created while it is on, or that enters the issues plane while it is on: a wisp promoted or made persistent) stores a snapshot of it as a row of `issue_versions`, in the same transaction, and `bd versions ` lists them. It is off by default, and it does not backfill: an issue already on the issues plane when it was turned on is legacy, and editing it records nothing (the write fence, design §16.2b). + +```bash +bd config set versioned-history.enabled true # turn it on +bd config set versioned-history.enabled false # turn it off +``` + +- **The switch belongs to the store.** It is a row of the database `config` table, so `bd config set` writes it there and not to `config.yaml`. Every store answers for itself: a write routed into another workspace (a prefix-routed `bd update` or `bd close`, or `bd create --repo`) reads the *target* store's row, never the row of the workspace you launched `bd` from. +- **It replicates.** The config table travels with `bd dolt push` and `bd dolt pull`, so enabling it on one clone turns it on for every clone that pulls. +- **Single writer only.** Keep one writer at a time per store, and never two disconnected clones recording. Two writers that each mint the same revision for an issue collide on merge, and the pull fails (`VersionedHistoryConfigurer` in `internal/storage/storage.go`, and [#6379](https://github.com/gastownhall/beads/issues/6379) item 4). Because the setting replicates, turning it on is a decision for the whole store, not for one clone. +- **Turning it on checks the store first.** `bd config set versioned-history.enabled true` reads every issue the store would version and refuses, writing nothing, while any holds a value a version could not record: a number outside the I-JSON exact-integer range (magnitude above 2^53-1, whatever its spelling, fractions and exponent forms included), in its metadata or in a gate's timeout. A timeout is held as nanoseconds, so a gate that waits longer than about 104.25 days (9007199254740991 ns) is refused. It prints how many issues, which ones (the first 20), what is wrong with each and which field it is in. With history on, a write that introduces such a value already fails at once and commits nothing, and so does a later write to a row that already holds one if that row participates in history (it was created, or entered the issues plane, while history was on); a legacy row, created before history was on, is skipped by the write fence and stays writable, but the check still names it, because it cannot see participation. The fix depends on the field. For a number in metadata, one command per issue, and it works with history on or off: `bd update --metadata '{"": ""}'` to replace the value, or `bd update --unset-metadata ` to remove the key. `bd update` cannot change a gate's timeout, so while history is off, remove the gate with `bd delete --force` (without `--force`, `bd delete` only previews, and removing the gate also unblocks anything it was blocking), or set the `timeout_ns` column of the `issues` table within the limit with `bd sql`, which needs a server-backed store and does not run on an embedded one. Then run the command again. **There is no override**: the check has no flag, and `bd config set versioned-history.enabled false` never runs it. +- **What that check does not cover.** It runs once, at the moment you turn the setting on. A writer that does not record versions (an older `bd`, `bd sql`, a pull from a clone that had history off) can add such a row at any time afterwards, including right after the switch is on, and, if that row participates in history, the first write to it then fails with the same refusal; the same fix applies. Checking and writing the setting in one transaction would narrow that window, not close it. Turning recording on through `BD_VERSIONED_HISTORY_ENABLED` or `config.yaml` does not go through the command at all, so nothing is checked ahead of time there; a write to such a row is refused when it happens. Duplicate keys in metadata are refused too, but Dolt's JSON column collapses them when it stores a document, so on a Dolt store neither the check nor a write can see one; that refusal is a defense for another backend that keeps them. +- **The environment can turn it on, never off.** `BD_VERSIONED_HISTORY_ENABLED=1`, or a hand-written `enabled: true` under `versioned-history:` in `config.yaml`, applies to every store the process opens, routed targets included. The environment and the row are OR'd, not ranked, so `BD_VERSIONED_HISTORY_ENABLED=0` does not turn off a store whose row says `true`. Turn it off where it was turned on: `bd config set versioned-history.enabled false`. +- **Server-backed modes read the same row.** Proxied-server mode, and `bd serve` against a server-mode workspace, write through a unit-of-work provider rather than a store. The provider reads the row from its own database when it is constructed, through a short read-only transaction that is rolled back, so `bd config set versioned-history.enabled true` covers those modes too. It does not fall back to the environment alone. +- **The Go SDK is not covered.** Writers built on the Go SDK (the `beads` package: `Open`, `OpenFromConfig`, `OpenGated`, `OpenBestAvailable`) do not apply the switch: a write they make to a store whose row says `true` records no version. The `bd` binary's store factories apply it, and the events journal's switch has the same boundary. Opening a store through the SDK and applying both switches is a separate change. +- **The value is read when a store is opened.** A long-lived process such as `bd serve` keeps the value it read at start until it is restarted, so after changing the setting, restart it. + ## Sync and Federation Beads syncs exclusively through Dolt remotes (`bd dolt push` / `bd dolt pull`) with cell-level merge. Use `bd export` for issue portability and `bd backup` for restorable database backups. @@ -498,6 +518,7 @@ Selected commonly-used variables: | `BD_DOLT_AUTO_COMMIT` | Override `dolt.auto-commit` (`on`/`off`) | | `BD_DOLT_AUTO_PUSH`, `BD_DOLT_AUTO_PUSH_INTERVAL`, `BD_DOLT_AUTO_PUSH_TIMEOUT` | Override auto-push settings | | `BD_BACKUP_ENABLED`, `BD_BACKUP_INTERVAL`, `BD_BACKUP_GIT_REPO` | Override backup settings | +| `BD_VERSIONED_HISTORY_ENABLED` | Turn on `versioned-history.enabled` for every store the process opens. `1` turns recording on; `0` does not turn it off for a store whose row says `true` (see [Versioned Issue History](#versioned-issue-history)) | | `BD_AGENT_PROFILE` | Override `agent.profile` | | `BD_AI_MODEL` | Override AI model | | `BD_FEDERATION_REMOTE`, `BD_FEDERATION_SOVEREIGNTY` | Override federation settings | diff --git a/internal/config/config.go b/internal/config/config.go index 6095bd983..ed5597347 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -263,6 +263,18 @@ func Initialize() error { // own deletion itself. Env: BD_EVENTS_JOURNAL_AUTO_PRUNE. v.SetDefault("events-journal-auto-prune", true) v.SetDefault("audit.enabled", false) + // versioned-history.enabled activates the dual-write issue-version history + // (issue_versions, issues.current_revision) that migrations 0067+ provision. + // Default OFF: with it off the writer short-circuits and a store is + // byte-identical to one built without the feature, which is the contract + // gastownhall/beads#6135 ships under. Env: BD_VERSIONED_HISTORY_ENABLED. + // + // The flag is single-writer-only until the version_id primary key lands + // (design section 16.3 steps 1-5): issue_versions.revision and + // issues.current_revision are per-store ordinals, so two disconnected + // clones can both mint revision 8 for the same issue. Do not enable it on + // a store that more than one writer mutates. + v.SetDefault("versioned-history.enabled", false) v.SetDefault("no-db", false) v.SetDefault("no-hooks", false) v.SetDefault("db", "") diff --git a/internal/httpclient/served_participation_generation_test.go b/internal/httpclient/served_participation_generation_test.go index fed38636d..22552b96b 100644 --- a/internal/httpclient/served_participation_generation_test.go +++ b/internal/httpclient/served_participation_generation_test.go @@ -178,6 +178,9 @@ func TestServedParticipationGenerationFence(t *testing.T) { t.Run("ProceedsForPromotedRecord", func(t *testing.T) { conformance.RunParticipationGenerationProceedsForPromotedRecord(t, ctx, fixture) }) + t.Run("FlagOffSkipsUpdateOnParticipatingRecord", func(t *testing.T) { + conformance.RunParticipationGenerationFlagOffSkipsUpdateOnParticipatingRecord(t, ctx, fixture) + }) t.Run("FenceIgnoresSchemaSkewHatch", func(t *testing.T) { conformance.RunParticipationGenerationFenceIgnoresSchemaSkewHatch(t, ctx, fixture) }) diff --git a/internal/storage/BUILD.bazel b/internal/storage/BUILD.bazel index 8c7bb43a7..73737e702 100644 --- a/internal/storage/BUILD.bazel +++ b/internal/storage/BUILD.bazel @@ -65,6 +65,7 @@ go_library( "storage.go", "sync.go", "version_control.go", + "version_lister.go", "versioned.go", "wake_defers.go", ], diff --git a/internal/storage/createbatchequiv/createbatchequiv.go b/internal/storage/createbatchequiv/createbatchequiv.go index 78a93af91..07944b0d0 100644 --- a/internal/storage/createbatchequiv/createbatchequiv.go +++ b/internal/storage/createbatchequiv/createbatchequiv.go @@ -92,7 +92,7 @@ func largeScenarios() []scenario { // recorded from the code BEFORE the batch-create fast paths existed // (ebe3b6bcb), so the check holds the current code — fast and per-row // bodies alike, including changes the fast-path switch does not gate — to -// what that code stored, apart from two deliberate behavior changes since. +// what that code stored, apart from three deliberate behavior changes since. // // First, the create role defaults an empty created_by to the request's actor // (issueops.PreparePublicCreateRequest), so apply.json's issues, wisps and @@ -107,6 +107,17 @@ func largeScenarios() []scenario { // their seeded rows instead (stampParticipation), so their dependency adds, // closes and reopens mint as they did before the fence, and their digests // did not move. +// +// Third, the issues_updated_at digests of small, depadd, import458 and +// import458-reject-stale were re-recorded when the update-shaped version +// mint stopped moving updated_at. Its UPDATE of current_revision used to +// fire the column's ON UPDATE CURRENT_TIMESTAMP, so a row minted a second +// time read the wall clock and dropped out of the compared set. Such a row +// now keeps the timestamp it has with versioned history off: the seeded one +// for depadd's eq-db1 and eq-dl, whose dependency adds mint again, and the +// import source's for the in-batch duplicates eq-n2 (small) and eq-b3 and +// eq-b250 (both import458 scenarios). apply and waitsfor did not move: their +// re-minted rows take the wall clock from their other writes anyway. const GoldenDirEnv = "CREATEBATCHEQUIV_GOLDEN_DIR" // Run seeds fresh databases, applies each scenario's batch through the fast diff --git a/internal/storage/createbatchequiv/golden/depadd.json b/internal/storage/createbatchequiv/golden/depadd.json index c3ad1dc71..667115076 100644 --- a/internal/storage/createbatchequiv/golden/depadd.json +++ b/internal/storage/createbatchequiv/golden/depadd.json @@ -33,8 +33,8 @@ "sha256": "27fb5cdc6779efe18023610f468f30f7f3c21c9b47fe2b2660b280ed132785d1" }, "issues_updated_at": { - "rows": 13, - "sha256": "7119d0e3827b68a653156835b19fbae4b2b497aa17743ca2abce9ce387870831" + "rows": 15, + "sha256": "bf6295c3ca28217417b2e545df8d756b756f3cea3eb3d0284dadb97b7c8c5377" }, "labels": { "rows": 0, diff --git a/internal/storage/createbatchequiv/golden/import458-reject-stale.json b/internal/storage/createbatchequiv/golden/import458-reject-stale.json index caa557ad6..997198eb4 100644 --- a/internal/storage/createbatchequiv/golden/import458-reject-stale.json +++ b/internal/storage/createbatchequiv/golden/import458-reject-stale.json @@ -33,8 +33,8 @@ "sha256": "039484a3e75d7cb2f0939177e2504fa514602d33a40112092978db6b2de7316d" }, "issues_updated_at": { - "rows": 400, - "sha256": "7b83c4335f54068965727a7230e71ad966f75fe6d23170963a142ab1654768cc" + "rows": 402, + "sha256": "80b591ea31a9821b44f006c8dca6d5390923b1e6d55689e0d776fecbb7cdf44b" }, "labels": { "rows": 1165, diff --git a/internal/storage/createbatchequiv/golden/import458.json b/internal/storage/createbatchequiv/golden/import458.json index f6cfacb92..46decb0f7 100644 --- a/internal/storage/createbatchequiv/golden/import458.json +++ b/internal/storage/createbatchequiv/golden/import458.json @@ -33,8 +33,8 @@ "sha256": "d4245c8b62b007795d9c7f293f30ac0f341ffc541cf50aa319a24c8cbed3505a" }, "issues_updated_at": { - "rows": 401, - "sha256": "a256b57e6a67ca0717d1200b4ad867abc1ccff8f9b088bc6c331c18fa6915c7f" + "rows": 403, + "sha256": "de6dbabb1bdd928fe633356cf3199f6124f440168c9ef9d56d76eea9fff0df02" }, "labels": { "rows": 1166, diff --git a/internal/storage/createbatchequiv/golden/small.json b/internal/storage/createbatchequiv/golden/small.json index 83aec24c6..da494c75e 100644 --- a/internal/storage/createbatchequiv/golden/small.json +++ b/internal/storage/createbatchequiv/golden/small.json @@ -33,8 +33,8 @@ "sha256": "742c88693fbef407c0648e75ccd42618cb7722ffc6b62546b6a04d3b8dc28254" }, "issues_updated_at": { - "rows": 41, - "sha256": "2b1590dbb503fb2a1dcc71f78261f593f3c9144c4530c80325428b21d5e2164d" + "rows": 42, + "sha256": "ea303585edc660d2e4ea6e982668fefe4f85efa8b43a6e3d8bcb65850a08b856" }, "labels": { "rows": 65, diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 506ffafed..4be1fec0b 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -289,6 +289,7 @@ go_test( "merge_strategy_test.go", "metadata_cas_contract_test.go", "metadata_merge_test.go", + "metadata_number_fidelity_test.go", "metadata_schema_test.go", "metadata_validation_test.go", "migration_0052_test.go", @@ -372,6 +373,7 @@ go_test( "version_reconciler_contract_test.go", "version_scope_completeness_test.go", "version_staging_completeness_test.go", + "versioned_read_contract_test.go", "versioned_test.go", "wisp_custom_type_tx_test.go", "wisp_delete_cascade_integration_test.go", diff --git a/internal/storage/dolt/dualwrite_participation_generation_contract_test.go b/internal/storage/dolt/dualwrite_participation_generation_contract_test.go index 5380b4db2..b81940b47 100644 --- a/internal/storage/dolt/dualwrite_participation_generation_contract_test.go +++ b/internal/storage/dolt/dualwrite_participation_generation_contract_test.go @@ -35,6 +35,9 @@ func TestParticipationGenerationFence(t *testing.T) { t.Run("ProceedsForPromotedRecord", func(t *testing.T) { conformance.RunParticipationGenerationProceedsForPromotedRecord(t, ctx, fixture) }) + t.Run("FlagOffSkipsUpdateOnParticipatingRecord", func(t *testing.T) { + conformance.RunParticipationGenerationFlagOffSkipsUpdateOnParticipatingRecord(t, ctx, fixture) + }) t.Run("WispPromotionBeginsHistory", func(t *testing.T) { conformance.RunParticipationGenerationWispPromotionBeginsHistory(t, ctx, fixture) }) diff --git a/internal/storage/dolt/metadata_number_fidelity_test.go b/internal/storage/dolt/metadata_number_fidelity_test.go new file mode 100644 index 000000000..ae0fb076f --- /dev/null +++ b/internal/storage/dolt/metadata_number_fidelity_test.go @@ -0,0 +1,272 @@ +package dolt + +import ( + "bytes" + "database/sql" + "encoding/json" + "errors" + "fmt" + "math/big" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// metadataNumberCorpus is what TestVersionedHistoryKeepsNumbersTheStoreDistinguishesApart writes +// through the store. It groups literals that share one nearest binary64 (0.1 and its longer +// spellings, 1 and its spellings, pi at several precisions), because those are the literals a +// store could tell apart and RFC 8785 cannot, and it includes the values outside the I-JSON +// exact-integer range that the mint has to refuse. The embedded leg carries the same list in +// embeddeddolt/metadata_number_fidelity_test.go; keep the two in step. +var metadataNumberCorpus = []string{ + "0.1", "0.10000000000000000555", "0.1000000000000000055511151231257827", + "1", "1.0", "1.00", "1e0", "10e-1", + "3.14159265358979323846", "3.141592653589793", "3.1415926535897932", + "0.3", "0.30000000000000004", + "0", "-0", "0.0", "1e-400", + "9007199254740991", + "9007199254740992", "9007199254740993", "9007199254740993.5", "9007199254740994", + "1727000000000000000", "1727000000000000123", + "1e300", "123456789012345678901234567890", +} + +// metadataNumbersOutsideTheRange are the corpus literals no store normalization can bring back +// inside 2^53-1: each is refused at the mint, and the write it belonged to must not survive. +var metadataNumbersOutsideTheRange = map[string]bool{ + "9007199254740992": true, "9007199254740993": true, "9007199254740993.5": true, "9007199254740994": true, + "1727000000000000000": true, "1727000000000000123": true, + "1e300": true, "123456789012345678901234567890": true, +} + +// TestVersionedHistoryKeepsNumbersTheStoreDistinguishesApart pins the safety property the +// number-admission rule rests on, against the real store rather than against what one version +// of it is known to do. +// +// RFC 8785 turns every number into its nearest double, so two literals that differ by less than +// a double can share one canonical form. That is harmless only while the store cannot hold them +// apart: if it stores both as the same value they are the same state. The rule was argued from a +// measurement of one Dolt. A Dolt that started keeping exact decimals would silently reopen the +// collision, so this asserts the property instead of the measurement, in four phases: +// +// 1. History OFF. The store accepts every literal, including the ones outside the I-JSON range, +// and the table logs what it actually holds for each. This is how a row comes to hold such a +// number without the mint ever having seen it. +// 2. History ON, two kinds of row. (a) The rows phase 1 made never participated in history +// (participation_generation is NULL), so design §16.2b's write fence skips an update-shaped +// write to them whole, before the mint's admission gate: the write succeeds, mints nothing +// and leaves the stamp NULL. (b) A row that DOES participate (created with history on, so +// stamped) and then comes to hold such a number by a write that does not mint is refused on +// its next write, and the write changes nothing. This is the hazard the switch's pre-enable +// check exists for. +// 3. History ON, fresh rows. Each literal is written and read back, and no two literals that +// read back as DIFFERENT numbers may have been both admitted with the SAME canonical form. +// A literal the mint refuses must leave neither an issue row nor a version row, because the +// mint runs in the writer's own transaction. +// +// The logged table (written, held or read back, canonical or refused, with the Dolt version) is +// the record of what this Dolt does today. +func TestVersionedHistoryKeepsNumbersTheStoreDistinguishesApart(t *testing.T) { + store, storeCleanup := setupTestStore(t) + defer storeCleanup() + ctx, cancel := testContext(t) + defer cancel() + configurer, ok := any(store).(storage.VersionedHistoryConfigurer) + if !ok { + t.Fatalf("%T does not implement storage.VersionedHistoryConfigurer", store) + } + defer configurer.SetVersionedHistoryEnabled(false) + + var doltVersion string + if err := store.db.QueryRowContext(ctx, `SELECT dolt_version()`).Scan(&doltVersion); err != nil { + doltVersion = "unknown (" + err.Error() + ")" + } + newIssue := func(id, lit string) *types.Issue { + return &types.Issue{ + ID: id, Title: "n " + lit, IssueType: types.TypeTask, Status: types.StatusOpen, + Metadata: json.RawMessage(`{"n":` + lit + `}`), + } + } + + // Phase 1: history off. What does the store hold for a number outside the range? + t.Logf("metadata number normalization, server leg, dolt %s", doltVersion) + configurer.SetVersionedHistoryEnabled(false) + var held, heldLiterals []string + for i, lit := range metadataNumberCorpus { + if !metadataNumbersOutsideTheRange[lit] { + continue + } + id := fmt.Sprintf("nfoff-%02d", i) + if err := store.CreateIssue(ctx, newIssue(id, lit), "actor"); err != nil { + t.Logf(" history off: %-34s the store refused it: %v", lit, err) + continue + } + issue, err := store.GetIssue(ctx, id) + if err != nil { + t.Fatalf("reading %s back: %v", id, err) + } + stored, _ := storedNumberAndCanonicalMetadata(t, issue.Metadata, nil) + t.Logf(" history off: %-34s held as %s", lit, stored) + held = append(held, id) + heldLiterals = append(heldLiterals, lit) + } + + rowState := func(id string) (versions int, revision int64, generation sql.NullInt64) { + t.Helper() + if err := store.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM issue_versions WHERE issue_id = ?`, id).Scan(&versions); err != nil { + t.Fatalf("counting the version rows of %s: %v", id, err) + } + if err := store.db.QueryRowContext(ctx, `SELECT current_revision, participation_generation FROM issues WHERE id = ?`, id).Scan(&revision, &generation); err != nil { + t.Fatalf("reading the revision and participation of %s: %v", id, err) + } + return versions, revision, generation + } + + // Phase 2a: history on, rows that never participated. Each was made with history off, so it is + // a legacy record (participation_generation NULL) and design §16.2b's write fence skips an + // update-shaped write to it whole, before the mint's admission gate: the write succeeds, + // nothing is versioned, nothing is refused and the stamp stays NULL, whatever the metadata holds. + configurer.SetVersionedHistoryEnabled(true) + for _, id := range held { + _, revisionBefore, _ := rowState(id) + if err := store.UpdateIssue(ctx, id, map[string]any{"title": "retitled"}, "actor"); err != nil { + t.Errorf("updating %s, a legacy row that holds a number outside the I-JSON range, = %v, want success: the fence skips a legacy record before the mint's admission gate", id, err) + continue + } + after, err := store.GetIssue(ctx, id) + if err != nil || after.Title != "retitled" { + t.Errorf("the update of legacy row %s did not land: title %q (err %v), want %q", id, after.Title, err, "retitled") + } + versions, revision, generation := rowState(id) + if versions != 0 || generation.Valid || revision != revisionBefore { + t.Errorf("the update of legacy row %s versioned or promoted it: version rows %d (want 0), participation_generation %v (want NULL), current_revision %d -> %d (want unchanged)", + id, versions, generation, revisionBefore, revision) + } + } + + // Phase 2b: history on, rows that participate. Each is created with history on holding an + // in-range stand-in, so the real create-shaped mint stamps it; the stamp is read first, because + // that is what makes this case about participation. History is then switched off and the row's + // metadata replaced through the store, which is how a row comes to hold such a number without the + // mint having seen it, and switched on again. The next write is refused and changes nothing. + for k, lit := range heldLiterals { + id := fmt.Sprintf("nfpart-%02d", k) + configurer.SetVersionedHistoryEnabled(true) + if err := store.CreateIssue(ctx, newIssue(id, "1"), "actor"); err != nil { + t.Fatalf("creating the participating row %s with history on: %v", id, err) + } + if _, _, generation := rowState(id); !generation.Valid { + t.Fatalf("%s was created with history on but participation_generation is NULL: the control failed, so this case would not be about participation", id) + } + configurer.SetVersionedHistoryEnabled(false) + if err := store.UpdateIssue(ctx, id, map[string]any{"metadata": json.RawMessage(`{"n":` + lit + `}`)}, "actor"); err != nil { + t.Fatalf("replacing the metadata of %s with %s while history is off: %v", id, lit, err) + } + configurer.SetVersionedHistoryEnabled(true) + before, err := store.GetIssue(ctx, id) + if err != nil { + t.Fatalf("reading %s: %v", id, err) + } + versionsBefore, revisionBefore, generationBefore := rowState(id) + err = store.UpdateIssue(ctx, id, map[string]any{"title": "retitled"}, "actor") + if !errors.Is(err, issueops.ErrIntegerNotRepresentable) { + t.Errorf("updating %s, a participating row that holds %s, = %v, want ErrIntegerNotRepresentable", id, lit, err) + } + after, getErr := store.GetIssue(ctx, id) + versions, revision, generation := rowState(id) + if getErr != nil || after.Title != before.Title || versions != versionsBefore || revision != revisionBefore || generation != generationBefore { + t.Errorf("the refused update of %s left a trace: title %q -> %q, version rows %d -> %d, current_revision %d -> %d, participation_generation %v -> %v (err %v)", + id, before.Title, after.Title, versionsBefore, versions, revisionBefore, revision, generationBefore, generation, getErr) + } + } + + // Phase 3: history on, fresh rows. The property. + configurer.SetVersionedHistoryEnabled(true) + type admittedLiteral struct{ written, storedAs, canonical string } + var admitted []admittedLiteral + for i, lit := range metadataNumberCorpus { + id := fmt.Sprintf("nfid-%02d", i) + err := store.CreateIssue(ctx, newIssue(id, lit), "actor") + if err != nil { + if !metadataNumbersOutsideTheRange[lit] { + t.Errorf("%s was refused although it is an ordinary number: %v", lit, err) + } else if !errors.Is(err, issueops.ErrIntegerNotRepresentable) { + t.Errorf("%s was refused with %v, want ErrIntegerNotRepresentable", lit, err) + } + if _, getErr := store.GetIssue(ctx, id); !errors.Is(getErr, storage.ErrNotFound) { + t.Errorf("%s was refused but its issue exists afterwards (GetIssue = %v): the refusal did not abort the write", lit, getErr) + } + var versions int + if scanErr := store.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM issue_versions WHERE issue_id = ?`, id).Scan(&versions); scanErr != nil || versions != 0 { + t.Errorf("%s was refused but left %d version rows (err %v)", lit, versions, scanErr) + } + t.Logf(" history on: %-34s refused", lit) + continue + } + if metadataNumbersOutsideTheRange[lit] { + t.Errorf("%s is outside the I-JSON exact-integer range and was written", lit) + } + issue, err := store.GetIssue(ctx, id) + if err != nil { + t.Fatalf("reading %s back: %v", id, err) + } + var durable []byte + if err := store.db.QueryRowContext(ctx, + `SELECT durable_state FROM issue_versions WHERE issue_id = ? ORDER BY revision DESC LIMIT 1`, id).Scan(&durable); err != nil { + t.Fatalf("reading the version of %s: %v", id, err) + } + stored, canonical := storedNumberAndCanonicalMetadata(t, issue.Metadata, durable) + t.Logf(" history on: %-34s read back as %-28s canonical metadata %s", lit, stored, canonical) + admitted = append(admitted, admittedLiteral{lit, stored, canonical}) + } + + if len(admitted) < 12 { + t.Fatalf("only %d of %d literals were admitted; the property is close to vacuous", len(admitted), len(metadataNumberCorpus)) + } + first := map[string]admittedLiteral{} + for _, a := range admitted { + prior, seen := first[a.canonical] + if !seen { + first[a.canonical] = a + continue + } + if !sameNumber(prior.storedAs, a.storedAs) { + t.Errorf("literals %s and %s read back from the store as different numbers (%s and %s) but share the canonical form %s: "+ + "two distinct stored states would carry one content token", prior.written, a.written, prior.storedAs, a.storedAs, a.canonical) + } + } +} + +// storedNumberAndCanonicalMetadata returns the number the store handed back in the issue's +// metadata, and, when durable is given, the metadata member of the canonical bytes the mint +// stored for it. +func storedNumberAndCanonicalMetadata(t *testing.T, readBack json.RawMessage, durable []byte) (stored, canonical string) { + t.Helper() + var back struct { + N json.Number `json:"n"` + } + dec := json.NewDecoder(bytes.NewReader(readBack)) + dec.UseNumber() + if err := dec.Decode(&back); err != nil { + t.Fatalf("decoding the metadata read back %s: %v", readBack, err) + } + if durable == nil { + return back.N.String(), "" + } + var version struct { + Metadata json.RawMessage `json:"metadata"` + } + if err := json.Unmarshal(durable, &version); err != nil { + t.Fatalf("decoding durable_state %s: %v", durable, err) + } + return back.N.String(), strings.TrimSpace(string(version.Metadata)) +} + +// sameNumber reports whether two JSON number literals denote the same value, compared exactly. +func sameNumber(a, b string) bool { + x, okA := new(big.Rat).SetString(a) + y, okB := new(big.Rat).SetString(b) + return okA && okB && x.Cmp(y) == 0 +} diff --git a/internal/storage/dolt/versioned.go b/internal/storage/dolt/versioned.go index ad0c4a7a6..ab93db846 100644 --- a/internal/storage/dolt/versioned.go +++ b/internal/storage/dolt/versioned.go @@ -33,6 +33,24 @@ func (s *DoltStore) History(ctx context.Context, issueID string) ([]*storage.His return result, err } +// ListVersions implements storage.VersionLister over issue_versions. +// +// Plain withReadTx, not the long-timeout variant History needs: this reads an +// ordinary user table by its composite primary key prefix, not the +// dolt_history_issues scan whose cost History has to budget for. +func (s *DoltStore) ListVersions(ctx context.Context, issueID string) ([]storage.IssueVersion, error) { + var result []storage.IssueVersion + err := s.withReadTx(ctx, func(tx *sql.Tx) error { + var err error + result, err = issueops.ListVersionsInTx(ctx, tx, issueID) + if err != nil { + return wrapQueryError("list issue versions", err) + } + return nil + }) + return result, err +} + // HistoricalIssueIDs reports which of ids appear in HEAD's committed history // of the issues table. Implements storage.HistoryPresence. // @@ -278,3 +296,9 @@ func isSafeCommitRef(s string) bool { } return true } + +// Compile-time proof that this leg serves versions. It lives here rather than +// in package storage because storage cannot import the legs. Dropping +// ListVersions is then a build failure rather than a silent demotion to +// "this backend cannot serve version history" at runtime. +var _ storage.VersionLister = (*DoltStore)(nil) diff --git a/internal/storage/dolt/versioned_read_contract_test.go b/internal/storage/dolt/versioned_read_contract_test.go new file mode 100644 index 000000000..79d8b3118 --- /dev/null +++ b/internal/storage/dolt/versioned_read_contract_test.go @@ -0,0 +1,337 @@ +package dolt + +import ( + "context" + "crypto/rand" + "database/sql" + "encoding/hex" + "fmt" + "os" + "testing" + "time" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// asOfReadHarness is this leg's Design D dispatch table (gastownhall/beads#5898 +// revision 9, gastownhall/beads#6136): the conformance suite +// invents storeID values to select among N genuinely isolated stores, and +// isolating them is entirely this harness's job -- AsOfReadInTx itself is +// storeID-oblivious (see the package doc on +// internal/storage/issueops/asof_read.go). Each storeID gets its own fresh +// *DoltStore, built lazily on first use (buildAsOfReadStore) and torn down +// via t.Cleanup on the TestAsOfReadContract-level *testing.T. created tracks +// which (storeID, id) pairs already have a base issues row, since mintAt's +// second and later calls for the same pair must only add a historical +// issue_versions row, never a second base create. +// +// store() cannot use setupConcurrentTestStore. AsOfReadFixture's hook types +// (Resolve/MintAt/MakeUnanswerable) carry no *testing.T of their own, so a +// store built lazily inside a hook -- itself invoked from inside a t.Run +// subtest closure, sometimes nested two deep (the restriction-loop case +// below) -- has access only to whichever *testing.T this harness was +// constructed with: necessarily the OUTER TestAsOfReadContract's t, captured +// before any t.Run. setupConcurrentTestStore calls t.Skip/t.Fatalf +// internally, and calling FailNow/SkipNow on a *testing.T from a goroutine +// other than the one running that exact test -- which a t.Run subtest's +// closure always is, relative to the outer test -- panics with "subtest may +// have called FailNow on a parent test" (go's testing package requires +// FailNow/SkipNow to run only on the goroutine executing that specific +// test). buildAsOfReadStore below is a Fatalf/Skip-free twin of +// setupConcurrentTestStore's body for exactly this reason: its failures come +// back as a plain error, which flows through the calling hook's own +// (..., error) return to conformance.RunAsOfReadXXX, which correctly +// Fatalfs using the SUBTEST's own *testing.T on the SUBTEST's own goroutine. +// Dolt-availability skipping itself is handled once, up front in +// TestAsOfReadContract's own body via skipIfNoDolt(t) -- safe there because +// that runs before any t.Run, in the one goroutine that actually owns t. +type asOfReadHarness struct { + t *testing.T + stores map[string]*DoltStore + created map[[2]string]bool +} + +func newAsOfReadHarness(t *testing.T) *asOfReadHarness { + return &asOfReadHarness{ + t: t, + stores: map[string]*DoltStore{}, + created: map[[2]string]bool{}, + } +} + +func (h *asOfReadHarness) store(ctx context.Context, storeID string) (*DoltStore, error) { + if s, ok := h.stores[storeID]; ok { + return s, nil + } + store, cleanup, err := buildAsOfReadStore(ctx) + if err != nil { + return nil, err + } + // t.Cleanup only appends to a mutex-guarded slice -- unlike Fatalf/Skip it + // never calls runtime.Goexit, so registering it on the outer h.t from + // inside a subtest's goroutine is safe. h.t's own cleanups run only after + // ALL of TestAsOfReadContract's t.Run calls have returned (none of them, + // nor their own nested t.Run calls, ever invoke t.Parallel), so every + // store this harness builds stays open for the whole test, matching + // created's own whole-test lifetime. + h.t.Cleanup(cleanup) + h.stores[storeID] = store + return store, nil +} + +// buildAsOfReadStore constructs one fresh, isolated *DoltStore -- the same +// shape setupConcurrentTestStore builds (own temp dir, own database, +// MaxOpenConns=2, issue_prefix seeded), minus every t.Fatalf/t.Skip call. +// The issue_prefix seed is NOT optional here, even though +// DoltStore.CreateIssue's single-issue path sets SkipPrefixValidation +// unconditionally (matches its own documented legacy behavior): that only +// skips validating an explicit ID's OWN prefix against the configured one. +// A separate, unconditional precondition -- issueops/helpers.go's +// resolvePrefix, "issue_prefix config is missing", wrapped in +// storage.ErrNotInitialized (bd-166) -- fails every write, including ones +// under SkipPrefixValidation, when no issue_prefix is configured at all. +// Confirmed the hard way: the first version of this function omitted the +// SetConfig call on the strength of the SkipPrefixValidation reasoning +// alone, and every MintAt in a real run failed with exactly that +// ErrNotInitialized. See the asOfReadHarness doc comment for why this +// function cannot take a *testing.T and call Fatalf/Skip the way +// setupConcurrentTestStore does. +// +// It also departs from setupConcurrentTestStore in when it releases +// testSem: setupConcurrentTestStore's callers build at most a couple of +// stores and hold the slot for the whole test via t.Cleanup, but this +// harness can lazily build many stores across one TestAsOfReadContract run +// (up to nine, across every RunAsOfReadXXX case), all held open +// simultaneously until h.t's own cleanups run at the very end. Holding one +// slot per store for the whole test against testSem's cap of 2 would make +// the third store's construction block on a slot only some OTHER test's +// completion could free -- an outright hang when this test runs alone (e.g. +// -run TestAsOfReadContract), since nothing else would ever be running to +// free one. testSem exists to bound concurrent SETUP load on the shared +// container, and this harness's own setups are already serialized (no +// subtest here calls t.Parallel), so acquiring only around New() and +// releasing immediately after -- rather than for the store's whole open +// lifetime -- still honors that purpose without the hang. +func buildAsOfReadStore(ctx context.Context) (*DoltStore, func(), error) { + acquireTestSlot() + defer releaseTestSlot() + + ctx, cancel := context.WithTimeout(ctx, testTimeout) + defer cancel() + + tmpDir, err := os.MkdirTemp("", "dolt-asof-test-*") + if err != nil { + return nil, nil, fmt.Errorf("as-of read fixture: create temp dir: %w", err) + } + + dbSuffix := make([]byte, 6) + if _, err := rand.Read(dbSuffix); err != nil { + os.RemoveAll(tmpDir) + return nil, nil, fmt.Errorf("as-of read fixture: generate db name: %w", err) + } + + cfg := &Config{ + Path: tmpDir, + CommitterName: "test", + CommitterEmail: "test@example.com", + Database: "testdb_asof_" + hex.EncodeToString(dbSuffix), + MaxOpenConns: 2, // matches setupConcurrentTestStore: avoid overwhelming the test container + CreateIfMissing: true, + } + + store, err := New(ctx, cfg) + if err != nil { + os.RemoveAll(tmpDir) + return nil, nil, fmt.Errorf("as-of read fixture: create Dolt store: %w", err) + } + store.db.SetMaxIdleConns(0) + store.db.SetConnMaxIdleTime(time.Second) + + if err := store.SetConfig(ctx, "issue_prefix", "asof"); err != nil { + store.Close() + os.RemoveAll(tmpDir) + return nil, nil, fmt.Errorf("as-of read fixture: set issue_prefix: %w", err) + } + + cleanup := func() { + store.Close() + os.RemoveAll(tmpDir) + } + return store, cleanup, nil +} + +// ensureCreated bare-creates id in storeID's store the first time it is +// asked for, giving RecordVersionAtInTx (called from mintAt) a base issues +// row to snapshot -- recordVersionAtInTx starts with GetIssueInTx, so it has +// nothing to read until this has run once. +func (h *asOfReadHarness) ensureCreated(ctx context.Context, storeID, id string) error { + key := [2]string{storeID, id} + if h.created[key] { + return nil + } + store, err := h.store(ctx, storeID) + if err != nil { + return err + } + issue := &types.Issue{ + ID: id, + Title: "as-of read fixture issue", + Status: types.StatusOpen, + Priority: 3, + IssueType: types.TypeTask, + } + if err := store.CreateIssue(ctx, issue, "asof-read-fixture"); err != nil { + return fmt.Errorf("as-of read fixture: bare-create %s: %w", id, err) + } + h.created[key] = true + return nil +} + +func (h *asOfReadHarness) resolve(ctx context.Context, storeID, id string, selector conformance.AsOfSelector) (conformance.AsOfReadResult, error) { + store, err := h.store(ctx, storeID) + if err != nil { + return conformance.AsOfReadResult{}, err + } + var result issueops.AsOfReadResult + err = store.withReadTx(ctx, func(tx *sql.Tx) error { + var err error + result, err = issueops.AsOfReadInTx(ctx, tx, storeID, id, toIssueopsSelector(selector)) + return err + }) + if err != nil { + return conformance.AsOfReadResult{}, err + } + return toConformanceResult(result), nil +} + +// mintAt is the two-phase implementation MintAt's contract needs: phase one +// (ensureCreated) runs only once per (storeID, id); phase two +// (RecordVersionAtInTx) runs every call and always supplies the row mintAt +// reports the address of. RecordVersionAtInTx deliberately does not gate on +// versionedHistoryEnabled (this harness never turns that on for its stores), +// so it always mints, and reading MAX(revision) back inside the same +// transaction (read-your-writes) is how we learn which revision it picked -- +// RecordVersionAtInTx itself reports no revision number. +func (h *asOfReadHarness) mintAt(ctx context.Context, storeID, id string, acceptedAt time.Time) (conformance.Address, error) { + if err := h.ensureCreated(ctx, storeID, id); err != nil { + return "", err + } + store, err := h.store(ctx, storeID) + if err != nil { + return "", err + } + var revision int64 + err = store.withWriteTx(ctx, func(tx *sql.Tx) error { + if err := issueops.RecordVersionAtInTx(ctx, tx, id, "asof-read-fixture", acceptedAt); err != nil { + return err + } + return tx.QueryRowContext(ctx, + `SELECT MAX(revision) FROM issue_versions WHERE issue_id = ?`, id, + ).Scan(&revision) + }) + if err != nil { + return "", err + } + return conformance.Address(issueops.AsOfVersionAddress(id, revision)), nil +} + +func (h *asOfReadHarness) makeUnanswerable(ctx context.Context, storeID, id string, restriction conformance.Restriction) error { + asOfRestriction, ok := conformanceToAsOfRestriction[restriction] + if !ok { + return fmt.Errorf("as-of read fixture: unmapped restriction %v", restriction) + } + store, err := h.store(ctx, storeID) + if err != nil { + return err + } + reason := fmt.Sprintf("as-of read fixture: made unanswerable as %s", restriction) + return store.withWriteTx(ctx, func(tx *sql.Tx) error { + return issueops.MarkLatestVersionRemovedInTx(ctx, tx, id, asOfRestriction, reason) + }) +} + +// conformanceToAsOfRestriction and its inverse are this leg's translation +// boundary between backend/conformance.Restriction (hyphenated String(), +// int-backed) and issueops.AsOfRestriction (underscored, string-backed -- +// its values ARE the removed_restriction column's contents). issueops must +// stay conformance-oblivious and conformance must stay backend-oblivious +// (internal/storage/issueops/asof_read.go's package doc), so this mapping is +// deliberately duplicated per leg rather than shared centrally. +var conformanceToAsOfRestriction = map[conformance.Restriction]issueops.AsOfRestriction{ + conformance.RestrictionLive: issueops.AsOfRestrictionLive, + conformance.RestrictionGoneRetention: issueops.AsOfRestrictionGoneRetention, + conformance.RestrictionGoneErasure: issueops.AsOfRestrictionGoneErasure, + conformance.RestrictionGoneReorganization: issueops.AsOfRestrictionGoneReorganization, + conformance.RestrictionUnknown: issueops.AsOfRestrictionUnknown, +} + +var asOfRestrictionToConformance = map[issueops.AsOfRestriction]conformance.Restriction{ + issueops.AsOfRestrictionLive: conformance.RestrictionLive, + issueops.AsOfRestrictionGoneRetention: conformance.RestrictionGoneRetention, + issueops.AsOfRestrictionGoneErasure: conformance.RestrictionGoneErasure, + issueops.AsOfRestrictionGoneReorganization: conformance.RestrictionGoneReorganization, + issueops.AsOfRestrictionUnknown: conformance.RestrictionUnknown, +} + +func toIssueopsSelector(selector conformance.AsOfSelector) issueops.AsOfSelector { + out := issueops.AsOfSelector{At: selector.At} + if selector.Version != nil { + out.Address = string(*selector.Version) + } + return out +} + +func toConformanceResult(result issueops.AsOfReadResult) conformance.AsOfReadResult { + if result.Refused { + return conformance.AsOfReadResult{ + Refusal: &conformance.AsOfReadRefusal{ + Restriction: asOfRestrictionToConformance[result.Restriction], + Reason: result.Reason, + }, + } + } + return conformance.AsOfReadResult{ + Address: conformance.Address(result.Address), + State: result.State, + } +} + +// TestAsOfReadContract wires this leg into the R7.1 as-of-read contract +// (gastownhall/beads#5898 revision 9, gastownhall/beads#6136) with a real +// Resolve/MintAt/MakeUnanswerable backed by one isolated *DoltStore per +// storeID (asOfReadHarness). +func TestAsOfReadContract(t *testing.T) { + // Must run before any t.Run: skipIfNoDolt calls t.Skip/t.Fatalf + // internally, and this is the one place in this file where that is safe + // -- the outer test's own goroutine, before any subtest closure exists. + // See the asOfReadHarness doc comment for why no other call in this file + // may do the same. + skipIfNoDolt(t) + + ctx := context.Background() + h := newAsOfReadHarness(t) + fixture := conformance.AsOfReadFixture{ + IssuePrefix: "asof", + Resolve: h.resolve, + MintAt: h.mintAt, + MakeUnanswerable: h.makeUnanswerable, + } + + t.Run("SelectorAcceptsEitherAnInstantOrAVersionAddress", func(t *testing.T) { + conformance.RunAsOfReadSelectorAcceptsEitherAnInstantOrAVersionAddress(t, ctx, fixture) + }) + t.Run("ReturnsTheLatestVersionAtOrBeforeTStrictlyExcludingLater", func(t *testing.T) { + conformance.RunAsOfReadReturnsTheLatestVersionAtOrBeforeTStrictlyExcludingLater(t, ctx, fixture) + }) + t.Run("ReportsTheAddressItServed", func(t *testing.T) { + conformance.RunAsOfReadReportsTheAddressItServed(t, ctx, fixture) + }) + t.Run("RefusesAsGoneOrUnknownNeverSubstitutingAnotherVersion", func(t *testing.T) { + conformance.RunAsOfReadRefusesAsGoneOrUnknownNeverSubstitutingAnotherVersion(t, ctx, fixture) + }) + t.Run("IsCompleteOnlyForTheAnsweringStoresHoldings", func(t *testing.T) { + conformance.RunAsOfReadIsCompleteOnlyForTheAnsweringStoresHoldings(t, ctx, fixture) + }) +} diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index 5b9d3687f..c7744f5a8 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -190,6 +190,7 @@ go_test( "merge_blockers_test.go", "merge_strategy_test.go", "metadata_cas_contract_test.go", + "metadata_number_fidelity_test.go", "migrate_consent_gate_test.go", "migrate_dep_rekey_dedup_test.go", "migrate_events_flip_test.go", @@ -242,6 +243,8 @@ go_test( "update_mergeops_test.go", "version_completeness_behaviour_test.go", "version_reconciler_contract_test.go", + "version_subsecond_test.go", + "versioned_read_contract_test.go", "workspace_config_contract_test.go", ], embed = [":embeddeddolt"], diff --git a/internal/storage/embeddeddolt/dualwrite_participation_generation_contract_test.go b/internal/storage/embeddeddolt/dualwrite_participation_generation_contract_test.go index 6c2b5aa88..f067d8481 100644 --- a/internal/storage/embeddeddolt/dualwrite_participation_generation_contract_test.go +++ b/internal/storage/embeddeddolt/dualwrite_participation_generation_contract_test.go @@ -40,6 +40,9 @@ func TestParticipationGenerationFence(t *testing.T) { t.Run("ProceedsForPromotedRecord", func(t *testing.T) { conformance.RunParticipationGenerationProceedsForPromotedRecord(t, ctx, fixture) }) + t.Run("FlagOffSkipsUpdateOnParticipatingRecord", func(t *testing.T) { + conformance.RunParticipationGenerationFlagOffSkipsUpdateOnParticipatingRecord(t, ctx, fixture) + }) t.Run("FenceIgnoresSchemaSkewHatch", func(t *testing.T) { conformance.RunParticipationGenerationFenceIgnoresSchemaSkewHatch(t, ctx, fixture) }) diff --git a/internal/storage/embeddeddolt/metadata_number_fidelity_test.go b/internal/storage/embeddeddolt/metadata_number_fidelity_test.go new file mode 100644 index 000000000..e538c9f9f --- /dev/null +++ b/internal/storage/embeddeddolt/metadata_number_fidelity_test.go @@ -0,0 +1,286 @@ +//go:build cgo + +package embeddeddolt_test + +import ( + "bytes" + "database/sql" + "encoding/json" + "errors" + "fmt" + "math/big" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// metadataNumberCorpus is what TestVersionedHistoryKeepsNumbersTheStoreDistinguishesApart writes +// through the store. It groups literals that share one nearest binary64 (0.1 and its longer +// spellings, 1 and its spellings, pi at several precisions), because those are the literals a +// store could tell apart and RFC 8785 cannot, and it includes the values outside the I-JSON +// exact-integer range that the mint has to refuse. The server leg carries the same list in +// dolt/metadata_number_fidelity_test.go; keep the two in step. +var metadataNumberCorpus = []string{ + "0.1", "0.10000000000000000555", "0.1000000000000000055511151231257827", + "1", "1.0", "1.00", "1e0", "10e-1", + "3.14159265358979323846", "3.141592653589793", "3.1415926535897932", + "0.3", "0.30000000000000004", + "0", "-0", "0.0", "1e-400", + "9007199254740991", + "9007199254740992", "9007199254740993", "9007199254740993.5", "9007199254740994", + "1727000000000000000", "1727000000000000123", + "1e300", "123456789012345678901234567890", +} + +// metadataNumbersOutsideTheRange are the corpus literals no store normalization can bring back +// inside 2^53-1: each is refused at the mint, and the write it belonged to must not survive. +var metadataNumbersOutsideTheRange = map[string]bool{ + "9007199254740992": true, "9007199254740993": true, "9007199254740993.5": true, "9007199254740994": true, + "1727000000000000000": true, "1727000000000000123": true, + "1e300": true, "123456789012345678901234567890": true, +} + +// embeddedDoltVersion names the engine this leg runs. The embedded engine answers +// dolt_version() with the placeholder "SET_BY_INIT", and a test binary's build info lists no +// dependencies, so the version of the module it is built from is read from go.mod; both are +// logged. Where go.mod is not reachable the reported string stands alone. +func embeddedDoltVersion(reported string) string { + _, file, _, ok := runtime.Caller(0) + if !ok { + return reported + } + data, err := os.ReadFile(filepath.Join(filepath.Dir(file), "..", "..", "..", "go.mod")) + if err != nil { + return reported + } + for _, line := range strings.Split(string(data), "\n") { + if fields := strings.Fields(line); len(fields) >= 2 && fields[0] == "github.com/dolthub/dolt/go" { + return fields[1] + " (go.mod; dolt_version() reports " + reported + ")" + } + } + return reported +} + +// TestVersionedHistoryKeepsNumbersTheStoreDistinguishesApart is the embedded leg of the safety +// property the number-admission rule rests on; the server leg's twin carries the full argument. +// +// In short: RFC 8785 turns every number into its nearest double, so two literals that differ by +// less than a double can share a canonical form, which is harmless only while the store cannot +// hold them apart. Whatever this Dolt does to numbers, it asserts in three phases that (1) with +// history OFF the store accepts a number outside the I-JSON range and the table logs what it +// holds, (2) with history ON a later write to such a row succeeds and mints nothing while the row +// never participated (design §16.2b's fence skips a legacy record whole, before the mint's +// admission gate), and is refused and changes nothing once the row participates, and +// (3) no two literals which read back as DIFFERENT numbers were both admitted with the SAME +// canonical form, and a literal the mint refuses leaves neither an issue row nor a version row. +func TestVersionedHistoryKeepsNumbersTheStoreDistinguishesApart(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "nf") + ctx := t.Context() + kit := newEmbeddedRoleFixtureKit(te, "nf") + + configurer, ok := any(te.store).(storage.VersionedHistoryConfigurer) + if !ok { + t.Fatalf("%T does not implement storage.VersionedHistoryConfigurer", te.store) + } + + var reported string + if err := kit.QueryScalar(ctx, `SELECT dolt_version()`, nil, &reported); err != nil { + reported = "unknown (" + err.Error() + ")" + } + newIssue := func(id, lit string) *types.Issue { + return &types.Issue{ + ID: id, Title: "n " + lit, IssueType: types.TypeTask, Status: types.StatusOpen, + Metadata: json.RawMessage(`{"n":` + lit + `}`), + } + } + + // Phase 1: history off. What does the store hold for a number outside the range? + t.Logf("metadata number normalization, embedded leg, dolt %s", embeddedDoltVersion(reported)) + configurer.SetVersionedHistoryEnabled(false) + var held, heldLiterals []string + for i, lit := range metadataNumberCorpus { + if !metadataNumbersOutsideTheRange[lit] { + continue + } + id := fmt.Sprintf("nf-off-%02d", i) + if err := te.store.CreateIssue(ctx, newIssue(id, lit), "actor"); err != nil { + t.Logf(" history off: %-34s the store refused it: %v", lit, err) + continue + } + issue, err := te.store.GetIssue(ctx, id) + if err != nil { + t.Fatalf("reading %s back: %v", id, err) + } + stored, _ := storedNumberAndCanonicalMetadata(t, issue.Metadata, nil) + t.Logf(" history off: %-34s held as %s", lit, stored) + held = append(held, id) + heldLiterals = append(heldLiterals, lit) + } + + rowState := func(id string) (versions int, revision int64, generation sql.NullInt64) { + t.Helper() + if err := kit.QueryScalar(ctx, `SELECT COUNT(*) FROM issue_versions WHERE issue_id = ?`, []any{id}, &versions); err != nil { + t.Fatalf("counting the version rows of %s: %v", id, err) + } + if err := kit.QueryScalar(ctx, `SELECT current_revision FROM issues WHERE id = ?`, []any{id}, &revision); err != nil { + t.Fatalf("reading the revision of %s: %v", id, err) + } + if err := kit.QueryScalar(ctx, `SELECT participation_generation FROM issues WHERE id = ?`, []any{id}, &generation); err != nil { + t.Fatalf("reading the participation of %s: %v", id, err) + } + return versions, revision, generation + } + + // Phase 2a: history on, rows that never participated. Each was made with history off, so it is + // a legacy record (participation_generation NULL) and design §16.2b's write fence skips an + // update-shaped write to it whole, before the mint's admission gate: the write succeeds, + // nothing is versioned, nothing is refused and the stamp stays NULL, whatever the metadata holds. + configurer.SetVersionedHistoryEnabled(true) + for _, id := range held { + _, revisionBefore, _ := rowState(id) + if err := te.store.UpdateIssue(ctx, id, map[string]any{"title": "retitled"}, "actor"); err != nil { + t.Errorf("updating %s, a legacy row that holds a number outside the I-JSON range, = %v, want success: the fence skips a legacy record before the mint's admission gate", id, err) + continue + } + after, err := te.store.GetIssue(ctx, id) + if err != nil || after.Title != "retitled" { + t.Errorf("the update of legacy row %s did not land: title %q (err %v), want %q", id, after.Title, err, "retitled") + } + versions, revision, generation := rowState(id) + if versions != 0 || generation.Valid || revision != revisionBefore { + t.Errorf("the update of legacy row %s versioned or promoted it: version rows %d (want 0), participation_generation %v (want NULL), current_revision %d -> %d (want unchanged)", + id, versions, generation, revisionBefore, revision) + } + } + + // Phase 2b: history on, rows that participate. Each is created with history on holding an + // in-range stand-in, so the real create-shaped mint stamps it; the stamp is read first, because + // that is what makes this case about participation. History is then switched off and the row's + // metadata replaced through the store, which is how a row comes to hold such a number without the + // mint having seen it, and switched on again. The next write is refused and changes nothing. + for k, lit := range heldLiterals { + id := fmt.Sprintf("nf-part-%02d", k) + configurer.SetVersionedHistoryEnabled(true) + if err := te.store.CreateIssue(ctx, newIssue(id, "1"), "actor"); err != nil { + t.Fatalf("creating the participating row %s with history on: %v", id, err) + } + if _, _, generation := rowState(id); !generation.Valid { + t.Fatalf("%s was created with history on but participation_generation is NULL: the control failed, so this case would not be about participation", id) + } + configurer.SetVersionedHistoryEnabled(false) + if err := te.store.UpdateIssue(ctx, id, map[string]any{"metadata": json.RawMessage(`{"n":` + lit + `}`)}, "actor"); err != nil { + t.Fatalf("replacing the metadata of %s with %s while history is off: %v", id, lit, err) + } + configurer.SetVersionedHistoryEnabled(true) + before, err := te.store.GetIssue(ctx, id) + if err != nil { + t.Fatalf("reading %s: %v", id, err) + } + versionsBefore, revisionBefore, generationBefore := rowState(id) + err = te.store.UpdateIssue(ctx, id, map[string]any{"title": "retitled"}, "actor") + if !errors.Is(err, issueops.ErrIntegerNotRepresentable) { + t.Errorf("updating %s, a participating row that holds %s, = %v, want ErrIntegerNotRepresentable", id, lit, err) + } + after, getErr := te.store.GetIssue(ctx, id) + versions, revision, generation := rowState(id) + if getErr != nil || after.Title != before.Title || versions != versionsBefore || revision != revisionBefore || generation != generationBefore { + t.Errorf("the refused update of %s left a trace: title %q -> %q, version rows %d -> %d, current_revision %d -> %d, participation_generation %v -> %v (err %v)", + id, before.Title, after.Title, versionsBefore, versions, revisionBefore, revision, generationBefore, generation, getErr) + } + } + + // Phase 3: history on, fresh rows. The property. + configurer.SetVersionedHistoryEnabled(true) + type admittedLiteral struct{ written, storedAs, canonical string } + var admitted []admittedLiteral + for i, lit := range metadataNumberCorpus { + id := fmt.Sprintf("nf-%02d", i) + err := te.store.CreateIssue(ctx, newIssue(id, lit), "actor") + if err != nil { + if !metadataNumbersOutsideTheRange[lit] { + t.Errorf("%s was refused although it is an ordinary number: %v", lit, err) + } else if !errors.Is(err, issueops.ErrIntegerNotRepresentable) { + t.Errorf("%s was refused with %v, want ErrIntegerNotRepresentable", lit, err) + } + if _, getErr := te.store.GetIssue(ctx, id); !errors.Is(getErr, storage.ErrNotFound) { + t.Errorf("%s was refused but its issue exists afterwards (GetIssue = %v): the refusal did not abort the write", lit, getErr) + } + var versions int + if scanErr := kit.QueryScalar(ctx, `SELECT COUNT(*) FROM issue_versions WHERE issue_id = ?`, []any{id}, &versions); scanErr != nil || versions != 0 { + t.Errorf("%s was refused but left %d version rows (err %v)", lit, versions, scanErr) + } + t.Logf(" history on: %-34s refused", lit) + continue + } + if metadataNumbersOutsideTheRange[lit] { + t.Errorf("%s is outside the I-JSON exact-integer range and was written", lit) + } + issue, err := te.store.GetIssue(ctx, id) + if err != nil { + t.Fatalf("reading %s back: %v", id, err) + } + var durable []byte + if err := kit.QueryScalar(ctx, + `SELECT durable_state FROM issue_versions WHERE issue_id = ? ORDER BY revision DESC LIMIT 1`, []any{id}, &durable); err != nil { + t.Fatalf("reading the version of %s: %v", id, err) + } + stored, canonical := storedNumberAndCanonicalMetadata(t, issue.Metadata, durable) + t.Logf(" history on: %-34s read back as %-28s canonical metadata %s", lit, stored, canonical) + admitted = append(admitted, admittedLiteral{lit, stored, canonical}) + } + + if len(admitted) < 12 { + t.Fatalf("only %d of %d literals were admitted; the property is close to vacuous", len(admitted), len(metadataNumberCorpus)) + } + first := map[string]admittedLiteral{} + for _, a := range admitted { + prior, seen := first[a.canonical] + if !seen { + first[a.canonical] = a + continue + } + if !sameNumber(prior.storedAs, a.storedAs) { + t.Errorf("literals %s and %s read back from the store as different numbers (%s and %s) but share the canonical form %s: "+ + "two distinct stored states would carry one content token", prior.written, a.written, prior.storedAs, a.storedAs, a.canonical) + } + } +} + +// storedNumberAndCanonicalMetadata returns the number the store handed back in the issue's +// metadata, and, when durable is given, the metadata member of the canonical bytes the mint +// stored for it. +func storedNumberAndCanonicalMetadata(t *testing.T, readBack json.RawMessage, durable []byte) (stored, canonical string) { + t.Helper() + var back struct { + N json.Number `json:"n"` + } + dec := json.NewDecoder(bytes.NewReader(readBack)) + dec.UseNumber() + if err := dec.Decode(&back); err != nil { + t.Fatalf("decoding the metadata read back %s: %v", readBack, err) + } + if durable == nil { + return back.N.String(), "" + } + var version struct { + Metadata json.RawMessage `json:"metadata"` + } + if err := json.Unmarshal(durable, &version); err != nil { + t.Fatalf("decoding durable_state %s: %v", durable, err) + } + return back.N.String(), strings.TrimSpace(string(version.Metadata)) +} + +// sameNumber reports whether two JSON number literals denote the same value, compared exactly. +func sameNumber(a, b string) bool { + x, okA := new(big.Rat).SetString(a) + y, okB := new(big.Rat).SetString(b) + return okA && okB && x.Cmp(y) == 0 +} diff --git a/internal/storage/embeddeddolt/store.go b/internal/storage/embeddeddolt/store.go index 842717273..7fa8f3ade 100644 --- a/internal/storage/embeddeddolt/store.go +++ b/internal/storage/embeddeddolt/store.go @@ -1165,6 +1165,17 @@ func (s *EmbeddedDoltStore) History(ctx context.Context, issueID string) ([]*sto return result, err } +// ListVersions implements storage.VersionLister over issue_versions. +func (s *EmbeddedDoltStore) ListVersions(ctx context.Context, issueID string) ([]storage.IssueVersion, error) { + var result []storage.IssueVersion + err := s.withConn(ctx, false, func(tx *sql.Tx) error { + var err error + result, err = issueops.ListVersionsInTx(ctx, tx, issueID) + return err + }) + return result, err +} + func (s *EmbeddedDoltStore) AsOf(ctx context.Context, issueID string, ref string) (*types.Issue, error) { var result *types.Issue err := s.withConn(ctx, false, func(tx *sql.Tx) error { @@ -1529,3 +1540,8 @@ func (s *EmbeddedDoltStore) GetStaleIssues(ctx context.Context, filter types.Sta }) return result, err } + +// Compile-time proof that this leg serves versions -- see the matching +// assertion in internal/storage/dolt/versioned.go for why it lives in the +// leg's own package. +var _ storage.VersionLister = (*EmbeddedDoltStore)(nil) diff --git a/internal/storage/embeddeddolt/version_completeness_behaviour_test.go b/internal/storage/embeddeddolt/version_completeness_behaviour_test.go index 455a196b4..d79c1afd0 100644 --- a/internal/storage/embeddeddolt/version_completeness_behaviour_test.go +++ b/internal/storage/embeddeddolt/version_completeness_behaviour_test.go @@ -5,6 +5,7 @@ package embeddeddolt_test import ( "encoding/json" "testing" + "time" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" @@ -218,6 +219,118 @@ func TestEmbeddedEveryAcceptedLifecycleMutationMintsOneVersion(t *testing.T) { p.revisionMatchesRows(b) } +// TestEmbeddedMintLeavesUpdatedAtWhereHistoryOffDoes pins that minting a +// version is not itself an edit. issues.updated_at carries ON UPDATE +// CURRENT_TIMESTAMP, so a mint whose own UPDATE left it unassigned would move +// it to the clock on every versioned write, including the label and +// dependency writes that never touch it with history off. bd import's +// stale-upsert check reads that column, so a moved updated_at would also turn +// the source's next real edit away as stale. Each scenario runs with history +// off first, as the control the history-on run must match. +func TestEmbeddedMintLeavesUpdatedAtWhereHistoryOffDoes(t *testing.T) { + skipUnlessEmbeddedDolt(t) + // Fixed instants well before any clock this runs under, so a mint that + // wrote the clock cannot pass for any of them. + aged := time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC) + sourceEdit := time.Date(2025, 2, 1, 0, 0, 0, 0, time.UTC) + laterSourceEdit := time.Date(2025, 3, 1, 0, 0, 0, 0, time.UTC) + + for _, history := range []struct { + name string + on bool + }{{"HistoryOff", false}, {"HistoryOn", true}} { + t.Run(history.name, func(t *testing.T) { + te := newTestEnv(t, "verts") + ctx := t.Context() + te.store.SetVersionedHistoryEnabled(history.on) + // With history on every step below must mint exactly one version, + // so a mint that never ran cannot pass for one that kept updated_at. + mints := 0 + if history.on { + mints = 1 + } + + a, b := "verts-a", "verts-b" + for _, id := range []string{a, b} { + if err := te.store.CreateIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + }, "seed"); err != nil { + t.Fatalf("CreateIssue(%s): %v", id, err) + } + } + age := func() { te.exec(t, ctx, "UPDATE issues SET updated_at = ? WHERE id = ?", aged, a) } + versions := func() int { + t.Helper() + var n int + te.queryScalar(t, ctx, "SELECT COUNT(*) FROM issue_versions WHERE issue_id = ?", []any{a}, &n) + return n + } + // check asserts a's updated_at and that the step minted as the + // history setting says, and returns the new version count. + check := func(step string, before int, want time.Time) int { + t.Helper() + var got time.Time + te.queryScalar(t, ctx, "SELECT updated_at FROM issues WHERE id = ?", []any{a}, &got) + if !got.UTC().Equal(want) { + t.Errorf("%s: updated_at for %s is %v, want %v", step, a, got.UTC(), want) + } + n := versions() + if n != before+mints { + t.Fatalf("%s: version rows for %s = %d, want %d", step, a, n, before+mints) + } + return n + } + + // Label and dependency writes never assign updated_at themselves. + age() + n := versions() + if err := te.store.AddLabel(ctx, a, "l1", "actor"); err != nil { + t.Fatalf("AddLabel: %v", err) + } + n = check("add label", n, aged) + if err := te.store.RemoveLabel(ctx, a, "l1", "actor"); err != nil { + t.Fatalf("RemoveLabel: %v", err) + } + n = check("remove label", n, aged) + if err := te.store.AddDependency(ctx, &types.Dependency{IssueID: a, DependsOnID: b, Type: types.DepBlocks}, "actor"); err != nil { + t.Fatalf("AddDependency: %v", err) + } + n = check("add dependency", n, aged) + if err := te.store.RemoveDependency(ctx, a, b, "actor"); err != nil { + t.Fatalf("RemoveDependency: %v", err) + } + n = check("remove dependency", n, aged) + + // bd import's options. The upsert stores the source's own + // updated_at, and the source's next edit is compared against it. + // Aged again so this half does not depend on the half above. + age() + var rejected []string + opts := storage.BatchCreateOptions{ + RejectStaleUpserts: true, + SkipDependencyValidationErrors: true, + OnStaleRejected: func(id string) { rejected = append(rejected, id) }, + } + importEdit := func(title string, at time.Time) { + t.Helper() + if err := te.store.CreateIssuesWithFullOptions(ctx, []*types.Issue{{ + ID: a, Title: title, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + CreatedAt: aged, UpdatedAt: at, + }}, "importer", opts); err != nil { + t.Fatalf("import %q: %v", title, err) + } + if len(rejected) != 0 { + t.Fatalf("import %q with source updated_at %v was rejected as stale (%v)", title, at, rejected) + } + } + importEdit("source edit", sourceEdit) + n = check("import", n, sourceEdit) + importEdit("later source edit", laterSourceEdit) + check("later import", n, laterSourceEdit) + }) + } +} + // TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges pins the // batch-create ordering fix: the one version minted at creation is minted // AFTER PersistDependenciesWithOptionsResult, so it carries the outgoing edge diff --git a/internal/storage/embeddeddolt/version_subsecond_test.go b/internal/storage/embeddeddolt/version_subsecond_test.go new file mode 100644 index 000000000..9b672caaf --- /dev/null +++ b/internal/storage/embeddeddolt/version_subsecond_test.go @@ -0,0 +1,99 @@ +//go:build cgo + +package embeddeddolt + +import ( + "context" + "database/sql" + "testing" + "time" +) + +// TestRecordVersionKeepsSubSecondChangeAt pins the WRITER half of migration +// 0069's contract: a sub-second change_at must survive through +// RecordVersionAtInTx into the row and back. +// +// 0069 widened issue_versions.change_at to DATETIME(6) precisely so that two +// versions minted inside the same second stay distinguishable, because +// --at is the only PORTABLE selector the read surface offers while +// revision remains a per-store ordinal and no durable version address has +// shipped. The column's own half is pinned by +// TestMigration0069ChangeAtSurvivesSubSecondPrecisionThroughDoltCLI in +// internal/storage/schema. The writer's half was NOT pinned, and that is +// exactly how it got defeated: recordVersionAtInTx carried an +// `at.Truncate(time.Second)` — correct while the column was precision 0, +// where Dolt rounds half-up — which kept flooring every value after the +// widening landed. A store with both applied stored four versions minted in +// one second as the identical 15:40:38.000000. +// +// So this test asserts the two things a caller actually depends on: the +// microseconds are there, and same-second versions differ. +func TestRecordVersionKeepsSubSecondChangeAt(t *testing.T) { + skipUnlessEmbeddedDolt(t) + + ctx := context.Background() + store, cleanup, err := buildAsOfReadStore(ctx) + if err != nil { + t.Fatalf("build store: %v", err) + } + t.Cleanup(cleanup) + + h := newAsOfReadHarness(t) + h.stores["subsecond"] = store + + const id = "vt-subsec" + // Two instants inside the SAME second, a quarter of a second apart. Under + // the old floor both landed on :07.000000; under a precision-0 column the + // second would additionally have rounded up to :08. + first := time.Date(2026, 9, 27, 11, 4, 7, 250000000, time.UTC) + second := time.Date(2026, 9, 27, 11, 4, 7, 750000000, time.UTC) + + if _, err := h.mintAt(ctx, "subsecond", id, first); err != nil { + t.Fatalf("mint at %s: %v", first, err) + } + if _, err := h.mintAt(ctx, "subsecond", id, second); err != nil { + t.Fatalf("mint at %s: %v", second, err) + } + + var stamps []time.Time + if err := store.withConn(ctx, false, func(tx *sql.Tx) error { + rows, err := tx.QueryContext(ctx, + `SELECT change_at FROM issue_versions WHERE issue_id = ? ORDER BY revision`, id) + if err != nil { + return err + } + defer func() { _ = rows.Close() }() + for rows.Next() { + var at time.Time + if err := rows.Scan(&at); err != nil { + return err + } + stamps = append(stamps, at) + } + return rows.Err() + }); err != nil { + t.Fatalf("read change_at back: %v", err) + } + + if len(stamps) < 2 { + t.Fatalf("got %d version rows for %s, want at least 2", len(stamps), id) + } + // The harness mints a creation version too, so take the last two. + got := stamps[len(stamps)-2:] + + for i, at := range got { + if at.Nanosecond() == 0 { + t.Errorf("change_at[%d] = %s has a zero fractional part; the sub-second component was dropped between the caller and the row. "+ + "If recordVersionAtInTx floors `at` again, or the column reverts to DATETIME(0), --at can no longer separate versions minted in the same second.", + i, at.Format("15:04:05.000000")) + } + } + if got[0].Equal(got[1]) { + t.Errorf("two versions minted 500ms apart both stored %s — same-second versions must stay distinguishable by change_at, "+ + "which is what migration 0069 widened the column for", got[0].Format("15:04:05.000000")) + } + if !got[1].After(got[0]) { + t.Errorf("change_at went backwards or sideways: %s then %s", + got[0].Format("15:04:05.000000"), got[1].Format("15:04:05.000000")) + } +} diff --git a/internal/storage/embeddeddolt/versioned_read_contract_test.go b/internal/storage/embeddeddolt/versioned_read_contract_test.go new file mode 100644 index 000000000..709e02e3c --- /dev/null +++ b/internal/storage/embeddeddolt/versioned_read_contract_test.go @@ -0,0 +1,309 @@ +//go:build cgo + +package embeddeddolt + +import ( + "context" + "database/sql" + "fmt" + "os" + "path/filepath" + "testing" + "time" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// asOfReadHarness is this leg's Design D dispatch table (gastownhall/beads#5898 +// revision 9, gastownhall/beads#6136): the conformance suite +// invents storeID values to select among N genuinely isolated stores, and +// isolating them is entirely this harness's job -- AsOfReadInTx itself is +// storeID-oblivious (see the package doc on +// internal/storage/issueops/asof_read.go). Each storeID gets its own fresh +// *EmbeddedDoltStore, built lazily on first use (buildAsOfReadStore) and torn +// down via t.Cleanup on the TestAsOfReadContract-level *testing.T. created +// tracks which (storeID, id) pairs already have a base issues row, since +// mintAt's second and later calls for the same pair must only add a +// historical issue_versions row, never a second base create. +// +// store() cannot use a *testing.T-tied fixture helper (e.g. +// newPristineEmbeddedDoltFixture): AsOfReadFixture's hook types +// (Resolve/MintAt/MakeUnanswerable) carry no *testing.T of their own, so a +// store built lazily inside a hook -- itself invoked from inside a t.Run +// subtest closure, sometimes nested two deep (the restriction-loop case +// below) -- has access only to whichever *testing.T this harness was +// constructed with: necessarily the OUTER TestAsOfReadContract's t, captured +// before any t.Run. Calling FailNow/SkipNow (t.Fatalf/t.Skip) on a *testing.T +// from a goroutine other than the one running that exact test -- which a +// t.Run subtest's closure always is, relative to the outer test -- panics +// with "subtest may have called FailNow on a parent test" (go's testing +// package requires FailNow/SkipNow to run only on the goroutine executing +// that specific test). buildAsOfReadStore below is a Fatalf/Skip-free +// construction path for exactly this reason: its failures come back as a +// plain error, which flows through the calling hook's own (..., error) +// return to conformance.RunAsOfReadXXX, which correctly Fatalfs using the +// SUBTEST's own *testing.T on the SUBTEST's own goroutine. Embedded-Dolt +// availability skipping itself is handled once, up front in +// TestAsOfReadContract's own body via skipUnlessEmbeddedDolt(t) -- safe there +// because that runs before any t.Run, in the one goroutine that actually owns +// t. See internal/storage/dolt/versioned_read_contract_test.go for this +// leg's sibling, which this file mirrors as closely as the two backends' +// store-construction APIs allow. +type asOfReadHarness struct { + t *testing.T + stores map[string]*EmbeddedDoltStore + created map[[2]string]bool +} + +func newAsOfReadHarness(t *testing.T) *asOfReadHarness { + return &asOfReadHarness{ + t: t, + stores: map[string]*EmbeddedDoltStore{}, + created: map[[2]string]bool{}, + } +} + +func (h *asOfReadHarness) store(ctx context.Context, storeID string) (*EmbeddedDoltStore, error) { + if s, ok := h.stores[storeID]; ok { + return s, nil + } + store, cleanup, err := buildAsOfReadStore(ctx) + if err != nil { + return nil, err + } + // t.Cleanup only appends to a mutex-guarded slice -- unlike Fatalf/Skip it + // never calls runtime.Goexit, so registering it on the outer h.t from + // inside a subtest's goroutine is safe. h.t's own cleanups run only after + // ALL of TestAsOfReadContract's t.Run calls have returned (none of them, + // nor their own nested t.Run calls, ever invoke t.Parallel), so every + // store this harness builds stays open for the whole test, matching + // created's own whole-test lifetime. + h.t.Cleanup(cleanup) + h.stores[storeID] = store + return store, nil +} + +// buildAsOfReadStore constructs one fresh, isolated *EmbeddedDoltStore: its +// own temp directory (so its own on-disk database, with nothing shared +// between stores -- unlike the dolt leg's single testcontainers server, an +// embedded store's isolation is just its own directory, so a fixed database +// name is safe to reuse across every call), issue_prefix seeded. Minus every +// t.Fatalf/t.Skip call; see the asOfReadHarness doc comment for why no call +// in this file may make one outside TestAsOfReadContract's own body. +func buildAsOfReadStore(ctx context.Context) (*EmbeddedDoltStore, func(), error) { + tmpDir, err := os.MkdirTemp("", "embeddeddolt-asof-test-*") + if err != nil { + return nil, nil, fmt.Errorf("as-of read fixture: create temp dir: %w", err) + } + + beadsDir := filepath.Join(tmpDir, ".beads") + store, err := Open(ctx, beadsDir, "asof", "main") + if err != nil { + os.RemoveAll(tmpDir) + return nil, nil, fmt.Errorf("as-of read fixture: open embedded Dolt store: %w", err) + } + + if err := store.SetConfig(ctx, "issue_prefix", "asof"); err != nil { + store.Close() + os.RemoveAll(tmpDir) + return nil, nil, fmt.Errorf("as-of read fixture: set issue_prefix: %w", err) + } + + cleanup := func() { + store.Close() + os.RemoveAll(tmpDir) + } + return store, cleanup, nil +} + +// ensureCreated bare-creates id in storeID's store the first time it is +// asked for, giving RecordVersionAtInTx (called from mintAt) a base issues +// row to snapshot -- recordVersionAtInTx starts with GetIssueInTx, so it has +// nothing to read until this has run once. +func (h *asOfReadHarness) ensureCreated(ctx context.Context, storeID, id string) error { + key := [2]string{storeID, id} + if h.created[key] { + return nil + } + store, err := h.store(ctx, storeID) + if err != nil { + return err + } + issue := &types.Issue{ + ID: id, + Title: "as-of read fixture issue", + Status: types.StatusOpen, + Priority: 3, + IssueType: types.TypeTask, + } + if err := store.CreateIssue(ctx, issue, "asof-read-fixture"); err != nil { + return fmt.Errorf("as-of read fixture: bare-create %s: %w", id, err) + } + h.created[key] = true + return nil +} + +func (h *asOfReadHarness) resolve(ctx context.Context, storeID, id string, selector conformance.AsOfSelector) (conformance.AsOfReadResult, error) { + store, err := h.store(ctx, storeID) + if err != nil { + return conformance.AsOfReadResult{}, err + } + var result issueops.AsOfReadResult + err = store.withConn(ctx, false, func(tx *sql.Tx) error { + var err error + result, err = issueops.AsOfReadInTx(ctx, tx, storeID, id, toIssueopsSelector(selector)) + return err + }) + if err != nil { + return conformance.AsOfReadResult{}, err + } + return toConformanceResult(result), nil +} + +// mintAt is the two-phase implementation MintAt's contract needs: phase one +// (ensureCreated) runs only once per (storeID, id); phase two +// (RecordVersionAtInTx) runs every call and always supplies the row mintAt +// reports the address of. RecordVersionAtInTx deliberately does not gate on +// versionedHistoryEnabled (this harness never turns that on for its stores), +// so it always mints, and reading MAX(revision) back inside the same +// transaction (read-your-writes) is how we learn which revision it picked -- +// RecordVersionAtInTx itself reports no revision number. +func (h *asOfReadHarness) mintAt(ctx context.Context, storeID, id string, acceptedAt time.Time) (conformance.Address, error) { + if err := h.ensureCreated(ctx, storeID, id); err != nil { + return "", err + } + store, err := h.store(ctx, storeID) + if err != nil { + return "", err + } + var revision int64 + err = store.withConn(ctx, true, func(tx *sql.Tx) error { + if err := issueops.RecordVersionAtInTx(ctx, tx, id, "asof-read-fixture", acceptedAt); err != nil { + return err + } + return tx.QueryRowContext(ctx, + `SELECT MAX(revision) FROM issue_versions WHERE issue_id = ?`, id, + ).Scan(&revision) + }) + if err != nil { + return "", err + } + return conformance.Address(issueops.AsOfVersionAddress(id, revision)), nil +} + +func (h *asOfReadHarness) makeUnanswerable(ctx context.Context, storeID, id string, restriction conformance.Restriction) error { + asOfRestriction, ok := conformanceToAsOfRestriction[restriction] + if !ok { + return fmt.Errorf("as-of read fixture: unmapped restriction %v", restriction) + } + store, err := h.store(ctx, storeID) + if err != nil { + return err + } + reason := fmt.Sprintf("as-of read fixture: made unanswerable as %s", restriction) + return store.withConn(ctx, true, func(tx *sql.Tx) error { + return issueops.MarkLatestVersionRemovedInTx(ctx, tx, id, asOfRestriction, reason) + }) +} + +// conformanceToAsOfRestriction and its inverse are this leg's translation +// boundary between backend/conformance.Restriction (hyphenated String(), +// int-backed) and issueops.AsOfRestriction (underscored, string-backed -- +// its values ARE the removed_restriction column's contents). issueops must +// stay conformance-oblivious and conformance must stay backend-oblivious +// (internal/storage/issueops/asof_read.go's package doc), so this mapping is +// deliberately duplicated per leg rather than shared centrally. +var conformanceToAsOfRestriction = map[conformance.Restriction]issueops.AsOfRestriction{ + conformance.RestrictionLive: issueops.AsOfRestrictionLive, + conformance.RestrictionGoneRetention: issueops.AsOfRestrictionGoneRetention, + conformance.RestrictionGoneErasure: issueops.AsOfRestrictionGoneErasure, + conformance.RestrictionGoneReorganization: issueops.AsOfRestrictionGoneReorganization, + conformance.RestrictionUnknown: issueops.AsOfRestrictionUnknown, +} + +var asOfRestrictionToConformance = map[issueops.AsOfRestriction]conformance.Restriction{ + issueops.AsOfRestrictionLive: conformance.RestrictionLive, + issueops.AsOfRestrictionGoneRetention: conformance.RestrictionGoneRetention, + issueops.AsOfRestrictionGoneErasure: conformance.RestrictionGoneErasure, + issueops.AsOfRestrictionGoneReorganization: conformance.RestrictionGoneReorganization, + issueops.AsOfRestrictionUnknown: conformance.RestrictionUnknown, +} + +func toIssueopsSelector(selector conformance.AsOfSelector) issueops.AsOfSelector { + out := issueops.AsOfSelector{At: selector.At} + if selector.Version != nil { + out.Address = string(*selector.Version) + } + return out +} + +func toConformanceResult(result issueops.AsOfReadResult) conformance.AsOfReadResult { + if result.Refused { + return conformance.AsOfReadResult{ + Refusal: &conformance.AsOfReadRefusal{ + Restriction: asOfRestrictionToConformance[result.Restriction], + Reason: result.Reason, + }, + } + } + return conformance.AsOfReadResult{ + Address: conformance.Address(result.Address), + State: result.State, + } +} + +// skipUnlessEmbeddedDolt is this file's own copy of the embeddeddolt_test +// package's helper of the same name (create_issue_test.go): that one is +// unexported and lives in the external test package, unreachable from here. +// This harness must live in-package (see the asOfReadHarness doc comment for +// why: raw DBTX access requires the unexported withConn), so it carries its +// own copy rather than importing across the package/package_test boundary. +// Reads the same BEADS_TEST_EMBEDDED_DOLT env var so `go test +// ./internal/storage/embeddeddolt/...` gates uniformly regardless of which +// half of the package a given test file lives in. +func skipUnlessEmbeddedDolt(t *testing.T) { + t.Helper() + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt tests") + } +} + +// TestAsOfReadContract wires this leg into the R7.1 as-of-read contract +// (gastownhall/beads#5898 revision 9, gastownhall/beads#6136) with a real +// Resolve/MintAt/MakeUnanswerable backed by one isolated *EmbeddedDoltStore +// per storeID (asOfReadHarness). +func TestAsOfReadContract(t *testing.T) { + // Must run before any t.Run: skipUnlessEmbeddedDolt calls t.Skip + // internally, and this is the one place in this file where that is safe + // -- the outer test's own goroutine, before any subtest closure exists. + // See the asOfReadHarness doc comment for why no other call in this file + // may do the same. + skipUnlessEmbeddedDolt(t) + + ctx := context.Background() + h := newAsOfReadHarness(t) + fixture := conformance.AsOfReadFixture{ + IssuePrefix: "asof", + Resolve: h.resolve, + MintAt: h.mintAt, + MakeUnanswerable: h.makeUnanswerable, + } + + t.Run("SelectorAcceptsEitherAnInstantOrAVersionAddress", func(t *testing.T) { + conformance.RunAsOfReadSelectorAcceptsEitherAnInstantOrAVersionAddress(t, ctx, fixture) + }) + t.Run("ReturnsTheLatestVersionAtOrBeforeTStrictlyExcludingLater", func(t *testing.T) { + conformance.RunAsOfReadReturnsTheLatestVersionAtOrBeforeTStrictlyExcludingLater(t, ctx, fixture) + }) + t.Run("ReportsTheAddressItServed", func(t *testing.T) { + conformance.RunAsOfReadReportsTheAddressItServed(t, ctx, fixture) + }) + t.Run("RefusesAsGoneOrUnknownNeverSubstitutingAnotherVersion", func(t *testing.T) { + conformance.RunAsOfReadRefusesAsGoneOrUnknownNeverSubstitutingAnotherVersion(t, ctx, fixture) + }) + t.Run("IsCompleteOnlyForTheAnsweringStoresHoldings", func(t *testing.T) { + conformance.RunAsOfReadIsCompleteOnlyForTheAnsweringStoresHoldings(t, ctx, fixture) + }) +} diff --git a/internal/storage/issueops/BUILD.bazel b/internal/storage/issueops/BUILD.bazel index 58cf1890f..138b27d5b 100644 --- a/internal/storage/issueops/BUILD.bazel +++ b/internal/storage/issueops/BUILD.bazel @@ -5,6 +5,7 @@ go_library( srcs = [ "aggregate.go", "as_of.go", + "asof_read.go", "batch_apply.go", "batchget.go", "batching.go", @@ -100,6 +101,8 @@ go_library( "version.go", "version_commit.go", "version_history.go", + "version_history_scan.go", + "version_list.go", "wake_defers.go", "wisp_filter_convert.go", "wisp_routing.go", @@ -134,6 +137,7 @@ go_test( "aggregate_authorize_notes_test.go", "aggregate_authorize_transfer_test.go", "as_of_test.go", + "asof_read_test.go", "batch_apply_test.go", "batchget_test.go", "batching_test.go", @@ -200,6 +204,9 @@ go_test( "update_validation_test.go", "version_completeness_test.go", "version_history_durable_state_test.go", + "version_history_integer_admission_test.go", + "version_history_number_admission_test.go", + "version_history_preenable_scan_test.go", "version_history_test.go", "wisp_routing_test.go", ], diff --git a/internal/storage/issueops/asof_read.go b/internal/storage/issueops/asof_read.go new file mode 100644 index 000000000..b71f1f724 --- /dev/null +++ b/internal/storage/issueops/asof_read.go @@ -0,0 +1,286 @@ +package issueops + +import ( + "context" + "database/sql" + "encoding/json" + "errors" + "fmt" + "strconv" + "strings" + "time" +) + +// This file implements R7.1 as-of read (gastownhall/beads#5898 revision 9, +// gastownhall/beads#6136; conformance suite: backend/conformance/ +// versioned_read_contract.go): given an instant T or a version address, it +// returns one issue's durable state as of that point -- the latest version +// accepted at or before T, with everything later excluded -- marking the +// address it served, or refuses as gone or unknown with the reason (R8, +// R20). It reads issue_versions rows minted by RecordVersionInTx / +// RecordVersionAtInTx (version_history.go); it adds no writer of its own for +// the mutation path. +// +// NOT part of this primitive (R7.1's own text): memory-plane consumer +// policy -- closed-before-T boundary, sibling exclusion, supersession +// exclusion (gastownhall/beads#5877). Those are layered on top by a +// consumer, not implemented here. +// +// ALL THREE LEGS SHARE THIS BODY, the same way CompareAndSetMetadataKeyInTx, +// R16's CompareAndSetVersionInTx, and R20's epoch_cas.go do: the two +// Dolt-backed stores wrap it in their own transaction, and the +// unit-of-work provider reaches it through its own leg-specific adapter. +// Each leg's own contract-test file is the translation boundary to and from +// backend/conformance's Address/Restriction vocabulary -- this file knows +// nothing about the conformance package (same package-boundary precedent as +// epoch_cas.go's EpochRestriction). +// +// removed_at / removed_reason (migration 0067, virgin and unused before this +// slice) plus removed_restriction (migration 0071, added by this slice) are +// the durable "this version row was removed" marker AsOfReadInTx reads. +// R7.1 only READS them here; a future R17 (not yet implemented on this +// branch) will be the production WRITER, via real hold/remove/erase +// primitives -- no collision, since this file never writes them outside the +// test-support seam below. +// +// STOREID IS ACCEPTED BUT NOT A FILTER, the same way CurrentEpochInTx +// accepts one: issue_versions has no store_id column, and a store's rows are +// already exactly the rows in that store's own database (one logical store +// per database, matching production reality). The parameter exists only so +// this function's signature matches AsOfReadFixture.Resolve's hook shape, +// which the conformance suite's own ad-hoc dispatch string requires (Design +// D: the suite invents storeID values to select among N genuinely isolated +// per-leg test stores; the leg fixture, not this function, does that +// isolation). + +// AsOfRestriction is this file's own local answer vocabulary for R7.1's +// refusal path -- deliberately not backend/conformance.Restriction (see the +// package-boundary note above). Unlike R20's EpochRestriction (int-backed: +// R20 never persists a restriction, it recomputes GoneReorganization at read +// time from an epoch comparison), this type is STRING-backed: it is written +// directly into issue_versions.removed_restriction and read back with a +// plain cast, no separate parse/format step needed. AsOfRestrictionLive +// exists only for API symmetry with EpochRestriction's shape -- it is never +// persisted (Live is the absence of a value, removed_at IS NULL, not a +// stored one) and AsOfReadInTx never returns it. +type AsOfRestriction string + +const ( + AsOfRestrictionLive AsOfRestriction = "live" + AsOfRestrictionGoneRetention AsOfRestriction = "gone_retention" + AsOfRestrictionGoneErasure AsOfRestriction = "gone_erasure" + AsOfRestrictionGoneReorganization AsOfRestriction = "gone_reorganization" + AsOfRestrictionUnknown AsOfRestriction = "unknown" +) + +// NormalizeRemovedRestriction is the one place a stored removed_restriction +// becomes an answer. The column is a free VARCHAR written by whatever produced +// the row, so only the values this file's vocabulary names are handed back. +// NULL, "", "live" (never persisted: Live is the absence of removed_at) and +// any other string say nothing about why the row was removed, and are reported +// as unknown rather than passed on as if they were an answer. +// +// Everything that says why a removed version is gone goes through it -- the +// as-of read, the version listing and the label bd versions prints -- so they +// cannot answer differently for the same row. It does not decide whether the +// row was removed; that is removed_at, which a caller reads first. +func NormalizeRemovedRestriction(stored string) AsOfRestriction { + restriction := AsOfRestriction(stored) + switch restriction { + case AsOfRestrictionGoneRetention, AsOfRestrictionGoneErasure, + AsOfRestrictionGoneReorganization, AsOfRestrictionUnknown: + return restriction + default: + return AsOfRestrictionUnknown + } +} + +// AsOfSelector selects a point in an issue's version history to read: either +// a literal instant (At) or a previously-served version address (Address) -- +// exactly one populated, mirroring the duality +// backend/conformance.AsOfSelector exposes to its own callers. This file +// owns its own copy rather than importing conformance's type (see the +// package-boundary note above). +type AsOfSelector struct { + At *time.Time + Address string +} + +// AsOfReadResult is AsOfReadInTx's answer. State and Address are populated +// only on a served (non-refused) result; Restriction and Reason only on a +// refused one. +type AsOfReadResult struct { + Address string + State map[string]any + Refused bool + Restriction AsOfRestriction + Reason string +} + +// AsOfVersionAddress is the address AsOfReadInTx serves for a resolved +// revision: a deterministic token over (issueID, revision). storeID is not +// part of the token -- a version address is meaningful only against its own +// issueID's rows, and Resolve is always called with issueID already known +// as its own separate parameter (matching the conformance fixture's own hook +// shape) -- but issueID is embedded anyway for readability/debuggability, +// per the design's stated rationale. +// +// Exported so each leg's AsOfReadFixture.MintAt can build the exact same +// token its own RecordVersionAtInTx write will resolve against, without +// duplicating the "asof:%s:%d" format in three separate packages. +func AsOfVersionAddress(issueID string, revision int64) string { + return fmt.Sprintf("asof:%s:%d", issueID, revision) +} + +// parseAsOfVersionAddress reverses AsOfVersionAddress. ok is false for any +// address not shaped like one this file mints, INCLUDING one whose embedded +// issueID does not match the issueID the caller separately supplied -- a +// version address for a different issue is never valid in that caller's +// context. +func parseAsOfVersionAddress(address, issueID string) (revision int64, ok bool) { + parts := strings.SplitN(address, ":", 3) + if len(parts) != 3 || parts[0] != "asof" || parts[1] != issueID { + return 0, false + } + rev, err := strconv.ParseInt(parts[2], 10, 64) + if err != nil { + return 0, false + } + return rev, true +} + +// resolveAsOfRevisionInTx resolves selector to the one issue_versions +// revision it names, for issueID: the version-address branch parses the +// revision straight out of the token, the instant branch is +// +// SELECT revision FROM issue_versions +// WHERE issue_id = ? AND change_at <= ? +// ORDER BY change_at DESC, revision DESC LIMIT 1 +// +// -- the latest version accepted at or before T, with everything later +// strictly excluded (R7.1's own wording); revision DESC is a deterministic +// tie-break for two versions minted in the same instant. found is false +// when no row resolves either way -- the address named nothing this store +// holds, or no version of issueID existed yet at T -- which the caller +// reports as Unknown, never substituting a neighboring version (R8). +func resolveAsOfRevisionInTx(ctx context.Context, tx DBTX, issueID string, selector AsOfSelector) (revision int64, found bool, err error) { + if selector.Address != "" { + rev, ok := parseAsOfVersionAddress(selector.Address, issueID) + if !ok { + return 0, false, nil + } + var exists int + err := tx.QueryRowContext(ctx, + `SELECT 1 FROM issue_versions WHERE issue_id = ? AND revision = ?`, issueID, rev, + ).Scan(&exists) + if errors.Is(err, sql.ErrNoRows) { + return 0, false, nil + } + if err != nil { + return 0, false, fmt.Errorf("as-of read: resolve address for %s: %w", issueID, err) + } + return rev, true, nil + } + + if selector.At == nil { + return 0, false, fmt.Errorf("as-of read: selector for %s has neither an instant nor a version address", issueID) + } + var rev int64 + err = tx.QueryRowContext(ctx, + `SELECT revision FROM issue_versions WHERE issue_id = ? AND change_at <= ? ORDER BY change_at DESC, revision DESC LIMIT 1`, + issueID, *selector.At, + ).Scan(&rev) + if errors.Is(err, sql.ErrNoRows) { + return 0, false, nil + } + if err != nil { + return 0, false, fmt.Errorf("as-of read: resolve instant for %s: %w", issueID, err) + } + return rev, true, nil +} + +// AsOfReadInTx answers R7.1's as-of read for issueID under selector: it +// resolves selector to one issue_versions row (resolveAsOfRevisionInTx), +// then reports that SAME row's own removed_at/removed_reason/ +// removed_restriction. Asking as-of an earlier T than when a later removal +// happened is therefore not a refusal -- the earlier row itself was never +// marked removed, only whichever row a later MarkLatestVersionRemovedInTx +// call touched was. +// +// storeID is accepted only for signature parity (see the package doc above); +// it plays no role in resolution or scoping. +func AsOfReadInTx(ctx context.Context, tx DBTX, storeID, issueID string, selector AsOfSelector) (AsOfReadResult, error) { + _ = storeID + + revision, found, err := resolveAsOfRevisionInTx(ctx, tx, issueID, selector) + if err != nil { + return AsOfReadResult{}, err + } + if !found { + return AsOfReadResult{ + Refused: true, + Restriction: AsOfRestrictionUnknown, + Reason: fmt.Sprintf("no version of %s found at or before the requested point", issueID), + }, nil + } + + var durableState []byte + var removedAt sql.NullTime + var removedReason sql.NullString + var removedRestriction sql.NullString + err = tx.QueryRowContext(ctx, + `SELECT durable_state, removed_at, removed_reason, removed_restriction + FROM issue_versions WHERE issue_id = ? AND revision = ?`, + issueID, revision, + ).Scan(&durableState, &removedAt, &removedReason, &removedRestriction) + if errors.Is(err, sql.ErrNoRows) { + return AsOfReadResult{ + Refused: true, + Restriction: AsOfRestrictionUnknown, + Reason: fmt.Sprintf("no version of %s found at or before the requested point", issueID), + }, nil + } + if err != nil { + return AsOfReadResult{}, fmt.Errorf("as-of read: load revision %d for %s: %w", revision, issueID, err) + } + + if removedAt.Valid { + return AsOfReadResult{ + Refused: true, + Restriction: NormalizeRemovedRestriction(removedRestriction.String), + Reason: removedReason.String, + }, nil + } + + state := map[string]any{} + if len(durableState) > 0 { + if err := json.Unmarshal(durableState, &state); err != nil { + return AsOfReadResult{}, fmt.Errorf("as-of read: unmarshal durable state for %s revision %d: %w", issueID, revision, err) + } + } + return AsOfReadResult{Address: AsOfVersionAddress(issueID, revision), State: state}, nil +} + +// MarkLatestVersionRemovedInTx is a test-support seam for R7.1's +// MakeUnanswerable fixture hook: it marks issueID's LATEST version row (by +// revision, not by time) as removed under restriction, independent of +// R17/R20's real (not yet implemented) hold/remove/erase primitives. +// "Latest" is correct for every AsOfReadContract case: each mints exactly +// one version before calling this, so the latest row and the row under test +// always coincide. Production code never calls this; only per-leg +// AsOfReadFixture.MakeUnanswerable implementations do. +func MarkLatestVersionRemovedInTx(ctx context.Context, tx DBTX, issueID string, restriction AsOfRestriction, reason string) error { + var revision int64 + if err := tx.QueryRowContext(ctx, + `SELECT revision FROM issue_versions WHERE issue_id = ? ORDER BY revision DESC LIMIT 1`, issueID, + ).Scan(&revision); err != nil { + return fmt.Errorf("as-of read: find latest version for %s: %w", issueID, err) + } + if _, err := tx.ExecContext(ctx, + `UPDATE issue_versions SET removed_at = ?, removed_reason = ?, removed_restriction = ? WHERE issue_id = ? AND revision = ?`, + time.Now().UTC(), reason, string(restriction), issueID, revision, + ); err != nil { + return fmt.Errorf("as-of read: mark version removed for %s: %w", issueID, err) + } + return nil +} diff --git a/internal/storage/issueops/asof_read_test.go b/internal/storage/issueops/asof_read_test.go new file mode 100644 index 000000000..eb7f1128a --- /dev/null +++ b/internal/storage/issueops/asof_read_test.go @@ -0,0 +1,173 @@ +package issueops + +import ( + "context" + "fmt" + "regexp" + "testing" + "time" + + "github.com/DATA-DOG/go-sqlmock" +) + +// TestAsOfReadMapsAnyStoredRestrictionOutsideTheVocabularyToUnknown is finding +// 5 of the #6661 review. removed_restriction is a free VARCHAR(30) written by +// whatever produced the row, and AsOfReadInTx handed it back as an +// AsOfRestriction by a plain cast. The type's doc says the answer is one of four +// values and that AsOfRestrictionLive is never returned (Live is the absence of +// removed_at, not a stored value), yet "live" and any other string passed +// straight through, so a caller switching on the result saw a value the +// vocabulary does not contain and could not tell it from an answer. +// +// A stored value outside the four says nothing about why the row was removed, +// so it is reported as unknown: this store cannot say. The hyphenated spelling +// is in the table because it is what the migration header used to tell a writer +// to store. +func TestAsOfReadMapsAnyStoredRestrictionOutsideTheVocabularyToUnknown(t *testing.T) { + const ( + issueID = "bd-1" + revision = int64(3) + reason = "retention window elapsed" + ) + removedAt := time.Date(2026, 9, 2, 0, 0, 0, 0, time.UTC) + + for _, tt := range []struct { + name string + stored any // the removed_restriction column; nil is SQL NULL + want AsOfRestriction + }{ + {"retention", "gone_retention", AsOfRestrictionGoneRetention}, + {"erasure", "gone_erasure", AsOfRestrictionGoneErasure}, + {"reorganization", "gone_reorganization", AsOfRestrictionGoneReorganization}, + {"unknown", "unknown", AsOfRestrictionUnknown}, + {"empty string", "", AsOfRestrictionUnknown}, + {"NULL", nil, AsOfRestrictionUnknown}, + {"live is never a stored answer", "live", AsOfRestrictionUnknown}, + {"hyphenated spelling of a real value", "gone-retention", AsOfRestrictionUnknown}, + {"arbitrary text", "quarantined", AsOfRestrictionUnknown}, + } { + t.Run(tt.name, func(t *testing.T) { + _, mock, tx := beginMockTx(t) + mock.ExpectQuery(regexp.QuoteMeta(`SELECT 1 FROM issue_versions WHERE issue_id = ? AND revision = ?`)). + WithArgs(issueID, revision). + WillReturnRows(sqlmock.NewRows([]string{"1"}).AddRow(1)) + mock.ExpectQuery(`SELECT durable_state, removed_at, removed_reason, removed_restriction\s+FROM issue_versions`). + WithArgs(issueID, revision). + WillReturnRows(sqlmock.NewRows([]string{"durable_state", "removed_at", "removed_reason", "removed_restriction"}). + AddRow([]byte("{}"), removedAt, reason, tt.stored)) + + got, err := AsOfReadInTx(context.Background(), tx, "store", issueID, + AsOfSelector{Address: AsOfVersionAddress(issueID, revision)}) + if err != nil { + t.Fatalf("AsOfReadInTx: %v", err) + } + if !got.Refused { + t.Fatalf("a version row with removed_at set must be refused, got %+v", got) + } + if got.Restriction != tt.want { + t.Errorf("stored removed_restriction %#v: Restriction = %q, want %q", tt.stored, got.Restriction, tt.want) + } + if got.Reason != reason { + t.Errorf("Reason = %q, want %q: the stored reason must survive the restriction being normalised", got.Reason, reason) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } + }) + } +} + +// TestListVersionsAgreesWithTheAsOfReadOnWhyAVersionWasRemoved is finding 3 of the +// #6661 review. AsOfReadInTx puts a stored removed_restriction through the four-value +// vocabulary: anything outside it, a NULL or empty column included, is unknown. The +// listing returned the column as stored, for every row, so bd versions and its JSON +// could name a restriction the as-of read never gives ("live", "quarantined", a +// hyphenated spelling), say nothing for a removed version the as-of read calls +// unknown, and attach a restriction to a version that was never removed. Both reads +// must answer the same question the same way: a removed version's restriction is one +// of the vocabulary's values, and a version that is not removed carries none. +// +// The as-of read is the oracle, run on the same stored value, so the two cannot drift +// apart again whichever one is changed. +func TestListVersionsAgreesWithTheAsOfReadOnWhyAVersionWasRemoved(t *testing.T) { + const ( + issueID = "bd-1" + revision = int64(3) + reason = "retention window elapsed" + ) + changeAt := time.Date(2026, 9, 1, 12, 0, 0, 0, time.UTC) + removedAt := time.Date(2026, 9, 2, 0, 0, 0, 0, time.UTC) + + for _, tt := range []struct { + name string + stored any // the removed_restriction column; nil is SQL NULL + }{ + {"retention", "gone_retention"}, + {"erasure", "gone_erasure"}, + {"reorganization", "gone_reorganization"}, + {"unknown", "unknown"}, + {"empty string", ""}, + {"NULL", nil}, + {"live is never a stored answer", "live"}, + {"hyphenated spelling of a real value", "gone-retention"}, + {"arbitrary text", "quarantined"}, + } { + for _, removed := range []bool{true, false} { + t.Run(fmt.Sprintf("%s/removed=%t", tt.name, removed), func(t *testing.T) { + // The listing selects COALESCE(removed_restriction, ''), so NULL reaches it as "". + listed, _ := tt.stored.(string) + var removedAtValue any // nil is SQL NULL: a version that was never removed + var storedReason string + if removed { + removedAtValue = removedAt + storedReason = reason + } + + _, mock, tx := beginMockTx(t) + mock.ExpectQuery(`FROM issue_versions\s+WHERE issue_id = \?\s+ORDER BY revision DESC`). + WithArgs(issueID). + WillReturnRows(sqlmock.NewRows([]string{ + "issue_id", "revision", "epoch", "change_actor", "change_agent", "change_message", + "change_at", "attribution_status", "removed_at", "removed_reason", "removed_restriction", "state_bytes", + }).AddRow(issueID, revision, int64(1), "", "", "", changeAt, "", removedAtValue, storedReason, listed, int64(0))) + mock.ExpectQuery(regexp.QuoteMeta(`SELECT 1 FROM issue_versions WHERE issue_id = ? AND revision = ?`)). + WithArgs(issueID, revision). + WillReturnRows(sqlmock.NewRows([]string{"1"}).AddRow(1)) + mock.ExpectQuery(`SELECT durable_state, removed_at, removed_reason, removed_restriction\s+FROM issue_versions`). + WithArgs(issueID, revision). + WillReturnRows(sqlmock.NewRows([]string{"durable_state", "removed_at", "removed_reason", "removed_restriction"}). + AddRow([]byte("{}"), removedAtValue, storedReason, tt.stored)) + + versions, err := ListVersionsInTx(context.Background(), tx, issueID) + if err != nil { + t.Fatalf("ListVersionsInTx: %v", err) + } + asOf, err := AsOfReadInTx(context.Background(), tx, "store", issueID, + AsOfSelector{Address: AsOfVersionAddress(issueID, revision)}) + if err != nil { + t.Fatalf("AsOfReadInTx: %v", err) + } + if len(versions) != 1 { + t.Fatalf("ListVersionsInTx returned %d versions, want 1", len(versions)) + } + got := versions[0].RemovedRestriction + + if removed { + if !asOf.Refused { + t.Fatalf("the as-of read of a removed version must be refused, got %+v", asOf) + } + if got != string(asOf.Restriction) { + t.Errorf("stored removed_restriction %#v: the listing says %q, the as-of read says %q; they must agree", tt.stored, got, asOf.Restriction) + } + return + } + if asOf.Refused { + t.Fatalf("the as-of read of a version that was not removed must be served, got %+v", asOf) + } + if got != "" { + t.Errorf("stored removed_restriction %#v on a version that was not removed: the listing says %q, want no restriction", tt.stored, got) + } + }) + } + } +} diff --git a/internal/storage/issueops/journal_completeness_test.go b/internal/storage/issueops/journal_completeness_test.go index 9af2f61d1..2147d46e2 100644 --- a/internal/storage/issueops/journal_completeness_test.go +++ b/internal/storage/issueops/journal_completeness_test.go @@ -203,17 +203,32 @@ var beadDMLExemptions = map[string]string{ // equivalents used by the uow leg — version_completeness_test.go pins the // set) immediately after those callers already invoke RecordEventInTx, so the // mutation itself is already journaled by the time RecordVersionInTx runs. - // Its own UPDATE issues SET current_revision = ? only advances a - // denormalized pointer to match the snapshot row it just inserted into - // issue_versions (not a bead table, so it can't anchor the exemption on - // its own) — bookkeeping for an already-journaled mutation, not a second - // mutation that needs its own emit. + // Its own UPDATE issues SET current_revision = ?, updated_at = updated_at + // only advances a denormalized pointer to match the snapshot row it just + // inserted into issue_versions (not a bead table, so it can't anchor the + // exemption on its own), leaving updated_at where the mutation put it — + // bookkeeping for an already-journaled mutation, not a second mutation + // that needs its own emit. "RecordVersionInTx": "advances the denormalized current_revision pointer to match a snapshot just inserted into issue_versions (not a bead table); called from the same entry points that already journal the mutation via RecordEventInTx, so this is bookkeeping for an already-journaled mutation, not a second one", // RecordVersionForCreateInTx is RecordVersionInTx's create-shaped sibling // (design §16.2b write-fence, be-h89oq): same entry points, same // already-journaled-by-the-caller bookkeeping, plus stamping // participation_generation on the row it just inserted into issue_versions. "RecordVersionForCreateInTx": "advances the denormalized current_revision pointer and stamps participation_generation to match a snapshot just inserted into issue_versions (not a bead table); called from the same create entry points that already journal the mutation via RecordEventInTx, so this is bookkeeping for an already-journaled mutation, not a second one", + + // RecordVersionAtInTx (R7.1 as-of read, gastownhall/beads#5898 rev 9, + // gastownhall/beads#6136) is RecordVersionInTx's test-support + // twin: it shares RecordVersionInTx's body (recordVersionAtInTx) but skips + // the versionedHistoryEnabled gate and takes an explicit historical + // timestamp, so an as-of-read conformance fixture can mint a version row + // dated in the past. It has no production caller and is not part of any + // mutationEntryPoints flow — only per-leg AsOfReadFixture.MintAt + // implementations call it, always directly, never from inside a journaled + // mutation. There is no separate bead mutation here for it to double-journal: + // the same "UPDATE issues SET current_revision only advances a pointer to + // a issue_versions snapshot, not a bead table" reasoning as RecordVersionInTx + // applies verbatim. + "RecordVersionAtInTx": "test-support twin of RecordVersionInTx (R7.1 as-of read); mints a historical-timestamped snapshot directly for conformance fixtures, with no production caller and no mutation of its own to journal — its UPDATE issues SET current_revision only advances a pointer to the issue_versions row it just inserted (not a bead table)", } // journalExemptMutations are mutation paths that deliberately do NOT journal diff --git a/internal/storage/issueops/row_lock_guard_test.go b/internal/storage/issueops/row_lock_guard_test.go index f7bc0dc13..9f1083a63 100644 --- a/internal/storage/issueops/row_lock_guard_test.go +++ b/internal/storage/issueops/row_lock_guard_test.go @@ -141,28 +141,32 @@ var funcNameExemptions = map[string]string{ // ExpectedVersion CAS reject a row it should still recognize. "resolveOneConflictRow": "whole-row `theirs` adoption: the adopted row_lock already vouches for the (identical) adopted content", - // recordVersionInTx (version_history.go) is the private seam both public - // entry points — RecordVersionInTx (update-shaped) and + // recordVersionAtInTx (version_history.go) is the shared body behind all + // three public entry points: RecordVersionInTx (update-shaped), // RecordVersionForCreateInTx (create-shaped, design §16.2b's write fence, - // be-h89oq) — delegate to; the three UPDATE issues statements this guard - // would otherwise flag live here, not in either public wrapper, so the - // exemption is keyed to this name. It runs immediately after the mutation - // it is snapshotting — create.go/update.go's own writes, or domain/db's - // Insert/Update — and those callers have already minted a fresh row_lock - // in that same INSERT/UPDATE, in the same transaction. Every UPDATE - // issues statement recordVersionInTx issues writes only bookkeeping - // columns: the update-shaped and create-shaped UPDATE issues SET - // current_revision = ... statements advance the denormalized pointer to - // the issue_versions row just inserted (the create-shaped one also stamps - // participation_generation in the same statement), and the no-history - // stamp sets participation_generation alone on a create-shaped row it does - // not version. None touches row_lock at all (not even to carry the - // existing value forward) because reminting here would bump the token a - // second time within one transaction, invalidating the RowVersion/CAS + // be-h89oq) and the R7.1 test-support twin RecordVersionAtInTx + // (gastownhall/beads#5898 rev 9, gastownhall/beads#6136). + // The three UPDATE issues statements this guard would otherwise flag live + // here, not in any public wrapper, so the exemption is keyed to this name. + // The two production entry points' callers run it immediately after the + // mutation it is snapshotting — create.go/update.go's own writes, or + // domain/db's Insert/Update — and those callers have already minted a + // fresh row_lock in that same INSERT/UPDATE, in the same transaction. + // RecordVersionAtInTx's fixture-only callers likewise run it only after a + // bare-create has already minted the row's row_lock, never as a row's + // first write. Every UPDATE issues statement this function issues writes + // only bookkeeping columns: the update-shaped and create-shaped UPDATE + // issues SET current_revision = ... statements advance the denormalized + // pointer to the issue_versions row just inserted (the create-shaped one + // also stamps participation_generation in the same statement), and the + // no-history stamp sets participation_generation alone on a create-shaped + // row it does not version. None touches row_lock at all (not even to carry + // the existing value forward) because reminting here would bump the token + // a second time within one transaction, invalidating the RowVersion/CAS // value the primary mutation's own caller already captured and could // cause a legitimate subsequent CAS check to fail as a spurious lost // update. - "recordVersionInTx": "runs after the primary mutation already minted a fresh row_lock in the same transaction; reminting here would double-bump the token and invalidate the RowVersion/CAS value already returned to that mutation's caller — covers all three of its UPDATE issues statements (the update-shaped and create-shaped current_revision advances and the no-history participation_generation stamp), since both public entry points delegate to this one private seam", + "recordVersionAtInTx": "runs after the primary mutation (or, for RecordVersionAtInTx's fixture-only callers, a bare-create) already minted a fresh row_lock in the same transaction; reminting here would double-bump the token and invalidate the RowVersion/CAS value already returned to that mutation's caller — covers all three of its UPDATE issues statements (the update-shaped and create-shaped current_revision advances and the no-history participation_generation stamp), since all three public entry points delegate to this one shared body", } // TestAllIssueRowWritesStampRowLock is the load-bearing completeness guard for diff --git a/internal/storage/issueops/version_completeness_test.go b/internal/storage/issueops/version_completeness_test.go index 3b9b08dbe..e9b484eb4 100644 --- a/internal/storage/issueops/version_completeness_test.go +++ b/internal/storage/issueops/version_completeness_test.go @@ -359,6 +359,62 @@ func sortedShapeKeys(m map[string]string) []string { return keys } +// versionTestSupportMints sit outside the shape split above, explicitly: they +// are neither create-shaped, dual-shape nor update-shaped. They are not +// mutation entry points (versionedEntryPoints) but the seam's test-support +// twin, exempt in versionExemptions, and they reach the shared body directly +// rather than through either entry. Each must reach neither +// RecordVersionForCreateInTx nor RecordVersionInTx: the unfenced mint +// (mintUnfenced) never stamps participation_generation, so a conformance +// fixture's controlled-timestamp mint cannot promote a legacy row, and it is +// not gated or fenced, so routing it through the update-shaped entry would +// silently stop the as-of fixtures from minting history-off rows. +var versionTestSupportMints = map[string]string{ + "RecordVersionAtInTx": "R7.1 as-of read fixtures' controlled-timestamp mint (mintUnfenced): no gate, no fence, no stamp", +} + +// TestTestSupportMintsStayOutsideTheShapeSplit pins versionTestSupportMints. +func TestTestSupportMintsStayOutsideTheShapeSplit(t *testing.T) { + fns, createShaped, updateShaped := versionShapedMints(t) + shapeListed := map[string]string{} + for _, entry := range versionedEntryPoints { + shapeListed[entry] = "versionedEntryPoints" + } + for _, entry := range versionCreateShapedEntryPoints { + shapeListed[entry] = "versionCreateShapedEntryPoints" + } + for _, site := range versionCreateShapedMintSites { + shapeListed[site] = "versionCreateShapedMintSites" + } + for entry := range versionDualShapeEntryPoints { + shapeListed[entry] = "versionDualShapeEntryPoints" + } + for site := range versionDualShapeMintSites { + shapeListed[site] = "versionDualShapeMintSites" + } + for _, name := range sortedShapeKeys(versionTestSupportMints) { + if versionTestSupportMints[name] == "" { + t.Errorf("%s has an empty test-support reason", name) + } + if _, defined := fns[name]; !defined { + t.Errorf("test-support mint %q not found in issueops — was it renamed? update versionTestSupportMints", name) + continue + } + if list, listed := shapeListed[name]; listed { + t.Errorf("test-support mint %q is listed in %s; it is the seam's twin, not a mutation entry point or mint site, and stays outside the shape split", name, list) + } + if createShaped[name] { + t.Errorf("test-support mint %q reaches RecordVersionForCreateInTx, so a fixture's mint can stamp participation_generation and promote a legacy record (R2.3)", name) + } + if updateShaped[name] { + t.Errorf("test-support mint %q reaches RecordVersionInTx, whose gate and fence would stop an as-of fixture from minting a history-off row", name) + } + if _, exempt := versionExemptions[name]; !exempt { + t.Errorf("test-support mint %q is not in versionExemptions; it is the seam's twin, not a mutation entry point", name) + } + } +} + // versionExemptions are exported functions the DML detector flags as writing a // work-bead table but which legitimately do NOT mint a version for the row // they write, each with a reason. The staleness check fails if any stops being @@ -433,6 +489,15 @@ var versionExemptions = map[string]string{ // bookkeeping half of the mint, not a mutation that needs its own. "RecordVersionInTx": "the seam itself; advances current_revision to match the row it just inserted", "RecordVersionForCreateInTx": "the seam's create-shaped sibling; stamps participation_generation and advances current_revision to match the row it just inserted", + + // RecordVersionAtInTx (R7.1 as-of read, gastownhall/beads#5898 rev 9, + // gastownhall/beads#6136) shares RecordVersionInTx's body + // (recordVersionAtInTx) but deliberately does not call RecordVersionInTx + // itself — it exists precisely to bypass RecordVersionInTx's + // versionedHistoryEnabled gate, so an as-of-read conformance fixture can + // mint a version dated in the past. It is the seam too, by the same + // reasoning as RecordVersionInTx's own entry above. + "RecordVersionAtInTx": "test-support twin of RecordVersionInTx, sharing its body; the seam itself, not a caller of it — advances current_revision to match the row it just inserted", } // versionNeverMints pins the deliberate NOT-versioned rulings from the other diff --git a/internal/storage/issueops/version_history.go b/internal/storage/issueops/version_history.go index cd2ae2319..202220502 100644 --- a/internal/storage/issueops/version_history.go +++ b/internal/storage/issueops/version_history.go @@ -1,15 +1,21 @@ package issueops import ( + "bytes" "context" "database/sql" "encoding/json" "errors" "fmt" + "io" + "math" + "strconv" "sync" "time" "github.com/gowebpki/jcs" + + "github.com/steveyegge/beads/internal/types" ) // Dual-write issue-version history records every accepted issue mutation as a @@ -81,6 +87,53 @@ import ( // costs no extra round-trip. Tracked as gastownhall/beads#6379 alongside // item 4. +// THE MINT'S REFUSAL IS AN INTEGRITY BACKSTOP, NOT AN ADMISSION POLICY, AND IT +// BINDS THE ROWS THAT PARTICIPATE IN HISTORY. +// canonicalDurableState refuses what it cannot record faithfully -- a number +// outside the I-JSON exact-integer range, wherever in the issue it sits (in +// metadata, or in a gate's timeout, which is written as nanoseconds), and +// duplicate keys -- and it does so inside the mutating transaction, so with +// history on a write that introduces such a value to a row that participates +// fails atomically and commits nothing: refusing at the mint IS refusing the +// write, for every store-mediated write to such a row. That is what keeps a +// version's content token meaning what it says. It is not a rule about what an +// issue may hold, which stays open while history is off, and that is the case to +// plan for: a row that participates in history (participation_generation is set), +// written while history was off or by a path that does not mint (such as bd sql, +// an older bd or a pull from a clone that had history off), that already holds +// such a value would fail every later write to it once history is on. `bd config +// set versioned-history.enabled true` therefore checks the store first, with the +// check the mint runs (CheckIssueVersionable, over every issue the mint would +// version), and refuses, writing nothing, while any of them holds a value it would +// refuse. +// +// A row that does not participate is outside all of this. Design §16.2b makes an +// update-shaped write to a legacy record (participation_generation NULL) a skip, +// not a refusal: the fence returns before the snapshot is canonicalized, so no +// version is minted and the gate never runs. Such a write succeeds whatever the +// row's metadata holds, exactly as it would with history off, because no version +// row, and so no content token, comes of it. A create-shaped mint never meets a +// legacy row (its call site has just inserted the row), so it runs the gate +// whenever it mints; a new row flagged no-history is stamped but not minted, so +// nothing is canonicalized for it. The switch's check cannot see participation, +// so it checks legacy rows too: an over-approximation that can only refuse more. +// +// That check ran once, at that moment. Nothing keeps a writer that does not mint +// from adding such a row afterwards, even right after the switch is on; if the +// row participates, the first write to it then fails with the same refusal, and +// the same fix applies. Making the check and the write of the setting one +// transaction would narrow that window, not close it. Turning history on through +// the environment or config.yaml does not pass through that command at all; for +// those planes this refusal is the control on the rows that participate, and +// finding the rows ahead of time is a bd doctor check (gastownhall/beads#6379 +// item 3). +// +// Duplicate keys are the exception to "the case to plan for". A Dolt JSON column +// collapses duplicate keys when it stores a document, so on a Dolt store neither +// the mint nor the scan can ever see one, and the refusal is a defense for a +// backend registered through backends.Lookup that keeps them. It is worth +// checking again whenever Dolt is upgraded. + var versionedHistoryTransactions sync.Map // map[DBTX]bool; entries live for one transaction // ScopeVersionedHistoryTransaction associates versioned-history activation @@ -179,11 +232,18 @@ func attributionStatusForActor(actor string) string { // (1.0 is 1, 1e300 is 1e+300), only the escapes RFC 8785 requires -- so the // same issue state always yields the same bytes, whatever encoding/json's // formatting happens to be. Numbers are canonicalized as IEEE-754 doubles -// (RFC 8785 section 3.2.2.3), so an integer past 2^53 is rounded here, once, -// by the writer, before anything hashes it: the stored bytes and the hashed -// bytes are the same bytes. types.Issue carries no such magnitudes (its -// integers are ordinals and priorities), but the rule is stated so nobody -// expects int64 fidelity from the token. +// (RFC 8785 section 3.2.2.3), so canonicalization alone would round a number +// past 2^53 here, once, by the writer, before anything hashes it; the admission +// gate (refuseUnrepresentableIntegers) refuses such a number instead, so the +// stored bytes and the hashed bytes are the same bytes and nothing is rounded +// silently. Two fields of types.Issue can reach that magnitude: Metadata, and +// Timeout, a time.Duration that encoding/json writes as a count of nanoseconds, +// so a gate timeout above 2^53-1 ns (about 104.25 days) is refused. Its other +// integers are priorities, ordinals and counts held in 32-bit columns. +// +// A refusal names the top-level field it was found in (nameRefusedField). The +// success path reads the document once; the field is located only after a +// refusal. // // Two properties worth stating because a hand-rolled canonicalizer would get // them wrong. First, jcs.Transform normalizes away encoding/json's HTML @@ -191,18 +251,95 @@ func attributionStatusForActor(actor string) string { // Go's escaping policy. Second, a snapshot that cannot be canonicalized // deliberately FAILS the mutation: types.Issue.Metadata is a json.RawMessage // passed through verbatim, RFC 8785 rejects duplicate keys, and this seam -// returns the error to its caller, which aborts the transaction. An issue -// whose metadata column already holds duplicate keys (reachable through bd -// sql or a hand-written JSONL import) therefore mutates fine with the flag -// off and becomes unmutatable with it on. Fail-closed is the policy for now -// -- bytes that could canonicalize two ways would not be a content token -- -// and whether to normalize such metadata first, or to find such rows with +// returns the error to its caller, which aborts the transaction. A backend that +// keeps duplicate keys in a metadata document would therefore let an issue +// mutate with the flag off and make it unmutatable with it on, if it participates +// in history; a legacy row stays writable, because the fence skips it before this +// function runs. Dolt does not keep them (see the note above), so on a Dolt +// store this refusal is never reached. Fail-closed is the policy for a backend +// that does -- bytes that could canonicalize two ways would not be a content +// token -- and whether to normalize such metadata first, or to find such rows with // bd doctor, is gastownhall/beads#6379 (item 3). func canonicalDurableState(issue any) ([]byte, error) { marshaled, err := json.Marshal(issue) if err != nil { return nil, err } + if err := refuseUnrepresentableIntegers(marshaled); err != nil { + return nil, nameRefusedField(marshaled, err) + } + canonical, err := jcsCanonicalize(marshaled) + if err != nil { + return nil, nameRefusedField(marshaled, err) + } + return canonical, nil +} + +// refusedFieldError is a refusal from canonicalDurableState that has been traced +// to the top-level field of the issue it was found in. It unwraps to the refusal +// itself, so errors.Is(err, ErrIntegerNotRepresentable) still answers. +type refusedFieldError struct { + field string + err error +} + +func (e *refusedFieldError) Error() string { return fmt.Sprintf("%v (field %q)", e.err, e.field) } + +func (e *refusedFieldError) Unwrap() error { return e.err } + +// nameRefusedField finds the top-level field of marshaled, one issue as a JSON +// object, that the two checks canonicalDurableState runs refuse on their own, and +// returns err naming it. It runs only after a refusal, so what a successful +// canonicalization costs does not change. A document it cannot attribute, one +// that is not an object or in which no single field is refused alone, comes back +// unchanged. +func nameRefusedField(marshaled []byte, err error) error { + dec := json.NewDecoder(bytes.NewReader(marshaled)) + if open, derr := dec.Token(); derr != nil || open != json.Delim('{') { + return err + } + for dec.More() { + keyToken, derr := dec.Token() + key, isString := keyToken.(string) + if derr != nil || !isString { + return err + } + var value json.RawMessage + if derr := dec.Decode(&value); derr != nil { + return err + } + if refuseUnrepresentableIntegers(value) != nil { + return &refusedFieldError{field: key, err: err} + } + if _, jerr := jcsCanonicalize(value); jerr != nil { + return &refusedFieldError{field: key, err: err} + } + } + return err +} + +// jcsCanonicalize is canonicalDurableState's RFC 8785 (JCS) step alone, below +// the admission gate rather than behind it. +// +// IT MUST EXIST SEPARATELY because refuseUnrepresentableIntegers and RFC 8785 +// disagree on purpose, not by oversight: RFC 8785 section 3.2.2.3 canonicalizes +// ANY number as a double, whatever its magnitude, while the admission gate +// refuses any number whose nearest double is past 2^53-1 outright (see +// refuseUnrepresentableIntegers's doc comment on why that rule has no +// notation carve-out). One of the RFC 8785 vectors this store pins conformance +// against, testdata/jcs/input/values.json, both ships from the spec's own +// reference suite AND contains 1E30 -- a value the gate refuses and RFC 8785 +// requires canonicalizing to 1e+30. Routing that vector through +// canonicalDurableState would force a choice between weakening spec-conformance +// coverage and putting a magnitude exception in the gate, and neither is +// right: the gate's refusal is a policy decision about what this store admits, +// not a claim about what JCS can canonicalize. jcsCanonicalize lets a caller +// ask the second question without the first, so TestCanonicalDurableStateIsJCS +// can hold the RFC 8785 vectors to the spec's own bytes -- including +// values.json -- while refuseUnrepresentableIntegers's refusal of the same +// magnitude is pinned separately, in the admission tests +// (version_history_integer_admission_test.go), where it belongs. +func jcsCanonicalize(marshaled []byte) ([]byte, error) { canonical, err := jcs.Transform(marshaled) if err != nil { return nil, fmt.Errorf("canonicalize (RFC 8785): %w", err) @@ -210,6 +347,129 @@ func canonicalDurableState(issue any) ([]byte, error) { return canonical, nil } +// ErrIntegerNotRepresentable is the refusal for a JSON number outside the I-JSON +// exact-integer range: one whose nearest binary64 has magnitude above 2^53-1, or +// that overflows binary64. +// +// The name is historical. It began as the refusal for an integer literal binary64 +// cannot hold exactly, and it now also covers a literal spelled as a fraction whose +// nearest double lies at or beyond 2^53, which RFC 8785 would round to an integer +// just the same. It is kept rather than renamed so that nothing matching it with +// errors.Is has to change. +var ErrIntegerNotRepresentable = errors.New("number is outside the I-JSON exact-integer range (magnitude above 2^53-1)") + +// maxExactJSONInteger is 2^53-1, the largest magnitude in the I-JSON +// interoperable integer range (RFC 7493 section 2.2), and BDP's admission law +// (gastownhall/bdp#21) makes a literal past it invalid rather than merely lossy. +// binary64 does represent 2^53 exactly, but 2^53 and 2^53+1 round to the same +// double, so from 2^53 up a literal can no longer be told from its neighbor. +const maxExactJSONInteger = 1<<53 - 1 + +// refuseUnrepresentableIntegers rejects marshaled if any JSON number literal in +// it is outside the I-JSON exact-integer range: the binary64 nearest to the +// literal has magnitude above 2^53-1, or the literal overflows binary64. Which of +// JSON's number forms spells the value -- bare integer, decimal or exponential -- +// does not matter. +// +// WHY REFUSE RATHER THAN ROUND. jcs.Transform canonicalizes numbers as +// IEEE-754 doubles (RFC 8785 section 3.2.2.3), so 9007199254740993 becomes +// 9007199254740992 on the way in. That is not a formatting difference, it is +// a DIFFERENT VALUE, and it collapses two distinct issue states onto one +// durable_state and therefore one content token -- the token can no longer +// tell them apart, and a reader comparing tokens concludes the second write +// was a no-op. Rounding is silent; refusing is not. +// +// This seam is the admission gate, one layer above RecordVersionInTx, which +// is where BDP puts it: an authority adapting an existing store "MUST map or +// refuse values outside this contract before it serves them as BDP +// Resources" (gastownhall/bdp#20 section 5.1). +// +// THE NEAREST DOUBLE, NOT THE LITERAL'S SPELLING. Each literal is classified +// with strconv.ParseFloat, which returns the correctly rounded binary64 in time +// linear in the length of the literal, and that is exactly the value RFC 8785 +// canonicalizes the literal to. So "the nearest double is past the range" and +// "canonicalization would move this literal past the range" are one question, +// and the gate is closed under its own canonicalization: whatever it admits +// canonicalizes to a form it also admits. A rule keyed to integer VALUES was +// not: 9007199254740991.5 is not an integer, so it was admitted, yet it rounds +// to 2^53 and its canonical bytes were then refused. Here 9007199254740993, +// 9007199254740993.0 and 9.007199254740993e15 are refused alike, and so is +// 9007199254740993.5, whose nearest double is 2^53+2. An overflow +// (strconv.ErrRange) is a refusal, not a syntax error; a magnitude too small for +// binary64 rounds to zero and is admitted. +// +// NOTHING IS BUILT PER EXPONENT. ParseFloat never constructs the number, so a +// literal such as 1e1000000 is an overflow refused in the time it takes to read +// it, not a million-digit integer built first and then compared. The gate runs +// inside the writer's transaction, so a classifier whose cost grew with an +// exponent would be a way to hold that transaction open with a few +// exponent-heavy metadata values. +// +// SCOPE. Ordinary fractions are admitted: 0.1 is canonicalized to its nearest +// double, which is how the store holds it. Whether two literals that share a +// nearest double can be told apart inside the store is a property of the store, +// not of this gate, and it is the one thing a Dolt upgrade could change: the +// store-level tests on each leg assert that no two literals the store reads back +// as different numbers are both admitted with one canonical form, and log what +// each Dolt does to a number, rather than assuming it. An integer past 2^53 is +// refused whatever the store does to it: a store may keep it exactly, where RFC +// 8785 would round it, and a store that already rounds it hands the gate an +// integer-valued double, which the rule refuses just the same. A JSON null is a +// token, not a number: only json.Number tokens are classified. Of types.Issue's +// own fields only Metadata (a json.RawMessage passed through verbatim) and +// Timeout (a time.Duration, written as nanoseconds) can reach this bound; its +// other integers are priorities, ordinals and counts held in 32-bit columns. +func refuseUnrepresentableIntegers(marshaled []byte) error { + dec := json.NewDecoder(bytes.NewReader(marshaled)) + dec.UseNumber() + for { + tok, err := dec.Token() + if err == io.EOF { + return nil + } + if err != nil { + return fmt.Errorf("scan for unrepresentable integers: %w", err) + } + num, ok := tok.(json.Number) + if !ok { + continue + } + lit := num.String() + f, err := strconv.ParseFloat(lit, 64) + if err != nil && !errors.Is(err, strconv.ErrRange) { + return fmt.Errorf("scan for unrepresentable integers: invalid JSON number literal %q: %w", lit, err) + } + if math.IsInf(f, 0) || math.Abs(f) > maxExactJSONInteger { + return fmt.Errorf("%w: %s", ErrIntegerNotRepresentable, lit) + } + } +} + +// CheckIssueVersionable reports whether issue could be minted into a version's +// durable_state. It is canonicalDurableState's own refusal -- the admission gate, +// then RFC 8785 canonicalization, over the whole marshaled issue -- so it refuses +// a number outside the I-JSON exact-integer range (ErrIntegerNotRepresentable) +// wherever in the issue it sits, in metadata or in a gate's timeout, and it also +// refuses duplicate keys, which RFC 8785 cannot canonicalize. The refusal names +// the top-level field it found the value in. +// +// It exists so that the check made when versioned history is switched on and +// the check the mint makes cannot disagree: FindUnversionable runs this over +// every issue the mint would version, and a store that passes it cannot then +// refuse the first write to a row it cleared. It is the one such check. A check +// over only part of an issue, its metadata say, can pass what the mint refuses (a +// gate's timeout can be past the range while its metadata is fine), so none may +// be exported beside it. +// +// The issue it is handed need not be the one the mint sees. The scan reads a +// lite projection, with no heavy text columns, no labels and no dependencies, +// and that is sound only because none of what it leaves out can hold a number. +// The tests in version_history_preenable_scan_test.go pin that. +func CheckIssueVersionable(issue *types.Issue) error { + _, err := canonicalDurableState(issue) + return err +} + // RecordVersionInTx mints one issue_versions row for issueID and advances // issues.current_revision to match, as of tx (read-your-writes within the // same transaction). A no-op when versioned history is disabled for tx, or @@ -239,8 +499,15 @@ func canonicalDurableState(issue any) ([]byte, error) { // for a create-shaped mutation instead: a brand-new row has no legacy state // to preserve, so that entry point stamps the column rather than checking // it. +// +// The skip is whole: nothing about a legacy record is read, canonicalized or +// admitted, so such a write is neither versioned nor refused (design §16.2b: a +// skip, not a refusal). func RecordVersionInTx(ctx context.Context, tx DBTX, issueID, actor string) error { - return recordVersionInTx(ctx, tx, issueID, actor, false) + if !versionedHistoryEnabled(tx) { + return nil + } + return recordVersionAtInTx(ctx, tx, issueID, actor, time.Now().UTC(), mintUpdate) } // RecordVersionForCreateInTx is RecordVersionInTx's create-shaped sibling @@ -261,48 +528,135 @@ func RecordVersionInTx(ctx context.Context, tx DBTX, issueID, actor string) erro // a wisp is never versioned (FR-8). That covers an ordinary wisp create and // an unflagged wisps-table row (import's "wisp_plane" marker) alike. func RecordVersionForCreateInTx(ctx context.Context, tx DBTX, issueID, actor string) error { - return recordVersionInTx(ctx, tx, issueID, actor, true) -} - -func recordVersionInTx(ctx context.Context, tx DBTX, issueID, actor string, isCreate bool) error { if !versionedHistoryEnabled(tx) { return nil } + return recordVersionAtInTx(ctx, tx, issueID, actor, time.Now().UTC(), mintCreate) +} + +// RecordVersionAtInTx is RecordVersionInTx's test-support twin: it mints a +// version row at a caller-chosen change_at instead of time.Now(), and -- +// deliberately -- does not gate on versionedHistoryEnabled at all, since its +// whole purpose is controlled-timestamp minting for conformance fixtures +// (R7.1's AsOfReadFixture.MintAt, backend/conformance/versioned_read_contract.go) +// regardless of a store's activation state. Production code never calls +// this; only per-leg as-of-read fixtures do, so their bare issue-create step +// can stay history-off (no unwanted real-time row) while still minting the +// exact versions a test needs at the instants it needs them. +// +// It bypasses design §16.2b's write fence exactly as it bypasses the gate: +// it ignores participation_generation and never stamps it, so it mints for +// a legacy row and a participating row alike and leaves the column as it +// found it. The fence and the stamp belong to the production entry points +// above, never to a fixture's controlled-timestamp mint. The plane rule is +// not the fence, and it does apply: a row the wisps table holds, or one +// flagged Ephemeral or NoHistory, mints nothing here either. +func RecordVersionAtInTx(ctx context.Context, tx DBTX, issueID, actor string, at time.Time) error { + return recordVersionAtInTx(ctx, tx, issueID, actor, at, mintUnfenced) +} + +// versionMintKind says how recordVersionAtInTx treats design §16.2b's write +// fence, the one thing RecordVersionInTx, RecordVersionForCreateInTx and +// RecordVersionAtInTx do not share. +type versionMintKind int + +const ( + // mintUpdate is an update-shaped mutation (RecordVersionInTx): the fence + // applies, and a record whose participation_generation is NULL mints nothing. + mintUpdate versionMintKind = iota + // mintCreate is a create-shaped mutation (RecordVersionForCreateInTx): a + // brand-new row has no legacy state to preserve, so the fence does not + // apply and the mint stamps participation_generation instead. + mintCreate + // mintUnfenced is RecordVersionAtInTx's test-support mint: no fence and no + // stamp, exactly as it behaved before the fence existed. Like the other + // two, it mints nothing for a row that is not on the issues plane. + mintUnfenced +) + +// recordVersionAtInTx is RecordVersionInTx, RecordVersionForCreateInTx and +// RecordVersionAtInTx's shared body, taking at where the callers differ: +// time.Now().UTC() for the production gate-checked paths, a caller-chosen +// instant for test minting; and kind, which says whether design §16.2b's +// write fence applies (mintUpdate), the mint stamps participation_generation +// (mintCreate) or neither (mintUnfenced). The no-op rules documented on +// RecordVersionInTx above (wisps) apply here too, since this is that +// function's entire mechanism minus the gate. +func recordVersionAtInTx(ctx context.Context, tx DBTX, issueID, actor string, at time.Time, kind versionMintKind) error { + // change_at is DATETIME(6) as of migration 0069, and this function + // deliberately does NOT floor `at` to the second. + // + // It used to. While 0067's plain DATETIME (precision 0) was in force that + // floor was load-bearing: Dolt's datetime(0) does not truncate sub-second + // input, it ROUNDS half-up, so a change_at of 13:32:04.77 stored as + // 13:32:05 -- strictly LATER than the instant it recorded. That breaks + // AsOfReadInTx's "latest version accepted at or before T" + // (resolveAsOfRevisionInTx, asof_read.go) for any T between the true + // instant and the rounded-up one. Flooring first left nothing to round. + // + // 0069 widened change_at and removed_at to DATETIME(6). That moves the + // rounding below the microsecond rather than removing it: GMS still rounds + // DATETIME(6) input to the microsecond (.1234567 is stored as .123457). + // The remainder is harmless to that lookup (resolveAsOfRevisionInTx), + // because the instant it compares against is rounded the same way, so a + // stored value still compares equal to the instant it was minted at. What the widening does + // make is the floor actively harmful: it threw away the microseconds the + // widening exists to keep. Two versions minted in the same second + // collapsed onto one stored value and became indistinguishable + // by change_at -- which defeats --at, the only PORTABLE selector the read + // surface offers while revision stays a local ordinal and no durable + // version address has shipped. Measured before this change on a store + // carrying both: four versions minted in one second all read back + // 15:40:38.000000. + // + // So the column's precision is now relied on rather than worked around. + // Pinned by TestRecordVersionKeepsSubSecondChangeAt below; the column's + // own half of the contract is pinned by + // TestMigration0069ChangeAtSurvivesSubSecondPrecisionThroughDoltCLI in + // internal/storage/schema. MarkLatestVersionRemovedInTx (asof_read.go) + // needs no matching change -- it already passes time.Now().UTC() straight + // through, so removed_at was never floored. issue, err := GetIssueInTx(ctx, tx, issueID) if err != nil { return fmt.Errorf("versioned history: snapshot %s: %w", issueID, err) } // A flagged row (Ephemeral || NoHistory) is never versioned, so an - // update-shaped call has nothing to do. A create-shaped one may still - // have to declare participation; see the IsWisp branch below. - if IsWisp(issue) && !isCreate { + // update-shaped call, or the test-support mint, has nothing to do. A + // create-shaped one may still have to declare participation; see the + // IsWisp branch below. + if IsWisp(issue) && kind != mintCreate { return nil } - // design §16.2b write fence. Read straight off the row rather than - // through types.Issue: participation_generation is dual-write - // bookkeeping, not part of the issue's own content model, so it stays - // out of the durable_state snapshot canonicalDurableState marshals - // below. isCreate skips the check entirely (a brand-new row has no - // legacy state to preserve, and this same function stamps the column - // itself further down); an update-shaped call against a NULL (legacy) - // value returns here, before either half of this seam's write runs. - // BD_IGNORE_SCHEMA_SKEW (internal/storage/schema/schema.go) has no - // bearing on this check: it downgrades CheckForwardDrift's - // migration-cursor refusal, a different plane than this column's real - // per-row data. - // // THE TABLE THAT HOLDS THE ROW decides its plane, not its flags. IsWisp // reads Ephemeral || NoHistory, and a wisps-table row can carry neither: // bd mol squash clears a wisp root's flag in place, and import's // "wisp_plane" marker routes an unflagged row to wisps. Such a row passes // IsWisp, but there is no issues row for it, so this read finds nothing. // No issues row means nothing to version (FR-8), for an update- or a - // create-shaped call alike: mint nothing and let the mutation proceed, - // rather than abort it on sql.ErrNoRows. GetIssueInTx resolved the - // snapshot above, so the row exists; ErrNoRows here can only mean it is - // in the wisps table. + // create-shaped call alike, and for the test-support mint too, since this + // is the plane rule and not the fence: mint nothing and let the mutation + // proceed, rather than abort it on sql.ErrNoRows. GetIssueInTx resolved + // the snapshot above, so the row exists; ErrNoRows here can only mean it + // is in the wisps table. + // + // The same read is design §16.2b's write fence. participation_generation + // is read straight off the row rather than through types.Issue: it is + // dual-write bookkeeping, not part of the issue's own content model, so it + // stays out of the durable_state snapshot canonicalDurableState marshals + // below. Only an update-shaped call (mintUpdate) acts on its value: a + // brand-new row (mintCreate) has no legacy state to preserve, and this + // same function stamps the column itself further down; the test-support + // mint (mintUnfenced) ignores it. An update-shaped call against a NULL + // (legacy) value returns here, before either half of this seam's write + // runs. The fence runs before the admission gate (canonicalDurableState) + // on purpose, so a legacy record is skipped whole: nothing about it is + // canonicalized, admitted or refused. + // BD_IGNORE_SCHEMA_SKEW (internal/storage/schema/schema.go) has no + // bearing on this check: it downgrades CheckForwardDrift's + // migration-cursor refusal, a different plane than this column's real + // per-row data. var participationGeneration sql.NullInt64 if err := tx.QueryRowContext(ctx, "SELECT participation_generation FROM issues WHERE id = ?", issueID, @@ -337,7 +691,7 @@ func recordVersionInTx(ctx context.Context, tx DBTX, issueID, actor string, isCr } return nil } - if !isCreate && !participationGeneration.Valid { + if kind == mintUpdate && !participationGeneration.Valid { return nil } @@ -389,12 +743,12 @@ func recordVersionInTx(ctx context.Context, tx DBTX, issueID, actor string, isCr `INSERT INTO issue_versions (issue_id, revision, epoch, durable_state, change_actor, change_agent, change_message, change_at, attribution_status) VALUES (?, ?, ?, ?, ?, NULL, NULL, ?, ?)`, - issueID, newRevision, epoch, durableState, actor, time.Now().UTC(), attributionStatusForActor(actor), + issueID, newRevision, epoch, durableState, actor, at, attributionStatusForActor(actor), ); err != nil { return fmt.Errorf("versioned history: insert version row for %s: %w", issueID, err) } - if isCreate { + if kind == mintCreate { // Stamps participation_generation in the same statement that advances // current_revision — design §16.2b's "positive declaration sourced // from store_epoch.epoch," minted here from the epoch already read @@ -410,8 +764,16 @@ func recordVersionInTx(ctx context.Context, tx DBTX, issueID, actor string, isCr } return nil } + // The same updated_at rule for every update-shaped mint. Advancing + // current_revision is bookkeeping, not an edit, but on its own it fires + // the ON UPDATE CURRENT_TIMESTAMP clause, so a label or dependency write + // that never touches updated_at would move it to the wall clock only + // while history is on, and an import's stale-upsert check would then + // reject the source's next edit. History on must leave updated_at + // exactly where history off does (the same reason #4298 and + // blocked_state.go's mark/unmark templates assign it to itself). if _, err := tx.ExecContext(ctx, - "UPDATE issues SET current_revision = ? WHERE id = ?", newRevision, issueID, + "UPDATE issues SET current_revision = ?, updated_at = updated_at WHERE id = ?", newRevision, issueID, ); err != nil { return fmt.Errorf("versioned history: advance current_revision for %s: %w", issueID, err) } diff --git a/internal/storage/issueops/version_history_durable_state_test.go b/internal/storage/issueops/version_history_durable_state_test.go index 957b55db1..c800ede2c 100644 --- a/internal/storage/issueops/version_history_durable_state_test.go +++ b/internal/storage/issueops/version_history_durable_state_test.go @@ -38,22 +38,63 @@ import ( // be the JCS form or the token would depend on which marshaler ran first. // Every case is also checked to be a fixed point of canonicalization, and a // duplicate-key input is refused rather than silently resolved. +// +// THE OFFICIAL VECTORS RUN BELOW THE ADMISSION GATE, through jcsCanonicalize +// rather than canonicalDurableState, and this is deliberate, not an +// inconsistency: testdata/jcs/input/values.json -- from the spec's own +// reference suite, not a case authored for this repo -- contains 1E30, an +// integer-valued literal past 2^53-1 that refuseUnrepresentableIntegers now +// refuses outright (see version_history.go's refuseUnrepresentableIntegers +// and jcsCanonicalize doc comments). RFC 8785 conformance is a claim about +// the canonicalizer, not about this store's admission policy, so weakening +// or dropping that vector to fit the gate would be the wrong trade; the gate +// refusing the same magnitude is pinned on its own terms in +// version_history_integer_admission_test.go. The two local cases below carry +// no such conflict and keep exercising the full gated seam, which is the +// more faithful check for them since production traffic never bypasses the +// gate. func TestCanonicalDurableStateIsJCS(t *testing.T) { t.Parallel() - cases := append(jcsTestVectors(t), - jcsCase{ + for _, tc := range jcsTestVectors(t) { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + marshaled, err := json.Marshal(tc.state) + if err != nil { + t.Fatalf("json.Marshal: %v", err) + } + got, err := jcsCanonicalize(marshaled) + if err != nil { + t.Fatalf("jcsCanonicalize: %v", err) + } + if !bytes.Equal(got, tc.want) { + t.Fatalf("jcsCanonicalize =\n %s\nwant\n %s", got, tc.want) + } + // canonical(canonical(x)) == canonical(x): the stored bytes are + // a fixed point, so re-canonicalizing what was read back can + // never change the token. + again, err := jcsCanonicalize(got) + if err != nil { + t.Fatalf("jcsCanonicalize over its own output: %v", err) + } + if !bytes.Equal(again, got) { + t.Fatalf("canonical form is not a fixed point:\n %s\n %s", got, again) + } + }) + } + + for _, tc := range []jcsCase{ + { name: "unicode and astral keys sort by UTF-16 code units", state: json.RawMessage(`{"b":1,"a":2,"é":3,"A":4,"😀":5}`), want: []byte(`{"A":4,"a":2,"b":1,"é":3,"😀":5}`), }, - jcsCase{ + { name: "html escapes are normalized away", state: map[string]any{"s": `&"'`}, want: []byte(`{"s":"&\"'"}`), }, - ) - for _, tc := range cases { + } { t.Run(tc.name, func(t *testing.T) { t.Parallel() got, err := canonicalDurableState(tc.state) @@ -118,9 +159,23 @@ func TestCanonicalDurableStateIsJCS(t *testing.T) { "id": "bd-1", "priority": json.Number("1.0"), "metadata": map[string]any{ - "weight": json.Number("1.50"), - "huge": json.Number("1E300"), - "ordinal": json.Number("9007199254740993"), // 2^53 + 1 + "weight": json.Number("1.50"), + // 1E-300, not 1E300: this test is about number FORM + // canonicalization (ES6 exponential notation), not the admission + // gate. 1E300 denotes an INTEGER value (10^300) past what + // binary64 holds exactly, so refuseUnrepresentableIntegers now + // refuses it -- see version_history_integer_admission_test.go. + // 1E-300 exercises the same exponential FORM without being an + // integer at all, so it stays in scope for this test. + "huge": json.Number("1E-300"), + // 2^53-1, the largest integer binary64 holds exactly. This was + // 2^53+1 until the admission gate began refusing literals that + // binary64 cannot hold: JCS would round such a literal to a + // DIFFERENT VALUE, collapsing two distinct states onto one + // durable_state. That case now lives in + // version_history_integer_admission_test.go, which asserts the + // refusal; this test stays about number FORM canonicalization. + "ordinal": json.Number("9007199254740991"), "tags": []any{json.Number("2.0"), "b", nil}, }, "raw": json.RawMessage(`{ "z" : 10.0e0 , "a" : true }`), @@ -132,9 +187,10 @@ func TestCanonicalDurableStateIsJCS(t *testing.T) { } // RFC 8785: keys sorted, no whitespace, numbers in ES6 Number::toString - // form (1.0 -> 1, 1.50 -> 1.5, 1E300 -> 1e+300, 2^53+1 -> the double it - // rounds to, 10.0e0 -> 10). - want := []byte(`{"id":"bd-1","metadata":{"huge":1e+300,"ordinal":9007199254740992,"tags":[2,"b",null],"weight":1.5},"priority":1,"raw":{"a":true,"z":10}}`) + // form (1.0 -> 1, 1.50 -> 1.5, 1E-300 -> 1e-300, 10.0e0 -> 10). An + // integer-valued literal past 2^53-1, any spelling, never reaches JCS now + // -- the admission gate refuses it first. + want := []byte(`{"id":"bd-1","metadata":{"huge":1e-300,"ordinal":9007199254740991,"tags":[2,"b",null],"weight":1.5},"priority":1,"raw":{"a":true,"z":10}}`) if !bytes.Equal(got, want) { t.Fatalf("canonicalDurableState =\n %s\nwant\n %s", got, want) } @@ -148,7 +204,7 @@ func TestCanonicalDurableStateIsJCS(t *testing.T) { if bytes.Equal(plain, got) { t.Fatalf("plain json.Marshal already equals the canonical form (%s); this test no longer proves canonicalization matters", plain) } - for _, nonCanonical := range []string{"1.0", "1.50", "1E300", "9007199254740993", "10.0e0"} { + for _, nonCanonical := range []string{"1.0", "1.50", "1E-300", "10.0e0"} { if !bytes.Contains(plain, []byte(nonCanonical)) { t.Errorf("plain marshal lost the non-canonical form %q, so the fixture no longer exercises it: %s", nonCanonical, plain) } diff --git a/internal/storage/issueops/version_history_integer_admission_test.go b/internal/storage/issueops/version_history_integer_admission_test.go new file mode 100644 index 000000000..9966afa88 --- /dev/null +++ b/internal/storage/issueops/version_history_integer_admission_test.go @@ -0,0 +1,186 @@ +package issueops + +import ( + "encoding/json" + "errors" + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +func issueWithMetadata(meta string) types.Issue { + return types.Issue{ID: "be-1", Title: "t", Metadata: json.RawMessage(meta)} +} + +// A metadata integer past 2^53 used to be ROUNDED by jcs.Transform, which +// collapsed two distinct issue states onto one durable_state and therefore +// one content token. It must be refused instead, so the loss is loud. +func TestCanonicalDurableStateRefusesIntegerPastBinary64(t *testing.T) { + _, err := canonicalDurableState(issueWithMetadata(`{"n":9007199254740993}`)) + if err == nil { + t.Fatal("canonicalDurableState(metadata 9007199254740993) = nil error, want a refusal") + } + if !errors.Is(err, ErrIntegerNotRepresentable) { + t.Fatalf("error = %v, want ErrIntegerNotRepresentable", err) + } +} + +// The defect this admission gate exists to prevent, stated directly: two +// DISTINCT states must never render to the same bytes. +func TestCanonicalDurableStateNeverCollapsesTwoDistinctStates(t *testing.T) { + // Before the admission gate, BOTH of these canonicalized, and both to + // the same bytes: 2^53+1 rounded down onto 2^53. One content token for + // two live states. Now the unrepresentable side is refused, so the pair + // can never be confused for one another. + a, errA := canonicalDurableState(issueWithMetadata(`{"n":9007199254740993}`)) + b, errB := canonicalDurableState(issueWithMetadata(`{"n":9007199254740991}`)) + if errA == nil && errB == nil && string(a) == string(b) { + t.Fatalf("two distinct states produced identical durable_state %s", a) + } + if errA == nil { + t.Fatalf("the unrepresentable state canonicalized instead of being refused: %s", a) + } + if errB != nil { + t.Fatalf("the representable state was refused: %v", errB) + } +} + +// The bound itself, both sides of it, bare-integer spelling. +func TestCanonicalDurableStateIntegerBoundary(t *testing.T) { + for _, tc := range []struct { + name string + meta string + refused bool + }{ + {"2^53-1 accepted", `{"n":9007199254740991}`, false}, + {"2^53 refused", `{"n":9007199254740992}`, true}, + {"2^53+1 refused", `{"n":9007199254740993}`, true}, + {"negative past bound refused", `{"n":-9007199254740993}`, true}, + {"beyond int64 refused", `{"n":18446744073709551616}`, true}, + } { + t.Run(tc.name, func(t *testing.T) { + _, err := canonicalDurableState(issueWithMetadata(tc.meta)) + if tc.refused && err == nil { + t.Fatalf("%s: nil error, want refusal", tc.meta) + } + if !tc.refused && err != nil { + t.Fatalf("%s: %v, want acceptance", tc.meta, err) + } + }) + } +} + +// Ordinary fractions must keep working: they are not exactly representable +// either, but ES6 shortest-round-trip returns them to the same binary64 +// value, so they neither round to a different value nor collide. +// +// 1e300 is deliberately NOT in this list: it is a mathematical integer +// (10^300) whose magnitude vastly exceeds 2^53-1, so it is refused like any +// other unrepresentable integer, regardless of spelling. See +// TestCanonicalDurableStateRefusesHugeExponentInteger below. +func TestCanonicalDurableStateStillAcceptsOrdinaryDecimals(t *testing.T) { + for _, meta := range []string{`{"n":0.1}`, `{"n":1.0}`, `{"n":-2.5}`, `{"n":0}`} { + if _, err := canonicalDurableState(issueWithMetadata(meta)); err != nil { + t.Fatalf("canonicalDurableState(%s) = %v, want acceptance", meta, err) + } + } +} + +// 1e300 denotes the integer 10^300 at a magnitude vastly past 2^53-1, and +// is subject to the same collision defect this gate +// exists to close -- binary64 spacing near 10^300 is roughly 2^248, so a wide +// range of distinct large integers near 10^300 collapse onto that one +// double, same failure mode as the 2^53 boundary, larger window. The +// spec for this predicate is "value not form, any spelling, no exceptions" +// (see version_history.go's refuseUnrepresentableIntegers doc comment), +// applied literally that refuses 1e300 -- but two pre-gate tests assumed it +// stayed accepted, one of which lived right here until this admission gate +// (see version_history_durable_state_test.go's "huge" fixture, moved off +// 1E300 for the same reason). +// +// RESOLVED: REFUSE 1e300. The rule applies +// literally, with no notation carve-out -- admitting a spelling this large +// while refusing 9007199254740993 would be indefensible, since the collision +// window near 10^300 (binary64 spacing ~2^248) is vastly larger than the one +// at the 2^53 boundary this gate was built to close. This is final, not +// provisional. +func TestCanonicalDurableStateRefusesHugeExponentInteger(t *testing.T) { + _, err := canonicalDurableState(issueWithMetadata(`{"n":1e300}`)) + if err == nil { + t.Fatal("canonicalDurableState(metadata 1e300) = nil error, want a refusal") + } + if !errors.Is(err, ErrIntegerNotRepresentable) { + t.Fatalf("error = %v, want ErrIntegerNotRepresentable", err) + } +} + +// THE GAP QA FOUND IN THE FIRST CUT OF THIS GATE. The first patch's +// refuseUnrepresentableIntegers skipped any literal containing '.', 'e', or +// 'E' outright, reasoning that non-integer literals round-trip losslessly +// through ES6 shortest-round-trip. That reasoning covers genuinely +// fractional literals (0.1, 1.5) but not integer-VALUED literals written in +// decimal or exponential FORM: 9007199254740993.0, 9.007199254740993e15, and +// 90071992547409930e-1 all denote the same integer VALUE as the bare literal +// 9007199254740993, and all three walked straight past the first gate and +// canonicalized to the identical colliding target 9007199254740992 -- the +// exact defect class this gate exists to close, just reachable through +// spelling instead of magnitude. +// +// The predicate under test must inspect the VALUE (the binary64 nearest +// to the literal) rather than the literal's spelling, so it must +// refuse and accept the same values regardless of which of the three JSON +// number forms (integer, decimal, exponent) carries them. This is pinned +// here for the bound itself; ordinary small decimals are covered separately +// by TestCanonicalDurableStateStillAcceptsOrdinaryDecimals. +func TestCanonicalDurableStateIntegerBoundaryAllSpellings(t *testing.T) { + for _, tc := range []struct { + name string + lit string + refused bool + }{ + // at the bound (2^53-1 = 9007199254740991): every spelling accepted. + {"at bound, bare", `9007199254740991`, false}, + {"at bound, decimal", `9007199254740991.0`, false}, + {"at bound, exponent", `9.007199254740991e15`, false}, + + // one past the bound (2^53+1 = 9007199254740993): every spelling + // refused. The decimal and exponent rows are the QA-found gap. + {"past bound, bare", `9007199254740993`, true}, + {"past bound, decimal (THE GAP)", `9007199254740993.0`, true}, + {"past bound, exponent (THE GAP)", `9.007199254740993e15`, true}, + {"past bound, exponent negative-exp form (THE GAP)", `90071992547409930e-1`, true}, + } { + t.Run(tc.name, func(t *testing.T) { + meta := `{"n":` + tc.lit + `}` + _, err := canonicalDurableState(issueWithMetadata(meta)) + if tc.refused && err == nil { + t.Fatalf("metadata %s: nil error, want refusal", meta) + } + if !tc.refused && err != nil { + t.Fatalf("metadata %s: %v, want acceptance", meta, err) + } + if tc.refused && !errors.Is(err, ErrIntegerNotRepresentable) { + t.Fatalf("metadata %s: error = %v, want ErrIntegerNotRepresentable", meta, err) + } + }) + } +} + +// Same property as TestCanonicalDurableStateNeverCollapsesTwoDistinctStates, +// spelled in decimal form -- the exact QA-found gap stated as a distinctness +// property rather than a table row: two states differing only in a +// decimal-spelled metadata integer past 2^53 must never render to the same +// bytes. +func TestCanonicalDurableStateNeverCollapsesDecimalSpellingOfDistinctStates(t *testing.T) { + a, errA := canonicalDurableState(issueWithMetadata(`{"n":9007199254740993.0}`)) + b, errB := canonicalDurableState(issueWithMetadata(`{"n":9007199254740991.0}`)) + if errA == nil && errB == nil && string(a) == string(b) { + t.Fatalf("two distinct states produced identical durable_state %s", a) + } + if errA == nil { + t.Fatalf("the unrepresentable state (9007199254740993.0) canonicalized instead of being refused: %s", a) + } + if errB != nil { + t.Fatalf("the representable state (9007199254740991.0) was refused: %v", errB) + } +} diff --git a/internal/storage/issueops/version_history_number_admission_test.go b/internal/storage/issueops/version_history_number_admission_test.go new file mode 100644 index 000000000..f68193aa9 --- /dev/null +++ b/internal/storage/issueops/version_history_number_admission_test.go @@ -0,0 +1,298 @@ +package issueops + +import ( + "errors" + "math" + "math/rand" + "runtime" + "strconv" + "strings" + "testing" +) + +// admitNumber runs the admission gate over one JSON number literal in the +// position the mint meets it: an object member of an issue's metadata. +func admitNumber(lit string) error { + return refuseUnrepresentableIntegers([]byte(`{"n":` + lit + `}`)) +} + +// The admission rule, stated once for the tests below: a JSON number literal +// is admitted into a durable_state if and only if the binary64 nearest to it +// has magnitude at most 2^53-1, whatever the literal's spelling (integer, +// decimal or exponent), and a literal that overflows binary64 is refused. +// Ordinary fractions such as 0.1 stay admitted, and so do magnitudes too small +// for binary64, which round to zero. + +// A fraction whose nearest double lies at or beyond 2^53 used to be admitted: +// the old rule looked only at literals that denote an integer VALUE, and +// 9007199254740993.5 is not one. jcs.Transform then rounds it to the integer +// 9007199254740994, so two distinct issue states shared one durable_state, +// which is exactly what the gate exists to prevent. Refusing must depend on +// where the nearest double falls, not on whether the literal looks integral. +func TestNumberAdmissionRefusesFractionsWhoseNearestDoubleIsPastTheBound(t *testing.T) { + for _, tc := range []struct { + lit string + refused bool + why string + }{ + {"9007199254740993.5", true, "nearest double is 2^53+2"}, + {"9007199254740994.5", true, "nearest double is 2^53+2"}, + {"9007199254740991.5", true, "a tie between 2^53-1 and 2^53 that rounds to the even 2^53"}, + {"9007199254740991.75", true, "nearest double is 2^53"}, + {"-9007199254740993.5", true, "magnitude decides, not sign"}, + {"9007199254740990.5", false, "a tie that rounds to the even 2^53-2, inside the range"}, + {"9007199254740991.25", false, "nearest double is 2^53-1, inside the range"}, + {"9007199254740991.0", false, "exactly 2^53-1"}, + } { + t.Run(tc.lit, func(t *testing.T) { + err := admitNumber(tc.lit) + if tc.refused { + if !errors.Is(err, ErrIntegerNotRepresentable) { + t.Fatalf("admitNumber(%s) = %v, want ErrIntegerNotRepresentable (%s)", tc.lit, err, tc.why) + } + if _, mintErr := canonicalDurableState(issueWithMetadata(`{"n":` + tc.lit + `}`)); !errors.Is(mintErr, ErrIntegerNotRepresentable) { + t.Fatalf("canonicalDurableState(metadata %s) = %v, want ErrIntegerNotRepresentable", tc.lit, mintErr) + } + return + } + if err != nil { + t.Fatalf("admitNumber(%s) = %v, want acceptance (%s)", tc.lit, err, tc.why) + } + }) + } +} + +// numberAdmissionCorpus is every literal the closure property is checked +// over: the boundary cases the rule was written around, every spelling of the +// values on either side of 2^53, and pseudo-random doubles in three renderings. +// The seed is fixed so a failure names a literal that reproduces. +func numberAdmissionCorpus() []string { + corpus := []string{ + "0", "-0", "0.0", "-0.0", "1", "1.0", "-2.5", "0.1", "3.14159265358979323846", "0.30000000000000004", + "5e-324", "4.9e-324", "2.2250738585072014e-308", "1e-7", "1e-400", "1e-100000000", + "9007199254740991", "9007199254740992", "9007199254740993", "9.007199254740991e15", "90071992547409930e-1", + "9007199254740993.5", "9007199254740994.5", "9007199254740991.5", "9007199254740990.5", + "1727000000000000000", "1727000000000000123", "1e21", "1e300", "1.7976931348623157e308", + "123456789012345678901234567890", "18446744073709551616", + } + // Every spelling of the integers and near-integers either side of the bound. + for delta := int64(-6); delta <= 6; delta++ { + base := strconv.FormatInt(int64(1)<<53+delta, 10) + for _, frac := range []string{"", ".0", ".25", ".5", ".75", ".999999"} { + corpus = append(corpus, base+frac, "-"+base+frac) + } + } + rng := rand.New(rand.NewSource(20260930)) + for _, scale := range []float64{1, 1e3, 1e9, 1e15, 1e-9} { + for range 60 { + f := (rng.Float64()*2 - 1) * scale + corpus = append(corpus, + strconv.FormatFloat(f, 'g', -1, 64), + strconv.FormatFloat(f, 'e', 17, 64), + strconv.FormatFloat(f, 'f', -1, 64)) + } + } + for range 200 { + f := math.Float64frombits(rng.Uint64()) + if math.IsNaN(f) || math.IsInf(f, 0) { + continue + } + corpus = append(corpus, strconv.FormatFloat(f, 'g', -1, 64), strconv.FormatFloat(f, 'e', -1, 64)) + } + return corpus +} + +// The gate must be closed under its own canonicalization: whatever it admits +// canonicalizes to a form it also admits. Otherwise a value can pass at the +// door and become unmintable one step later, which is what 9007199254740991.5 +// did (it rounds to 2^53, and the canonical bytes are then refused). Checked as +// a property over a corpus, not by example. +func TestNumberAdmissionIsClosedUnderCanonicalization(t *testing.T) { + admitted := 0 + for _, lit := range numberAdmissionCorpus() { + doc := []byte(`{"n":` + lit + `}`) + if err := refuseUnrepresentableIntegers(doc); err != nil { + continue // refused literals are outside the property + } + admitted++ + canonical, err := jcsCanonicalize(doc) + if err != nil { + t.Errorf("literal %s: jcsCanonicalize: %v", lit, err) + continue + } + if err := refuseUnrepresentableIntegers(canonical); err != nil { + t.Errorf("literal %s is admitted but its canonical form %s is refused: %v", lit, canonical, err) + } + } + if admitted < 100 { + t.Fatalf("only %d corpus literals were admitted; the property is close to vacuous", admitted) + } +} + +// The classification must not depend on how large an exponent the literal +// spells. The old rule parsed every literal as an exact rational, so a huge +// exponent either cost a million-digit integer or fell outside what the +// parser accepts and came back as an invalid literal, a different class of +// error from the refusal it is. Overflow is a refusal; a magnitude too small +// for binary64 is admitted. +func TestNumberAdmissionClassifiesByMagnitudeWhateverTheExponent(t *testing.T) { + for _, tc := range []struct { + lit string + refused bool + }{ + {"1e100000000", true}, + {"1e1000000", true}, + {"1e309", true}, + {"-1e400", true}, + {"1.7976931348623159e308", true}, + {"1e-100000000", false}, + {"1e-400", false}, + {"-1e-400", false}, + {"5e-324", false}, + } { + t.Run(tc.lit, func(t *testing.T) { + err := admitNumber(tc.lit) + switch { + case tc.refused && !errors.Is(err, ErrIntegerNotRepresentable): + t.Fatalf("admitNumber(%s) = %v, want ErrIntegerNotRepresentable", tc.lit, err) + case !tc.refused && err != nil: + t.Fatalf("admitNumber(%s) = %v, want acceptance", tc.lit, err) + } + }) + } +} + +// The gate allocates nothing proportional to an exponent. Measured as an +// allocation count and never as wall time, so a loaded machine cannot flake it, +// and it is what distinguishes a classifier that builds a million-digit integer +// from one that does not. +// +// The count is exact only without the race detector. Under it sync.Pool drops +// one Put in four at random (sync/pool.go), and the refusal builds its error with +// fmt.Errorf, whose printer comes from a pool, so a call now and then allocates a +// printer the pool would have supplied: a literal that costs 22 allocations reads +// 23, or 24, on a run where a printer was dropped, and comparing two +// independently averaged samples with a strict > is a coin flip. That noise only +// ever ADDS allocations, so the fewest allocations over many single-call samples +// is the count without it, and it is the same on every run. The comparison stays +// strict, with no margin to tune, and it loses no sensitivity: a classifier that +// builds per-exponent state does so on every call, so it raises the fewest too. +func TestNumberAdmissionAllocationsDoNotGrowWithTheExponent(t *testing.T) { + // The chance that every one of these samples is noisy is under one in 10^19. + const samples = 32 + fewest := func(lit string) float64 { + doc := []byte(`{"n":` + lit + `}`) + least := math.Inf(1) + for range samples { + least = math.Min(least, testing.AllocsPerRun(1, func() { _ = refuseUnrepresentableIntegers(doc) })) + } + return least + } + baseline := fewest("1e400") + for _, lit := range []string{"1e1000000", "1e100000000", "-1e1000000"} { + if got := fewest(lit); got > baseline { + t.Errorf("%s allocates %v times per call against %v for 1e400: allocation grows with the exponent", lit, got, baseline) + } + } +} + +// TestNumberAdmissionBytesDoNotGrowWithTheExponent is the count bound's twin, and finding 6 +// of the #6661 review. The count bound passes a classifier that builds the number it is +// asked about: one big-integer allocation is one allocation whether it holds 400 bits or a +// hundred million. What grows with the exponent is then the size of the allocation, so this +// bounds bytes: the fewest bytes one call allocates for a literal with a huge exponent is +// no more than for a small one, plus a margin and a few bytes for each byte of literal (a +// refusal names the literal in its message). +// +// The fewest over many single-call samples, as in the count bound and for the same reason: +// whatever else allocates while a sample runs only ever ADDS bytes, so the minimum is the +// cost of the call itself. TotalAlloc is read before and after each call, with GOMAXPROCS +// pinned to one for the measurement as testing.AllocsPerRun does, so a background +// goroutine does not add to a sample. +func TestNumberAdmissionBytesDoNotGrowWithTheExponent(t *testing.T) { + const ( + samples = 32 + smallLiteral = "1e400" + // What the bound allows beyond the small literal's bytes: a margin, and a few + // bytes for each byte a longer literal adds to the refusal's message. + slackBytes = 1024 + perLitByte = 16 + ) + defer runtime.GOMAXPROCS(runtime.GOMAXPROCS(1)) + fewest := func(lit string) uint64 { + doc := []byte(`{"n":` + lit + `}`) + least := uint64(math.MaxUint64) + var before, after runtime.MemStats + for range samples { + runtime.ReadMemStats(&before) + _ = refuseUnrepresentableIntegers(doc) + runtime.ReadMemStats(&after) + least = min(least, after.TotalAlloc-before.TotalAlloc) + } + return least + } + baseline := fewest(smallLiteral) + for _, lit := range []string{"1e1000000", "1e100000000", "-1e1000000"} { + limit := baseline + slackBytes + perLitByte*uint64(max(len(lit)-len(smallLiteral), 0)) + if got := fewest(lit); got > limit { + t.Errorf("%s allocates %d bytes per call against %d for %s (limit %d): bytes grow with the exponent", lit, got, baseline, smallLiteral, limit) + } + } +} + +// The sentinel names the range it enforces. Its old wording said a literal +// was not exactly representable, which no longer describes what is refused: +// 9007199254740993.5 is a fraction, and the rule is about magnitude. The +// identifier keeps its historical name so nothing that matches it with +// errors.Is has to change; the text is what a person reads. +func TestIntegerNotRepresentableTextNamesTheIJSONRange(t *testing.T) { + const want = "number is outside the I-JSON exact-integer range (magnitude above 2^53-1)" + got := ErrIntegerNotRepresentable.Error() + if got != want { + t.Fatalf("ErrIntegerNotRepresentable.Error() = %q, want %q", got, want) + } + if !strings.Contains(got, "I-JSON") { + t.Errorf("sentinel text %q does not name I-JSON", got) + } + if strings.Contains(got, "cannot represent") { + t.Errorf("sentinel text %q still uses the old wording", got) + } + err := admitNumber("9007199254740993") + if err == nil || err.Error() != want+": 9007199254740993" { + t.Fatalf("refusal text = %v, want the sentinel followed by the offending literal", err) + } +} + +// Nothing the store already accepted becomes admitted or refused differently. +// Dolt does not hand a literal back the way it was written: an integer up to +// int64 is stored exactly and every other number is stored as a double, so what +// the mint sees is the right-hand column. Each row pins the verdict on that +// column, which is what production meets. The verdicts are the ones the rule +// had before it changed, so this stays green through the change and fails only +// if the new rule moves one. +func TestNumberAdmissionKeepsTheVerdictOnEveryLiteralAsTheStoreReturnsIt(t *testing.T) { + for _, tc := range []struct { + written string + returned string // what a Dolt JSON column hands back for it + refused bool + }{ + {"9007199254740993", "9007199254740993", true}, + {"1727000000000000123", "1727000000000000123", true}, + {"9007199254740993.5", "9007199254740994", true}, + {"1.0", "1", false}, + {"0.1", "0.1", false}, + {"3.14159265358979323846", "3.141592653589793", false}, + {"1e300", "1" + strings.Repeat("0", 300), true}, + {"123456789012345678901234567890", "123456789012345680000000000000", true}, + } { + t.Run(tc.written, func(t *testing.T) { + err := admitNumber(tc.returned) + if tc.refused && !errors.Is(err, ErrIntegerNotRepresentable) { + t.Fatalf("admitNumber(%s), the store's rendering of %s = %v, want a refusal", tc.returned, tc.written, err) + } + if !tc.refused && err != nil { + t.Fatalf("admitNumber(%s), the store's rendering of %s = %v, want acceptance", tc.returned, tc.written, err) + } + }) + } +} diff --git a/internal/storage/issueops/version_history_preenable_scan_test.go b/internal/storage/issueops/version_history_preenable_scan_test.go new file mode 100644 index 000000000..74f57b903 --- /dev/null +++ b/internal/storage/issueops/version_history_preenable_scan_test.go @@ -0,0 +1,680 @@ +package issueops + +import ( + "context" + "database/sql/driver" + "encoding/json" + "errors" + "fmt" + "go/ast" + "go/parser" + "go/token" + "io/fs" + "reflect" + "regexp" + "sort" + "strings" + "testing" + "time" + + "github.com/DATA-DOG/go-sqlmock" + + "github.com/steveyegge/beads/internal/storage/sqlbuild" + "github.com/steveyegge/beads/internal/types" +) + +// With versioned history on, the mint runs in the same transaction as the +// mutation that reaches it, so refusing at the mint IS refusing the write: a +// number outside the I-JSON range aborts the writer's own transaction and +// nothing it did is committed. The mint reads the whole issue, so the number +// can be in any field that holds one: in metadata, or in a gate's timeout, +// which encoding/json writes as a count of nanoseconds. The mint's reads run +// first and its two writes, the version row and the current_revision advance, +// come last, so a refusal must arrive after the reads and before either write. +// +// The mock scripts exactly the reads. It expects no INSERT into issue_versions +// and no UPDATE of issues.current_revision, so a mint that wrote before it +// refused would make the mock fail the call with an unexpected-statement error +// instead of the refusal asserted here. The caller's own rollback closes the +// transaction, as every store does when the mint returns an error. +// +// The refusal names the top-level field the number was found in, as well as the +// literal, so a person who reads it knows which part of the issue to fix. +func TestRecordVersionInTxRefusesANumberOutsideTheRangeAndWritesNothing(t *testing.T) { + for _, tc := range []struct { + name string + column string + value driver.Value + literal string + field string + }{ + {"a number in metadata", "metadata", `{"ts":1727000000000000000}`, "1727000000000000000", "metadata"}, + {"a gate timeout past 2^53-1 nanoseconds", "timeout_ns", int64(2600 * time.Hour), "9360000000000000", "timeout"}, + } { + t.Run(tc.name, func(t *testing.T) { + const id = "bd-9" + _, mock, tx := beginMockTx(t) + defer ScopeVersionedHistoryTransaction(tx, true)() + + values := issueRowValues(id, "carries an out-of-range number") + for i, col := range issueColumns() { + if col == tc.column { + values[i] = tc.value + } + } + mock.ExpectQuery(regexp.QuoteMeta("SELECT " + IssueSelectColumns + " FROM issues " + sqlbuild.LeaseJoin("issues") + " WHERE id = ?")). + WithArgs(id). + WillReturnRows(issueRows().AddRow(values...)) + mock.ExpectQuery(regexp.QuoteMeta("SELECT label FROM labels WHERE issue_id = ? ORDER BY label")). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"label"})) + // design §16.2b's write fence reads participation_generation right after the + // snapshot. A non-NULL value is a participating row, so the mint goes on to the + // refusal this test pins; a NULL one would be a legacy row and mint nothing. + mock.ExpectQuery(regexp.QuoteMeta("SELECT participation_generation FROM issues WHERE id = ?")). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"participation_generation"}).AddRow(1)) + mock.ExpectQuery(regexp.QuoteMeta("SELECT 1 FROM wisps LIMIT 1")). + WillReturnRows(sqlmock.NewRows([]string{"1"})) + mock.ExpectQuery(`FROM dependencies WHERE issue_id IN`). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"issue_id", "depends_on_id", "type", "created_at", "created_by", "metadata", "thread_id"})) + mock.ExpectQuery(regexp.QuoteMeta("SELECT epoch FROM store_epoch WHERE id = 1")). + WillReturnRows(sqlmock.NewRows([]string{"epoch"}).AddRow(1)) + mock.ExpectQuery(regexp.QuoteMeta("SELECT COALESCE(MAX(revision), 0) + 1 FROM issue_versions WHERE issue_id = ?")). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"next"}).AddRow(1)) + mock.ExpectRollback() + + err := RecordVersionInTx(context.Background(), tx, id, "actor") + if !errors.Is(err, ErrIntegerNotRepresentable) { + t.Fatalf("RecordVersionInTx = %v, want ErrIntegerNotRepresentable", err) + } + if !strings.Contains(err.Error(), id) { + t.Errorf("refusal %q does not name the issue %s", err, id) + } + if !strings.Contains(err.Error(), tc.literal) { + t.Errorf("refusal %q does not name the offending literal %s", err, tc.literal) + } + if !strings.Contains(err.Error(), fmt.Sprintf("%q", tc.field)) { + t.Errorf("refusal %q does not name the field %q the number was found in", err, tc.field) + } + if rbErr := tx.Rollback(); rbErr != nil { + t.Fatalf("rolling back the writer's transaction: %v", rbErr) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("the mint read or wrote something other than the scripted reads: %v", err) + } + }) + } +} + +// A legacy row (participation_generation NULL) holding the same out-of-range number is outside +// the mint's refusal: design §16.2b's write fence returns right after the snapshot, so the +// admission gate never runs and RecordVersionInTx returns nil. +// +// The mock scripts the snapshot, the labels and the fence's read returning NULL, and nothing after +// it. A mint that read the dependencies, the epoch or a revision, or wrote, before it decided to +// skip would fail the call with an unexpected-statement error instead of returning nil, so this is +// the cheapest pin of the fence's order against the gate. +func TestRecordVersionInTxSkipsALegacyRowBeforeTheAdmissionGate(t *testing.T) { + const id = "bd-9" + _, mock, tx := beginMockTx(t) + defer ScopeVersionedHistoryTransaction(tx, true)() + + values := issueRowValues(id, "a legacy row that carries an out-of-range number") + for i, col := range issueColumns() { + if col == "metadata" { + values[i] = `{"ts":1727000000000000000}` + } + } + mock.ExpectQuery(regexp.QuoteMeta("SELECT " + IssueSelectColumns + " FROM issues " + sqlbuild.LeaseJoin("issues") + " WHERE id = ?")). + WithArgs(id). + WillReturnRows(issueRows().AddRow(values...)) + mock.ExpectQuery(regexp.QuoteMeta("SELECT label FROM labels WHERE issue_id = ? ORDER BY label")). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"label"})) + mock.ExpectQuery(regexp.QuoteMeta("SELECT participation_generation FROM issues WHERE id = ?")). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"participation_generation"}).AddRow(nil)) + mock.ExpectRollback() + + if err := RecordVersionInTx(context.Background(), tx, id, "actor"); err != nil { + t.Fatalf("RecordVersionInTx on a legacy row = %v, want nil: the fence skips it before the admission gate", err) + } + if rbErr := tx.Rollback(); rbErr != nil { + t.Fatalf("rolling back the writer's transaction: %v", rbErr) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("the mint read or wrote something after the fence skipped the row: %v", err) + } +} + +// TestRecordVersionAtInTxKeepsThePlaneRuleButNotTheFence pins what the +// test-support mint (RecordVersionAtInTx, mintUnfenced) does on the paths #6943 +// gave the shared body. +// +// - A row the wisps table holds, flags clear: the plane rule is not the fence, +// so the unfenced mint honours it like the other two kinds. The issues-table +// read finds nothing and the mint returns nil having written nothing, where +// before it would have inserted a version row no issues row points at. +// - An issues-table row flagged NoHistory: never versioned, and the unfenced +// mint never declares participation, so it returns right after the snapshot, +// without reading or stamping participation_generation. +// - A legacy row (participation_generation NULL), flags clear: the fence does +// not apply, so it mints, and the closing UPDATE advances current_revision +// alone, with no stamp. +// +// The mock scripts each sequence exactly, so an extra read or any write fails +// the call with an unexpected-statement error. +func TestRecordVersionAtInTxKeepsThePlaneRuleButNotTheFence(t *testing.T) { + const id = "bd-7" + at := time.Date(2026, 9, 1, 12, 0, 0, 0, time.UTC) + issuesSnapshot := regexp.QuoteMeta("SELECT " + IssueSelectColumns + " FROM issues " + sqlbuild.LeaseJoin("issues") + " WHERE id = ?") + participation := regexp.QuoteMeta("SELECT participation_generation FROM issues WHERE id = ?") + + t.Run("a row the wisps table holds mints nothing", func(t *testing.T) { + _, mock, tx := beginMockTx(t) + mock.ExpectQuery(issuesSnapshot).WithArgs(id).WillReturnRows(issueRows()) + mock.ExpectQuery(regexp.QuoteMeta("SELECT " + IssueSelectColumns + " FROM wisps " + sqlbuild.LeaseJoin("wisps") + " WHERE id = ?")). + WithArgs(id). + WillReturnRows(issueRows().AddRow(issueRowValues(id, "a squashed wisp root, flags clear")...)) + mock.ExpectQuery(regexp.QuoteMeta("SELECT label FROM wisp_labels WHERE issue_id = ? ORDER BY label")). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"label"})) + mock.ExpectQuery(participation).WithArgs(id).WillReturnRows(sqlmock.NewRows([]string{"participation_generation"})) + mock.ExpectRollback() + + if err := RecordVersionAtInTx(context.Background(), tx, id, "actor", at); err != nil { + t.Fatalf("RecordVersionAtInTx on a wisps-table row = %v, want nil", err) + } + if err := tx.Rollback(); err != nil { + t.Fatalf("rollback: %v", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("the unfenced mint did something after finding no issues row: %v", err) + } + }) + + t.Run("an issues row flagged no-history is neither minted nor stamped", func(t *testing.T) { + _, mock, tx := beginMockTx(t) + values := issueRowValues(id, "no-history, on the issues plane") + for i, col := range issueColumns() { + if col == "no_history" { + values[i] = int64(1) + } + } + mock.ExpectQuery(issuesSnapshot).WithArgs(id).WillReturnRows(issueRows().AddRow(values...)) + mock.ExpectQuery(regexp.QuoteMeta("SELECT label FROM labels WHERE issue_id = ? ORDER BY label")). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"label"})) + mock.ExpectRollback() + + if err := RecordVersionAtInTx(context.Background(), tx, id, "actor", at); err != nil { + t.Fatalf("RecordVersionAtInTx on a no-history row = %v, want nil", err) + } + if err := tx.Rollback(); err != nil { + t.Fatalf("rollback: %v", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("the unfenced mint read or wrote something for a no-history row: %v", err) + } + }) + + t.Run("a legacy row is minted and left unstamped", func(t *testing.T) { + _, mock, tx := beginMockTx(t) + mock.ExpectQuery(issuesSnapshot).WithArgs(id). + WillReturnRows(issueRows().AddRow(issueRowValues(id, "legacy, history was off when it was made")...)) + mock.ExpectQuery(regexp.QuoteMeta("SELECT label FROM labels WHERE issue_id = ? ORDER BY label")). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"label"})) + mock.ExpectQuery(participation).WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"participation_generation"}).AddRow(nil)) + mock.ExpectQuery(regexp.QuoteMeta("SELECT 1 FROM wisps LIMIT 1")). + WillReturnRows(sqlmock.NewRows([]string{"1"})) + mock.ExpectQuery(`FROM dependencies WHERE issue_id IN`). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"issue_id", "depends_on_id", "type", "created_at", "created_by", "metadata", "thread_id"})) + mock.ExpectQuery(regexp.QuoteMeta("SELECT epoch FROM store_epoch WHERE id = 1")). + WillReturnRows(sqlmock.NewRows([]string{"epoch"}).AddRow(1)) + mock.ExpectQuery(regexp.QuoteMeta("SELECT COALESCE(MAX(revision), 0) + 1 FROM issue_versions WHERE issue_id = ?")). + WithArgs(id). + WillReturnRows(sqlmock.NewRows([]string{"next"}).AddRow(1)) + mock.ExpectExec(`INSERT INTO issue_versions`). + WithArgs(id, int64(1), 1, sqlmock.AnyArg(), "actor", at, sqlmock.AnyArg()). + WillReturnResult(sqlmock.NewResult(0, 1)) + mock.ExpectExec(regexp.QuoteMeta("UPDATE issues SET current_revision = ?, updated_at = updated_at WHERE id = ?")). + WithArgs(int64(1), id). + WillReturnResult(sqlmock.NewResult(0, 1)) + mock.ExpectRollback() + + if err := RecordVersionAtInTx(context.Background(), tx, id, "actor", at); err != nil { + t.Fatalf("RecordVersionAtInTx on a legacy row = %v, want nil", err) + } + if err := tx.Rollback(); err != nil { + t.Fatalf("rollback: %v", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("the unfenced mint did not mint a legacy row exactly once, unstamped: %v", err) + } + }) +} + +// refusalClass names what an admission verdict is, so two functions can be +// compared for agreement without depending on their exact wording. +func refusalClass(err error) string { + switch { + case err == nil: + return "admitted" + case errors.Is(err, ErrIntegerNotRepresentable): + return "number outside the I-JSON range" + case strings.Contains(err.Error(), "Duplicate key"): + return "duplicate key" + default: + return "other refusal" + } +} + +func issueWithTimeout(timeout time.Duration) types.Issue { + return types.Issue{ID: "be-1", Title: "t", IssueType: types.TypeGate, Timeout: timeout} +} + +// The check the pre-enable scan runs and the check the mint runs must never +// disagree, or a store could pass the scan, be switched on, and then refuse the +// first write to the row the scan cleared. So the check is the mint's own two +// steps behind one exported name, and this pins that they agree on every shape +// of issue. +// +// The domain is the WHOLE issue, because that is what the mint canonicalizes and +// what the scan hands the check. Metadata covers a number past the range, a +// fraction rounding past it, duplicate keys at the top level and nested, +// ordinary values, null, empty and an empty object, and text that is not JSON at +// all. The other field that can reach the range is a gate's timeout, a +// time.Duration that encoding/json writes as nanoseconds: 2^53-1 ns is about +// 104.25 days, so 2500h is admitted and 2600h is not. +func TestCheckIssueVersionableAgreesWithTheMint(t *testing.T) { + const maxExact = 1<<53 - 1 // 9007199254740991 ns, the last whole nanosecond count in range + for _, tc := range []struct { + name string + issue types.Issue + want string + }{ + {"number past the range", issueWithMetadata(`{"ts":1727000000000000000}`), "number outside the I-JSON range"}, + {"fraction rounding past the range", issueWithMetadata(`{"n":9007199254740993.5}`), "number outside the I-JSON range"}, + {"overflowing exponent", issueWithMetadata(`{"n":1e400}`), "number outside the I-JSON range"}, + {"number nested in an array", issueWithMetadata(`{"a":[1,{"b":-9007199254740993}]}`), "number outside the I-JSON range"}, + {"duplicate key", issueWithMetadata(`{"k":1,"k":2}`), "duplicate key"}, + {"nested duplicate key", issueWithMetadata(`{"a":{"k":1,"k":2}}`), "duplicate key"}, + {"ordinary values", issueWithMetadata(`{"a":1,"b":[0.1,2,"x"],"c":{"d":null}}`), "admitted"}, + {"number at the bound", issueWithMetadata(`{"n":9007199254740991}`), "admitted"}, + {"tiny magnitude", issueWithMetadata(`{"n":1e-400}`), "admitted"}, + {"null", issueWithMetadata(`null`), "admitted"}, + {"empty object", issueWithMetadata(`{}`), "admitted"}, + {"empty", issueWithMetadata(``), "admitted"}, + {"text that is not JSON", issueWithMetadata(`{"a":`), "other refusal"}, + + {"no timeout", issueWithTimeout(0), "admitted"}, + {"timeout 2500h", issueWithTimeout(2500 * time.Hour), "admitted"}, + {"timeout 2600h", issueWithTimeout(2600 * time.Hour), "number outside the I-JSON range"}, + {"timeout 8760h", issueWithTimeout(8760 * time.Hour), "number outside the I-JSON range"}, + {"negative timeout -2600h", issueWithTimeout(-2600 * time.Hour), "number outside the I-JSON range"}, + {"timeout of exactly 2^53-1 ns", issueWithTimeout(time.Duration(maxExact)), "admitted"}, + {"timeout of exactly 2^53 ns", issueWithTimeout(time.Duration(maxExact + 1)), "number outside the I-JSON range"}, + {"timeout in range beside ordinary metadata", func() types.Issue { + issue := issueWithTimeout(time.Hour) + issue.Metadata = json.RawMessage(`{"ok":1}`) + return issue + }(), "admitted"}, + } { + t.Run(tc.name, func(t *testing.T) { + check := CheckIssueVersionable(&tc.issue) + _, mint := canonicalDurableState(tc.issue) + if got := refusalClass(check); got != tc.want { + t.Errorf("CheckIssueVersionable = %v (%s), want %s", check, got, tc.want) + } + if got := refusalClass(mint); got != tc.want { + t.Errorf("canonicalDurableState = %v (%s), want %s", mint, got, tc.want) + } + if refusalClass(check) != refusalClass(mint) { + t.Errorf("the pre-enable check and the mint disagree: check %v, mint %v", check, mint) + } + }) + } +} + +// The scan runs the check over every issue the mint would version and skips +// the ones it would not, by the mint's own rule (IsWisp): ephemeral rows and +// no-history rows are never versioned, so a poisoned value in one can never +// abort a write and is not an obstacle to switching history on. The scan +// reports what it found in the order it was given, names each offender with +// the refusal the mint would return and the field it was found in, and finds +// nothing in a clean store. +// +// A gate whose timeout is over 105 days is the case that motivated checking the +// whole issue: its timeout is past 2^53-1 ns, so, if it participates in history, +// the mint refuses every write to it, and a scan that looked only at metadata +// cleared the switch over it. +func TestFindUnversionableFindsWhatTheMintWouldRefuse(t *testing.T) { + oversize := json.RawMessage(`{"ts":1727000000000000000}`) + overlong := 105 * 24 * time.Hour + issues := []*types.Issue{ + {ID: "bd-1", Metadata: oversize}, + {ID: "bd-2", Metadata: json.RawMessage(`{"k":1,"k":2}`)}, + {ID: "bd-3", Metadata: json.RawMessage(`{"ok":1}`)}, + {ID: "bd-4", Ephemeral: true, Metadata: oversize}, + {ID: "bd-5", NoHistory: true, Metadata: oversize}, + {ID: "bd-6"}, + {ID: "bd-7", Metadata: json.RawMessage(`{"n":9007199254740993.5}`)}, + {ID: "bd-8", Ephemeral: true, WispPlaneOverride: boolPtr(false), Metadata: oversize}, + {ID: "bd-9", WispPlaneOverride: boolPtr(true), Metadata: oversize}, + {ID: "bd-10", IssueType: types.TypeGate, Timeout: overlong}, + {ID: "bd-11", IssueType: types.TypeGate, Timeout: 2500 * time.Hour}, + {ID: "bd-12", IssueType: types.TypeGate, Ephemeral: true, Timeout: overlong}, + {ID: "bd-13", IssueType: types.TypeGate, Timeout: -overlong}, + } + + got := FindUnversionable(issues) + + want := []struct { + id string + class string + field string + }{ + {"bd-1", "number outside the I-JSON range", "metadata"}, + {"bd-2", "duplicate key", "metadata"}, + {"bd-7", "number outside the I-JSON range", "metadata"}, + // The override decides which plane the mint routes an issue to, and the + // scan follows it: bd-8 is versioned despite Ephemeral, bd-9 is not. + {"bd-8", "number outside the I-JSON range", "metadata"}, + {"bd-10", "number outside the I-JSON range", "timeout"}, + {"bd-13", "number outside the I-JSON range", "timeout"}, + } + if len(got) != len(want) { + t.Fatalf("FindUnversionable found %d issues %v, want %d", len(got), unversionableIDs(got), len(want)) + } + for i, w := range want { + if got[i].ID != w.id { + t.Fatalf("offender %d is %s, want %s (order of the input); got %v", i, got[i].ID, w.id, unversionableIDs(got)) + } + if class := refusalClass(got[i].Err); class != w.class { + t.Errorf("%s: refusal class %q (%v), want %q", w.id, class, got[i].Err, w.class) + } + if got[i].Field != w.field { + t.Errorf("%s: field %q, want %q", w.id, got[i].Field, w.field) + } + if !strings.Contains(got[i].Err.Error(), fmt.Sprintf("%q", w.field)) { + t.Errorf("%s: refusal %q does not name the field %q", w.id, got[i].Err, w.field) + } + } + + clean := []*types.Issue{ + {ID: "bd-1", Metadata: json.RawMessage(`{"ok":1,"f":0.1}`)}, + {ID: "bd-2"}, + {ID: "bd-3", Metadata: json.RawMessage(`null`)}, + {ID: "bd-4", Ephemeral: true, Metadata: oversize}, + {ID: "bd-5", IssueType: types.TypeGate, Timeout: 2500 * time.Hour}, + {ID: "bd-6", IssueType: types.TypeGate, Ephemeral: true, Timeout: overlong}, + } + if found := FindUnversionable(clean); len(found) != 0 { + t.Fatalf("a store with nothing the mint would refuse reported %v", unversionableIDs(found)) + } + if found := FindUnversionable(nil); len(found) != 0 { + t.Fatalf("no issues reported %v", unversionableIDs(found)) + } +} + +func unversionableIDs(found []UnversionableIssue) []string { + ids := make([]string, 0, len(found)) + for _, f := range found { + ids = append(ids, f.ID) + } + return ids +} + +// The scan does not hand the check the issue the mint sees. It reads a lite +// projection (the heavy text columns are left out, labels are not loaded) and +// never loads dependencies, which the mint adds before it canonicalizes. That is +// sound only while everything the scan leaves out is free of numbers: a number in +// a field the scan never loads is one it can never refuse and the mint will. The +// next three tests are how that stays true. They fail, with the way out, the day +// a field is added that breaks it. + +// fieldByJSONName finds the exported field of the struct type typ that +// encoding/json writes under name. +func fieldByJSONName(typ reflect.Type, name string) (reflect.StructField, bool) { + for i := 0; i < typ.NumField(); i++ { + f := typ.Field(i) + if !f.IsExported() { + continue + } + tag := f.Tag.Get("json") + if tag == "-" { + continue + } + jsonName := strings.Split(tag, ",")[0] + if jsonName == "" { + jsonName = f.Name + } + if jsonName == name { + return f, true + } + } + return reflect.StructField{}, false +} + +// heavyColumnProblems maps each column to its field of the issue type and reports +// every one that is missing or that can hold a number. A string or a slice of +// strings cannot: encoding/json writes both without a JSON number in them. +func heavyColumnProblems(columns []string, issue reflect.Type) []string { + var problems []string + for _, col := range columns { + f, ok := fieldByJSONName(issue, col) + if !ok { + problems = append(problems, fmt.Sprintf("column %q has no %s field written under that name: map it, then say whether it can hold a number", col, issue)) + continue + } + numberFree := f.Type.Kind() == reflect.String || + (f.Type.Kind() == reflect.Slice && f.Type.Elem().Kind() == reflect.String) + if !numberFree { + problems = append(problems, fmt.Sprintf("column %q is %s.%s (%s), which can hold a number the scan never reads but the mint refuses: load it in the scan or classify it lite", col, issue, f.Name, f.Type)) + } + } + return problems +} + +// TestTheScanProjectionOmitsOnlyNumberFreeFields pins that every column the +// scan's lite projection leaves out (HeavyDropList) is a string or a list of +// strings on the issue. +func TestTheScanProjectionOmitsOnlyNumberFreeFields(t *testing.T) { + if problems := heavyColumnProblems(HeavyDropList, reflect.TypeOf(types.Issue{})); len(problems) > 0 { + t.Fatalf("the scan's projection leaves out a column that can hold a number:\n %s", strings.Join(problems, "\n ")) + } +} + +// The guard above would pass vacuously if it never found anything to refuse, so +// it is run once against columns that must be refused: numbers, metadata (a +// verbatim JSON document), a column that maps to nothing, and the columns that +// really are number-free. +func TestTheHeavyColumnGuardRefusesWhatCanHoldANumber(t *testing.T) { + issue := reflect.TypeOf(types.Issue{}) + for _, col := range []string{"priority", "estimated_minutes", "timeout", "metadata", "created_at_unix"} { + if problems := heavyColumnProblems([]string{col}, issue); len(problems) != 1 { + t.Errorf("column %q: got %d problems %v, want exactly one", col, len(problems), problems) + } + } + for _, col := range []string{"description", "design", "acceptance_criteria", "notes", "waiters", "payload"} { + if problems := heavyColumnProblems([]string{col}, issue); len(problems) != 0 { + t.Errorf("column %q is number-free but was refused: %v", col, problems) + } + } + problems := heavyColumnProblems([]string{"priority"}, issue) + if len(problems) == 1 && !strings.Contains(problems[0], "load it in the scan or classify it lite") { + t.Errorf("the refusal %q does not say how to resolve it", problems[0]) + } +} + +var ( + timeType = reflect.TypeOf(time.Time{}) + rawMessageType = reflect.TypeOf(json.RawMessage(nil)) +) + +// numberBearingPaths walks typ the way encoding/json does and returns the path of +// everything that can write a JSON number or a value this walk cannot see into: +// an integer, unsigned integer or float (a time.Duration is an int64), a +// json.RawMessage, an interface, a map. It recurses through slices, arrays, +// pointers and structs, skips what json:"-" leaves out and the unexported fields, +// and treats a time.Time as the string it is written as. +func numberBearingPaths(typ reflect.Type, path string, onPath map[reflect.Type]bool) []string { + switch typ { + case timeType: + return nil + case rawMessageType: + return []string{path + " is a json.RawMessage, passed through verbatim"} + } + switch typ.Kind() { + case reflect.Bool, reflect.String: + return nil + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, + reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr, + reflect.Float32, reflect.Float64: + return []string{fmt.Sprintf("%s is a %s", path, typ)} + case reflect.Interface, reflect.Map: + return []string{fmt.Sprintf("%s is a %s", path, typ.Kind())} + case reflect.Pointer, reflect.Slice, reflect.Array: + return numberBearingPaths(typ.Elem(), path, onPath) + case reflect.Struct: + if onPath[typ] { + return nil + } + onPath[typ] = true + defer delete(onPath, typ) + var found []string + for i := 0; i < typ.NumField(); i++ { + f := typ.Field(i) + if !f.IsExported() || f.Tag.Get("json") == "-" { + continue + } + found = append(found, numberBearingPaths(f.Type, path+"."+f.Name, onPath)...) + } + return found + default: + return []string{fmt.Sprintf("%s is a %s, which this walk cannot classify", path, typ.Kind())} + } +} + +// TestWhatTheMintAddsAndTheScanDoesNotLoadCarriesNoNumber covers the other half +// of what the scan leaves out. The mint adds the issue's dependencies before it +// canonicalizes, and the scan reads without labels, so a number anywhere in a +// Dependency, a Comment, a BondRef or a label would be refused by the mint and +// invisible to the scan. None carries one today; a field that could (a Weight +// int on Dependency) fails here, and the person who adds it decides whether the +// scan has to load it. +func TestWhatTheMintAddsAndTheScanDoesNotLoadCarriesNoNumber(t *testing.T) { + for _, tc := range []struct { + name string + typ reflect.Type + }{ + {"types.Dependency", reflect.TypeOf(types.Dependency{})}, + {"types.Comment", reflect.TypeOf(types.Comment{})}, + {"types.BondRef", reflect.TypeOf(types.BondRef{})}, + {"a label", reflect.TypeOf("")}, + {"the labels of an issue", reflect.TypeOf([]string(nil))}, + } { + t.Run(tc.name, func(t *testing.T) { + if found := numberBearingPaths(tc.typ, tc.name, map[reflect.Type]bool{}); len(found) > 0 { + t.Fatalf("a field the mint writes and the scan does not load can hold a number:\n %s", strings.Join(found, "\n ")) + } + }) + } +} + +// The walk is run against shapes it must flag and shapes it must not, for the +// same reason as the guard above. +func TestTheNumberFreeWalkFlagsEveryNumberBearingShape(t *testing.T) { + type weighted struct { + Weight int `json:"weight"` + } + type unsigned struct{ N uint32 } + type scored struct{ Score float64 } + type waiting struct{ Wait time.Duration } + type verbatim struct{ Blob json.RawMessage } + type anything struct{ Value any } + type keyed struct{ Extra map[string]string } + type nested struct{ Deps []*weighted } + type tagged struct { + Name string `json:"name"` + Tags []string `json:"tags"` + At time.Time `json:"at"` + Skipped int `json:"-"` + _ int + } + type recursive struct{ Next *recursive } + + for name, typ := range map[string]reflect.Type{ + "an int": reflect.TypeOf(weighted{}), + "an unsigned integer": reflect.TypeOf(unsigned{}), + "a float": reflect.TypeOf(scored{}), + "a time.Duration": reflect.TypeOf(waiting{}), + "a json.RawMessage": reflect.TypeOf(verbatim{}), + "an interface": reflect.TypeOf(anything{}), + "a map": reflect.TypeOf(keyed{}), + "an int behind slice+pointer": reflect.TypeOf(nested{}), + } { + if found := numberBearingPaths(typ, "x", map[reflect.Type]bool{}); len(found) == 0 { + t.Errorf("%s was not flagged", name) + } + } + for name, typ := range map[string]reflect.Type{ + "strings, a time, a skipped int and an unexported int": reflect.TypeOf(tagged{}), + "a type that contains itself": reflect.TypeOf(recursive{}), + } { + if found := numberBearingPaths(typ, "x", map[reflect.Type]bool{}); len(found) != 0 { + t.Errorf("%s was flagged: %v", name, found) + } + } +} + +// There is ONE check, CheckIssueVersionable, and the scan and the mint both go +// through it. A check over a part of the issue, the metadata alone as there once +// was, can pass what the mint refuses, and an exported one invites reuse by the +// next caller (a doctor check, say) that then inherits the gap. So no exported +// function in the package may be named for versionability beside the check and +// the scan that calls it. +func TestNoExportedCheckOverASubPartOfTheIssueExistsBesideCheckIssueVersionable(t *testing.T) { + fset := token.NewFileSet() + pkgs, err := parser.ParseDir(fset, ".", func(fi fs.FileInfo) bool { + return !strings.HasSuffix(fi.Name(), "_test.go") + }, parser.SkipObjectResolution) + if err != nil { + t.Fatalf("parse issueops package: %v", err) + } + allowed := map[string]bool{"CheckIssueVersionable": true, "FindUnversionable": true} + seen := map[string]bool{} + var stray []string + for _, pkg := range pkgs { + for _, file := range pkg.Files { + for _, decl := range file.Decls { + fn, ok := decl.(*ast.FuncDecl) + if !ok || fn.Recv != nil || !fn.Name.IsExported() { + continue + } + if !strings.Contains(strings.ToLower(fn.Name.Name), "versionable") { + continue + } + seen[fn.Name.Name] = true + if !allowed[fn.Name.Name] { + stray = append(stray, fmt.Sprintf("%s (%s)", fn.Name.Name, fset.Position(fn.Pos()))) + } + } + } + } + sort.Strings(stray) + if len(stray) > 0 { + t.Errorf("an exported check over part of the issue exists beside CheckIssueVersionable: %s", strings.Join(stray, ", ")) + } + for name := range allowed { + if !seen[name] { + t.Errorf("%s was not found: this guard no longer sees the check it protects", name) + } + } +} diff --git a/internal/storage/issueops/version_history_scan.go b/internal/storage/issueops/version_history_scan.go new file mode 100644 index 000000000..855869325 --- /dev/null +++ b/internal/storage/issueops/version_history_scan.go @@ -0,0 +1,53 @@ +package issueops + +import ( + "errors" + + "github.com/steveyegge/beads/internal/types" +) + +// UnversionableIssue names one issue that RecordVersionInTx would refuse to +// version, and the refusal it would return. Field is the top-level field of the +// issue the refusal was found in ("metadata", "timeout"), empty when the refusal +// does not name one. +type UnversionableIssue struct { + ID string + Field string + Err error +} + +// FindUnversionable runs CheckIssueVersionable over every issue in issues that +// the mint would version and returns the ones it refuses, in the order they were +// given. It is the core of the check made when versioned history is switched on: +// with history on, a write that introduces a value the mint refuses aborts in its +// own transaction, but a row that already holds one -- written while history was +// off, or by a path that does not mint -- would, if it participates in history, +// fail every later write to it. The check is over the whole issue, as the mint's +// is, and not over its metadata alone: a gate's timeout past 2^53-1 nanoseconds is +// refused too. +// +// It skips the rows the mint never versions by IsWisp: ephemeral and no-history +// rows are never versioned, so a value they hold can never abort a write. It +// cannot skip legacy rows, because participation is not on types.Issue, so on a +// store with the write fence (design §16.2b) it is a deliberate +// over-approximation: the mint skips a legacy row whatever it holds, and this +// still refuses over it, which is the safe direction because it can only refuse +// more. It does no I/O; the caller loads the rows, which is what lets one +// function serve the direct store and the proxied route. +func FindUnversionable(issues []*types.Issue) []UnversionableIssue { + var found []UnversionableIssue + for _, issue := range issues { + if issue == nil || IsWisp(issue) { + continue + } + if err := CheckIssueVersionable(issue); err != nil { + var named *refusedFieldError + field := "" + if errors.As(err, &named) { + field = named.field + } + found = append(found, UnversionableIssue{ID: issue.ID, Field: field, Err: err}) + } + } + return found +} diff --git a/internal/storage/issueops/version_list.go b/internal/storage/issueops/version_list.go new file mode 100644 index 000000000..fdfc9f2f9 --- /dev/null +++ b/internal/storage/issueops/version_list.go @@ -0,0 +1,77 @@ +package issueops + +import ( + "context" + "database/sql" + "fmt" + + "github.com/steveyegge/beads/internal/storage" +) + +// ListVersionsInTx returns every retained version of issueID, newest first. +// +// No subquery wrapper here, deliberately: HistoryInTx wraps its select +// because dolt_history_issues is a Dolt SYSTEM table and the planner's +// max1Row optimization mis-assumes WHERE id=? yields one row. issue_versions +// is an ordinary user table with a composite (issue_id, revision) primary +// key, so that optimization does not apply and the wrapper would only cost a +// materialization. +// +// An issue with no versions returns an empty slice and no error -- that is a +// truthful "nothing retained here", and it is the CALLER's job to distinguish +// it from "the feature is off" and from "no such issue", which it cannot do +// from this result alone. +// +// A removed version's RemovedRestriction is one of the as-of read's four values: +// whatever else the column holds, a NULL or empty one included, is reported as +// unknown, exactly as AsOfReadInTx reports it. A version that was not removed +// carries none. +func ListVersionsInTx(ctx context.Context, tx DBTX, issueID string) ([]storage.IssueVersion, error) { + rows, err := tx.QueryContext(ctx, ` + SELECT + issue_id, revision, epoch, + COALESCE(change_actor, '') AS change_actor, + COALESCE(change_agent, '') AS change_agent, + COALESCE(change_message, '') AS change_message, + change_at, + COALESCE(attribution_status, '') AS attribution_status, + removed_at, + COALESCE(removed_reason, '') AS removed_reason, + COALESCE(removed_restriction, '') AS removed_restriction, + COALESCE(LENGTH(durable_state), 0) AS state_bytes + FROM issue_versions + WHERE issue_id = ? + ORDER BY revision DESC + `, issueID) + if err != nil { + return nil, fmt.Errorf("failed to list issue versions: %w", err) + } + defer func() { _ = rows.Close() }() + + var out []storage.IssueVersion + for rows.Next() { + var v storage.IssueVersion + var removedAt sql.NullTime + if err := rows.Scan( + &v.IssueID, &v.Revision, &v.Epoch, + &v.ChangeActor, &v.ChangeAgent, &v.ChangeMessage, &v.ChangeAt, + &v.AttributionStatus, + &removedAt, &v.RemovedReason, &v.RemovedRestriction, + &v.StateBytes, + ); err != nil { + return nil, fmt.Errorf("failed to scan issue version: %w", err) + } + if removedAt.Valid { + t := removedAt.Time + v.RemovedAt = &t + v.RemovedRestriction = string(NormalizeRemovedRestriction(v.RemovedRestriction)) + } else { + v.RemovedRestriction = "" + } + out = append(out, v) + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("failed to read issue versions: %w", err) + } + return out, nil +} diff --git a/internal/storage/leg_contract_wiring_test.go b/internal/storage/leg_contract_wiring_test.go index 78ac5200b..00fdcdd46 100644 --- a/internal/storage/leg_contract_wiring_test.go +++ b/internal/storage/leg_contract_wiring_test.go @@ -33,6 +33,30 @@ var neverSatisfiedTags = map[string]bool{"ignore": true, "never": true} const importerOneAccessorWaiverReason = "Importer has one accessor (uow.ImporterSource); " + "the store legs run bd import through the raw seam and implement no Importer role" +// asOfReadNoHTTPWireWaiverReason covers the five R7.1 as-of read contracts +// (backend/conformance/versioned_read_contract.go). Unlike the +// participation-generation cases, which run on this leg because their subject +// is a create or an update the client can send +// (internal/httpclient/served_participation_generation_test.go), the subject +// here is the read itself, AsOfReadFixture.Resolve, and it has no form on the +// wire: +// - R7.1's read is issueops.AsOfReadInTx. No publicops role reaches it and no +// wire operation serves it, so there is nothing for the client to call. +// - The wire carries neither selector (an instant, or a previously served +// version address) nor the gone/unknown refusal the read answers with. +// - The client's only point-in-time read, storage.DoltStorage.AsOf, is a +// Dolt commit-ref read, not R7.1, and it is on the client's unsupported +// allowlist (D8: v0 publishes no history surface). +// +// Binding Resolve to the reference store behind the served env would run the +// cases green, but as a property of that store, never of the client; the store +// legs already pin it. The waiver stands until the wire publishes an as-of +// read. +const asOfReadNoHTTPWireWaiverReason = "the read under test has no form on the http wire: R7.1's as-of read " + + "(issueops.AsOfReadInTx) is reached by no publicops role and no wire operation, the wire carries neither an " + + "instant/version-address selector nor a gone/unknown refusal, and the client's only point-in-time read, " + + "DoltStorage.AsOf, is a Dolt commit-ref read on its unsupported allowlist (D8)" + // The waivers of the three legs registered here. Each is registered beside its // leg rather than written into one literal, so a leg registered elsewhere // brings its waivers the same way; see unwiredContractEntrypoints. @@ -57,6 +81,12 @@ func init() { "RunIssueOperationsCreateParentChildRecomputesWaitsForClosure": stagingWaiverReason, }) registerContractLegWaivers("http", map[string]string{ + "RunAsOfReadIsCompleteOnlyForTheAnsweringStoresHoldings": asOfReadNoHTTPWireWaiverReason, + "RunAsOfReadRefusesAsGoneOrUnknownNeverSubstitutingAnotherVersion": asOfReadNoHTTPWireWaiverReason, + "RunAsOfReadReportsTheAddressItServed": asOfReadNoHTTPWireWaiverReason, + "RunAsOfReadReturnsTheLatestVersionAtOrBeforeTStrictlyExcludingLater": asOfReadNoHTTPWireWaiverReason, + "RunAsOfReadSelectorAcceptsEitherAnInstantOrAVersionAddress": asOfReadNoHTTPWireWaiverReason, + "RunCycleDetectorIncludeTracksFindsTheMoleculeRootShape": cycleTracksWireGapWaiverReason, "RunCycleDetectorIncludeTracksIgnoresAPureTracksLoop": cycleTracksWireGapWaiverReason, "RunCycleDetectorIncludeTracksWalksEachEdgeInItsStoredDirection": cycleTracksWireGapWaiverReason, diff --git a/internal/storage/schema/BUILD.bazel b/internal/storage/schema/BUILD.bazel index 09be66e8f..f81aca9bc 100644 --- a/internal/storage/schema/BUILD.bazel +++ b/internal/storage/schema/BUILD.bazel @@ -120,6 +120,7 @@ go_library( "migrations/0069_widen_issue_versions_datetime_precision.up.sql", "migrations/ignored/0028_add_wisps_participation_generation.up.sql", "migrations/0070_add_participation_generation.up.sql", + "migrations/0071_add_removed_restriction.up.sql", ], importpath = "github.com/steveyegge/beads/internal/storage/schema", visibility = ["//:__subpackages__"], @@ -154,6 +155,7 @@ go_test( "migration_0068_add_attribution_status_test.go", "migration_0069_datetime_precision_test.go", "migration_0070_add_participation_generation_test.go", + "migration_0071_add_removed_restriction_test.go", "migration_body_drain_test.go", "migration_freeze_test.go", "migration_watchdog_test.go", @@ -169,6 +171,7 @@ go_test( deps = [ "//internal/debug", "//internal/storage/depid", + "//internal/storage/issueops", "//internal/storage/rowid", "//internal/testutil", "@com_github_data_dog_go_sqlmock//:go-sqlmock", diff --git a/internal/storage/schema/cli_migrations.go b/internal/storage/schema/cli_migrations.go index 8b595c17a..1d161cbc6 100644 --- a/internal/storage/schema/cli_migrations.go +++ b/internal/storage/schema/cli_migrations.go @@ -168,6 +168,15 @@ func cliCompatibleMigrationSQL(name, sqlText string) string { // substitute ALTERs wisps directly, the same as 0067's, so it // belongs on cliSubstituteAssumesWispTables too. return cliMigration0070AddParticipationGeneration + case "0071_add_removed_restriction.up.sql": + // Direct DDL for the same reason as 0067/0068: the source migration's + // PREPARE guard (an INFORMATION_SCHEMA probe) is what makes the raw + // .up.sql idempotent when replayed onto an already-migrated store, + // and the 2.2.x CLI no-ops a prepared ADD COLUMN. issue_versions is + // always present here -- 0067 runs earlier in the same fresh-bundle + // series -- and never carries removed_restriction yet, so the column + // always needs adding on a fresh bundle. + return cliMigration0071AddRemovedRestriction default: return sqlText } @@ -318,6 +327,12 @@ ALTER TABLE issue_versions MODIFY COLUMN removed_at DATETIME(6);` const cliMigration0070AddParticipationGeneration = `ALTER TABLE issues ADD COLUMN participation_generation BIGINT NULL; ALTER TABLE wisps ADD COLUMN participation_generation BIGINT NULL;` +// cliMigration0071AddRemovedRestriction is 0071 with its one guarded PREPARE +// block replaced by the direct ALTER it would run on a fresh database. +// issue_versions is created earlier in the same series by 0067 and never +// carries removed_restriction yet, so the column always needs adding here. +const cliMigration0071AddRemovedRestriction = `ALTER TABLE issue_versions ADD COLUMN removed_restriction VARCHAR(30);` + const cliMigration0041SplitDependenciesTarget = `DELETE FROM dolt_nonlocal_tables; CALL DOLT_COMMIT('-Am', 'disable nonlocal tables for fk migrations'); SET FOREIGN_KEY_CHECKS = 0; diff --git a/internal/storage/schema/migration_0069_datetime_precision_test.go b/internal/storage/schema/migration_0069_datetime_precision_test.go index ea6bcc8cd..503be1b6d 100644 --- a/internal/storage/schema/migration_0069_datetime_precision_test.go +++ b/internal/storage/schema/migration_0069_datetime_precision_test.go @@ -39,6 +39,12 @@ const migration0069Down = "0069_widen_issue_versions_datetime_precision.down.sql const migration0069ChangeAtDatetimeGuard = "@issue_versions_change_at_needs_widen" const migration0069RemovedAtDatetimeGuard = "@issue_versions_removed_at_needs_widen" +// TestLatestVersionIncludesMigration0069 (pinning LatestVersion() == 69) is +// superseded by TestLatestVersionIncludesMigration0070 +// (migration_0070_add_participation_generation_test.go) now that 0070 claims the +// next free slot; only one such pin lives at a time, matching how this +// test itself already superseded 0068's own version. + // TestMigration0069WidensChangeAtAndRemovedAtPrecision is the pure-Go, // DB-independent half of the pin, mirroring // TestMigration0068AddsAttributionStatus's shape: it checks the frozen diff --git a/internal/storage/schema/migration_0070_add_participation_generation_test.go b/internal/storage/schema/migration_0070_add_participation_generation_test.go index e6d5a7369..0993347f2 100644 --- a/internal/storage/schema/migration_0070_add_participation_generation_test.go +++ b/internal/storage/schema/migration_0070_add_participation_generation_test.go @@ -33,20 +33,11 @@ import ( const migration0070Up = "0070_add_participation_generation.up.sql" const migration0070Down = "0070_add_participation_generation.down.sql" -// TestLatestVersionIncludesMigration0070 pins the real next free slot this -// migration claims, superseding 0069's own version of this test -// (LatestVersion() moved from 69 to 70 the moment this migration file was -// added). Deliberately a hardcoded literal for the same reason 0067's, -// 0068's, and 0069's were: LatestVersion() drifting to 70 for the wrong -// reason (an unrelated migration landing first) should still be caught by -// this test failing to explain why 70 is participation_generation-shaped, -// which the CLI test below checks. -func TestLatestVersionIncludesMigration0070(t *testing.T) { - const want = 70 - if got := LatestVersion(); got != want { - t.Fatalf("LatestVersion() = %d, want %d (issues/wisps participation_generation migration slot claimed by be-h89oq)", got, want) - } -} +// TestLatestVersionIncludesMigration0070 (pinning LatestVersion() == 70) is +// superseded by TestLatestVersionIncludesMigration0071 +// (migration_0071_add_removed_restriction_test.go) now that 0071 claims the +// next free slot; only one such pin lives at a time, matching how this +// test itself already superseded 0069's own version. // TestMigration0070AddsParticipationGeneration is the pure-Go, DB-independent // half of the pin, mirroring TestMigration0069WidensChangeAtAndRemovedAtPrecision's diff --git a/internal/storage/schema/migration_0071_add_removed_restriction_test.go b/internal/storage/schema/migration_0071_add_removed_restriction_test.go new file mode 100644 index 000000000..2b4da39ac --- /dev/null +++ b/internal/storage/schema/migration_0071_add_removed_restriction_test.go @@ -0,0 +1,171 @@ +package schema + +import ( + "os" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/testutil" +) + +// R7.1 as-of read (gastownhall/beads#5898 revision 9, gastownhall/beads#6136) +// adds one nullable column: issue_versions gains +// removed_restriction VARCHAR(30). issue_versions already carries removed_at +// and removed_reason (migration 0067), virgin and unused until this slice; +// AsOfReadInTx (internal/storage/issueops/asof_read.go) reads all three as +// the durable "this version row was removed" marker. removed_restriction +// carries the categorical restriction removed_at's presence alone cannot: +// gone_retention / gone_erasure / gone_reorganization / unknown, never +// live -- live is the absence of a value, not a stored one. + +const migration0071Up = "0071_add_removed_restriction.up.sql" +const migration0071Down = "0071_add_removed_restriction.down.sql" + +// TestLatestVersionIncludesMigration0071 pins the real next free slot this +// phase claims, superseding 0070's own version of this test +// (migration_0070_add_participation_generation_test.go). Deliberately a +// hardcoded literal for the same reason 0067's, 0068's, 0069's and 0070's +// were: LatestVersion() drifting to 71 for the wrong reason should still be +// caught by this test failing to explain why 71 is removed-restriction-shaped, +// which the CLI test below checks. +func TestLatestVersionIncludesMigration0071(t *testing.T) { + const want = 71 + if got := LatestVersion(); got != want { + t.Fatalf("LatestVersion() = %d, want %d (the migration slot claimed for issue_versions.removed_restriction)", got, want) + } +} + +// TestMigration0071AddsRemovedRestriction is a pure-Go, DB-independent check +// of the frozen migration bytes themselves — it runs even where no `dolt` +// binary is available. +// +// It pins the same guarded-PREPARE shape 0067/0068 use and for the same +// reason: the guard is what makes a raw replay of this file onto an +// already-migrated store a no-op, and Dolt accepts no unprepared conditional +// ADD COLUMN. The cost is the same pre-2.3 CLI hazard those migrations carry, +// so this migration needs the same direct-DDL override in +// cliCompatibleMigrationSQL, policed by the same two guard tests +// (TestBundleMigrationsWithPreparedALTERAreOverriddenOrJustified and +// TestAllMigrationsSQLUsesDirectDDLForKnownCLIIncompatibilities). +func TestMigration0071AddsRemovedRestriction(t *testing.T) { + upSQL, err := MigrationSQL(migration0071Up) + if err != nil { + t.Fatalf("MigrationSQL(%s) error = %v, want the migration file to exist", migration0071Up, err) + } + for _, want := range []string{ + "ALTER TABLE issue_versions ADD COLUMN removed_restriction VARCHAR(30)", + "COLUMN_NAME = 'removed_restriction'", + "@issue_versions_rr_needs_add", + } { + if !strings.Contains(upSQL, want) { + t.Errorf("0071 up migration missing %q\nfull SQL:\n%s", want, upSQL) + } + } + if !strings.Contains(strings.ToUpper(upSQL), "PREPARE STMT FROM @SQL") { + t.Error("0071 up migration must keep its guarded PREPARE block — it is what makes a raw .up.sql replay onto an already-migrated store a no-op, and Dolt accepts no unprepared conditional ADD COLUMN. Unwrapping it also invalidates cliMigration0071AddRemovedRestriction.") + } + // The bundle override is what keeps the PREPARE above off the pre-2.3 + // CLI path. Assert it directly rather than trusting the two schema_test + // assertions to stay pointed at this migration. + const wantDirectDDL = "ALTER TABLE issue_versions ADD COLUMN removed_restriction VARCHAR(30);" + if !strings.Contains(cliCompatibleMigrationSQL(migration0071Up, upSQL), wantDirectDDL) { + t.Errorf("0071's CLI bundle substitute missing direct DDL %q", wantDirectDDL) + } + if cliSubstituteAssumesWispTables(migration0071Up) { + t.Error("0071's CLI substitute touches only issue_versions, which has no wisps-side counterpart table — it must not be listed in cliSubstituteAssumesWispTables") + } + + // down.sql files are not part of the embedded FS (only migrations/*.up.sql + // is //go:embed'd — see mainSource.files), so unlike the up side above, + // this reads straight from disk by package-relative path, matching + // TestMigration0067AddsVersionedBeadsSchema's / TestMigration0068AddsAttributionStatus's + // precedent. + downBytes, err := os.ReadFile("migrations/" + migration0071Down) + if err != nil { + t.Fatalf("read %s: %v, want the migration file to exist", migration0071Down, err) + } + downSQL := string(downBytes) + for _, want := range []string{ + "ALTER TABLE issue_versions DROP COLUMN removed_restriction", + "COLUMN_NAME = 'removed_restriction'", + } { + if !strings.Contains(downSQL, want) { + t.Errorf("0071 down migration missing %q\nfull SQL:\n%s", want, downSQL) + } + } + // Only migrations/*.up.sql is embedded into the CLI fresh bundle + // (mainSource.files), so the pre-2.3 prepared-DDL hazard never reaches a + // down migration and the guard is free — 0060's/0067's/0068's downs are + // the precedent. + if !strings.Contains(strings.ToUpper(downSQL), "PREPARE STMT FROM @SQL") { + t.Error("0071 down migration must guard its DROP COLUMN the way the up migration guards its ADD COLUMN, so a partially-applied or already-rolled-back workspace rolls back safely") + } +} + +// TestMigration0071HeaderSpellsTheVocabularyTheCodeStores pins the header's +// vocabulary to the constants that store, read and label the column (finding 6 +// of the #6661 review). The header is the one place a writer learns which +// strings to store, and a writer follows it literally: it said gone-retention, +// the code compares gone_retention, so a row written from the header was not +// recognised and rendered without a label. Deriving the expected spellings +// from the constants keeps the two from drifting apart again. +// +// Pure Go, like TestMigration0071AddsRemovedRestriction: it reads the frozen +// bytes and needs no `dolt` binary. +func TestMigration0071HeaderSpellsTheVocabularyTheCodeStores(t *testing.T) { + upSQL, err := MigrationSQL(migration0071Up) + if err != nil { + t.Fatalf("MigrationSQL(%s) error = %v, want the migration file to exist", migration0071Up, err) + } + for _, stored := range []issueops.AsOfRestriction{ + issueops.AsOfRestrictionGoneRetention, + issueops.AsOfRestrictionGoneErasure, + issueops.AsOfRestrictionGoneReorganization, + issueops.AsOfRestrictionUnknown, + } { + if !strings.Contains(upSQL, string(stored)) { + t.Errorf("0071 header never spells the stored value %q", stored) + } + if hyphenated := strings.ReplaceAll(string(stored), "_", "-"); hyphenated != string(stored) && strings.Contains(upSQL, hyphenated) { + t.Errorf("0071 header spells %q as %q; the code stores and compares the underscore spelling, so a writer following the header stores a value that renders unlabeled", stored, hyphenated) + } + } +} + +// TestMigration0071AddsRemovedRestrictionThroughDoltCLI applies the full +// migration bundle through a real `dolt` binary (skipped without one — see +// testutil.RequireDoltBinary) and checks the shape acceptance criteria a +// pure-Go SQL-text check cannot: actual column type/nullability as Dolt +// reports it, and that the column stays NULL for a row that never sets it +// (Live is the absence of a value) while round-tripping any of the four +// restriction strings a row does set it to. +func TestMigration0071AddsRemovedRestrictionThroughDoltCLI(t *testing.T) { + testutil.RequireDoltBinary(t) + + dir := t.TempDir() + runDoltCommand(t, dir, "init", "--name", "test", "--email", "test@example.com") + runDoltSQL(t, dir, AllMigrationsSQL()) + + requireDoltColumnShape(t, dir, "issue_versions", "removed_restriction", "varchar(30)", "YES") + requireDoltNoRows(t, dir, "SELECT issue_id FROM issue_versions", "issue_versions") + + runDoltSQL(t, dir, `INSERT INTO issue_versions (issue_id, revision, epoch, change_at, attribution_status) VALUES ('iv-1', 1, 1, '2026-09-01 00:00:00', 'claimed')`) + // issue_id rides along as a non-NULL anchor column: queryDoltCSV trims the + // whole `dolt sql -r csv` output before parsing, and a row whose ONLY + // selected column is NULL renders as a bare blank line, which + // strings.TrimSpace swallows when it's the trailing line -- silently + // turning "one row, NULL value" into "zero rows" and passing the wrong + // assertion for the wrong reason. Selecting issue_id alongside keeps the + // row's CSV line non-blank so it survives the trim. + rows := queryDoltCSV(t, dir, `SELECT issue_id, removed_restriction FROM issue_versions WHERE issue_id = 'iv-1'`) + if len(rows) != 1 || rows[0]["removed_restriction"] != "" { + t.Fatalf("removed_restriction for a row that never set it = %v, want NULL/empty (Live is the absence of a value)", rows) + } + + runDoltSQL(t, dir, `INSERT INTO issue_versions (issue_id, revision, epoch, change_at, attribution_status, removed_at, removed_reason, removed_restriction) VALUES ('iv-2', 1, 1, '2026-09-01 00:00:00', 'claimed', '2026-09-02 00:00:00', 'retention window elapsed', 'gone_retention')`) + rows = queryDoltCSV(t, dir, `SELECT issue_id, removed_restriction FROM issue_versions WHERE issue_id = 'iv-2'`) + if len(rows) != 1 || rows[0]["removed_restriction"] != "gone_retention" { + t.Fatalf("removed_restriction round-trip failed post-migration: %v", rows) + } +} diff --git a/internal/storage/schema/migrations/0071_add_removed_restriction.down.sql b/internal/storage/schema/migrations/0071_add_removed_restriction.down.sql new file mode 100644 index 000000000..5c3f8bde6 --- /dev/null +++ b/internal/storage/schema/migrations/0071_add_removed_restriction.down.sql @@ -0,0 +1,16 @@ +-- Reverse of 0071: drop issue_versions.removed_restriction. +-- +-- Guarded on INFORMATION_SCHEMA the same way 0067/0068's downs are, so a +-- partially-applied or already-rolled-back workspace rolls back safely. +-- Only migrations/*.up.sql is embedded into the CLI fresh bundle, so the +-- PREPARE hazard (cli_prepared_ddl.go) never reaches this file. +SET @issue_versions_rr_has = ( + SELECT COUNT(*) FROM INFORMATION_SCHEMA.COLUMNS + WHERE TABLE_SCHEMA = DATABASE() + AND TABLE_NAME = 'issue_versions' + AND COLUMN_NAME = 'removed_restriction' +); +SET @sql = IF(@issue_versions_rr_has > 0, + 'ALTER TABLE issue_versions DROP COLUMN removed_restriction', + 'SELECT 1'); +PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt; diff --git a/internal/storage/schema/migrations/0071_add_removed_restriction.up.sql b/internal/storage/schema/migrations/0071_add_removed_restriction.up.sql new file mode 100644 index 000000000..e47123ed2 --- /dev/null +++ b/internal/storage/schema/migrations/0071_add_removed_restriction.up.sql @@ -0,0 +1,49 @@ +-- Migration 0071: R7.1 as-of read (gastownhall/beads#5898 revision 9, +-- gastownhall/beads#6136); conformance suite: backend/conformance/ +-- versioned_read_contract.go. +-- +-- issue_versions already carries removed_at DATETIME and removed_reason +-- VARCHAR(255) (migration 0067), virgin and unused until now: R7.1's +-- AsOfReadInTx (internal/storage/issueops/asof_read.go) reads them as the +-- durable "this version row was removed" marker on the resolved row, and +-- refuses a served answer whenever removed_at is set (R8: never silently +-- substitute a neighboring/surviving/current version). +-- +-- removed_restriction VARCHAR(30) is the one column this migration adds: the +-- categorical restriction (gone_retention / gone_erasure / +-- gone_reorganization / unknown) that removed_at's presence alone cannot +-- carry. Nullable and NULL for every existing row -- Live is the absence of +-- a value, never a stored one -- and populated only alongside removed_at, +-- never independently. 30 chars comfortably fits the longest local +-- constant (internal/storage/issueops/asof_read.go's AsOfRestriction, +-- "gone_reorganization", 19 bytes) with headroom, matching change_agent's +-- own no-tight-fit precedent elsewhere in this table. +-- +-- Forward-compatible with R17 (not yet implemented on this branch): R17's +-- eventual real hold/remove/erase primitives will be the production WRITER +-- of these three columns; R7.1 only reads them here. No future collision -- +-- this migration only adds the one column R17 will also need, it does not +-- claim the writing role. +-- +-- Guarded the same way 0067/0068 guard their ADD COLUMNs (see 0067's header +-- for the full explanation): no MariaDB-only IF NOT EXISTS on Dolt 2.2.3's +-- ADD COLUMN, so an INFORMATION_SCHEMA probe + PREPARE is the only +-- replay-safe shape. Needs a CLI-bundle direct-DDL override +-- (cliMigration0071AddRemovedRestriction in cli_migrations.go), the same +-- dolthub/dolt#11345 escape hatch 0067/0068 use, guarded by +-- TestBundleMigrationsWithPreparedALTERAreOverriddenOrJustified. +-- +-- No wisps twin: issue_versions has no wisps-side counterpart table (design +-- section 16.3), so cliSubstituteAssumesWispTables does not apply here +-- either, matching 0068. +SET @issue_versions_rr_needs_add = ( + SELECT IF(COUNT(*) = 0, 1, 0) + FROM INFORMATION_SCHEMA.COLUMNS + WHERE TABLE_SCHEMA = DATABASE() + AND TABLE_NAME = 'issue_versions' + AND COLUMN_NAME = 'removed_restriction' +); +SET @sql = IF(@issue_versions_rr_needs_add = 1, + 'ALTER TABLE issue_versions ADD COLUMN removed_restriction VARCHAR(30)', + 'SELECT 1'); +PREPARE stmt FROM @sql; EXECUTE stmt; DEALLOCATE PREPARE stmt; diff --git a/internal/storage/schema/schema_test.go b/internal/storage/schema/schema_test.go index c1ffe2b30..b63c0b553 100644 --- a/internal/storage/schema/schema_test.go +++ b/internal/storage/schema/schema_test.go @@ -1974,6 +1974,9 @@ func TestAllMigrationsSQLUsesDirectDDLForKnownCLIIncompatibilities(t *testing.T) // 0070: two-plane prepared ADD COLUMN, same shape as 0067. "ALTER TABLE issues ADD COLUMN participation_generation BIGINT NULL;", "ALTER TABLE wisps ADD COLUMN participation_generation BIGINT NULL;", + // 0071: single-plane prepared ADD COLUMN, same shape as 0068's + // attribution_status (no wisps twin -- issue_versions has none). + "ALTER TABLE issue_versions ADD COLUMN removed_restriction VARCHAR(30);", } { if !strings.Contains(got, want) { t.Fatalf("AllMigrationsSQL missing direct CLI DDL %q", want) @@ -2010,6 +2013,9 @@ func TestAllMigrationsSQLUsesDirectDDLForKnownCLIIncompatibilities(t *testing.T) // 0070 guards both planes' ALTERs the same way, mirroring 0067's pair. "@issues_pg_needs_add", "@wisps_pg_needs_add", + // 0071 guards its ALTER the same way; only its source text carries + // this probe. + "@issue_versions_rr_needs_add", } { if strings.Contains(got, forbidden) { t.Fatalf("AllMigrationsSQL contains source prepared-DDL guard %q", forbidden) diff --git a/internal/storage/uow/BUILD.bazel b/internal/storage/uow/BUILD.bazel index ac9440eec..96c3d988c 100644 --- a/internal/storage/uow/BUILD.bazel +++ b/internal/storage/uow/BUILD.bazel @@ -175,6 +175,7 @@ go_test( "version_reconcile_boot_test.go", "version_reconcile_integration_test.go", "version_reconciler_contract_test.go", + "versioned_read_contract_test.go", "workspace_config_contract_test.go", ], data = [ diff --git a/internal/storage/uow/dualwrite_participation_generation_contract_test.go b/internal/storage/uow/dualwrite_participation_generation_contract_test.go index 4f2a585df..42cf07f8e 100644 --- a/internal/storage/uow/dualwrite_participation_generation_contract_test.go +++ b/internal/storage/uow/dualwrite_participation_generation_contract_test.go @@ -33,6 +33,9 @@ func TestParticipationGenerationFence(t *testing.T) { t.Run("ProceedsForPromotedRecord", func(t *testing.T) { conformance.RunParticipationGenerationProceedsForPromotedRecord(t, ctx, fixture) }) + t.Run("FlagOffSkipsUpdateOnParticipatingRecord", func(t *testing.T) { + conformance.RunParticipationGenerationFlagOffSkipsUpdateOnParticipatingRecord(t, ctx, fixture) + }) t.Run("FenceIgnoresSchemaSkewHatch", func(t *testing.T) { conformance.RunParticipationGenerationFenceIgnoresSchemaSkewHatch(t, ctx, fixture) }) diff --git a/internal/storage/uow/versioned_read_contract_test.go b/internal/storage/uow/versioned_read_contract_test.go new file mode 100644 index 000000000..947f2997f --- /dev/null +++ b/internal/storage/uow/versioned_read_contract_test.go @@ -0,0 +1,404 @@ +package uow + +import ( + "context" + "crypto/rand" + "encoding/hex" + "fmt" + "os/exec" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/require" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" + "github.com/steveyegge/beads/internal/storage/domain" + storageissueops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/testutil" + "github.com/steveyegge/beads/internal/types" +) + +// asOfReadHarness is this leg's Design D dispatch table (gastownhall/beads#5898 +// revision 9, gastownhall/beads#6136): the conformance suite +// invents storeID values to select among N genuinely isolated stores, and +// isolating them is entirely this harness's job -- AsOfReadInTx itself is +// storeID-oblivious (see the package doc on +// internal/storage/issueops/asof_read.go). Each storeID gets its own fresh +// UnitOfWorkProvider, built lazily on first use (provider) and torn down via +// t.Cleanup on the TestAsOfReadContract-level *testing.T. created tracks +// which (storeID, id) pairs already have a base issues row, since mintAt's +// second and later calls for the same pair must only add a historical +// issue_versions row, never a second base create. +// +// UNLIKE the dolt leg's buildAsOfReadStore (one shared testcontainer, N +// randomly-named databases) or the embeddeddolt leg's buildAsOfReadStore (N +// fully independent on-disk databases, isolation free), this leg spawns one +// real dolt sql-server SUBPROCESS per NewDoltServerUOWProvider call unless +// that call adopts an already-running one. Booting N subprocesses -- one per +// storeID, up to seven across a full TestAsOfReadContract run -- would be +// needlessly heavy, so this harness follows the same "one shared server, many +// databases" shape the dolt leg uses: TestAsOfReadContract does the ONE +// heavyweight, Fatal-permitted setup (dolt/bd binaries, HOME, port, +// storeRootDir, server config, log path) once in its own body, before any +// t.Run, and hands the resulting coordinates to newAsOfReadHarness. provider() +// then calls NewDoltServerUOWProvider -- a pure, Fatal-free function -- against +// those SAME coordinates with a fresh, randomly-named database per storeID. +// proxy.GetCreateDatabaseProxyServerEndpoint's adoption semantics (given the +// same rootDir, a later call dials the already-running server instead of +// spawning a duplicate, regardless of which database name it carries -- see +// TestNewDoltServerUOWProvider_ConcurrentInstantiation) mean only ONE server +// process is ever spawned, no matter how many storeIDs a case invents. +// +// storeID itself is used ONLY as a map key into h.providers here -- never as +// the database name. The conformance suite's storeID strings are built from +// free-form tags (asOfReadStore in backend/conformance/ +// versioned_read_contract.go: fixture.IssuePrefix + "-asof-store-" + tag, +// where tag can itself embed a Restriction's hyphenated String()), so they are +// not assumed to be safe SQL/Dolt identifiers. Generating an independent +// random per-store database name (asOfReadDatabaseName, matching the dolt +// leg's own dbSuffix approach) sidesteps that question entirely rather than +// sanitizing storeID. +// +// provider() cannot use newTestUOWProvider or newUOWRoleFixtureProvider. +// AsOfReadFixture's hook types (Resolve/MintAt/MakeUnanswerable) carry no +// *testing.T of their own, so a provider built lazily inside a hook -- itself +// invoked from inside a t.Run subtest closure, sometimes nested two deep (the +// restriction-loop case) -- has access only to whichever *testing.T this +// harness was constructed with: necessarily the OUTER TestAsOfReadContract's +// t, captured before any t.Run. Both of those helpers call testify's +// require.NoError / t.Fatalf internally, and calling FailNow/SkipNow on a +// *testing.T from a goroutine other than the one running that exact test -- +// which a t.Run subtest's closure always is, relative to the outer test -- +// panics with "subtest may have called FailNow on a parent test" (go's +// testing package requires FailNow/SkipNow to run only on the goroutine +// executing that specific test). provider() is a Fatalf/Skip-free path for +// exactly this reason: its failures come back as a plain error, which flows +// through the calling hook's own (..., error) return to +// conformance.RunAsOfReadXXX, which correctly Fatalfs using the SUBTEST's own +// *testing.T on the SUBTEST's own goroutine. Also unlike +// newUOWRoleFixtureProvider (one provider for a whole role suite, matching +// roleFixtureKit's frozen shape), this harness needs N independently-isolated +// providers -- roleFixtureKit is deliberately not reused here; see this +// file's own note by conformanceToAsOfRestriction on why issueops must stay +// conformance-oblivious, the same boundary that keeps this harness's own +// mapping from living centrally. +// +// See internal/storage/dolt/versioned_read_contract_test.go and +// internal/storage/embeddeddolt/versioned_read_contract_test.go for this +// leg's siblings, which this file mirrors as closely as the three backends' +// construction APIs allow. +type asOfReadHarness struct { + t *testing.T + storeRootDir string + doltBin string + cfgPath string + logPath string + providers map[string]UnitOfWorkProvider + created map[[2]string]bool +} + +func newAsOfReadHarness(t *testing.T, storeRootDir, doltBin, cfgPath, logPath string) *asOfReadHarness { + return &asOfReadHarness{ + t: t, + storeRootDir: storeRootDir, + doltBin: doltBin, + cfgPath: cfgPath, + logPath: logPath, + providers: map[string]UnitOfWorkProvider{}, + created: map[[2]string]bool{}, + } +} + +// provider returns storeID's UnitOfWorkProvider, building it on first use +// against a fresh, randomly-named database on the shared server. See the +// asOfReadHarness doc comment for why this function -- and everything it +// calls -- must never call *testing.T's Fatalf/Skip. +func (h *asOfReadHarness) provider(ctx context.Context, storeID string) (UnitOfWorkProvider, error) { + if p, ok := h.providers[storeID]; ok { + return p, nil + } + database, err := asOfReadDatabaseName() + if err != nil { + return nil, err + } + provider, err := NewDoltServerUOWProvider( + ctx, + h.storeRootDir, + database, + h.logPath, + h.cfgPath, + proxy.BackendLocalServer, + "root", + "", + h.doltBin, + 0, + 0, + false, + "", + ) + if err != nil { + return nil, fmt.Errorf("as-of read fixture: new provider for store %q (database %q): %w", storeID, database, err) + } + // t.Cleanup only appends to a mutex-guarded slice -- unlike Fatalf/Skip it + // never calls runtime.Goexit, so registering it on the outer h.t from + // inside a subtest's goroutine is safe. h.t's own cleanups run only after + // ALL of TestAsOfReadContract's t.Run calls have returned (none of them, + // nor their own nested t.Run calls, ever invoke t.Parallel), so every + // provider this harness builds stays open for the whole test. These + // per-provider Close cleanups are registered AFTER (so, per t.Cleanup's + // LIFO order, run BEFORE) TestAsOfReadContract's own shared + // verifiedShutdownCleanup (proxy.Shutdown) cleanup, so every provider is + // closed while the server they share is still up. + h.t.Cleanup(func() { _ = provider.Close(context.Background()) }) + + if err := RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) { + return "bd: set issue prefix", uw.ConfigUseCase().SetConfig(ctx, "issue_prefix", "asof") + }); err != nil { + return nil, fmt.Errorf("as-of read fixture: set issue_prefix for store %q: %w", storeID, err) + } + + h.providers[storeID] = provider + return provider, nil +} + +// asOfReadDatabaseName mints a fresh, unique database name, independent of +// any storeID's own content -- see the asOfReadHarness doc comment for why +// storeID itself is never used as one. +func asOfReadDatabaseName() (string, error) { + suffix := make([]byte, 6) + if _, err := rand.Read(suffix); err != nil { + return "", fmt.Errorf("as-of read fixture: generate db name: %w", err) + } + return "asof_" + hex.EncodeToString(suffix), nil +} + +// ensureCreated bare-creates id in storeID's store the first time it is +// asked for, giving RecordVersionAtInTx (called from mintAt) a base issues +// row to snapshot -- recordVersionAtInTx starts with GetIssueInTx, so it has +// nothing to read until this has run once. This goes through the real +// IssueUseCase, the same production entry point batch_creator.go and +// issue_operations.go use, unlike resolve/mintAt/makeUnanswerable below, +// which reach the shared issueops body directly (see the package doc on +// internal/storage/issueops/asof_read.go: "ALL THREE LEGS SHARE THIS BODY"). +func (h *asOfReadHarness) ensureCreated(ctx context.Context, storeID, id string) error { + key := [2]string{storeID, id} + if h.created[key] { + return nil + } + provider, err := h.provider(ctx, storeID) + if err != nil { + return err + } + issue := &types.Issue{ + ID: id, + Title: "as-of read fixture issue", + Status: types.StatusOpen, + Priority: 3, + IssueType: types.TypeTask, + } + err = RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) { + params := domain.CreateIssueParams{ + Issue: issue, + ExplicitID: issue.ID, + CreateOnly: true, + } + _, err := uw.IssueUseCase().CreateIssue(ctx, params, "asof-read-fixture") + return "as-of read fixture: bare-create " + issue.ID, err + }) + if err != nil { + return fmt.Errorf("as-of read fixture: bare-create %s: %w", id, err) + } + h.created[key] = true + return nil +} + +func (h *asOfReadHarness) resolve(ctx context.Context, storeID, id string, selector conformance.AsOfSelector) (conformance.AsOfReadResult, error) { + provider, err := h.provider(ctx, storeID) + if err != nil { + return conformance.AsOfReadResult{}, err + } + result, err := RunTxRead(ctx, provider, func(ctx context.Context, uw UnitOfWork) (storageissueops.AsOfReadResult, error) { + runner, err := importStatementRunner(uw) + if err != nil { + return storageissueops.AsOfReadResult{}, err + } + return storageissueops.AsOfReadInTx(ctx, runner, storeID, id, toIssueopsSelector(selector)) + }) + if err != nil { + return conformance.AsOfReadResult{}, err + } + return toConformanceResult(result), nil +} + +// mintAt is the two-phase implementation MintAt's contract needs: phase one +// (ensureCreated) runs only once per (storeID, id); phase two +// (RecordVersionAtInTx) runs every call and always supplies the row mintAt +// reports the address of. RecordVersionAtInTx deliberately does not gate on +// versionedHistoryEnabled (this harness never turns that on for its stores), +// so it always mints, and reading MAX(revision) back inside the same +// transaction (read-your-writes) is how we learn which revision it picked -- +// RecordVersionAtInTx itself reports no revision number. +// +// The commit message returned to RunTxResult is deliberately non-empty: an +// empty commitMsg makes RunTxResultWithin skip calling uw.Commit entirely +// (tx.go), so the write would never persist past this attempt's implicit +// rollback -- exactly the RunTxEphemeral hazard its own doc comment warns +// against for versioned tables. +func (h *asOfReadHarness) mintAt(ctx context.Context, storeID, id string, acceptedAt time.Time) (conformance.Address, error) { + if err := h.ensureCreated(ctx, storeID, id); err != nil { + return "", err + } + provider, err := h.provider(ctx, storeID) + if err != nil { + return "", err + } + revision, err := RunTxResult(ctx, provider, func(ctx context.Context, uw UnitOfWork) (int64, string, error) { + runner, err := importStatementRunner(uw) + if err != nil { + return 0, "", err + } + if err := storageissueops.RecordVersionAtInTx(ctx, runner, id, "asof-read-fixture", acceptedAt); err != nil { + return 0, "", err + } + var rev int64 + if err := runner.QueryRowContext(ctx, + `SELECT MAX(revision) FROM issue_versions WHERE issue_id = ?`, id, + ).Scan(&rev); err != nil { + return 0, "", err + } + return rev, "as-of read fixture: mint version for " + id, nil + }) + if err != nil { + return "", err + } + return conformance.Address(storageissueops.AsOfVersionAddress(id, revision)), nil +} + +func (h *asOfReadHarness) makeUnanswerable(ctx context.Context, storeID, id string, restriction conformance.Restriction) error { + asOfRestriction, ok := conformanceToAsOfRestriction[restriction] + if !ok { + return fmt.Errorf("as-of read fixture: unmapped restriction %v", restriction) + } + provider, err := h.provider(ctx, storeID) + if err != nil { + return err + } + reason := fmt.Sprintf("as-of read fixture: made unanswerable as %s", restriction) + return RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) { + runner, err := importStatementRunner(uw) + if err != nil { + return "", err + } + if err := storageissueops.MarkLatestVersionRemovedInTx(ctx, runner, id, asOfRestriction, reason); err != nil { + return "", err + } + return "as-of read fixture: mark " + id + " removed", nil + }) +} + +// conformanceToAsOfRestriction and its inverse are this leg's translation +// boundary between backend/conformance.Restriction (hyphenated String(), +// int-backed) and issueops.AsOfRestriction (underscored, string-backed -- +// its values ARE the removed_restriction column's contents). issueops must +// stay conformance-oblivious and conformance must stay backend-oblivious +// (internal/storage/issueops/asof_read.go's package doc), so this mapping is +// deliberately duplicated per leg rather than shared centrally. +var conformanceToAsOfRestriction = map[conformance.Restriction]storageissueops.AsOfRestriction{ + conformance.RestrictionLive: storageissueops.AsOfRestrictionLive, + conformance.RestrictionGoneRetention: storageissueops.AsOfRestrictionGoneRetention, + conformance.RestrictionGoneErasure: storageissueops.AsOfRestrictionGoneErasure, + conformance.RestrictionGoneReorganization: storageissueops.AsOfRestrictionGoneReorganization, + conformance.RestrictionUnknown: storageissueops.AsOfRestrictionUnknown, +} + +var asOfRestrictionToConformance = map[storageissueops.AsOfRestriction]conformance.Restriction{ + storageissueops.AsOfRestrictionLive: conformance.RestrictionLive, + storageissueops.AsOfRestrictionGoneRetention: conformance.RestrictionGoneRetention, + storageissueops.AsOfRestrictionGoneErasure: conformance.RestrictionGoneErasure, + storageissueops.AsOfRestrictionGoneReorganization: conformance.RestrictionGoneReorganization, + storageissueops.AsOfRestrictionUnknown: conformance.RestrictionUnknown, +} + +func toIssueopsSelector(selector conformance.AsOfSelector) storageissueops.AsOfSelector { + out := storageissueops.AsOfSelector{At: selector.At} + if selector.Version != nil { + out.Address = string(*selector.Version) + } + return out +} + +func toConformanceResult(result storageissueops.AsOfReadResult) conformance.AsOfReadResult { + if result.Refused { + return conformance.AsOfReadResult{ + Refusal: &conformance.AsOfReadRefusal{ + Restriction: asOfRestrictionToConformance[result.Restriction], + Reason: result.Reason, + }, + } + } + return conformance.AsOfReadResult{ + Address: conformance.Address(result.Address), + State: result.State, + } +} + +// TestAsOfReadContract wires this leg into the R7.1 as-of-read contract +// (gastownhall/beads#5898 revision 9, gastownhall/beads#6136) with a real +// Resolve/MintAt/MakeUnanswerable backed by one isolated UnitOfWorkProvider +// per storeID (asOfReadHarness). +func TestAsOfReadContract(t *testing.T) { + // Everything through logPath below is the same one-time, Fatal-permitted + // setup newTestUOWProvider does, run here directly instead of through + // that helper: newTestUOWProvider both sets up AND builds one provider + // bound to a fixed "beads" database, but this harness needs the setup + // coordinates kept separate so provider() can build N providers, one per + // storeID, lazily from inside subtest closures. See the asOfReadHarness + // doc comment for why provider() must stay Fatalf/Skip-free, which is + // exactly why this block -- the one place in this file where Fatalf/Skip + // is safe -- must run here, before any t.Run. + testutil.RequireDoltBinary(t) + bin, err := exec.LookPath("dolt") + require.NoError(t, err) + + bdBin := buildBDBinary(t) + prev := proxy.ResolveExecutable + proxy.ResolveExecutable = func() (string, error) { return bdBin, nil } + t.Cleanup(func() { proxy.ResolveExecutable = prev }) + + t.Setenv("HOME", t.TempDir()) + + port, err := proxy.PickFreePort() + require.NoError(t, err) + storeRootDir := t.TempDir() + shutdownOnInterrupt(t, storeRootDir) + verifiedShutdownCleanup(t, storeRootDir) + cfgPath := writeServerConfig(t, port) + logPath := filepath.Join(t.TempDir(), "server.log") + + ctx := context.Background() + h := newAsOfReadHarness(t, storeRootDir, bin, cfgPath, logPath) + fixture := conformance.AsOfReadFixture{ + IssuePrefix: "asof", + Resolve: h.resolve, + MintAt: h.mintAt, + MakeUnanswerable: h.makeUnanswerable, + } + + t.Run("SelectorAcceptsEitherAnInstantOrAVersionAddress", func(t *testing.T) { + conformance.RunAsOfReadSelectorAcceptsEitherAnInstantOrAVersionAddress(t, ctx, fixture) + }) + t.Run("ReturnsTheLatestVersionAtOrBeforeTStrictlyExcludingLater", func(t *testing.T) { + conformance.RunAsOfReadReturnsTheLatestVersionAtOrBeforeTStrictlyExcludingLater(t, ctx, fixture) + }) + t.Run("ReportsTheAddressItServed", func(t *testing.T) { + conformance.RunAsOfReadReportsTheAddressItServed(t, ctx, fixture) + }) + t.Run("RefusesAsGoneOrUnknownNeverSubstitutingAnotherVersion", func(t *testing.T) { + conformance.RunAsOfReadRefusesAsGoneOrUnknownNeverSubstitutingAnotherVersion(t, ctx, fixture) + }) + t.Run("IsCompleteOnlyForTheAnsweringStoresHoldings", func(t *testing.T) { + conformance.RunAsOfReadIsCompleteOnlyForTheAnsweringStoresHoldings(t, ctx, fixture) + }) +} diff --git a/internal/storage/version_lister.go b/internal/storage/version_lister.go new file mode 100644 index 000000000..df0946780 --- /dev/null +++ b/internal/storage/version_lister.go @@ -0,0 +1,76 @@ +package storage + +import ( + "context" + "time" +) + +// IssueVersion is one accepted, immutable durable state of one issue, as +// issue_versions records it. +// +// Revision is a PER-STORE ORDINAL, not a portable address: two disconnected +// clones can both hold revision 8 for the same issue (migration 0067's own +// header says so, and the flag is single-writer-only until the version_id +// primary key lands). Callers must not present it as a citable address. +// +// Its JSON key is "local_revision", NOT "revision", for two reasons that +// compound. types.Issue already ships `json:"revision"` (types.go:1206) and +// that is the row-lock CAS token, an unrelated thing -- and it is a STRING +// there while this is an int64, so a script that learned one shape breaks on +// the other. The name also carries the non-citability with the value, where a +// machine reader sees it, instead of only in a footer no script parses. It +// needs no rename when version_id lands beside it. (bee-ghosttrack, #5898 +// consumer read, 2026-09-20.) +// +// ChangeAgent is read from issue_versions.change_agent (migration 0067) but +// is deliberately NOT part of the JSON shape, which is what `bd versions +// --json` prints. Provenance V2 (gastownhall/beads#7510) replaces the ad hoc +// agent field with an actor chain, so publishing it now would make a public +// contract out of a field that is on its way out. The column and the field +// stay; only the JSON key is withheld. +// +// It lives here rather than in issueops for the same reason HistoryEntry +// does: issueops imports storage, so the shared row type has to sit on this +// side of that edge. +type IssueVersion struct { + IssueID string `json:"issue_id"` + Revision int64 `json:"local_revision"` + Epoch int64 `json:"epoch"` + ChangeActor string `json:"change_actor"` + ChangeAgent string `json:"-"` + ChangeMessage string `json:"change_message"` + ChangeAt time.Time `json:"change_at"` + AttributionStatus string `json:"attribution_status"` + RemovedAt *time.Time `json:"removed_at"` + RemovedReason string `json:"removed_reason"` + RemovedRestriction string `json:"removed_restriction"` + StateBytes int64 `json:"state_bytes"` +} + +// Removed reports whether this version is no longer served -- retention, +// erasure or an epoch reorganization removed it. A removed version keeps its +// row (so the gap is legible) but its durable state is gone, and callers must +// render it as a restriction rather than as an empty version. +func (v IssueVersion) Removed() bool { return v.RemovedAt != nil } + +// VersionLister is implemented by backends that can serve an issue's +// versioned history from the issue_versions table (migration 0067+). +// +// This is a narrow capability rather than a method on Storage on purpose: a +// backend that has no issue_versions table -- a proxied client, a no-db +// workspace, a non-Dolt backend -- should not be forced to implement it, and +// a caller that asserts for it learns the truth instead of receiving a stub +// that answers "no versions" for a store that simply cannot know. Same +// reasoning as ExternalRefHistoryQuerier in history_viewer.go. +// +// Holding this capability says the backend can READ versions. It says nothing +// about whether version recording is switched on -- that is +// VersionedHistoryConfigurer, and a store can legitimately implement both +// while the feature is off, in which case ListVersions truthfully returns +// nothing. +type VersionLister interface { + // ListVersions returns every retained version of issueID, newest first. + // An empty result means no retained versions, never "unsupported" -- + // unsupported is the failed type assertion. + ListVersions(ctx context.Context, issueID string) ([]IssueVersion, error) +} diff --git a/internal/versionedhistory/BUILD.bazel b/internal/versionedhistory/BUILD.bazel new file mode 100644 index 000000000..250b9fd91 --- /dev/null +++ b/internal/versionedhistory/BUILD.bazel @@ -0,0 +1,92 @@ +load("@rules_go//go:def.bzl", "go_library", "go_test") + +go_library( + name = "versionedhistory", + srcs = ["activation.go"], + importpath = "github.com/steveyegge/beads/internal/versionedhistory", + visibility = ["//:__subpackages__"], + deps = [ + "//internal/config", + "//internal/debug", + "//internal/storage", + "//internal/storage/uow", + "//issueops", + ], +) + +go_test( + name = "versionedhistory_test", + srcs = ["activation_test.go"], + embed = [":versionedhistory"], + deps = [ + "//internal/config", + "//internal/debug", + "//internal/storage", + "//internal/storage/domain", + "//internal/storage/externaldeps", + "//internal/storage/uow", + "//internal/telemetry", + "//issueops", + ], +) + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +# --- end repo_files --- diff --git a/internal/versionedhistory/activation.go b/internal/versionedhistory/activation.go new file mode 100644 index 000000000..d187b2758 --- /dev/null +++ b/internal/versionedhistory/activation.go @@ -0,0 +1,350 @@ +// Package versionedhistory decides whether dual-write issue-version history is on +// for one store or unit-of-work provider, and applies that decision where the +// thing is constructed. +// +// It is a package rather than helpers inside cmd/bd for the same reason as +// internal/eventsjournal: bd doctor's repair handlers live in their own package +// and cannot import package main, and a repair that mutates issues needs exactly +// the same rule as the command surface. There is ONE implementation of that rule, +// here. +// +// THE SWITCH IS A PROPERTY OF THE STORE. `bd config set versioned-history.enabled +// true` writes a row of the workspace's own config table, and a store answers for +// itself: a write routed into rig B reads rig B's row, never the launching +// workspace's. The environment (BD_VERSIONED_HISTORY_ENABLED) and a hand-edited +// config.yaml value are read through viper, are process-wide, and can only turn +// recording ON for every store the process opens; nothing but the row turns a +// store off. The two are OR'd rather than ranked, deliberately: the hazard is a +// write that fails to record, never one that records when it needn't, so either +// plane saying yes is enough and neither can silently switch the other off. +// +// Activation is applied by the FACTORIES that construct a store or provider +// (ActivateStore, ActivateProvider), never by the commands that use one. Applying +// it at the root pre-run reached only the store a command opens for its own +// workspace, and missed every store a routed update, a routed close, `bd create +// --repo` or the direct-mode open builds for another one -- which then recorded no +// history while the command reported success. See +// cmd/bd/versioned_history_construction_test.go for the structural guard. +package versionedhistory + +import ( + "context" + "fmt" + "os" + "strconv" + "strings" + + "github.com/steveyegge/beads/internal/config" + "github.com/steveyegge/beads/internal/debug" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/uow" + "github.com/steveyegge/beads/issueops" +) + +// ConfigKey is the one spelling of the switch, used for the store-config read +// below and for the `bd config set` line the help text and the off-refusal tell +// people to run. They must not drift. +const ConfigKey = "versioned-history.enabled" + +// Enabled reports whether version recording is on for this store, reading BOTH +// planes bd stores configuration in. +// +// This exists because the two disagreed. `bd config set versioned-history.enabled +// true` -- the command the help text and the off-refusal both name -- writes the +// workspace DATABASE, because the key is not a yaml-only key (config.IsYamlOnlyKey, +// and "versioned-history." is not among the prefixes at yaml_config.go). +// config.GetBool reads viper, which is yaml and env only. So the advertised command +// wrote one plane and the reader read the other: the user ran exactly what they +// were told, was told it worked, and the feature stayed off. Configured on, +// actually off -- the GH#536 class, and the precise failure this command was added +// to end. (bee-ghosttrack, #6661 review, finding 1.) +// +// The store plane is authoritative in the sense that matters: it makes the switch +// a property of the STORE rather than of whoever is calling, so two clients of one +// dolt sql-server agree about whether a write records a version. That closes the +// silent-holes case in the same review's finding 2 -- a client with recording off +// advances nothing, and the next enabled write mints a version that absorbs the +// unrecorded change under its own actor, producing contiguous revisions and a +// gapless-LOOKING history that is not one. +// +// The two are OR'd rather than ranked, deliberately. The hazard is a write that +// fails to record, never one that records when it needn't, so either plane saying +// yes is enough and neither can silently switch the other off. That keeps +// BD_VERSIONED_HISTORY_ENABLED=1 working for a one-off run without letting an unset +// (or =0) env var countermand a store that is recording. +// +// A store that cannot answer is not an error: proxied and no-db workspaces have no +// settings plane to read. Those fall back to viper alone rather than failing the +// command, because a store-config read is not worth breaking every bd invocation +// over. +// +// Nothing in production calls Enabled today. `bd versions` reports StoreSetting +// alone, because it only reads: the process-wide planes cannot make a store +// record, so they cannot speak for it. Activation uses EnabledForStore. Enabled +// stays, with its tests, as the ungated form of the combined rule. +func Enabled(ctx context.Context, st storage.DoltStorage) bool { + if config.GetBool(ConfigKey) { + // Already on via env or yaml; skip the store read entirely. This is also + // what keeps the common path cheap when someone is driving the feature from + // the environment. + return true + } + return StoreSetting(ctx, st) +} + +// StoreSetting reads the switch from the store's own settings plane, answering +// false for every reason a read can fail to produce a true. +// +// "EVERY reason" INCLUDES A PANIC, and that is the whole point of the defer below +// rather than an oversight to tidy away. The factories call this on every store +// they hand back, before anything has established that the store at the bottom of +// the chain can answer a config question at all -- so a store that cannot is not a +// hypothetical, it is a store that takes the process down. The shape that found +// it: a test double embedding a nil storage.DoltStorage, whose promoted +// WorkspaceConfig SIGSEGVs on call. The blast radius of letting that through is not +// one command; it is every bd invocation against any store whose settings plane is +// absent or half-built, for a read whose answer only ever decides whether to record +// versions. (bee-ghosttrack, #6661 second review.) +// +// A CAPABILITY ASSERTION ON WorkspaceConfig CANNOT REPLACE THIS, because +// WorkspaceConfig is declared on storage.DoltStorage itself +// (internal/storage/storage.go): every store satisfies an +// interface{ WorkspaceConfig() ... } assertion, including the one that panics when +// the method is actually called. That assertion would pass and the panic would +// happen anyway. +// +// AN ASSERTION ON storage.VersionedHistoryConfigurer IS DIFFERENT, and it is the +// primary guard now -- see EnabledForStore below. SetVersionedHistoryEnabled is NOT +// part of DoltStorage, so a store that embeds a nil DoltStorage does not promote it +// and does not satisfy that interface. (bee-ghosttrack, #6661 third review.) +// +// The recover stays anyway, because the guard does not cover every caller: `bd +// versions` calls StoreSetting directly, where the read is the point and no +// capability gate precedes it, and because a store that DOES implement the +// capability can still have a half-built settings plane. Guard first, recover as +// the backstop. +// +// A typed nil needs no separate guard for the same reason it needs no reflection: +// if its WorkspaceConfig panics, the recover answers false; if it returns (nil, nil) +// instead, the cfg == nil check answers false. Both arms are already here. +// +// The answer when the store cannot be read is false -- recording off, which is what +// the feature ships as. That is a real tradeoff and not a free one: a store that IS +// recording, whose settings read panics, is read as not recording, and a write +// during that window advances nothing. It is still the right side to fail to, +// because the alternative is that bd does not run at all, and a store too broken to +// answer a settings query is not one that was successfully recording versions a +// moment ago. +// +// WHAT IS AND IS NOT DIAGNOSED. An absent row is "" with a nil error by contract +// (WorkspaceConfig.GetSetting) and is the normal state of every workspace that never +// ran `bd config set`: it stays silently off. A read that ERRORS is different -- the +// row may well be there -- and resolving it to off with nothing to say so is the +// "set but silently off" shape this feature exists to end. Those go to debug.Logf, +// not to stderr: this runs on every invocation, and a diagnostic on a path the user +// did not ask about would be noise on all of them. +func StoreSetting(ctx context.Context, st storage.DoltStorage) (enabled bool) { + if st == nil { + return false + } + defer func() { + if r := recover(); r != nil { + // Deliberately swallowed, not re-raised and not logged: this runs on + // every invocation and its answer is a default, so the honest report is + // the false below rather than noise on a path the user did not ask + // about. The named return makes that explicit. + // + // This is the opposite choice from the repo's other non-test recover + // (internal/storage/dolt/transaction.go), which rolls back and then + // re-panics, and the difference is not stylistic: there a caller must + // not proceed over a half-applied transaction, so the panic has to keep + // traveling. Here the caller can proceed perfectly well, because what it + // wanted was a boolean with a documented default. + enabled = false + } + }() + cfg, err := st.WorkspaceConfig() + if err != nil { + debug.Logf("versioned history: %s left off: no settings plane readable on %T: %v\n", ConfigKey, st, err) + return false + } + if cfg == nil { + return false + } + res, err := cfg.GetSetting(ctx, issueops.GetSettingRequest{Key: ConfigKey}) + if err != nil { + debug.Logf("versioned history: %s left off: reading the setting from %T failed: %v\n", ConfigKey, st, err) + return false + } + return parseSettingBool(res.Value) +} + +// parseSettingBool reads the stored string the way `bd config set` writes it. +// Unset is "" and a nil error by contract (WorkspaceConfig.GetSetting), which lands +// here as false -- the correct default for a flag that ships off. +func parseSettingBool(v string) bool { + b, err := strconv.ParseBool(strings.TrimSpace(v)) + return err == nil && b +} + +// ValueEnables reports whether a stored value of the switch turns recording on, +// reading it the way every reader of the row does. `bd config set` uses it to +// decide whether the value it was just given deserves the replication and +// single-writer caveat, so that answer cannot drift from what the stores do with +// the same string. +func ValueEnables(v string) bool { return parseSettingBool(v) } + +// EnabledForStore answers the switch for a freshly constructed store, which is the +// caller that runs on EVERY store bd opens. +// +// It differs from Enabled in one deliberate way: it will not read the store's +// settings plane unless the store could actually act on the answer. Activation's +// whole use for the result is Apply, which needs a storage.VersionedHistoryConfigurer; +// a store that is not one cannot record versions however the question is answered, +// so the read is pure cost and, for a store whose settings plane is absent or +// half-built, pure risk. This is what took CI red at 32d4f966e: the read panicked at +// wiring time on a test double embedding a nil storage.DoltStorage, which is a panic +// in a path whose answer only ever decides whether to record versions. +// +// The ordering matters and is not interchangeable with Apply's own assertion. The +// viper plane is read FIRST and unconditionally, so a store that is +// set-but-unsupported still reaches Apply with enabled=true and still gets the +// warning it prints. Only the STORE-plane read is gated. The one case that loses its +// warning is a store whose own settings plane says true but which lacks the +// capability -- and `bd config set` could not have written that plane on such a +// store in the first place. +// +// (bee-ghosttrack, #6661 third review, blocking finding 1.) +func EnabledForStore(ctx context.Context, st storage.DoltStorage) bool { + if config.GetBool(ConfigKey) { + return true + } + if _, canRecord := st.(storage.VersionedHistoryConfigurer); !canRecord { + return false + } + return StoreSetting(ctx, st) +} + +// ProviderSetting reads the switch from a unit-of-work provider's own database, +// answering false for every reason a read can fail to produce a true. It is the +// provider-path twin of StoreSetting: a provider has no storage.DoltStorage to ask +// for a settings plane, so the row is read the way the rest of the unit-of-work +// layer reads one -- through a unit of work's ConfigUseCase, as +// internal/storage/uow/issue_operations.go does for claim.pools. +// +// The read opens one unit of work, asks for one config row, and rolls it back: a +// pooled connection, START TRANSACTION, a SELECT, a ROLLBACK. It starts no server +// and creates no schema (the provider constructors have already opened the pool and +// run initSchema by the time this is called) and commits nothing. That is the whole +// cost of "each provider answers for its own database", and it is why this is read +// at construction, once, rather than falling back to viper alone: a provider that +// silently ignored the row would record nothing for a workspace that asked, while +// every command reported success. +// +// The same failure policy as StoreSetting applies, including the recover. +func ProviderSetting(ctx context.Context, p uow.UnitOfWorkProvider) (enabled bool) { + if p == nil { + return false + } + defer func() { + if r := recover(); r != nil { + enabled = false + } + }() + unit, err := p.NewUOW(ctx) + if err != nil { + debug.Logf("versioned history: %s left off: opening a unit of work on %T to read it failed: %v\n", ConfigKey, p, err) + return false + } + // Rolled back, never committed: this is a read. + defer unit.Close(ctx) + raw, err := unit.ConfigUseCase().GetConfig(ctx, ConfigKey) + if err != nil { + debug.Logf("versioned history: %s left off: reading the setting through %T failed: %v\n", ConfigKey, p, err) + return false + } + return parseSettingBool(raw) +} + +// EnabledForProvider is EnabledForStore for a provider: viper first, then the +// capability check, then the provider's own row. +func EnabledForProvider(ctx context.Context, p uow.UnitOfWorkProvider) bool { + if config.GetBool(ConfigKey) { + return true + } + if _, canRecord := p.(storage.VersionedHistoryConfigurer); !canRecord { + return false + } + return ProviderSetting(ctx, p) +} + +// ActivateStore applies a freshly opened store's own configured switch to it. It is +// written to be used as a deferred rewrite of a factory's named results: +// +// func newX(ctx context.Context, ...) (s storage.DoltStorage, err error) { +// defer func() { s, err = versionedhistory.ActivateStore(ctx, s, err) }() +// ... open and return ... +// } +// +// The shape is deliberate, and is internal/eventsjournal's. It keeps the open and +// its activation in ONE function body, so "constructs a store" and "activates +// history on it" are the same syntactic unit -- which is what lets the construction +// guard check the property structurally instead of chasing a call graph. +// +// The store is activated RAW: every factory hands back the concrete store, before +// any decorator wraps it. That ordering is load-bearing, because activation is a +// type assertion and none of the decorators (telemetry, externaldeps, hooks) +// forwards SetVersionedHistoryEnabled -- asserting against a wrapper would fail +// silently and leave history off while the config said it was on. It is pinned by +// TestApplyReachesOnlyTheRawStore. +// +// A failed open passes through untouched, so the typed-nil store never reaches the +// activation. A store that cannot version at all is returned as-is without its +// settings plane being read: there is nothing to activate, and the factories run for +// every command that opens a store. +func ActivateStore(ctx context.Context, s storage.DoltStorage, err error) (storage.DoltStorage, error) { + if err != nil || s == nil { + return s, err + } + Apply(s, EnabledForStore(ctx, s)) + return s, nil +} + +// ActivateProvider is ActivateStore for a unit-of-work provider -- bd's second write +// plumbing, with its own transactions and its own activation switch, which +// proxied-server mode and `bd serve` against a server-mode workspace write through +// instead of a DoltStorage. +func ActivateProvider(ctx context.Context, p uow.UnitOfWorkProvider, err error) (uow.UnitOfWorkProvider, error) { + if err != nil || p == nil { + return p, err + } + Apply(p, EnabledForProvider(ctx, p)) + return p, nil +} + +// Apply binds a resolved answer to one plumbing -- a store or a unit-of-work +// provider. Activation is per instance rather than process-global: a process can +// hold several plumbings at once (a routed create holds two), and enabling history +// on one must not enable it on the rest. +// +// It only ever turns recording ON. A false answer touches nothing, so flag-off is +// byte-identical to a build without the feature, which is the contract #6135 ships +// under. +// +// A plumbing that does not implement the capability is not an error: proxied and +// no-db backends legitimately do not. But it is not silently ignored either -- +// asking for history and not getting it is exactly the case a user must be told +// about, so it warns. +func Apply(target any, enabled bool) { + if !enabled { + return + } + configurer, ok := target.(storage.VersionedHistoryConfigurer) + if !ok { + fmt.Fprintf(os.Stderr, + "warning: versioned-history.enabled is set, but this storage backend (%T) does not support version history; it stays off\n", + target) + return + } + configurer.SetVersionedHistoryEnabled(true) +} diff --git a/internal/versionedhistory/activation_test.go b/internal/versionedhistory/activation_test.go new file mode 100644 index 000000000..a043f6b16 --- /dev/null +++ b/internal/versionedhistory/activation_test.go @@ -0,0 +1,776 @@ +package versionedhistory + +import ( + "context" + "errors" + "fmt" + "io" + "os" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/config" + "github.com/steveyegge/beads/internal/debug" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/storage/externaldeps" + "github.com/steveyegge/beads/internal/storage/uow" + "github.com/steveyegge/beads/internal/telemetry" + "github.com/steveyegge/beads/issueops" +) + +// The resolver's tests. Most of them moved here from cmd/bd with the code they +// test, assertions unchanged: that is what shows the move preserved #6661's +// behavior. The rest are the new claims -- each store (and each provider) answers +// for itself, the environment is process-wide and can only turn recording on, and +// a read that errors is not the same as a row that is absent. + +const envVar = "BD_VERSIONED_HISTORY_ENABLED" + +// viperEnv sets the switch's environment variable and re-initialises viper, since +// config.GetBool answers false until viper is initialised and so would make every +// env-driven case below pass whatever the code did with the env. +func viperEnv(t *testing.T, value string) { + t.Helper() + t.Setenv(envVar, value) + // Nothing from the machine's own config may decide these cases. + t.Setenv("HOME", t.TempDir()) + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + t.Chdir(t.TempDir()) + if err := config.Initialize(); err != nil { + t.Fatalf("config.Initialize(): %v", err) + } + t.Cleanup(config.ResetForTesting) +} + +// captureStderr runs fn with os.Stderr redirected and returns what it wrote. With +// verbose set, debug.Logf writes too. +func captureStderr(t *testing.T, verbose bool, fn func()) string { + t.Helper() + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + orig := os.Stderr + os.Stderr = w + debug.SetVerbose(verbose) + func() { + defer func() { + debug.SetVerbose(false) + os.Stderr = orig + _ = w.Close() + }() + fn() + }() + out, err := io.ReadAll(r) + if err != nil { + t.Fatal(err) + } + return string(out) +} + +// --- fakes ----------------------------------------------------------------- + +// settingsPlane is a workspace settings plane with one canned answer. +type settingsPlane struct { + issueops.WorkspaceConfig + value string + err error + panicMsg string + reads int +} + +func (p *settingsPlane) GetSetting(_ context.Context, req issueops.GetSettingRequest) (issueops.SettingResult, error) { + p.reads++ + if p.panicMsg != "" { + panic(p.panicMsg) + } + return issueops.SettingResult{Key: req.Key, Value: p.value}, p.err +} + +// configProbeStore is a raw store at the bottom of a decorator chain whose +// settings plane behaves however a case needs: panicking, erroring, absent, or +// answering. It counts the reads so a test can prove it was actually asked, rather +// than passing because the switch short-circuited before reaching it. +// +// Every other method promotes from the embedded nil interface and panics if +// called, which is deliberate: nothing here should be calling them. +type configProbeStore struct { + storage.DoltStorage + cfg issueops.WorkspaceConfig + err error + panicMsg string + calls int + enabled bool + sets int +} + +func (s *configProbeStore) WorkspaceConfig() (issueops.WorkspaceConfig, error) { + s.calls++ + if s.panicMsg != "" { + panic(s.panicMsg) + } + return s.cfg, s.err +} + +// SetVersionedHistoryEnabled makes this store a storage.VersionedHistoryConfigurer, +// which is what earns it the store-plane read at activation time. Without it this +// store would be skipped before WorkspaceConfig was ever called, and every +// calls-count assertion below would pass for the wrong reason. +func (s *configProbeStore) SetVersionedHistoryEnabled(enabled bool) { + s.enabled = enabled + s.sets++ +} + +// nonConfigurerStore is the shape the capability guard exists for, and it is not a +// hypothetical: it is cmd/bd's own fakeProbingDoltStore, which embeds a nil +// storage.DoltStorage and documents that only its own methods may be called. +// SetVersionedHistoryEnabled is not part of DoltStorage, so nothing promotes it and +// this type does not satisfy storage.VersionedHistoryConfigurer -- while +// WorkspaceConfig IS part of DoltStorage, so it promotes, type-checks, and SIGSEGVs +// on call. +type nonConfigurerStore struct { + storage.DoltStorage + calls int +} + +func (s *nonConfigurerStore) WorkspaceConfig() (issueops.WorkspaceConfig, error) { + s.calls++ + panic("WorkspaceConfig on a store with no settings plane") +} + +// planeOnlyStore can answer a settings query but cannot record versions, which is +// what `bd versions` may be pointed at: the read is the point there. +type planeOnlyStore struct { + storage.DoltStorage + cfg issueops.WorkspaceConfig + calls int +} + +func (s *planeOnlyStore) WorkspaceConfig() (issueops.WorkspaceConfig, error) { + s.calls++ + return s.cfg, nil +} + +// nilDerefStore reproduces the exact shape that broke CI: the method body touches +// the receiver, so a TYPED NIL of this type is non-nil as an interface and panics +// when called. +type nilDerefStore struct { + storage.DoltStorage + cfg issueops.WorkspaceConfig +} + +func (s *nilDerefStore) WorkspaceConfig() (issueops.WorkspaceConfig, error) { + return s.cfg, nil +} + +// configUseCaseProbe is a unit of work's config use case with one canned answer. +type configUseCaseProbe struct { + domain.ConfigUseCase + value string + err error + reads int + keys []string +} + +func (c *configUseCaseProbe) GetConfig(_ context.Context, key string) (string, error) { + c.reads++ + c.keys = append(c.keys, key) + return c.value, c.err +} + +// unitProbe is a unit of work that records how it was finished: a settings read +// must be rolled back (Close), never committed. +type unitProbe struct { + uow.UnitOfWork + cfg *configUseCaseProbe + closed int + committed int +} + +func (u *unitProbe) ConfigUseCase() domain.ConfigUseCase { return u.cfg } +func (u *unitProbe) Close(context.Context) { u.closed++ } +func (u *unitProbe) Commit(context.Context, string) error { + u.committed++ + return nil +} + +// providerProbe is a unit-of-work provider that is a VersionedHistoryConfigurer. +type providerProbe struct { + uow.UnitOfWorkProvider + unit *unitProbe + newErr error + panicMsg string + newCalls int + enabled bool + sets int +} + +func (p *providerProbe) NewUOW(context.Context) (uow.UnitOfWork, error) { + p.newCalls++ + if p.panicMsg != "" { + panic(p.panicMsg) + } + if p.newErr != nil { + return nil, p.newErr + } + return p.unit, nil +} + +func (p *providerProbe) SetVersionedHistoryEnabled(enabled bool) { + p.enabled = enabled + p.sets++ +} + +func newProviderProbe(value string, err error) *providerProbe { + return &providerProbe{unit: &unitProbe{cfg: &configUseCaseProbe{value: value, err: err}}} +} + +// nonConfigurerProvider cannot record versions: it does not implement the capability. +type nonConfigurerProvider struct { + uow.UnitOfWorkProvider + newCalls int +} + +func (p *nonConfigurerProvider) NewUOW(context.Context) (uow.UnitOfWork, error) { + p.newCalls++ + panic("NewUOW on a provider that cannot record versions") +} + +// --- the resolver, moved from cmd/bd with its assertions unchanged ---------- + +// TestStoreSettingAnswersFalseForAStoreThatCannotAnswer pins the contract the +// function's own doc states, across every way a store can fail to produce a true +// -- including the two that are panics rather than errors, which is the pair that +// took the process down. +func TestStoreSettingAnswersFalseForAStoreThatCannotAnswer(t *testing.T) { + ctx := context.Background() + var typedNil *nilDerefStore + + for _, tt := range []struct { + name string + store storage.DoltStorage + want bool + }{ + {"nil store", nil, false}, + {"typed nil store", typedNil, false}, + {"WorkspaceConfig panics", &configProbeStore{panicMsg: "no settings plane on this store"}, false}, + {"WorkspaceConfig errors", &configProbeStore{err: fmt.Errorf("server unreachable")}, false}, + {"WorkspaceConfig returns no plane", &configProbeStore{}, false}, + {"GetSetting panics", &configProbeStore{cfg: &settingsPlane{panicMsg: "settings table missing"}}, false}, + {"GetSetting errors", &configProbeStore{cfg: &settingsPlane{err: fmt.Errorf("read failed")}}, false}, + {"unset is off", &configProbeStore{cfg: &settingsPlane{value: ""}}, false}, + {"stored false", &configProbeStore{cfg: &settingsPlane{value: "false"}}, false}, + {"unparseable is off", &configProbeStore{cfg: &settingsPlane{value: "yes please"}}, false}, + {"stored true", &configProbeStore{cfg: &settingsPlane{value: "true"}}, true}, + {"stored 1", &configProbeStore{cfg: &settingsPlane{value: "1"}}, true}, + {"stored true with whitespace", &configProbeStore{cfg: &settingsPlane{value: " true\n"}}, true}, + } { + t.Run(tt.name, func(t *testing.T) { + // No recover here on purpose: a panic escaping the call under test + // must fail this test, not be absorbed by it. + if got := StoreSetting(ctx, tt.store); got != tt.want { + t.Errorf("StoreSetting = %v, want %v", got, tt.want) + } + }) + } +} + +// TestParseSettingBool pins how the stored string is read back. `bd config set` +// stores whatever the user typed, verbatim, so this is the whole translation +// between that and the boolean the factories activate on. +func TestParseSettingBool(t *testing.T) { + for _, tt := range []struct { + in string + want bool + }{ + {"", false}, + {"true", true}, + {"TRUE", true}, + {"True", true}, + {"1", true}, + {"t", true}, + {" true ", true}, + {"false", false}, + {"0", false}, + {"off", false}, + {"on", false}, + {"nonsense", false}, + } { + t.Run(fmt.Sprintf("%q", tt.in), func(t *testing.T) { + if got := parseSettingBool(tt.in); got != tt.want { + t.Errorf("parseSettingBool(%q) = %v, want %v", tt.in, got, tt.want) + } + }) + } +} + +// TestEnabledForStoreSkipsTheStoreReadForANonConfigurer is the capability GUARD's +// test, and the regression test for the CI failure as it actually happened. +// +// The store read is only ever used to decide whether to call +// SetVersionedHistoryEnabled, so a store that cannot record versions has nothing to +// gain from being asked -- and, if its settings plane is absent or half-built, +// everything to lose. That is what took Test (macos-latest) and PR Core (wrapper +// timing) red at 32d4f966e, via cmd/bd's own fakeProbingDoltStore. +// +// The assertion that matters is calls == 0: not "it survived" (the recover would +// deliver that too, which is exactly how this regressed the first time) but "it was +// never asked". +func TestEnabledForStoreSkipsTheStoreReadForANonConfigurer(t *testing.T) { + raw := &nonConfigurerStore{} + if _, isConfigurer := storage.DoltStorage(raw).(storage.VersionedHistoryConfigurer); isConfigurer { + t.Fatal("nonConfigurerStore satisfies VersionedHistoryConfigurer; this test cannot exercise the guard it claims to") + } + if EnabledForStore(context.Background(), raw) { + t.Error("EnabledForStore = true for a store that cannot record versions") + } + if raw.calls != 0 { + t.Fatalf("WorkspaceConfig called %d times on a store that cannot record versions, want 0", raw.calls) + } +} + +// TestEnabledForStoreReadsTheStoreWhenItCanRecord is the other half: the guard must +// not be so eager that it stops reading the plane that finding 1 existed to start +// reading. A store that CAN record still gets asked, and a stored true still turns +// recording on. +func TestEnabledForStoreReadsTheStoreWhenItCanRecord(t *testing.T) { + raw := &configProbeStore{cfg: &settingsPlane{value: "true"}} + if got := EnabledForStore(context.Background(), raw); !got { + t.Fatal("EnabledForStore = false for a configurer whose store plane says true") + } + if raw.calls != 1 { + t.Fatalf("store read %d times, want 1", raw.calls) + } +} + +// TestEnabledReadsTheRowWithoutTheCapabilityGate: `bd versions` is the one caller +// where the read is the point, so it must ask a store that cannot record too. +func TestEnabledReadsTheRowWithoutTheCapabilityGate(t *testing.T) { + raw := &planeOnlyStore{cfg: &settingsPlane{value: "true"}} + if _, isConfigurer := storage.DoltStorage(raw).(storage.VersionedHistoryConfigurer); isConfigurer { + t.Fatal("planeOnlyStore satisfies VersionedHistoryConfigurer; this test cannot exercise what it claims to") + } + if !Enabled(context.Background(), raw) { + t.Error("Enabled = false for a store whose row says true") + } + if raw.calls != 1 { + t.Errorf("store read %d times, want 1", raw.calls) + } +} + +// --- each store answers for itself; the environment only turns recording on -- + +// TestActivateStoreReadsTheStoresOwnRow is D1 at the unit level: two stores opened +// by one process each read ITS OWN row. The one whose row says on records; the one +// with no row does not, and does not inherit the other's answer. +func TestActivateStoreReadsTheStoresOwnRow(t *testing.T) { + ctx := context.Background() + on := &configProbeStore{cfg: &settingsPlane{value: "true"}} + absent := &configProbeStore{cfg: &settingsPlane{value: ""}} + off := &configProbeStore{cfg: &settingsPlane{value: "false"}} + + for name, s := range map[string]*configProbeStore{"on": on, "absent": absent, "off": off} { + got, err := ActivateStore(ctx, s, nil) + if err != nil || got != storage.DoltStorage(s) { + t.Fatalf("%s: ActivateStore = (%v, %v), want the same store and no error", name, got, err) + } + } + if !on.enabled { + t.Error("the store whose row says true was not activated") + } + if absent.enabled || off.enabled { + t.Errorf("a store with no row (%v) or a false row (%v) was activated by another store's answer", absent.enabled, off.enabled) + } +} + +// TestActivateStoreReadsTheRowExactlyOncePerOpen: with the factories activating, +// wireStorageDecorators no longer does, so each store is activated and its row read +// once per open -- not twice. +func TestActivateStoreReadsTheRowExactlyOncePerOpen(t *testing.T) { + plane := &settingsPlane{value: "true"} + raw := &configProbeStore{cfg: plane} + if _, err := ActivateStore(context.Background(), raw, nil); err != nil { + t.Fatal(err) + } + if raw.calls != 1 || plane.reads != 1 { + t.Errorf("settings plane fetched %d time(s) and row read %d time(s), want 1 and 1", raw.calls, plane.reads) + } + if raw.sets != 1 || !raw.enabled { + t.Errorf("SetVersionedHistoryEnabled called %d time(s), enabled=%v; want exactly one call turning it on", raw.sets, raw.enabled) + } +} + +func TestActivateStorePassesAFailedOpenThrough(t *testing.T) { + openErr := errors.New("open failed") + + got, err := ActivateStore(context.Background(), nil, openErr) + if got != nil || !errors.Is(err, openErr) { + t.Errorf("ActivateStore(nil, err) = (%v, %v), want (nil, the open error)", got, err) + } + + // A store handed back alongside an error is not touched or read either. + raw := &configProbeStore{cfg: &settingsPlane{value: "true"}} + got, err = ActivateStore(context.Background(), raw, openErr) + if got != storage.DoltStorage(raw) || !errors.Is(err, openErr) { + t.Errorf("ActivateStore(store, err) = (%v, %v), want the store and the open error unchanged", got, err) + } + if raw.calls != 0 || raw.sets != 0 { + t.Errorf("a failed open still touched the store: %d settings reads, %d activations", raw.calls, raw.sets) + } +} + +// TestActivateStoreSurvivesAStoreThatCannotAnswer is the regression test for the CI +// failure itself, at the layer that now owns it. This is the RECOVER's test: the +// store here IS a configurer, so the capability guard lets it through to the read, +// and only the recover keeps the panic from escaping. +func TestActivateStoreSurvivesAStoreThatCannotAnswer(t *testing.T) { + raw := &configProbeStore{panicMsg: "WorkspaceConfig on a store with no settings plane"} + got, err := ActivateStore(context.Background(), raw, nil) + if err != nil || got != storage.DoltStorage(raw) { + t.Fatalf("ActivateStore = (%v, %v), want the store back and no error", got, err) + } + if raw.calls != 1 { + t.Fatalf("WorkspaceConfig called %d times, want 1; this test is not exercising the read it claims to", raw.calls) + } + if raw.enabled { + t.Error("a store whose settings plane panicked was switched on") + } +} + +func TestActivateStoreLeavesAStoreThatCannotRecordUntouched(t *testing.T) { + raw := &nonConfigurerStore{} + got, err := ActivateStore(context.Background(), raw, nil) + if err != nil || got != storage.DoltStorage(raw) { + t.Fatalf("ActivateStore = (%v, %v), want the store back and no error", got, err) + } + if raw.calls != 0 { + t.Errorf("a store that cannot record was asked for its settings plane %d time(s), want 0", raw.calls) + } +} + +// TestEnvironmentIsProcessWideAndShortCircuitsTheRead is D3 for stores: viper says +// on, so every store the process opens is activated -- including one whose own row +// says nothing -- and the row is not even read. +func TestEnvironmentIsProcessWideAndShortCircuitsTheRead(t *testing.T) { + viperEnv(t, "1") + + absent := &configProbeStore{cfg: &settingsPlane{value: ""}} + if _, err := ActivateStore(context.Background(), absent, nil); err != nil { + t.Fatal(err) + } + if !absent.enabled { + t.Error("the environment said on but a store with no row was not activated") + } + if absent.calls != 0 { + t.Errorf("the settings plane was read %d time(s) although the environment had already answered, want 0", absent.calls) + } +} + +// TestEnvironmentZeroNeverSwitchesOffAStoreWhoseRowIsOn is D3's other half, and +// the rule that OR is never ranked: =0 is not "off". A store whose own row says on +// keeps recording, and a store with no row stays quiet. The only way off is the row. +func TestEnvironmentZeroNeverSwitchesOffAStoreWhoseRowIsOn(t *testing.T) { + viperEnv(t, "0") + ctx := context.Background() + + on := &configProbeStore{cfg: &settingsPlane{value: "true"}} + absent := &configProbeStore{cfg: &settingsPlane{value: ""}} + for _, s := range []*configProbeStore{on, absent} { + if _, err := ActivateStore(ctx, s, nil); err != nil { + t.Fatal(err) + } + } + if !on.enabled { + t.Error("BD_VERSIONED_HISTORY_ENABLED=0 switched off a store whose own row says true") + } + if absent.enabled { + t.Error("BD_VERSIONED_HISTORY_ENABLED=0 switched ON a store with no row") + } +} + +// --- D7: an error is not an absent row -------------------------------------- + +func TestAnAbsentRowIsSilentlyOffButAReadErrorIsDiagnosed(t *testing.T) { + ctx := context.Background() + + t.Run("absent row: off, and nothing to say", func(t *testing.T) { + raw := &configProbeStore{cfg: &settingsPlane{value: ""}} + var got bool + out := captureStderr(t, true, func() { got = StoreSetting(ctx, raw) }) + if got { + t.Error("an absent row resolved to on") + } + if out != "" { + t.Errorf("an absent row is the normal state of every workspace and must not be diagnosed, got: %q", out) + } + }) + + t.Run("GetSetting error: off, and diagnosed at debug level", func(t *testing.T) { + raw := &configProbeStore{cfg: &settingsPlane{err: errors.New("read failed: table config is locked")}} + var got bool + out := captureStderr(t, true, func() { got = StoreSetting(ctx, raw) }) + if got { + t.Error("a failed row read resolved to on") + } + for _, want := range []string{ConfigKey, "table config is locked"} { + if !strings.Contains(out, want) { + t.Errorf("the diagnostic does not mention %q: %q", want, out) + } + } + }) + + t.Run("WorkspaceConfig error: off, and diagnosed at debug level", func(t *testing.T) { + raw := &configProbeStore{err: errors.New("server unreachable")} + var got bool + out := captureStderr(t, true, func() { got = StoreSetting(ctx, raw) }) + if got { + t.Error("an unreadable settings plane resolved to on") + } + if !strings.Contains(out, "server unreachable") { + t.Errorf("the diagnostic does not carry the cause: %q", out) + } + }) + + t.Run("the diagnostic is debug-level, never unconditional stderr", func(t *testing.T) { + raw := &configProbeStore{cfg: &settingsPlane{err: errors.New("read failed")}} + out := captureStderr(t, false, func() { _ = StoreSetting(ctx, raw) }) + if out != "" { + t.Errorf("this runs on every invocation and must be silent unless debugging, got: %q", out) + } + }) +} + +// --- Apply: the raw-store ordering, and the set-but-unsupported warning ----- + +// TestApplyReachesOnlyTheRawStore pins the ORDERING that activation depends on. +// +// The capability is reached by type assertion, and none of the decorators +// (telemetry, externaldeps, hooks) forwards SetVersionedHistoryEnabled. So applying +// the config after any wrap would assert against a wrapper, fail silently, and +// leave versioned history OFF while the operator's config said it was on -- a wrong +// answer rather than an empty one. That is why the factories, which hand back the +// concrete store before anything wraps it, are where activation lives. +// +// The second half of this test is the part that actually guards the ordering: it +// proves the wrapped chain does NOT satisfy the capability, so if a decorator ever +// starts forwarding, the constraint this test exists for has changed. +func TestApplyReachesOnlyTheRawStore(t *testing.T) { + raw := &configProbeStore{} + Apply(raw, true) + if !raw.enabled { + t.Error("Apply(raw, true) did not enable versioned history on the raw store") + } + + wrap := func(s storage.DoltStorage) storage.DoltStorage { + return externaldeps.New( + telemetry.WrapStorage(s), + func(externaldeps.ProjectName) (string, bool) { return "", false }, + func(context.Context, string) (storage.DoltStorage, error) { return nil, errors.New("unused") }, + ) + } + if _, ok := wrap(&configProbeStore{}).(storage.VersionedHistoryConfigurer); ok { + // Fatal, not Skip. A skip here returns BEFORE the assertion below -- the + // half that actually guards the ordering -- so the case would report + // "not applicable" while checking nothing. + t.Fatal("a decorator now forwards SetVersionedHistoryEnabled; the ordering constraint has changed and this test needs rewriting rather than silently passing") + } + + behindWrap := &configProbeStore{} + captureStderr(t, false, func() { Apply(wrap(behindWrap), true) }) + if behindWrap.enabled { + t.Error("the capability was somehow reached through the decorators; this test's premise is stale") + } +} + +// TestApplyOffIsANoop pins that the disabled path touches nothing at all -- +// flag-off must be byte-identical to a build without the feature, which is the +// contract #6135 ships under. +func TestApplyOffIsANoop(t *testing.T) { + raw := &configProbeStore{} + out := captureStderr(t, true, func() { Apply(raw, false) }) + if raw.sets != 0 || raw.enabled { + t.Errorf("Apply(raw, false) called SetVersionedHistoryEnabled %d time(s), enabled=%v; flag-off must be a no-op", raw.sets, raw.enabled) + } + if out != "" { + t.Errorf("Apply(raw, false) wrote %q", out) + } +} + +// TestApplyWarnsWhenTheBackendCannotRecord: asking for history and not getting it is +// exactly the case a user must be told about. The wording is #6661's, unchanged. +func TestApplyWarnsWhenTheBackendCannotRecord(t *testing.T) { + out := captureStderr(t, false, func() { Apply(&nonConfigurerStore{}, true) }) + for _, want := range []string{ + "warning: versioned-history.enabled is set, but this storage backend (", + "does not support version history; it stays off", + } { + if !strings.Contains(out, want) { + t.Errorf("the warning is missing %q: %q", want, out) + } + } +} + +// TestConfigKeyIsNotYamlOnly is D5's pin at the unit level. versioned-history.* must +// NOT become a yaml-only key: `bd config set versioned-history.enabled true` keeps +// writing the database row exactly as #6661 wired it, and a yaml-only key would move +// the switch off the store. +func TestConfigKeyIsNotYamlOnly(t *testing.T) { + if ConfigKey != "versioned-history.enabled" { + t.Errorf("ConfigKey = %q; the key is documented, and read back by `bd versions`, under this exact spelling", ConfigKey) + } + if config.IsYamlOnlyKey(ConfigKey) { + t.Errorf("%s is a yaml-only key, so `bd config set` would write config.yaml and no store would ever read it", ConfigKey) + } +} + +// --- K1: the provider path --------------------------------------------------- + +// TestActivateProviderReadsItsOwnRow is K1's central claim. A unit-of-work provider +// has no storage.DoltStorage to ask, so its flag comes from viper OR its own row -- +// read through a unit of work's ConfigUseCase -- and NOT from viper alone. +func TestActivateProviderReadsItsOwnRow(t *testing.T) { + p := newProviderProbe("true", nil) + got, err := ActivateProvider(context.Background(), p, nil) + if err != nil || got != uow.UnitOfWorkProvider(p) { + t.Fatalf("ActivateProvider = (%v, %v), want the provider back and no error", got, err) + } + if !p.enabled { + t.Fatal("a provider whose database row says true was not activated: the flag was resolved from viper alone") + } + if p.unit.cfg.reads != 1 || len(p.unit.cfg.keys) != 1 || p.unit.cfg.keys[0] != ConfigKey { + t.Errorf("config rows read = %v (%d read(s)), want exactly one read of %q", p.unit.cfg.keys, p.unit.cfg.reads, ConfigKey) + } + // A settings read is a read: rolled back, never committed. + if p.unit.closed != 1 || p.unit.committed != 0 { + t.Errorf("unit of work closed %d time(s) and committed %d time(s), want closed once and never committed", p.unit.closed, p.unit.committed) + } +} + +func TestActivateProviderLeavesAnAbsentOrFalseRowOff(t *testing.T) { + for _, value := range []string{"", "false", "0", "on", "nonsense"} { + t.Run(fmt.Sprintf("%q", value), func(t *testing.T) { + p := newProviderProbe(value, nil) + captureStderr(t, true, func() { _, _ = ActivateProvider(context.Background(), p, nil) }) + if p.enabled { + t.Errorf("a provider whose row is %q was activated", value) + } + if p.unit.closed != 1 { + t.Errorf("unit of work closed %d time(s), want 1", p.unit.closed) + } + }) + } + t.Run("an absent row is silent", func(t *testing.T) { + p := newProviderProbe("", nil) + if out := captureStderr(t, true, func() { _, _ = ActivateProvider(context.Background(), p, nil) }); out != "" { + t.Errorf("an absent row must not be diagnosed, got %q", out) + } + }) +} + +// TestActivateProviderReadFailureIsOffAndDiagnosed is D7 for providers: an error is +// not an absent row. It resolves to off -- but not silently. +func TestActivateProviderReadFailureIsOffAndDiagnosed(t *testing.T) { + t.Run("GetConfig errors", func(t *testing.T) { + p := newProviderProbe("", errors.New("no such table: config")) + out := captureStderr(t, true, func() { _, _ = ActivateProvider(context.Background(), p, nil) }) + if p.enabled { + t.Error("a failed row read activated the provider") + } + for _, want := range []string{ConfigKey, "no such table: config"} { + if !strings.Contains(out, want) { + t.Errorf("the diagnostic does not mention %q: %q", want, out) + } + } + if p.unit.closed != 1 { + t.Errorf("unit of work closed %d time(s) after a failed read, want 1", p.unit.closed) + } + }) + t.Run("NewUOW errors", func(t *testing.T) { + p := &providerProbe{newErr: errors.New("connection refused")} + out := captureStderr(t, true, func() { _, _ = ActivateProvider(context.Background(), p, nil) }) + if p.enabled { + t.Error("a provider that could not open a unit of work was activated") + } + if !strings.Contains(out, "connection refused") { + t.Errorf("the diagnostic does not carry the cause: %q", out) + } + }) + t.Run("silent unless debugging", func(t *testing.T) { + p := newProviderProbe("", errors.New("no such table: config")) + if out := captureStderr(t, false, func() { _, _ = ActivateProvider(context.Background(), p, nil) }); out != "" { + t.Errorf("this runs on every invocation and must be silent unless debugging, got: %q", out) + } + }) +} + +func TestActivateProviderSurvivesAProviderThatPanics(t *testing.T) { + p := &providerProbe{panicMsg: "NewUOW on a half-built provider"} + got, err := ActivateProvider(context.Background(), p, nil) + if err != nil || got != uow.UnitOfWorkProvider(p) { + t.Fatalf("ActivateProvider = (%v, %v), want the provider back and no error", got, err) + } + if p.newCalls != 1 { + t.Fatalf("NewUOW called %d times, want 1; this test is not exercising the read it claims to", p.newCalls) + } + if p.enabled { + t.Error("a provider whose read panicked was activated") + } +} + +func TestActivateProviderSkipsTheReadForANonConfigurer(t *testing.T) { + p := &nonConfigurerProvider{} + if _, ok := uow.UnitOfWorkProvider(p).(storage.VersionedHistoryConfigurer); ok { + t.Fatal("nonConfigurerProvider satisfies VersionedHistoryConfigurer; this test cannot exercise the guard it claims to") + } + got, err := ActivateProvider(context.Background(), p, nil) + if err != nil || got != uow.UnitOfWorkProvider(p) { + t.Fatalf("ActivateProvider = (%v, %v), want the provider back and no error", got, err) + } + if p.newCalls != 0 { + t.Errorf("a provider that cannot record was asked for a unit of work %d time(s), want 0", p.newCalls) + } +} + +func TestActivateProviderPassesAFailedOpenThrough(t *testing.T) { + openErr := errors.New("open failed") + got, err := ActivateProvider(context.Background(), nil, openErr) + if got != nil || !errors.Is(err, openErr) { + t.Errorf("ActivateProvider(nil, err) = (%v, %v), want (nil, the open error)", got, err) + } + p := newProviderProbe("true", nil) + got, err = ActivateProvider(context.Background(), p, openErr) + if got != uow.UnitOfWorkProvider(p) || !errors.Is(err, openErr) { + t.Errorf("ActivateProvider(provider, err) = (%v, %v), want the provider and the open error unchanged", got, err) + } + if p.newCalls != 0 || p.sets != 0 { + t.Errorf("a failed open still touched the provider: %d unit(s) of work opened, %d activation(s)", p.newCalls, p.sets) + } +} + +// TestProviderEnvironmentIsProcessWideAndOnlyTurnsRecordingOn is D3 for providers. +func TestProviderEnvironmentIsProcessWideAndOnlyTurnsRecordingOn(t *testing.T) { + t.Run("=1 activates without reading the row", func(t *testing.T) { + viperEnv(t, "1") + p := newProviderProbe("", nil) + if _, err := ActivateProvider(context.Background(), p, nil); err != nil { + t.Fatal(err) + } + if !p.enabled { + t.Error("the environment said on but the provider was not activated") + } + if p.newCalls != 0 { + t.Errorf("a unit of work was opened %d time(s) although the environment had already answered, want 0", p.newCalls) + } + }) + t.Run("=0 never switches off a provider whose row is on", func(t *testing.T) { + viperEnv(t, "0") + p := newProviderProbe("true", nil) + if _, err := ActivateProvider(context.Background(), p, nil); err != nil { + t.Fatal(err) + } + if !p.enabled { + t.Error("BD_VERSIONED_HISTORY_ENABLED=0 switched off a provider whose own row says true") + } + }) +} diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index b7e0b6570..410728ae0 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -290,6 +290,7 @@ release_cross_build( "//internal/utils:utils", "//internal/validation:validation", "//internal/versioncheck:versioncheck", + "//internal/versionedhistory:versionedhistory", "//internal/workapi/storecounter:storecounter", "//internal/workapi/storequerier:storequerier", "//internal/workapi/storereader:storereader", diff --git a/tools/bazel/go_srcs.py b/tools/bazel/go_srcs.py index 4b685968a..db584cf99 100755 --- a/tools/bazel/go_srcs.py +++ b/tools/bazel/go_srcs.py @@ -69,6 +69,7 @@ import sys # Packages whose own Go source a test scans. PACKAGES = ( + ".", # //cmd/bd:bd_test (events-journal construction scan: the Go SDK opens) "backend", # //backend:backend_test (public alias census) "backend/conformance", # //backend/conformance, //internal/storage "beadserrors", # role facade alias targets From 5c9c02ab77a3e52a861e45f0b9712628eba8ad34 Mon Sep 17 00:00:00 2001 From: Jim Wordelman Date: Sun, 18 Oct 2026 21:20:57 +0000 Subject: [PATCH 02/41] Phase 3 (R20): epoch-transition enforcement + minted-address CAS - gastownhall/beads#6136 (#6664) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(conformance): un-skip R16 CAS + R20 epoch tests with honest not-implemented stubs Wires all 10 ExpectedRevisionFixture-gated Run* functions and both EpochFixture-gated Run* functions to real (non-nil) hooks on all three storage legs (dolt, embeddeddolt, uow), replacing the Phase 0 all-nil fixtures that made these cases t.Skip. Every wired hook is an unconditional "not implemented yet" stub rather than a partial-real implementation. This is deliberate: every Run* function under test calls its primary hook (CompareAndSetVersion, or CurrentEpoch/MintUnderEpoch) first, before any other hook, so a stub that errors unconditionally on that first call guarantees deterministic RED with zero risk of an accidental pass -- a concern that's concrete, not hypothetical, since at least one case (RunExpectedRevisionAcceptsAWriteNamingNoVersion) would go green under a partially-real read/unconditional-write strategy backed by Phase 1's already-persisted current_revision/store_epoch columns. Each case now fails for its documented reason: most via a direct t.Fatalf quoting the stub's "not implemented" error, and two (RefusalIsDistinguishableFromNotFound, RefusalIsDistinguishableFromValidationFailure) via a clean want/got mismatch against the sentinel errors they expect. Explicitly out of scope and untouched: CrossRecordInvariantFixture (R16.1) and the non-epoch RetentionFixture cases (R17) -- confirmed via test run that TestRetentionContract still reports PASS with all 13 subtests SKIP on every leg. TestEveryLegWiresEveryRoleContract (internal/storage/leg_contract_wiring_test.go) is a source-level AST naming check independent of hook nil-ness; all 12 entrypoints were already called by every leg pre-change, so this lock was satisfied before and remains satisfied after. Sentinel error messages reference be-x5jqd.3 (R16 CAS implementation) and be-x5jqd.4 (R20 epoch enforcement implementation), the two builder children this un-skip unblocks. Verified via go build ./..., go vet ./..., and go test -tags cgo across all three legs: all 10 R16 + 2 R20 subtests FAIL for the documented reason, TestRetentionContract unaffected. Bead: be-x5jqd.1 * feat: green — Implement R20 epoch-transition enforcement (refs be-x5jqd.4) * fix(epoch): honor R20-n's retained-mapping exception in StillServes/Resolve StillServesInTx and ResolveEpochInTx only ever checked whether the literal queried address's own row matched the current epoch. Per R20-n (#5898 rev 9), a version that survives an epoch transition keeps its prior-epoch address resolving once its id has been carried forward into the current epoch (via CurrentAddressForInTx or a fresh MintUnderEpochInTx) — that carried-forward row wasn't being consulted, so a legitimately retained address would wrongly stop resolving. Add mintedIDHasAddressAtEpochInTx and consult it as a fallback in both functions when the queried address's own row is stale. Addresses whose id was never carried forward are unaffected (row.mintedID has no row at the current epoch, same false/GoneReorganization answer as before). Round-2 response to be-rskwc's causal_disproof finding: the merge-base commit FAILed honestly on this defect, this branch's tdd_green SKIPped instead (attribution_evidence). This fix is real and verified via a new sqlmock unit test (TestStillServesInTxHonorsARetainedMapping et al.), but does not by itself flip TestEpochContract/EpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed from SKIP to PASS: that test's own construction mints two records identically and never carries either forward before comparing StillServes on both, so no backend implementation can make them differ (escalation follows on a needs-mayor bead). refs be-x5jqd.4, be-rskwc round 2 * test(feat): red — harden EpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed (refs be-9fiaw) Per mayor ruling on be-9fiaw: fix the test rather than waive it. The fixture already lets record-a survive an epoch bump by minting it again under the new epoch; record-b, never re-minted, does not survive. That asymmetry was previously only discovered after the fact (skip if StillServes agreed for both addresses), which meant the case could pass by observing either polarity. Assert the concrete expected values instead: servesA must be true, servesB must be false. This makes the case deterministically exercise both halves of R20-n, and closes a latent gap where a backend with the polarity reversed could have silently passed via the old discover-then-swap logic. * fix(schema): renumber epoch_minted_addresses to 0070, linearizing our own slot collision (refs be-hs42e.7) R20's epoch_minted_addresses and R7.1's removed_restriction both claimed migration slot 0069 on parallel sibling branches. Neither contained the other, so nothing in either branch's own CI could see the union: each was green alone, and the second to merge would panic checkNoDuplicateVersions at store open -- before any command's RunE, so every bd invocation bricks rather than one package reddening. Same hazard bee reproduced for #6358 vs #6008. This linearizes them instead of leaving it to merge order. R7.1 keeps 0069: it is published beneath #6661 and cannot move without rewriting a branch a reviewer is reading. R20 stops being a sibling, rebases onto R7.1, and takes the next contiguous slot. Contiguity is why the renumber could not be done pre-emptively in either branch alone: TestAllMigrationsSQLAppliesThroughDoltCLIAndRecordsLatestVersion asserts COUNT(*) == LatestVersion(), so a jump to a far slot fails until the gap is filled. The order had to be settled first. - migrations/0069_add_epoch_minted_addresses.{up,down}.sql -> 0070_* - migration_0069_add_epoch_minted_addresses_test.go -> migration_0070_*, with TestLatestVersionIncludesMigration0069 -> ...0070 (want 70) - 0069_add_removed_restriction_test.go retires its own LatestVersion pin in favour of 0070's, keeping the one-pin-at-a-time chain 0068 -> 0069 -> 0070 cli_migrations.go needs no change: its only override is R7.1's guarded PREPARE, which keeps 0069. The epoch migration is a plain unguarded CREATE TABLE IF NOT EXISTS and needs none. Verified: TestEmbeddedMigrationSourcesHaveNoDuplicateVersions, TestLatestVersionIncludesMigration0070, TestAllMigrationsSQLAppliesThroughDoltCLIAndRecordsLatestVersion (applies the full 0001..0070 chain through the dolt CLI), TestBundleMigrationsWithPreparedALTERAreOverriddenOrJustified and TestAllMigrationsSQLUsesDirectDDLForKnownCLIIncompatibilities all pass. go build ./... and go vet ./... clean. Co-Authored-By: Claude Opus 5 (1M context) * test(feat): red — Fix PR #6664: restore R16 honest-skip (stale be-x5jqd.3 premise) + address bee's 2 blockers on epoch_cas.go (refs be-ah0jf) * feat: green — Fix PR #6664: restore R16 honest-skip (stale be-x5jqd.3 premise) + address bee's 2 blockers on epoch_cas.go (refs be-ah0jf) * test(feat): red — Review: Fix PR #6664: restore R16 honest-skip (stale be-x5jqd.3 premise) + address bee's 2 blockers on epoch_cas.go (refs be-yj55d) * feat: green — Review: Fix PR #6664: restore R16 honest-skip (stale be-x5jqd.3 premise) + address bee's 2 blockers on epoch_cas.go (refs be-yj55d) * docs(epoch): cite epoch_minted_addresses as migration 0071 Upstream's 0069_widen_issue_versions_datetime_precision (#6675) took slot 0069 first, so removed_restriction moved to 0070 and epoch_minted_addresses to 0071. The three epoch_cas.go headers still named 0070. Rebase note (maintainer): the migration is now 0072, per the landing-order ruling (#6943's participation_generation is 0070 and #6661's removed_restriction is 0071); the commit "chore(schema): renumber epoch_minted_addresses to 0072 after #6661's 0071" updates these three headers. Co-Authored-By: Claude Opus 5.5 (1M context) * test(feat): red — move #6664's Dolt server leg to post-transaction publish for store_epoch and epoch_minted_addresses (latent lost-update pattern) Tests only. On the commit before this one they fail for these reasons: - the server leg publishes store_epoch / epoch_minted_addresses inside the open SQL transaction (DOLT_ADD before COMMIT) for BumpEpoch, MintUnderEpoch and CurrentAddressFor: the lost-update hazard documented on doltAddAndCommitInTx - MintUnderEpoch and CurrentAddressFor stage only epoch_minted_addresses, so the first mint leaves store_epoch modified in the working set, on both legs (#6664 Major 2, reproduced with Status() on a real server and embedded) - BumpEpoch must surface a Dolt commit that fails after its SQL commit, while the mint paths keep their rows - ensureStoreEpochRow seeds with INSERT IGNORE and reads the row back TestMintOverlappingABumpCommitsAnAddressThatIsAlreadyGone characterizes current behaviour and passes now: a mint overlapping a bump commits an address the bump already voided. Its doc comment records why that is left as it is. * feat: green — move #6664's Dolt server leg to post-transaction publish for store_epoch and epoch_minted_addresses (latent lost-update pattern) Server leg (internal/storage/dolt/epoch_cas.go): BumpEpoch, MintUnderEpoch and CurrentAddressFor no longer stage and Dolt-commit inside the open SQL transaction. The two mint paths publish through runIssueOperationTxWithMessage, which commits the SQL transaction first. BumpEpoch commits it, then publishes with doltAddAndCommitPostTx and RETURNS a failed Dolt commit: a bump that is committed but unpublished must not read as success. Both legs: a mint stages store_epoch with epoch_minted_addresses, because the first mint inserts the singleton row (#6664 Major 2). The set lives in issueops.EpochMintDirtyTables so the legs cannot drift from what the body writes. issueops: ensureStoreEpochRow seeds with INSERT IGNORE and reads the row back, as version_history.go does for the same row. Not changed, on purpose: a mint overlapping a concurrent bump can still commit an address the bump already voided. Dolt has no lock for it and the only conflicting write would overwrite bumped_at/bumped_reason; see the doc comment on TestMintOverlappingABumpCommitsAnAddressThatIsAlreadyGone. One comment in that test file is reworded; no test logic changed since the red commit. * chore(bazel): bazel-sync (gazelle) for the two new epoch test files Only the two go_test srcs entries for the test files this branch adds: epoch_cas_publish_test.go (dolt) and epoch_cas_status_test.go (embeddeddolt). gazelle wanted nothing else for them (no deps change). * test(conformance): add the optional EpochFixture.LoseVersion capability Record survival explicitly instead of inferring it (gastownhall/beads#6664, review 5360880888, Major 1). Once an address stays served unless its own row says the store lost it, the only way for the contract to produce a Gone address is an explicit loss, so the contract needs a way to say "the store lost this Version". LoseVersion is additive and optional: no existing field changes signature, and a nil LoseVersion makes the cases that need one skip with a named reason. It is its own commit so that it can be judged, or replaced, on its own: the same semantics could ride on BumpEpoch taking a lost set, and only the plumbing would differ. Nothing uses it yet. * chore(epoch): add signature-only stubs for the survival rework The tests that follow exercise BumpEpochCarryingInTx, LoseVersionInTx, ErrEpochAddressNotServed and the BumpEpochCarrying and LoseVersion store methods (gastownhall/beads#6664, review 5360880888). Stubs give those names so the tree builds and the tests fail for their real, behavioural reasons instead of failing to compile. No logic: ErrEpochAddressNotServed is a sentinel, BumpEpochCarrying advances the counter exactly as BumpEpoch does and carries nothing, and LoseVersion returns "not implemented". The next commit adds the tests; the one after replaces these bodies. * test(feat): red — record survival explicitly in epoch_minted_addresses, make CurrentAddressFor read-only, carry on a token-scheme change Tests for the survival rework of gastownhall/beads#6664 (review 5360880888: Major 1, Minors 3 and 4). Every test is written before any production change. - Contract: the earlier mint-driven R20-n case is restated as three named phases (a survivor and a lost Version, loss is terminal, survival needs no upkeep), plus three new cases for a token-scheme change (carry the live lineages and only those, a carried address keeps resolving across later bumps, a lost address stays Gone across a scheme change). All three legs register them and wire LoseVersion and the carrying bump. - Shared body: unit tests for the row-based resolver (exactly the address's row and the epoch, fail closed on a row from a future epoch), a read-only CurrentAddressFor with typed errors, LoseVersion, a bump that writes only store_epoch, a mint that never hands back a lost address, and the per-database counter (Minor 3). Minor 4 gets a guard that no address in the test file is written by hand. - Real engine: the carry on the embedded leg (one carry row per live id, the lineage root, the collision rule, a row from a future epoch, a rewound counter, database-wide, lost lineages), the publication sets of every writer, and on the server leg the overlap of a mint with a bump (now a Live survivor) and a deterministic mint-after-loss interleaving that must not revive the lost address. - Migration 0071 byte pin and a real-dolt shape check for carried_from, gone_at_epoch, the carried_from index and the gone-not-before-mint CHECK. These fail on the current code, and some do not build yet because BumpEpochCarryingInTx, LoseVersionInTx, ErrEpochAddressNotServed and the store methods they exercise do not exist. * feat: green — record survival explicitly in epoch_minted_addresses, make CurrentAddressFor read-only, carry on a token-scheme change The survival rework of gastownhall/beads#6664 (review 5360880888: Major 1, Minors 3 and 4). An address's status is now a function of its own row and the current epoch, never inferred from other rows. - Migration 0071 (edited in place; no new slot, no ALTER): the table gains carried_from, gone_at_epoch, an index on carried_from and a CHECK that a loss epoch is not before the mint epoch. - Shared body: Live unless gone_at_epoch is set, and a row from an epoch that has not happened yet fails closed; the chain query is deleted. CurrentAddressFor only reads and returns typed errors. LoseVersion marks a whole lineage lost, once, guarded by gone_at_epoch IS NULL. BumpEpochCarrying (token-scheme change) writes one carry row per live lineage naming the lineage root, resolves a shared-id collision by carrying the newest and losing the rest, and fails if a row already sits at the new epoch. A mint only inserts its own root row and never hands back an address that is not served. A restore or reinit bump still writes only store_epoch. - Both Dolt legs: BumpEpochCarrying publishes store_epoch and epoch_minted_addresses, LoseVersion publishes epoch_minted_addresses and returns a failed publication, and CurrentAddressFor runs on the read path and publishes nothing. The per-database epoch counter (Minor 3) and the requirement that a caller of the carrying bump hold the workspace gate exclusively are documented where they apply. One change to tests written in the previous commit: the three real-server publication probes that only need a clean working set now open a database of their own, as the overlap tests do, instead of queueing on the shared test database's schema-init lock. Its five-second acquire timeout ran out under load and failed two of them; no assertion changed. * test(dolt): run the epoch contract on a database of its own TestEpochContract set up through setupTestStore, which puts every parallel test on the shared test database and queues their schema initialization on one lock with a five-second acquire timeout. Under host load a setup that waits behind another parallel test's runs out of it and fails at its first line with "schema migration lock unavailable: timeout". That failed the unchanged first-mint test twice in the dolt lane once this change had added tests to it, and the contract's own setup is exposed to the same timeout. The contract now opens a database of its own, the way the overlap tests and the publication probes already do, so this change adds no contention on the shared database's lock. No assertion changed. * docs: replace internal tracker ids in the epoch comments and test names with plain descriptions Comment and text edits only; no behavior changes and no line is added or removed (21 lines are replaced one for one). Each replaced line named an internal tracker id in a header, a test comment, a failure message or the migration header; it now says what it meant in plain words or cites the upstream issue or pull request. The EpochFixture API lines and the already published commit messages are left as they were. * ci: list this PR's embedded storage tests in the embedded shard manifest #7156 made the embedded-Dolt tier's shard manifests exact-once (50 cmd shards, 15 storage shards), so every embedded test must be listed. This PR adds eleven embedded storage tests and no embedded cmd tests; this commit is exactly what the generators write for them: python3 scripts/ci/gen_embedded_cmd_shard_manifest.py 50 --weights=duration --write python3 scripts/ci/gen_embedded_storage_shard_manifest.py 15 --weights=duration --write Incremental: 11 lines added, none moved; the cmd manifest is unchanged. Rebase note (maintainer): regenerated on current main, whose embedded storage manifest uses a 40-shard block; the generator's incremental output placed these tests at shards 7, 9, 11, 20, 21, 28, 31, 32, 33, 34 and 36 of 40, one test per shard (gen_embedded_storage_shard_manifest.py 40 --weights=duration --write; --check passes). * chore(bazel): bazel-sync for the epoch sources and migration 0071 The BUILD files did not list epoch_cas.go in the dolt, embeddeddolt and issueops libraries, issueops' epoch_cas_test.go, or migration 0071 and its test, so Bazel could not compile those packages' tests. This is exactly the patch make bazel-sync writes (gazelle plus go_srcs.py; the PR's BUILD-sync CI step printed it): 4 files, 6 lines, nothing else. Rebase note (maintainer): the migration is now 0072, per the landing-order ruling (#6943's participation_generation is 0070 and #6661's removed_restriction is 0071); the commit "chore(schema): renumber epoch_minted_addresses to 0072 after #6661's 0071" renames these BUILD entries. On the rebased base, this commit's internal/storage/schema/BUILD.bazel hunk is merged with the base's 0070 and 0071 entries. * chore(schema): renumber epoch_minted_addresses to 0072 after #6661's 0071 Migrations are numbered by landing order. #6943 (participation_generation, the versioned-history write fence) owns main-plane 0070 and #6661 (removed_restriction) owns 0071, so this PR's epoch_minted_addresses migration moves to the next free slot, 0072. The rename is done here, as one maintainer commit, rather than inside the replayed commits, so the history shows exactly what changed relative to the PR as reviewed. - migrations/0071_add_epoch_minted_addresses.{up,down}.sql become 0072_*. Neither header names its own number, so the SQL and its comments are unchanged. Neither file exists on main, #6943 or #6661, so this is not an edit to a shipped migration. - cli_migrations.go needs nothing: the migration is a plain CREATE TABLE IF NOT EXISTS with no CLI-bundle override. - migration_0071_add_epoch_minted_addresses_test.go becomes migration_0072_add_epoch_minted_addresses_test.go; its consts, tests and messages say 0072, it owns the one LatestVersion pin (72), and its slot history now records the move past #6943's 0070. TestLatestVersionIncludesMigration0071 in #6661's removed_restriction test was already retired to a pointer comment by the rebase's resolution of "renumber epoch_minted_addresses to 0070"; that comment now names TestLatestVersionIncludesMigration0072 and its file. - BUILD.bazel embedsrcs/srcs, and the three epoch_cas.go headers (dolt, embeddeddolt, issueops) that cite the migration number. The intermediate commits of the rebase are not expected to build, for two reasons: - a duplicate migration version: the PR's epoch_minted_addresses migration shares 0069 with main's 0069_widen_issue_versions_datetime_precision until the PR's own "renumber epoch_minted_addresses to 0070", and from there shares 0071 with #6661's removed_restriction until this commit; - a helper-name collision: from the PR's "feat: green — Fix PR #6664: restore R16 honest-skip ... (refs be-ah0jf)" on, internal/storage/issueops declares readStoreEpochInTx twice, once in #6943's version_history.go and once in this PR's epoch_cas.go, until "fix(epoch): rename #6664's read-only store_epoch read past #6943's readStoreEpochInTx" later in this stack. This commit ends the duplicate version, but it still does not compile on its own because of the collision; the stack builds from the rename commit on. Checked at the tip of the stack, where the rename has landed: go test ./internal/storage/schema/... with the dolt CLI lane (including TestAllMigrationsSQLAppliesThroughDoltCLIAndRecordsLatestVersion and TestMigration0072RecordsSurvivalInTheAddressTableThroughDoltCLI), and scripts/check-migration-hygiene.sh with BASE_SHA=origin/main and BASE_SHA=vh/6661-rebase. Co-Authored-By: Claude Opus 5.5 (1M context) * test(epoch): waive #6664's three new epoch contracts on main's http leg main gained an HTTP storage leg (#7247) after this PR's base, and TestEveryLegWiresEveryRoleContract requires every leg to name or waive every role contract. This PR adds three EpochFixture cases (RunTokenSchemeChangeCarriesLiveAddressesAndOnlyThose, RunACarriedAddressKeepsResolvingAcrossLaterBumps and RunALostAddressStaysGoneAcrossASchemeChange); dolt, embeddeddolt and uow wire them, and the http leg did not name them. They are waived on the http leg with the existing retentionEpochNoRoleWaiverReason, which already covers every other RetentionFixture/EpochFixture case there: EpochFixture is raw storage hooks (CurrentEpoch, BumpEpoch, MintUnderEpoch, StillServes, Resolve, CurrentAddressFor, LoseVersion) with no publicops role behind them, and internal/httpclient is a remote client with no epoch counter or address table of its own. The constant's doc comment now counts eighteen cases. Test-only; no production code changes. After this, the http leg's only unnamed entrypoints are #6943's seven RunParticipationGeneration* contracts, whose waivers belong to #6943. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(epoch): rename #6664's read-only store_epoch read past #6943's readStoreEpochInTx #6943's "a create stamps participation even when the row is flagged no-history" (7c00d65fa6) adds readStoreEpochInTx to version_history.go: the mint's read of store_epoch, which SEEDS the singleton row when it is absent. This PR's epoch_cas.go already declared a readStoreEpochInTx of its own, so the package stopped compiling once #6664 was stacked on that commit. The two cannot be folded into one. This PR's helper is deliberately read-only: an absent row answers epoch 1 without writing, so CurrentEpochInTx, StillServesInTx, ResolveEpochInTx, CurrentAddressForInTx and LoseVersionInTx never write from a read (review item B4, pinned by TestCurrentAddressForNeverReseedsAnAbsentStoreEpochRow and TestCurrentAddressForWritesAndPublishesNothing). Calling #6943's seeding read from those paths would break that. So #6664's helper becomes readStoreEpochWithoutSeedingInTx, and its doc comment says why it is not readStoreEpochInTx. The five call sites and the ensureStoreEpochRow comment that names it follow. No behavior changes on either side: #6943's mint path keeps its seeding read, and this PR's read-only paths keep theirs. (ensureStoreEpochRow, which BumpEpochInTx and MintUnderEpochInTx use, now has the same body as #6943's readStoreEpochInTx; it is left as it is rather than folded in here.) Co-Authored-By: Claude Opus 5.5 (1M context) * test(storage): say exactly why the retention and epoch contracts stay waived on http All but four of the participation-generation contracts (#6943's write fence) now run over the served wire (internal/httpclient/served_participation_generation_test.go): their subject is a create or an update, which the client can send, and the fixture reads the reference store's raw columns. The four stay waived, because the mutation each needs has no form on the wire. This checked whether the eighteen RetentionFixture/EpochFixture cases waived on the http leg, including the three this PR adds, can be wired the same way. They cannot, cheaply or otherwise, without a new wire operation: - every EpochFixture hook (CurrentEpoch, BumpEpoch, MintUnderEpoch, StillServes, Resolve, CurrentAddressFor, LoseVersion) is the operation under test, not a seed or a raw read. Those operations are methods on the concrete *DoltStore and *EmbeddedDoltStore over issueops' *InTx bodies, on no storage interface, no publicops role and no wire operation. Every call path into them starts in a test: the methods are called only from each store leg's own _test.go files (its EpochFixture, which the conformance contracts drive, and its epoch tests), and the *InTx bodies only from those methods, from each other and from tests (the uow leg's fixture through a test-only type assertion, and issueops' and the dolt leg's own tests). No command, role or server handler reaches them, so no client mutation touches epoch state either; - RetentionFixture's hooks (Resolve, Remove, Hold, ForceRemove, Erase, Mint) are implemented by no backend yet (every store leg wires them nil and its cases skip), and the wire has no removal, hold, erasure or gone/unknown resolve to bind them to. Binding the epoch hooks to the reference store behind the served env would pass, but as a property of that store, which the store legs already pin, never of the client. So the waivers stay; retentionEpochNoRoleWaiverReason's comment and reason string now say the above, in the shape #6661 gave asOfReadNoHTTPWireWaiverReason, instead of the earlier general "raw storage hooks with no publicops role" wording. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(epoch): report applied-but-unpublished epoch writes as indeterminate, bound address length, verify dup-key mints Maintainer fixup for the #6664 review (adopt-pr attempt 1, iteration 1). Major 1, applied writes reported as failed. An epoch writer whose SQL commit has landed, or may have, now wraps storage.ErrCommitIndeterminate on both legs and hands back what it applied, so nothing replays a bump that has already advanced the counter: - embedded BumpEpoch and BumpEpochCarrying return the new epoch with the error (failedBump), and MintUnderEpoch its address; - server bumps wrap the sentinel on every Dolt failure after the SQL commit, including a failed staged-set read (unpublishedBump), and so does LoseVersion; an indeterminate SQL commit returns the epoch or address it may have committed, and a definite (typed MySQL) commit error returns neither. Tests: TestEpochWritesReportAnUnpublishedCommitAsIndeterminate (new, embedded, through an afterSQLCommit seam in runTransactionWithMessage); on the server leg TestEpochWritesMarkAFailedStagedReadIndeterminate, TestEpochWritesReturnWhatAnIndeterminateSQLCommitMayHaveApplied and TestEpochWritesReturnNothingWhenTheSQLCommitFailsDefinitely, and the sentinel asserted in every publication-failure test. Major 2, address overflow. MintUnderEpochInTx refuses a pair whose address would outgrow epoch_minted_addresses.address (VARCHAR(255)) at the widest epoch the counter can reach, with a typed ErrEpochAddressTooLong, so a carry can never fail on a lineage that gained an epoch digit. Major 3, duplicate-key mint. A duplicate key from the mint's INSERT is no longer success: the row is read again and the mint stands only if this transaction sees it served; a Gone row fails the mint with ErrEpochAddressNotServed, an unreadable one with the duplicate-key error. Measured on dolt-sql-server 2.2.0, the stale-snapshot INSERT does not report a duplicate key at all: its COMMIT fails with Error 1213 and the retried mint answers ErrEpochAddressNotServed (TestAMintRacingTheMintAndLossOfItsAddressConflictsAtCommit). Minors: - absent store_epoch row: sqlmock cases pin that CurrentEpoch, StillServes, Resolve, CurrentAddressFor and LoseVersion never seed it; switching any of the five to the seeding reader now fails the test; - the server publication tests mint their address instead of writing it by hand, and TestEpochCASTestsBuildEveryAddressWithEpochAddress scans that file too (issueops_test gains //internal/storage/dolt:go_srcs as data); - stale text: the duplicate-key and carry comments, the issueops header's CompareAndSetVersionInTx citation, the embedded header's parity claim, and the 0072 header's "upserts the same row" (0072 is past the frozen floor). Declined, for the owner: - a scheme-change carry voids an older live lineage re-minted after a restore: rule (a), document and pin it, or rule (b), resolve the loser to the winner's carry, is a design call about R20-n, not a review fix; - the uow epoch fixtures' in-transaction DOLT_COMMIT('-Am'): test-only fixtures with no production caller; routing them through RunTxScoped is follow-up work. Validation: go build ./...; go vet; nogo (bazel --config=nogo) on issueops, dolt and embeddeddolt; issueops epoch tests; the dolt epoch tests against a real dolt-sql-server (56 pass) and the fake-backed publication tests; embedded epoch tests with BEADS_TEST_EMBEDDED_DOLT=1; uow epoch contract; the schema package; the address guard under bazel in the local sandbox. All 14 overlay mutants of the fixes are killed. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Jim Wordelman Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Julian Knutsen Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com> --- .../scripts/embedded-storage-test-shards.txt | 12 + .../conformance/retention_epoch_contract.go | 453 +++++-- .../versioned_history_wiring_test.go | 9 + internal/storage/dolt/BUILD.bazel | 2 + internal/storage/dolt/epoch_cas.go | 248 ++++ .../storage/dolt/epoch_cas_publish_test.go | 949 ++++++++++++++ .../dolt/expected_revision_contract_test.go | 20 +- .../dolt/retention_epoch_contract_test.go | 141 +- internal/storage/embeddeddolt/BUILD.bazel | 3 + internal/storage/embeddeddolt/epoch_cas.go | 197 +++ .../epoch_cas_publish_internal_test.go | 141 ++ .../embeddeddolt/epoch_cas_status_test.go | 562 ++++++++ .../expected_revision_contract_test.go | 20 +- .../retention_epoch_contract_test.go | 125 +- internal/storage/embeddeddolt/transaction.go | 6 + internal/storage/issueops/BUILD.bazel | 3 + internal/storage/issueops/epoch_cas.go | 680 ++++++++++ internal/storage/issueops/epoch_cas_test.go | 1141 +++++++++++++++++ internal/storage/leg_contract_wiring_test.go | 63 +- internal/storage/schema/BUILD.bazel | 2 + ...ation_0071_add_removed_restriction_test.go | 18 +- ...on_0072_add_epoch_minted_addresses_test.go | 184 +++ .../0072_add_epoch_minted_addresses.down.sql | 1 + .../0072_add_epoch_minted_addresses.up.sql | 64 + .../uow/expected_revision_contract_test.go | 20 +- .../uow/retention_epoch_contract_test.go | 205 ++- 26 files changed, 5135 insertions(+), 134 deletions(-) create mode 100644 internal/storage/dolt/epoch_cas.go create mode 100644 internal/storage/dolt/epoch_cas_publish_test.go create mode 100644 internal/storage/embeddeddolt/epoch_cas.go create mode 100644 internal/storage/embeddeddolt/epoch_cas_publish_internal_test.go create mode 100644 internal/storage/embeddeddolt/epoch_cas_status_test.go create mode 100644 internal/storage/issueops/epoch_cas.go create mode 100644 internal/storage/issueops/epoch_cas_test.go create mode 100644 internal/storage/schema/migration_0072_add_epoch_minted_addresses_test.go create mode 100644 internal/storage/schema/migrations/0072_add_epoch_minted_addresses.down.sql create mode 100644 internal/storage/schema/migrations/0072_add_epoch_minted_addresses.up.sql diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index d127ad9a6..cef22090a 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -205,6 +205,7 @@ 40 7 TestEmbeddedIssueOperationsUpdateMetadataReplaceClearsAndValidates 40 7 TestEmbeddedLenientOpenWarning_DataBehind 40 7 TestEmbeddedRelationsLeavesTheCallersRequestAlone +40 7 TestLoseVersionLeavesTheEpochTablesCommitted 40 7 TestMergeAndSettleWithStrategy_ResolvesDeclinedConflict 40 8 TestDeleterContract @@ -217,6 +218,7 @@ 40 8 TestGetIssueCommentsPageEmbedded 40 8 TestListWisps +40 9 TestBumpEpochCarryingNeitherCarriesNorServesARowFromAFutureEpoch 40 9 TestEmbeddedDependencyEditorRefusesCrossPlaneCycle 40 9 TestEmbeddedDependencyEditorSameTypeReAddIsIdempotent 40 9 TestEmbeddedDependencyEditorSnapshotsTheRequest @@ -237,6 +239,7 @@ 40 10 TestRunInTransactionPostCommitStageFailureIsIndeterminate 40 10 TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPasswordAmbient +40 11 TestABumpUnderOneStoreIDMovesTheEpochSeenThroughEveryStoreID 40 11 TestEmbeddedBatchCreateFirstVersionCarriesCreationTimeEdges 40 11 TestEmbeddedClaimerRefusalMessagesCarryTheirFragments 40 11 TestEmbeddedGuardedUpdateMintsOnceAfterItsPatches @@ -338,6 +341,7 @@ 40 19 TestRemoteVerbsPresentStoredPeerCredentials 40 19 TestRenameLabelMintsOneVersionPerTouchedBead +40 20 TestBumpEpochCarryingCarriesOnlyTheLineageHoldingTheNewestRow 40 20 TestCreateIssue 40 20 TestEmbeddedCheckedWrappersRouteTheirPreconditions 40 20 TestEmbeddedIssueOperationsCreateRoutesInfraTypesToWisps @@ -349,6 +353,7 @@ 40 20 TestHeartbeatRejectsWispEmbedded 40 20 TestIgnoredCursorHealNeverResurrectsRolledBackVersions +40 21 TestBumpEpochCarryingCarriesTheLiveRowsOfEveryStoreInTheDatabase 40 21 TestCreateRejectsCrossTableIDCollision 40 21 TestEmbeddedBatchCloserItemsAnswerToTheCloseGuards 40 21 TestEmbeddedDependencyEditorAcceptsATypeOutsideTheConstants @@ -435,6 +440,7 @@ 40 27 TestRemoteGitDataRefOnEngine 40 27 TestSweeperContract +40 28 TestBumpEpochCarryingDoesNotCarryALostLineage 40 28 TestEmbeddedBatchCloserForceBypassesOnlyClosePolicy 40 28 TestEmbeddedBatchCloserSettlesTheDependersOfWhatItClosed 40 28 TestEmbeddedClaimerClaimsAnUnassignedOpenIssueAndAnswersTheBareRow @@ -472,6 +478,7 @@ 40 30 TestLegacyTrackedIgnoredCursorUnwedgesPull 40 30 TestUpdateMergeOps_DoltEngine_TwoIssues +40 31 TestBumpEpochCarryingWritesOneCarryRowPerLiveIDNamingTheLineageRoot 40 31 TestDualWriteContract 40 31 TestEmbeddedDependencyEditorAcceptsAForeignRepoTarget 40 31 TestEmbeddedDependencyEditorMaintainsBlockedStateAcrossPlanes @@ -484,6 +491,7 @@ 40 31 TestReadyCounterContract 40 31 TestWithPeerAuth_KeyMismatchFailsClosed +40 32 TestCurrentAddressForNeverReseedsAnAbsentStoreEpochRow 40 32 TestEmbeddedClaimerGrantsALiveLeaseOnTheRowItWonAndLeavesARefusedOneAlone 40 32 TestEmbeddedDependencyEditorRefusesAMissingLocalTarget 40 32 TestEmbeddedDirtyTablesGate_BlocksReopenThenWorkingSetReconcileRecovers @@ -504,11 +512,13 @@ 40 33 TestEmbeddedRelationsResolvesTheAnchorIDExactly 40 33 TestEmbeddedSyncCommitsConfigBeforeMerge 40 33 TestEventsAutoPruneThrottleSkipsTheSecondPass +40 33 TestFirstMintLeavesTheEpochTablesCommitted 40 33 TestGetDependentRecordsEmbedded 40 33 TestPruneEventsBatchDeletesTheLowestSeqsOnly 40 33 TestTreeWalkerContract 40 34 TestAddLabel +40 34 TestBumpEpochCarryingLeavesTheEpochTablesCommitted 40 34 TestDecryptPassword_KeyMismatchNamesTheLocalKey 40 34 TestEmbeddedBlockedRecheckRecordsUnblockingWrites 40 34 TestEmbeddedDependencyEditorAttributesItsEventsToTheActor @@ -532,6 +542,7 @@ 40 35 TestReadyWorkWithMissingDeferredParentTables 40 35 TestReclaimExpiredLeaseSurvivesRestartEmbedded +40 36 TestBumpEpochCarryingFailsAndRollsBackWhenARowAlreadyExistsAtTheNewEpoch 40 36 TestCommenterContract 40 36 TestConcurrentOpenReturnsCached 40 36 TestEmbeddedBatchCloserAllRefusedBatchRecordsNoHistory @@ -554,6 +565,7 @@ 40 37 TestEmbeddedMergeAndSettleReportsOperatorConflicts 40 37 TestEmbeddedReaderListBriefDropsTheFreeFormTextAndNothingElse 40 37 TestEmbeddedReaderListNaturalNumericIDSortTrimsAfterTheFetch +40 37 TestEpochWritesReportAnUnpublishedCommitAsIndeterminate 40 37 TestIgnoredCursorHealRespectsOverrideUnderBroaderPattern 40 38 TestCreateIssues diff --git a/backend/conformance/retention_epoch_contract.go b/backend/conformance/retention_epoch_contract.go index 472b7a896..b2c0d100e 100644 --- a/backend/conformance/retention_epoch_contract.go +++ b/backend/conformance/retention_epoch_contract.go @@ -769,7 +769,21 @@ type EpochFixture struct { // Version now resolves to under the new epoch. A nil hook means this // backend cannot report that, and the case that needs it skips with // that reason. + // + // It only reads: it mints nothing and changes no state. It returns the + // address it was given when nothing carried the Version to a new one, + // and an error for an address the store does not serve. CurrentAddressFor func(ctx context.Context, storeID string, oldAddress Address) (Address, error) + + // LoseVersion records that storeID no longer serves the Version named by + // address. That address, and every address the same Version was carried + // to or from by a token-scheme change, resolves GoneReorganization from + // then on, in every later epoch: an epoch voids what the store lost, + // never what it still serves (R20-n). It is idempotent, never revives + // anything, and returns an error for an address storeID never minted. A + // nil LoseVersion means this backend cannot record a loss, and the cases + // that need one skip with that reason. + LoseVersion func(ctx context.Context, storeID string, address Address) error } // RunAnEpochBumpIsTriggeredOnlyByRestoreReinitOrSchemeChange pins R20-m: a @@ -819,94 +833,258 @@ func RunAnEpochBumpIsTriggeredOnlyByRestoreReinitOrSchemeChange(t *testing.T, ct // RunEpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed pins R20-n — "Rev7 // tightening... the clause most likely to be gotten wrong" in the -// architecture doc's own words. A prior-epoch Address goes -// GoneReorganization UNLESS the backend still serves that Version, in which -// case it must stay Live AND additionally report the Address it now -// resolves to under the new epoch. +// architecture doc's own words: an epoch voids what the store lost, never +// what it still serves. // -// StillServes is an ORACLE here, not a control: this case cannot make a real -// backend keep serving one address and drop another, since Phase 0 wires no -// real backend. It asks StillServes what the fixture (once real) decided, -// and checks the rest of the contract's promise against that decision for -// both outcomes. +// Survival is RECORDED, never inferred (gastownhall/beads#6664, bee-ghosttrack +// review 5360880888, Major 1). An earlier form of this case made a Version +// survive by minting it again in every later epoch, and treated a Version +// nobody minted again as lost. That reads loss off the absence of a mint, so +// it could not express a Version the store lost after minting it, and it +// voided every survivor that nobody happened to mint again. Here a restore or +// a destructive reinit leaves every address row alone: an address the store +// still serves stays Live at its own address with no upkeep, and the only way +// an address goes Gone is LoseVersion saying so. +// +// Each phase is a named subtest, so a backend that cannot record a loss skips +// only the phases that need LoseVersion, and a failure names its phase. func RunEpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed(t *testing.T, ctx context.Context, fixture EpochFixture) { t.Helper() - if fixture.MintUnderEpoch == nil { - t.Skip("this backend has no way to mint a Version under a known epoch (MintUnderEpoch is nil)") - } - if fixture.BumpEpoch == nil { - t.Skip("this backend has no epoch concept yet (BumpEpoch is nil)") - } - if fixture.StillServes == nil { - t.Skip("this backend cannot report whether it still serves a prior-epoch Version (StillServes is nil)") - } - if fixture.Resolve == nil { - t.Skip("this backend does not yet report retention state (Resolve is nil)") - } - if fixture.CurrentAddressFor == nil { - t.Skip("this backend cannot report a still-served Version's new Address (CurrentAddressFor is nil)") - } - store := epochStore(fixture, "voids") + epochSkipUnlessReady(t, fixture, false) - addrA, err := fixture.MintUnderEpoch(ctx, store, "record-a") - if err != nil { - t.Fatalf("MintUnderEpoch(record-a): %v", err) + // survivor-and-lost: of two Versions minted before a transition, the one + // the store loses goes Gone and the other stays Live at the address it + // already had, with no re-mint, for both triggers that leave addresses + // alone. + t.Run("survivor-and-lost", func(t *testing.T) { + if fixture.LoseVersion == nil { + t.Skip(epochLoseVersionSkip) + } + for _, trigger := range []EpochBumpTrigger{EpochBumpTriggerRestore, EpochBumpTriggerDestructiveReinit} { + t.Run(trigger.String(), func(t *testing.T) { + store := epochStore(fixture, "survivor-"+trigger.String()) + addrA := epochMintAddress(t, ctx, fixture, store, "record-a") + addrB := epochMintAddress(t, ctx, fixture, store, "record-b") + + newEpoch := epochBumpTo(t, ctx, fixture, store, trigger) + epochLoseAddress(t, ctx, fixture, store, addrB) + + epochRequireLive(t, ctx, fixture, store, addrA, newEpoch, + "record-a was never lost, so the transition leaves it served at its own address with no re-mint (R20-n's exception)") + epochRequireGone(t, ctx, fixture, store, addrB, newEpoch, + "record-b was lost, so the transition voids it (R20-n)") + + current := epochCurrentAddress(t, ctx, fixture, store, addrA) + epochRequireLive(t, ctx, fixture, store, current, newEpoch, + "the address CurrentAddressFor reports for a survivor must be served") + epochRequireLive(t, ctx, fixture, store, addrA, newEpoch, + "asking CurrentAddressFor must not retire or move the survivor's address") + + if lostCurrent, err := fixture.CurrentAddressFor(ctx, store, addrB); err == nil { + t.Errorf("CurrentAddressFor(a lost address) = %s with no error, want an error: a lost Version has no current address", lostCurrent) + } + epochRequireGone(t, ctx, fixture, store, addrB, newEpoch, + "CurrentAddressFor on a lost address must not bring it back") + }) + } + }) + + // gone-is-terminal: once an address is lost it stays lost. A later mint of + // the same id starts a new lineage under a new address, and neither that + // mint nor CurrentAddressFor revives the lost one (bee-ghosttrack's + // assertion for Major 1: a read-only lookup must not mint a replacement + // that makes a lost address look served). + t.Run("gone-is-terminal", func(t *testing.T) { + if fixture.LoseVersion == nil { + t.Skip(epochLoseVersionSkip) + } + store := epochStore(fixture, "terminal") + lost := epochMintAddress(t, ctx, fixture, store, "record-g") + epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerRestore) + epochLoseAddress(t, ctx, fixture, store, lost) + laterEpoch := epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerRestore) + + fresh := epochMintAddress(t, ctx, fixture, store, "record-g") + if fresh == lost { + t.Fatalf("MintUnderEpoch(record-g) returned %s, the address already lost: a lost address is never handed out again", lost) + } + epochRequireLive(t, ctx, fixture, store, fresh, laterEpoch, + "a fresh mint of the id is a new lineage and is served") + epochRequireGone(t, ctx, fixture, store, lost, laterEpoch, + "a later mint of the same id must not revive a lost address") + + if replacement, err := fixture.CurrentAddressFor(ctx, store, lost); err == nil { + t.Errorf("CurrentAddressFor(a lost address) = %s with no error, want an error: asking where a lost address resolves must not mint a replacement", replacement) + } + epochRequireGone(t, ctx, fixture, store, lost, laterEpoch, + "CurrentAddressFor on a lost address must not revive it") + + // Losing it a second time is a no-op that leaves the fresh lineage + // alone. + epochLoseAddress(t, ctx, fixture, store, lost) + epochRequireLive(t, ctx, fixture, store, fresh, laterEpoch, + "losing the old address again must leave the fresh lineage served") + }) + + // survival-needs-no-upkeep: with nothing lost, an address stays Live across + // any number of counter-only transitions without being minted again, and a + // later mint of the same id neither retires it nor revives anything. This + // is the phase the earlier, mint-driven form failed: it answered Gone for + // every survivor nobody minted again. + t.Run("survival-needs-no-upkeep", func(t *testing.T) { + store := epochStore(fixture, "upkeep") + kept := epochMintAddress(t, ctx, fixture, store, "record-c") + epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerRestore) + epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerDestructiveReinit) + finalEpoch := epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerRestore) + + epochRequireLive(t, ctx, fixture, store, kept, finalEpoch, + "nothing was lost, so three counter-only transitions leave the address served without a re-mint") + + fresh := epochMintAddress(t, ctx, fixture, store, "record-c") + if fresh == kept { + t.Fatalf("MintUnderEpoch(record-c) returned %s again after three bumps, want a new address for the new epoch", kept) + } + epochRequireLive(t, ctx, fixture, store, kept, finalEpoch, + "a later mint of the same id must not retire a served address") + epochRequireLive(t, ctx, fixture, store, fresh, finalEpoch, + "the fresh mint is served") + + current := epochCurrentAddress(t, ctx, fixture, store, kept) + epochRequireLive(t, ctx, fixture, store, current, finalEpoch, + "the current address of a survivor must be served") + epochRequireLive(t, ctx, fixture, store, kept, finalEpoch, + "asking for the current address must not change the survivor") + }) +} + +// RunTokenSchemeChangeCarriesLiveAddressesAndOnlyThose pins R20-n's retained +// mapping for the one trigger that changes how addresses are encoded. A +// token-scheme-change bump gives every Version the store still serves a new +// address under the new epoch and keeps the old address resolving, so a holder +// of the old address is neither stranded nor pointed at something else. A +// Version the store already lost gets no new address and stays Gone. +func RunTokenSchemeChangeCarriesLiveAddressesAndOnlyThose(t *testing.T, ctx context.Context, fixture EpochFixture) { + t.Helper() + epochSkipUnlessReady(t, fixture, true) + store := epochStore(fixture, "scheme-change") + + addrA := epochMintAddress(t, ctx, fixture, store, "record-a") + addrB := epochMintAddress(t, ctx, fixture, store, "record-b") + // record-b is lost in the epoch it was minted in, which a store may do. + epochLoseAddress(t, ctx, fixture, store, addrB) + + newEpoch := epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerTokenSchemeChange) + + epochRequireLive(t, ctx, fixture, store, addrA, newEpoch, + "record-a is still served after the scheme change, through the retained mapping") + epochRequireGone(t, ctx, fixture, store, addrB, newEpoch, + "record-b was lost before the scheme change, which must not carry it") + + carried := epochCurrentAddress(t, ctx, fixture, store, addrA) + if carried == addrA { + t.Errorf("CurrentAddressFor(%s) = the same address after a token-scheme change, want a new one: a scheme change re-encodes every served address, which is what tells a carry from a no-op", addrA) } - addrB, err := fixture.MintUnderEpoch(ctx, store, "record-b") - if err != nil { - t.Fatalf("MintUnderEpoch(record-b): %v", err) + epochRequireLive(t, ctx, fixture, store, carried, newEpoch, + "the carried address must be served under the new epoch") + epochRequireLive(t, ctx, fixture, store, addrA, newEpoch, + "the old address must keep resolving after its carried address is looked up") + + if again := epochCurrentAddress(t, ctx, fixture, store, carried); again != carried { + t.Errorf("CurrentAddressFor(%s) = %s, want the carried address itself: from the newest address the mapping is already current", carried, again) + } + if lostCurrent, err := fixture.CurrentAddressFor(ctx, store, addrB); err == nil { + t.Errorf("CurrentAddressFor(a lost address) = %s with no error, want an error: a lost Version is not carried", lostCurrent) + } +} + +// RunACarriedAddressKeepsResolvingAcrossLaterBumps pins that a carried mapping +// is recorded once and then simply persists: later counter-only transitions do +// not change it, and a second token-scheme change extends it to the newest +// address without breaking any older one. Every address of the Version points +// directly at the newest address (one hop from any member), never along a +// chain. +func RunACarriedAddressKeepsResolvingAcrossLaterBumps(t *testing.T, ctx context.Context, fixture EpochFixture) { + t.Helper() + epochSkipUnlessReady(t, fixture, false) + store := epochStore(fixture, "carried-later") + + addrA := epochMintAddress(t, ctx, fixture, store, "record-a") + epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerRestore) + epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerTokenSchemeChange) + addrA1 := epochCurrentAddress(t, ctx, fixture, store, addrA) + if addrA1 == addrA { + t.Fatalf("CurrentAddressFor(%s) = the same address after a token-scheme change, want a new one", addrA) } - newEpoch, err := fixture.BumpEpoch(ctx, store, EpochBumpTriggerRestore) - if err != nil { - t.Fatalf("BumpEpoch: %v", err) + // Two counter-only transitions, with nothing minted in between. + epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerRestore) + counterEpoch := epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerDestructiveReinit) + epochRequireLive(t, ctx, fixture, store, addrA, counterEpoch, + "the original address must still resolve after counter-only transitions") + epochRequireLive(t, ctx, fixture, store, addrA1, counterEpoch, + "the carried address must still resolve after counter-only transitions") + if got := epochCurrentAddress(t, ctx, fixture, store, addrA); got != addrA1 { + t.Errorf("CurrentAddressFor(%s) = %s, want %s: a counter-only transition does not change the mapping", addrA, got, addrA1) } - servesA, err := fixture.StillServes(ctx, store, addrA) - if err != nil { - t.Fatalf("StillServes(%s): %v", addrA, err) + // A second scheme change carries the Version again. + secondEpoch := epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerTokenSchemeChange) + addrA2 := epochCurrentAddress(t, ctx, fixture, store, addrA) + if addrA2 == addrA || addrA2 == addrA1 { + t.Fatalf("CurrentAddressFor(%s) = %s after a second scheme change, want an address different from %s and %s", addrA, addrA2, addrA, addrA1) } - servesB, err := fixture.StillServes(ctx, store, addrB) - if err != nil { - t.Fatalf("StillServes(%s): %v", addrB, err) + for _, member := range []Address{addrA, addrA1, addrA2} { + epochRequireLive(t, ctx, fixture, store, member, secondEpoch, + "every address of a carried Version must resolve after a second scheme change") } - if servesA == servesB { - t.Skip("this scenario needs one still-served and one no-longer-served address to exercise both halves of R20-n; StillServes reported the same answer for both, so this fixture gives this case nothing to contrast") + if got := epochCurrentAddress(t, ctx, fixture, store, addrA1); got != addrA2 { + t.Errorf("CurrentAddressFor(%s) = %s, want %s: every address of the Version points straight at the newest one", addrA1, got, addrA2) } - stillServed, noLongerServed := addrA, addrB - if servesB && !servesA { - stillServed, noLongerServed = addrB, addrA + finalEpoch := epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerRestore) + epochRequireLive(t, ctx, fixture, store, addrA2, finalEpoch, + "the newest carried address must still resolve after a later restore") +} + +// RunALostAddressStaysGoneAcrossASchemeChange pins that loss is terminal even +// across a token-scheme change. Losing a Version loses every address of its +// lineage at once, and a later scheme change must not carry a lost lineage +// back into service. +func RunALostAddressStaysGoneAcrossASchemeChange(t *testing.T, ctx context.Context, fixture EpochFixture) { + t.Helper() + epochSkipUnlessReady(t, fixture, true) + store := epochStore(fixture, "lost-scheme-change") + + addrA := epochMintAddress(t, ctx, fixture, store, "record-a") + epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerTokenSchemeChange) + addrA1 := epochCurrentAddress(t, ctx, fixture, store, addrA) + lostEpoch := epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerRestore) + + // One lineage, two addresses: losing either loses both. + epochLoseAddress(t, ctx, fixture, store, addrA) + epochRequireGone(t, ctx, fixture, store, addrA, lostEpoch, + "the original address of a lost Version must be Gone") + epochRequireGone(t, ctx, fixture, store, addrA1, lostEpoch, + "the carried address of a lost Version must be Gone too") + + schemeEpoch := epochBumpTo(t, ctx, fixture, store, EpochBumpTriggerTokenSchemeChange) + epochRequireGone(t, ctx, fixture, store, addrA, schemeEpoch, + "a later scheme change must not carry a lost lineage back into service") + epochRequireGone(t, ctx, fixture, store, addrA1, schemeEpoch, + "a later scheme change must not carry a lost lineage back into service") + for _, lostAddress := range []Address{addrA, addrA1} { + if current, err := fixture.CurrentAddressFor(ctx, store, lostAddress); err == nil { + t.Errorf("CurrentAddressFor(%s) = %s with no error, want an error: a lost lineage has no current address", lostAddress, current) + } } - voidedAnswer, err := fixture.Resolve(ctx, store, noLongerServed) - if err != nil { - t.Fatalf("Resolve(no-longer-served): %v", err) - } - if voidedAnswer.Restriction != RestrictionGoneReorganization { - t.Errorf("Resolve(a prior-epoch Address no longer served) = %s, want RestrictionGoneReorganization (R20-n)", voidedAnswer.Restriction) - } - if voidedAnswer.Epoch == nil { - t.Fatal("Resolve(a prior-epoch Address no longer served).Epoch = nil, want the current epoch populated (R20-n: the answer must name the current epoch, not just Restriction alone)") - } else if *voidedAnswer.Epoch != newEpoch { - t.Errorf("Resolve(a prior-epoch Address no longer served).Epoch = %d, want the current epoch %d", *voidedAnswer.Epoch, newEpoch) - } - - keptAnswer, err := fixture.Resolve(ctx, store, stillServed) - if err != nil { - t.Fatalf("Resolve(still-served): %v", err) - } - if keptAnswer.Restriction != RestrictionLive { - t.Errorf("Resolve(a prior-epoch Address the backend still serves) = %s, want RestrictionLive (R20-n's exception)", keptAnswer.Restriction) - } - - newAddr, err := fixture.CurrentAddressFor(ctx, store, stillServed) - if err != nil { - t.Fatalf("CurrentAddressFor(%s): %v", stillServed, err) - } - if newAddr == "" { - t.Error("CurrentAddressFor(a still-served prior-epoch Address) returned an empty Address, want a real one under the new epoch") + fresh := epochMintAddress(t, ctx, fixture, store, "record-a") + if fresh == addrA || fresh == addrA1 { + t.Fatalf("MintUnderEpoch(record-a) returned %s, an address of the lost lineage: a lost address is never handed out again", fresh) } + epochRequireLive(t, ctx, fixture, store, fresh, schemeEpoch, + "a fresh mint of the id after the loss is a new lineage and is served") } // --- fixture helpers ------------------------------------------------------- @@ -953,6 +1131,137 @@ func epochStore(fixture EpochFixture, tag string) string { return fixture.IssuePrefix + "-epoch-store-" + tag } +// epochLoseVersionSkip is the skip reason of every phase that needs +// EpochFixture.LoseVersion. +const epochLoseVersionSkip = "this backend cannot record that the store lost a Version (LoseVersion is nil)" + +// epochSkipUnlessReady skips the case when the fixture lacks a capability the +// epoch cases need. The cases that record a loss also need LoseVersion. The +// all-nil wiring test runs every case against an empty fixture, so each case +// must call this before it touches any of them. +func epochSkipUnlessReady(t *testing.T, fixture EpochFixture, needsLoseVersion bool) { + t.Helper() + if fixture.MintUnderEpoch == nil { + t.Skip("this backend has no way to mint a Version under a known epoch (MintUnderEpoch is nil)") + } + if fixture.BumpEpoch == nil { + t.Skip("this backend has no epoch concept yet (BumpEpoch is nil)") + } + if fixture.StillServes == nil { + t.Skip("this backend cannot report whether it still serves a prior-epoch Version (StillServes is nil)") + } + if fixture.Resolve == nil { + t.Skip("this backend does not yet report retention state (Resolve is nil)") + } + if fixture.CurrentAddressFor == nil { + t.Skip("this backend cannot report a still-served Version's new Address (CurrentAddressFor is nil)") + } + if needsLoseVersion && fixture.LoseVersion == nil { + t.Skip(epochLoseVersionSkip) + } +} + +// epochMintAddress mints id on store, failing the case if minting fails. +func epochMintAddress(t *testing.T, ctx context.Context, fixture EpochFixture, store, id string) Address { + t.Helper() + address, err := fixture.MintUnderEpoch(ctx, store, id) + if err != nil { + t.Fatalf("MintUnderEpoch(%s, %s): %v", store, id, err) + } + return address +} + +// epochBumpTo bumps store's epoch for trigger and returns the epoch it +// reports. Epochs are captured from the return value, never assumed: the +// counter is one per database, shared by every store the suite uses. +func epochBumpTo(t *testing.T, ctx context.Context, fixture EpochFixture, store string, trigger EpochBumpTrigger) int { + t.Helper() + epoch, err := fixture.BumpEpoch(ctx, store, trigger) + if err != nil { + t.Fatalf("BumpEpoch(%s, %s): %v", store, trigger, err) + } + return epoch +} + +// epochLoseAddress records that store lost the Version behind address. +func epochLoseAddress(t *testing.T, ctx context.Context, fixture EpochFixture, store string, address Address) { + t.Helper() + if err := fixture.LoseVersion(ctx, store, address); err != nil { + t.Fatalf("LoseVersion(%s, %s): %v", store, address, err) + } +} + +// epochCurrentAddress asks where address now resolves, failing the case on an +// error or an empty answer. +func epochCurrentAddress(t *testing.T, ctx context.Context, fixture EpochFixture, store string, address Address) Address { + t.Helper() + current, err := fixture.CurrentAddressFor(ctx, store, address) + if err != nil { + t.Fatalf("CurrentAddressFor(%s, %s): %v", store, address, err) + } + if current == "" { + t.Fatalf("CurrentAddressFor(%s, %s) returned an empty Address, want a real one", store, address) + } + return current +} + +// epochRequireLive asserts that address is served at epoch: StillServes is true +// and Resolve answers RestrictionLive naming that epoch. why says what the +// assertion is checking, for the failure message. +func epochRequireLive(t *testing.T, ctx context.Context, fixture EpochFixture, store string, address Address, epoch int, why string) { + t.Helper() + serves, err := fixture.StillServes(ctx, store, address) + if err != nil { + t.Fatalf("StillServes(%s): %v", address, err) + } + if !serves { + t.Errorf("StillServes(%s) = false, want true: %s", address, why) + } + answer, err := fixture.Resolve(ctx, store, address) + if err != nil { + t.Fatalf("Resolve(%s): %v", address, err) + } + if answer.Restriction != RestrictionLive { + t.Errorf("Resolve(%s) = %s, want RestrictionLive: %s", address, answer.Restriction, why) + } + epochRequireAnswerEpoch(t, answer, address, epoch, why) +} + +// epochRequireGone asserts that address is no longer served at epoch: +// StillServes is false and Resolve answers RestrictionGoneReorganization +// naming that epoch. +func epochRequireGone(t *testing.T, ctx context.Context, fixture EpochFixture, store string, address Address, epoch int, why string) { + t.Helper() + serves, err := fixture.StillServes(ctx, store, address) + if err != nil { + t.Fatalf("StillServes(%s): %v", address, err) + } + if serves { + t.Errorf("StillServes(%s) = true, want false: %s", address, why) + } + answer, err := fixture.Resolve(ctx, store, address) + if err != nil { + t.Fatalf("Resolve(%s): %v", address, err) + } + if answer.Restriction != RestrictionGoneReorganization { + t.Errorf("Resolve(%s) = %s, want RestrictionGoneReorganization: %s", address, answer.Restriction, why) + } + epochRequireAnswerEpoch(t, answer, address, epoch, why) +} + +// epochRequireAnswerEpoch asserts that a Live or Gone answer names the current +// epoch (R20-n: the answer carries the epoch, not just the Restriction). +func epochRequireAnswerEpoch(t *testing.T, answer RetentionAnswer, address Address, epoch int, why string) { + t.Helper() + if answer.Epoch == nil { + t.Errorf("Resolve(%s).Epoch = nil, want the current epoch %d: %s", address, epoch, why) + return + } + if *answer.Epoch != epoch { + t.Errorf("Resolve(%s).Epoch = %d, want the current epoch %d: %s", address, *answer.Epoch, epoch, why) + } +} + // sameRetentionAnswer compares two RetentionAnswer values by content rather // than identity, so a case can assert "this did not change" without // depending on the concrete Restriction the backend happened to pick. diff --git a/backend/conformance/versioned_history_wiring_test.go b/backend/conformance/versioned_history_wiring_test.go index 08eed2b53..b022b716e 100644 --- a/backend/conformance/versioned_history_wiring_test.go +++ b/backend/conformance/versioned_history_wiring_test.go @@ -129,5 +129,14 @@ func TestVersionedHistoryPhase0RunsInFullSkipMode(t *testing.T) { t.Run("EpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed", func(t *testing.T) { RunEpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed(t, ctx, fixture) }) + t.Run("TokenSchemeChangeCarriesLiveAddressesAndOnlyThose", func(t *testing.T) { + RunTokenSchemeChangeCarriesLiveAddressesAndOnlyThose(t, ctx, fixture) + }) + t.Run("ACarriedAddressKeepsResolvingAcrossLaterBumps", func(t *testing.T) { + RunACarriedAddressKeepsResolvingAcrossLaterBumps(t, ctx, fixture) + }) + t.Run("ALostAddressStaysGoneAcrossASchemeChange", func(t *testing.T) { + RunALostAddressStaysGoneAcrossASchemeChange(t, ctx, fixture) + }) }) } diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 4be1fec0b..773a53d6b 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -29,6 +29,7 @@ go_library( "dependency_editor.go", "edgereader.go", "ephemeral_routing.go", + "epoch_cas.go", "errors.go", "events.go", "federation.go", @@ -221,6 +222,7 @@ go_test( "edge_reader_contract_test.go", "empty_commit_test.go", "ephemeral_routing_test.go", + "epoch_cas_publish_test.go", "errors_test.go", "event_commit_boundary_test.go", "events_since_test.go", diff --git a/internal/storage/dolt/epoch_cas.go b/internal/storage/dolt/epoch_cas.go new file mode 100644 index 000000000..e455fda27 --- /dev/null +++ b/internal/storage/dolt/epoch_cas.go @@ -0,0 +1,248 @@ +package dolt + +import ( + "context" + "database/sql" + "errors" + "fmt" + + storeops "github.com/steveyegge/beads/internal/storage/issueops" +) + +// CurrentEpoch, BumpEpoch, BumpEpochCarrying, MintUnderEpoch, LoseVersion, +// StillServes, Resolve and CurrentAddressFor give this leg R20's +// epoch-transition enforcement +// (gastownhall/beads#5898 revision 9, #6136), +// backed by store_epoch (migration 0067) and epoch_minted_addresses +// (migration 0072). +// +// UNLIKE MetadataCAS, THIS ROLE HAS NO PUBLIC issueops INTERFACE: the +// conformance contract's own seven epoch hooks are bare function types (see +// backend/conformance.EpochFixture) — so there is no role accessor to +// return one of, and these are direct methods on *DoltStore instead of a +// wrapper struct. +// +// These also have no domain-repository exposure: the unit-of-work leg +// reaches internal/storage/issueops's epoch Tx functions directly rather +// than through the domain issue repository (see internal/storage/uow's own +// epoch wiring), so nothing forces these types out of issueops into the +// root storage package. Method signatures here use plain types plus +// storeops.EpochResolveResult directly; backend/conformance's own +// Address/EpochBumpTrigger/RetentionAnswer vocabulary is translated only in +// this leg's contract-test file, per internal/storage/issueops/epoch_cas.go's +// header. +// +// Method names match backend/conformance.EpochFixture's own field names +// verbatim. +// +// PUBLICATION ORDER. The four writers +// commit their SQL transaction FIRST and stage and Dolt-commit afterwards, +// never from inside the open transaction. doltAddAndCommitInTx builds the Dolt +// commit from the transaction's BEGIN-time snapshot, so under concurrent +// writers it writes every row changed in the meantime back to its old value, +// and store_epoch and epoch_minted_addresses are append-only: a row lost that +// way is never rewritten (the HAZARD block on that helper). +// +// AN ERROR RETURNED AFTER THE SQL COMMIT MUST NOT BE REPLAYED. Every error a +// writer returns once its SQL commit has landed, or may have, wraps +// ErrCommitIndeterminate, the storage-wide no-replay sentinel: a replayed bump +// advances the counter a second time. A bump returns the epoch it committed, or +// would have, with that error, and a mint its address; the caller learns which +// by reading CurrentEpoch or StillServes, never by writing again. The embedded +// leg does the same. + +// CurrentEpoch reports storeID's current epoch generation. +// +// The epoch counter is per database, not per store. storeID appears only in +// error text; one bump advances the epoch seen through every storeID that +// shares this database. +func (s *DoltStore) CurrentEpoch(ctx context.Context, storeID string) (int, error) { + var epoch int + if err := s.withReadTx(ctx, func(tx *sql.Tx) error { + var err error + epoch, err = storeops.CurrentEpochInTx(ctx, tx, storeID) + return err + }); err != nil { + return 0, err + } + return epoch, nil +} + +// BumpEpoch advances storeID's epoch generation by one for reason. +// +// A Dolt commit that fails after the SQL commit is RETURNED, where the mint +// paths below log it and carry on: a bump is rare and +// operator-driven, and its caller (the restore trigger) must not report a +// restore complete while the bump is committed but unpublished. In that case +// the bump HAS applied: the epoch returned is the new one, the row is durable +// in the working set and rides the next Dolt commit, and the error wraps +// ErrCommitIndeterminate so that nothing bumps again to retry. A SQL commit +// whose result was lost is returned the same way, with the epoch it would have +// committed (failedBump). +// +// The epoch counter is per database, not per store. storeID appears only in +// error text; one bump advances the epoch seen through every storeID that +// shares this database. +func (s *DoltStore) BumpEpoch(ctx context.Context, storeID, reason string) (int, error) { + var epoch int + if err := s.withRetryTx(ctx, func(tx *sql.Tx) error { + var err error + epoch, err = storeops.BumpEpochInTx(ctx, tx, storeID, reason) + return err + }); err != nil { + return failedBump(storeID, epoch, err) + } + if err := s.doltAddAndCommitPostTx(ctx, []string{"store_epoch"}, + fmt.Sprintf("bd: bump epoch for %s (%s)", storeID, reason)); err != nil { + return epoch, unpublishedBump(storeID, epoch, err) + } + return epoch, nil +} + +// failedBump is what a bump returns when its SQL transaction reports an error. +// An error wrapping ErrCommitIndeterminate came from the commit of a body that +// ran to completion, so the commit may have landed: the epoch that body set is +// returned with it, and the caller reads CurrentEpoch to learn which. Any other +// error means nothing landed. +func failedBump(storeID string, epoch int, err error) (int, error) { + if errors.Is(err, ErrCommitIndeterminate) { + return epoch, fmt.Errorf("epoch bump for %s may have applied (epoch %d): %w", storeID, epoch, err) + } + return 0, err +} + +// unpublishedBump is the error for a bump whose SQL commit landed and whose +// Dolt commit then failed. It wraps ErrCommitIndeterminate whatever the cause: +// doltAddAndCommitPostTx returns some failures without it (a staged-set read +// that fails, a breaker that refuses the commit), and those causes are no more +// replayable than the rest once the counter has moved. +func unpublishedBump(storeID string, epoch int, err error) error { + return fmt.Errorf("epoch bump for %s applied (epoch %d) but its Dolt commit failed: %w: %w", storeID, epoch, err, ErrCommitIndeterminate) +} + +// BumpEpochCarrying is the token-scheme-change bump: BumpEpoch, and also the +// carry of every live address in the database to the new epoch +// (storeops.BumpEpochCarryingInTx). It writes store_epoch and +// epoch_minted_addresses, so it publishes both. It surfaces a failed Dolt +// commit exactly as BumpEpoch does, and the bump has applied in that case: the +// epoch returned is the new one and the caller must not bump again to retry. +// A lost SQL commit result is returned as BumpEpoch returns it (failedBump). +// +// The epoch counter is per database, not per store. storeID appears only in +// error text; one bump advances the epoch seen through every storeID that +// shares this database. +// +// CALLERS MUST HOLD THE WORKSPACE GATE (internal/workspacegate) EXCLUSIVELY FOR +// THE BUMP, as a restore does: see storeops.BumpEpochCarryingInTx. +func (s *DoltStore) BumpEpochCarrying(ctx context.Context, storeID, reason string) (int, error) { + var epoch int + if err := s.withRetryTx(ctx, func(tx *sql.Tx) error { + var err error + epoch, err = storeops.BumpEpochCarryingInTx(ctx, tx, storeID, reason) + return err + }); err != nil { + return failedBump(storeID, epoch, err) + } + if err := s.doltAddAndCommitPostTx(ctx, []string{"store_epoch", "epoch_minted_addresses"}, + fmt.Sprintf("bd: carry addresses to epoch %d for %s (%s)", epoch, storeID, reason)); err != nil { + return epoch, unpublishedBump(storeID, epoch, err) + } + return epoch, nil +} + +// MintUnderEpoch mints id's address under storeID's current epoch. +// +// It publishes like every issue mutation, through +// runIssueOperationTxWithMessage: a Dolt commit that fails after the SQL commit +// is logged, not returned. The mint is durable in the working set and rides the +// next Dolt commit, and mints are frequent and automated, so failing one over a +// missing history commit would only turn a harmless gap into caller errors. A +// SQL commit whose result was lost is returned, wrapping +// ErrCommitIndeterminate, together with the address the mint may have +// committed; StillServes says whether it did. +// +// A mint whose transaction overlaps a BumpEpoch reads the epoch before the bump +// and commits an address minted under the epoch the bump replaced. That address +// is a Live survivor: a restore or reinit bump writes no address rows, and an +// address's status is read from its own row. TestMintOverlappingABumpLeavesALiveSurvivor +// records that interleaving, and TestAMintAfterAConcurrentLossNeverRevivesTheLostAddress +// the one that must never bring a lost address back. +func (s *DoltStore) MintUnderEpoch(ctx context.Context, storeID, id string) (string, error) { + var address string + if err := s.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) { + var err error + address, err = storeops.MintUnderEpochInTx(ctx, tx, storeID, id) + if err != nil { + return nil, "", err + } + return storeops.EpochMintDirtyTables(), + fmt.Sprintf("bd: mint %s under epoch for %s", id, storeID), nil + }); err != nil { + if errors.Is(err, ErrCommitIndeterminate) { + return address, err + } + return "", err + } + return address, nil +} + +// StillServes reports whether address is still served under storeID's +// current epoch. +func (s *DoltStore) StillServes(ctx context.Context, storeID, address string) (bool, error) { + var serves bool + if err := s.withReadTx(ctx, func(tx *sql.Tx) error { + var err error + serves, err = storeops.StillServesInTx(ctx, tx, storeID, address) + return err + }); err != nil { + return false, err + } + return serves, nil +} + +// Resolve answers R20's epoch-only restriction for address. +func (s *DoltStore) Resolve(ctx context.Context, storeID, address string) (storeops.EpochResolveResult, error) { + var result storeops.EpochResolveResult + if err := s.withReadTx(ctx, func(tx *sql.Tx) error { + var err error + result, err = storeops.ResolveEpochInTx(ctx, tx, storeID, address) + return err + }); err != nil { + return storeops.EpochResolveResult{}, err + } + return result, nil +} + +// CurrentAddressFor reports the address oldAddress now resolves to: the newest +// address of its lineage, or oldAddress itself when nothing carried it. It only +// reads, so it runs on this leg's read path and publishes nothing. +func (s *DoltStore) CurrentAddressFor(ctx context.Context, storeID, oldAddress string) (string, error) { + var address string + if err := s.withReadTx(ctx, func(tx *sql.Tx) error { + var err error + address, err = storeops.CurrentAddressForInTx(ctx, tx, storeID, oldAddress) + return err + }); err != nil { + return "", err + } + return address, nil +} + +// LoseVersion records that storeID no longer serves the Version named by +// address (storeops.LoseVersionInTx). It writes epoch_minted_addresses, so it +// publishes that table, and unlike a mint it RETURNS a failed Dolt commit: a +// loss its caller believes recorded must not be left unpublished. The loss has +// applied in that case, and the error wraps ErrCommitIndeterminate like a +// bump's, although replaying a loss is a no-op. +func (s *DoltStore) LoseVersion(ctx context.Context, storeID, address string) error { + if err := s.withRetryTx(ctx, func(tx *sql.Tx) error { + return storeops.LoseVersionInTx(ctx, tx, storeID, address) + }); err != nil { + return err + } + if err := s.doltAddAndCommitPostTx(ctx, []string{"epoch_minted_addresses"}, + fmt.Sprintf("bd: lose version %s for %s", address, storeID)); err != nil { + return fmt.Errorf("loss of %s for %s recorded but its Dolt commit failed: %w: %w", address, storeID, err, ErrCommitIndeterminate) + } + return nil +} diff --git a/internal/storage/dolt/epoch_cas_publish_test.go b/internal/storage/dolt/epoch_cas_publish_test.go new file mode 100644 index 000000000..3928b1059 --- /dev/null +++ b/internal/storage/dolt/epoch_cas_publish_test.go @@ -0,0 +1,949 @@ +package dolt + +import ( + "context" + "database/sql" + "database/sql/driver" + "errors" + "fmt" + "slices" + "strings" + "sync/atomic" + "testing" + "time" + + mysql "github.com/go-sql-driver/mysql" + + storeops "github.com/steveyegge/beads/internal/storage/issueops" +) + +// These tests pin WHERE and HOW this leg publishes the R20 epoch tables +// (store_epoch, epoch_minted_addresses) to Dolt history +// (gastownhall/beads#6664). +// +// The first group runs on the sequence-capturing driver from +// dolt_commit_ordering_test.go, so it needs no running Dolt server: the +// ordering contract and the staged-table lists are properties of the SQL +// this leg sends, not of Dolt. The tests after them run against a real server: +// the first reproduces bee-ghosttrack's own probe (dolt_status after the first +// mint), the next three apply it to the carrying bump, to a recorded loss and to +// the read-only CurrentAddressFor, and the last three choreograph how a mint +// behaves beside a concurrent bump, beside a concurrent loss, and beside another +// transaction's mint and loss of its own address on that server. + +const epochPublishStoreID = "epoch-store" +const epochPublishMintedID = "record-a" + +// epochPublishAddress is the address of epochPublishMintedID under epoch 1, the +// one the fake serves as minted. It is minted on a fake of its own rather than +// written by hand: the fake serves a row at any address, so a hand-written one +// would go stale unnoticed if the encoding moved +// (TestEpochCASTestsBuildEveryAddressWithEpochAddress in issueops scans this +// file for one). +func epochPublishAddress() string { + f := newEpochFake() + db := sql.OpenDB(f.connector) + defer func() { _ = db.Close() }() + address, err := (&DoltStore{db: db}).MintUnderEpoch(context.Background(), epochPublishStoreID, epochPublishMintedID) + if err != nil { + panic(fmt.Sprintf("mint %s on a fresh epoch fake: %v", epochPublishMintedID, err)) + } + return address +} + +// epochFake is a *DoltStore's driver for the epoch methods: it serves the +// reads the shared epoch Tx functions make, keeps the singleton's epoch moving +// when the bump's UPDATE runs, and can refuse every DOLT_COMMIT. +type epochFake struct { + rec *seqRecorder + connector *seqConnector + epoch atomic.Int64 + // publishErr, when set before the call under test, fails every + // CALL DOLT_COMMIT: the SQL transaction is fine, only publication is not. + publishErr error + // stagedReadErr, when set, fails the staged-set read stageAndCommit makes + // between DOLT_ADD and DOLT_COMMIT, a failure it returns without + // ErrCommitIndeterminate because nothing has reached Dolt history yet. + stagedReadErr error +} + +func newEpochFake() *epochFake { + f := &epochFake{rec: &seqRecorder{}} + f.epoch.Store(1) + f.connector = &seqConnector{rec: f.rec} + f.connector.queryErr = func(query string) error { + switch { + case strings.Contains(query, "UPDATE store_epoch"): + f.epoch.Add(1) + case strings.Contains(query, "DOLT_COMMIT") && f.publishErr != nil: + return f.publishErr + case strings.Contains(query, "FROM dolt_status WHERE staged") && f.stagedReadErr != nil: + return f.stagedReadErr + } + return nil + } + f.connector.rows = func(query string) (driver.Rows, bool) { + switch { + case strings.Contains(query, "FROM store_epoch"): + return &valRows{cols: []string{"epoch"}, vals: [][]driver.Value{{f.epoch.Load()}}}, true + case strings.Contains(query, "FROM epoch_minted_addresses WHERE address"): + // Every address is already minted, so CurrentAddressFor has an + // old address to look up and MintUnderEpoch has nothing to insert. + // A read that asks for the survival columns gets a Live row nothing + // carried; a read that does not gets the three-column shape. + if strings.Contains(query, "carried_from") { + return &valRows{ + cols: []string{"store_id", "minted_id", "minted_epoch", "carried_from", "gone_at_epoch"}, + vals: [][]driver.Value{{epochPublishStoreID, epochPublishMintedID, int64(1), nil, nil}}, + }, true + } + return &valRows{ + cols: []string{"store_id", "minted_id", "minted_epoch"}, + vals: [][]driver.Value{{epochPublishStoreID, epochPublishMintedID, int64(1)}}, + }, true + } + return nil, false + } + return f +} + +// epochArgConnector is the sequence-capturing connector plus the table each +// DOLT_ADD stages, which the sequence recorder drops. +type epochArgConnector struct{ *seqConnector } + +func (c epochArgConnector) Connect(ctx context.Context) (driver.Conn, error) { + conn, err := c.seqConnector.Connect(ctx) + if err != nil { + return nil, err + } + return &epochArgConn{seqConn: conn.(*seqConn)}, nil +} + +type epochArgConn struct{ *seqConn } + +func (c *epochArgConn) QueryContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Rows, error) { + if strings.Contains(query, "DOLT_ADD") && len(args) == 1 { + c.parent.rec.add(c.id, fmt.Sprintf("STAGE %v", args[0].Value)) + } + return c.seqConn.QueryContext(ctx, query, args) +} + +// epochPublishCall is one epoch method that writes. +type epochPublishCall struct { + name string + call func(ctx context.Context, s *DoltStore) error + // staged is the exact set of tables the call stages for its Dolt commit, + // sorted. + staged []string + // logsPublishFailure is true for the call that carries on past a failed Dolt + // commit; every other call returns it. + logsPublishFailure bool +} + +func epochPublishCalls() []epochPublishCall { + return []epochPublishCall{ + { + name: "BumpEpoch", + staged: []string{"store_epoch"}, + call: func(ctx context.Context, s *DoltStore) error { + _, err := s.BumpEpoch(ctx, epochPublishStoreID, "restore") + return err + }, + }, + { + name: "BumpEpochCarrying", + staged: []string{"epoch_minted_addresses", "store_epoch"}, + call: func(ctx context.Context, s *DoltStore) error { + _, err := s.BumpEpochCarrying(ctx, epochPublishStoreID, "token-scheme-change") + return err + }, + }, + { + name: "MintUnderEpoch", + staged: []string{"epoch_minted_addresses", "store_epoch"}, + logsPublishFailure: true, + call: func(ctx context.Context, s *DoltStore) error { + _, err := s.MintUnderEpoch(ctx, epochPublishStoreID, epochPublishMintedID) + return err + }, + }, + { + name: "LoseVersion", + staged: []string{"epoch_minted_addresses"}, + call: func(ctx context.Context, s *DoltStore) error { + return s.LoseVersion(ctx, epochPublishStoreID, epochPublishAddress()) + }, + }, + } +} + +func epochPublishIndex(events []string, pred func(string) bool) int { + for i, ev := range events { + if pred(ev) { + return i + } + } + return -1 +} + +// TestEpochWritesPublishToDoltOnlyAfterTheSQLCommit is the ordering contract +// of dolt_commit_ordering_test.go applied to the epoch methods. Staging and +// Dolt-committing store_epoch or epoch_minted_addresses inside the still-open +// SQL transaction is the lost-update hazard documented on doltAddAndCommitInTx +// (LatentLabsSpace/NEXUS#92), and it costs more here than on issues: both +// tables are append-only, so a row reverted to its BEGIN-time value is never +// rewritten. +func TestEpochWritesPublishToDoltOnlyAfterTheSQLCommit(t *testing.T) { + for _, tc := range epochPublishCalls() { + t.Run(tc.name, func(t *testing.T) { + f := newEpochFake() + db := sql.OpenDB(f.connector) + defer func() { _ = db.Close() }() + + if err := tc.call(context.Background(), &DoltStore{db: db}); err != nil { + t.Fatalf("%s: %v", tc.name, err) + } + + events := f.rec.snapshot() + commitIdx := epochPublishIndex(events, func(ev string) bool { return ev == "COMMIT" }) + addIdx := epochPublishIndex(events, func(ev string) bool { return strings.Contains(ev, "DOLT_ADD") }) + doltCommitIdx := epochPublishIndex(events, func(ev string) bool { return strings.Contains(ev, "DOLT_COMMIT") }) + if commitIdx == -1 || addIdx == -1 || doltCommitIdx == -1 { + t.Fatalf("%s: want a SQL COMMIT, a DOLT_ADD and a DOLT_COMMIT; events: %v", tc.name, events) + } + if addIdx < commitIdx { + t.Errorf("%s: DOLT_ADD ran INSIDE the open transaction (index %d < COMMIT index %d) — lost-update ordering hazard; events: %v", + tc.name, addIdx, commitIdx, events) + } + if doltCommitIdx < commitIdx { + t.Errorf("%s: DOLT_COMMIT ran INSIDE the open transaction (index %d < COMMIT index %d) — lost-update ordering hazard; events: %v", + tc.name, doltCommitIdx, commitIdx, events) + } + }) + } +} + +// TestEpochWritesStageExactlyTheTablesTheyChange is the server-leg half of +// gastownhall/beads#6664 Major 2 (bee-ghosttrack, review 5360880888): the +// first mint on a store lazily inserts store_epoch's singleton +// (ensureStoreEpochRow), so a mint that stages only epoch_minted_addresses +// leaves the epoch row modified and uncommitted, and the Dolt commit holds +// minted rows without the epoch row they depend on. It states the rule for +// every writer, with exact sets: a restore or reinit bump changes only the +// counter, the carrying bump and a mint change both tables, and a recorded loss +// changes only address rows. A writer that stages less leaves rows behind in +// the working set, and one that stages more commits rows it did not write. +func TestEpochWritesStageExactlyTheTablesTheyChange(t *testing.T) { + for _, tc := range epochPublishCalls() { + t.Run(tc.name, func(t *testing.T) { + f := newEpochFake() + db := sql.OpenDB(epochArgConnector{f.connector}) + defer func() { _ = db.Close() }() + + if err := tc.call(context.Background(), &DoltStore{db: db}); err != nil { + t.Fatalf("%s: %v", tc.name, err) + } + + var staged []string + for _, ev := range f.rec.snapshot() { + if table, ok := strings.CutPrefix(ev, "STAGE "); ok { + staged = append(staged, table) + } + } + slices.Sort(staged) + staged = slices.Compact(staged) + if !slices.Equal(staged, tc.staged) { + t.Errorf("%s staged %v, want exactly %v (Major 2: a table the call writes must be committed with it, and one it does not write must not be)", tc.name, staged, tc.staged) + } + }) + } +} + +// TestBumpEpochSurfacesAPublicationFailureAfterTheSQLCommit pins how a bump +// reports a failed publication: a bump whose SQL transaction +// committed but whose Dolt commit failed must tell its caller, because a +// restore trigger must not report a restore complete over an unpublished +// bump. The bump itself must survive: it is durable in the working set, so a +// failed history commit is an error to report, never a reason to roll back. +func TestBumpEpochSurfacesAPublicationFailureAfterTheSQLCommit(t *testing.T) { + f := newEpochFake() + f.publishErr = errors.New("dolt commit refused") + db := sql.OpenDB(f.connector) + defer func() { _ = db.Close() }() + + epoch, err := (&DoltStore{db: db}).BumpEpoch(context.Background(), epochPublishStoreID, "restore") + + requirePublicationFailureReported(t, "BumpEpoch", f, err, f.publishErr) + if epoch != 2 { + t.Errorf("BumpEpoch returned epoch %d with the publication error, want 2: the bump applied, the error only says its Dolt commit did not", epoch) + } +} + +// requirePublicationFailureReported checks what a writer that reports a failed +// Dolt commit must show once its SQL transaction committed: the transaction was +// committed and never rolled back, and the error it returned wraps the failure +// (cause) and ErrCommitIndeterminate, so that nothing replays a write that has +// already applied. +func requirePublicationFailureReported(t *testing.T, name string, f *epochFake, err, cause error) { + t.Helper() + events := f.rec.snapshot() + if !slices.Contains(events, "COMMIT") || slices.Contains(events, "ROLLBACK") { + t.Errorf("%s: a failed Dolt commit must not undo the write: want the SQL transaction COMMITTED and never rolled back; events: %v", name, events) + } + if err == nil { + t.Fatalf("%s = nil error although its Dolt commit failed: the caller would report success for a write that is committed but unpublished; events: %v", name, events) + } + if !errors.Is(err, cause) { + t.Errorf("%s error = %v, want it to wrap the Dolt commit failure %v", name, err, cause) + } + if !errors.Is(err, ErrCommitIndeterminate) { + t.Errorf("%s error = %v, want it to wrap ErrCommitIndeterminate: the write has applied, and a caller that takes the error for a write that did not would replay it", name, err) + } +} + +// TestBumpEpochCarryingSurfacesAPublicationFailureAfterTheSQLCommit holds the +// carrying bump to the rule BumpEpoch is held to above. It writes the address +// rows of every live Version as well as the counter, and a token-scheme change +// must not report success over a bump that is committed but unpublished. The +// bump applied, so the epoch comes back with the error and the caller must not +// bump again to retry. +func TestBumpEpochCarryingSurfacesAPublicationFailureAfterTheSQLCommit(t *testing.T) { + f := newEpochFake() + f.publishErr = errors.New("dolt commit refused") + db := sql.OpenDB(f.connector) + defer func() { _ = db.Close() }() + + epoch, err := (&DoltStore{db: db}).BumpEpochCarrying(context.Background(), epochPublishStoreID, "token-scheme-change") + + requirePublicationFailureReported(t, "BumpEpochCarrying", f, err, f.publishErr) + if epoch != 2 { + t.Errorf("BumpEpochCarrying returned epoch %d with the publication error, want 2: the bump applied, the error only says its Dolt commit did not", epoch) + } +} + +// TestLoseVersionSurfacesAPublicationFailureAfterTheSQLCommit pins the same rule +// for a recorded loss: a loss its caller believes recorded must not be left +// unpublished. Unlike a mint, a loss is rare and deliberate, so a failed Dolt +// commit is an error to report, never one to log. +func TestLoseVersionSurfacesAPublicationFailureAfterTheSQLCommit(t *testing.T) { + f := newEpochFake() + f.publishErr = errors.New("dolt commit refused") + db := sql.OpenDB(f.connector) + defer func() { _ = db.Close() }() + + err := (&DoltStore{db: db}).LoseVersion(context.Background(), epochPublishStoreID, epochPublishAddress()) + + requirePublicationFailureReported(t, "LoseVersion", f, err, f.publishErr) +} + +// TestEpochWritesMarkAFailedStagedReadIndeterminate holds the writers that +// return a failed publication to the no-replay rule for the failure +// stageAndCommit itself returns as a plain error: its staged-set read before +// DOLT_COMMIT. Nothing has reached Dolt history at that point, but the write's +// SQL commit has landed, so a bump replayed on that error would advance the +// counter a second time. The error must say so as a failed DOLT_COMMIT does. +func TestEpochWritesMarkAFailedStagedReadIndeterminate(t *testing.T) { + for _, tc := range epochPublishCalls() { + if tc.logsPublishFailure { + continue // a mint logs a failed publication and returns nil + } + t.Run(tc.name, func(t *testing.T) { + f := newEpochFake() + f.stagedReadErr = errors.New("staged-set read failed") + db := sql.OpenDB(f.connector) + defer func() { _ = db.Close() }() + + err := tc.call(context.Background(), &DoltStore{db: db}) + + requirePublicationFailureReported(t, tc.name, f, err, f.stagedReadErr) + }) + } +} + +// TestEpochWritesReturnWhatAnIndeterminateSQLCommitMayHaveApplied pins what each +// writer hands back when its SQL commit's result is lost (ErrCommitIndeterminate, +// which withRetryTx never retries): the commit may have landed, so a bump +// returns the epoch it would have committed and a mint its address, as a bump +// does when only its Dolt commit fails. The caller then reads CurrentEpoch or +// StillServes to learn which; a zero value would read as nothing applied and +// invite the replay the sentinel forbids. +func TestEpochWritesReturnWhatAnIndeterminateSQLCommitMayHaveApplied(t *testing.T) { + ctx := context.Background() + open := func(t *testing.T) (*DoltStore, *epochFake) { + t.Helper() + f := newEpochFake() + f.connector.commitErr = func() error { return testConnectionLoss } + db := sql.OpenDB(f.connector) + t.Cleanup(func() { _ = db.Close() }) + return &DoltStore{db: db}, f + } + requireIndeterminate := func(t *testing.T, name string, f *epochFake, err error) { + t.Helper() + if !errors.Is(err, ErrCommitIndeterminate) || !errors.Is(err, testConnectionLoss) { + t.Fatalf("%s = %v, want the lost commit response wrapped in ErrCommitIndeterminate", name, err) + } + if events := f.rec.snapshot(); slices.ContainsFunc(events, func(ev string) bool { return strings.Contains(ev, "DOLT_COMMIT") }) { + t.Errorf("%s published a SQL commit whose result was lost; events: %v", name, events) + } + } + + t.Run("BumpEpoch", func(t *testing.T) { + s, f := open(t) + epoch, err := s.BumpEpoch(ctx, epochPublishStoreID, "restore") + requireIndeterminate(t, "BumpEpoch", f, err) + if epoch != 2 { + t.Errorf("BumpEpoch returned epoch %d with an indeterminate commit, want 2, the epoch it may have committed", epoch) + } + }) + t.Run("BumpEpochCarrying", func(t *testing.T) { + s, f := open(t) + epoch, err := s.BumpEpochCarrying(ctx, epochPublishStoreID, "token-scheme-change") + requireIndeterminate(t, "BumpEpochCarrying", f, err) + if epoch != 2 { + t.Errorf("BumpEpochCarrying returned epoch %d with an indeterminate commit, want 2, the epoch it may have committed", epoch) + } + }) + t.Run("MintUnderEpoch", func(t *testing.T) { + s, f := open(t) + address, err := s.MintUnderEpoch(ctx, epochPublishStoreID, epochPublishMintedID) + requireIndeterminate(t, "MintUnderEpoch", f, err) + if want := epochPublishAddress(); address != want { + t.Errorf("MintUnderEpoch returned address %q with an indeterminate commit, want %s, the address it may have committed", address, want) + } + }) + t.Run("LoseVersion", func(t *testing.T) { + s, f := open(t) + requireIndeterminate(t, "LoseVersion", f, s.LoseVersion(ctx, epochPublishStoreID, epochPublishAddress())) + }) +} + +// TestEpochWritesReturnNothingWhenTheSQLCommitFailsDefinitely is the other side +// of that rule: a commit error the server answered with (a typed MySQL error, +// which withRetryTx neither retries nor marks indeterminate) means nothing +// landed. Each body ran to the end and computed an epoch or an address, and +// neither may come back with the error. +func TestEpochWritesReturnNothingWhenTheSQLCommitFailsDefinitely(t *testing.T) { + ctx := context.Background() + refused := &mysql.MySQLError{Number: 1105, Message: "commit refused"} + open := func(t *testing.T) *DoltStore { + t.Helper() + f := newEpochFake() + f.connector.commitErr = func() error { return refused } + db := sql.OpenDB(f.connector) + t.Cleanup(func() { _ = db.Close() }) + return &DoltStore{db: db} + } + requireDefinite := func(t *testing.T, name string, err error) { + t.Helper() + if !errors.Is(err, refused) || errors.Is(err, ErrCommitIndeterminate) { + t.Fatalf("%s = %v, want the refused commit, not marked ErrCommitIndeterminate", name, err) + } + } + + t.Run("BumpEpoch", func(t *testing.T) { + epoch, err := open(t).BumpEpoch(ctx, epochPublishStoreID, "restore") + requireDefinite(t, "BumpEpoch", err) + if epoch != 0 { + t.Errorf("BumpEpoch returned epoch %d with a refused commit, want 0: nothing applied", epoch) + } + }) + t.Run("BumpEpochCarrying", func(t *testing.T) { + epoch, err := open(t).BumpEpochCarrying(ctx, epochPublishStoreID, "token-scheme-change") + requireDefinite(t, "BumpEpochCarrying", err) + if epoch != 0 { + t.Errorf("BumpEpochCarrying returned epoch %d with a refused commit, want 0: nothing applied", epoch) + } + }) + t.Run("MintUnderEpoch", func(t *testing.T) { + address, err := open(t).MintUnderEpoch(ctx, epochPublishStoreID, epochPublishMintedID) + requireDefinite(t, "MintUnderEpoch", err) + if address != "" { + t.Errorf("MintUnderEpoch returned address %q with a refused commit, want none: nothing applied", address) + } + }) +} + +// TestMintPathsKeepTheirRowsWhenPublicationFails is the other half of that +// rule: swallow-and-log stays acceptable for the frequent, automated mint +// paths, so a failed Dolt commit does not fail a mint whose SQL transaction +// committed (the runIssueOperationTxWithMessage contract: a nil error means +// the data is durable in the working set). +func TestMintPathsKeepTheirRowsWhenPublicationFails(t *testing.T) { + for _, tc := range epochPublishCalls() { + if !tc.logsPublishFailure { + continue // the bumps and the loss surface the failure instead, see above + } + t.Run(tc.name, func(t *testing.T) { + f := newEpochFake() + f.publishErr = errors.New("dolt commit refused") + db := sql.OpenDB(f.connector) + defer func() { _ = db.Close() }() + + err := tc.call(context.Background(), &DoltStore{db: db}) + + events := f.rec.snapshot() + if !slices.Contains(events, "COMMIT") || slices.Contains(events, "ROLLBACK") { + t.Errorf("%s: a failed Dolt commit must not roll the mint back: want the SQL transaction COMMITTED; events: %v", tc.name, events) + } + if err != nil { + t.Errorf("%s = %v, want nil: the mint's data is durable, so a failed history commit is logged, not returned", tc.name, err) + } + }) + } +} + +// TestCurrentAddressForWritesAndPublishesNothing pins that CurrentAddressFor is a +// read on this leg: it answers from the address's own row, the carry rows and +// the counter, and sends no INSERT, UPDATE or DELETE and no DOLT_ADD or +// DOLT_COMMIT. The read path opens an ordinary transaction and rolls it back, so +// a write it made would vanish without an error; the statements it sends are the +// only place a stray write shows. +func TestCurrentAddressForWritesAndPublishesNothing(t *testing.T) { + f := newEpochFake() + db := sql.OpenDB(f.connector) + defer func() { _ = db.Close() }() + + address := epochPublishAddress() + current, err := (&DoltStore{db: db}).CurrentAddressFor(context.Background(), epochPublishStoreID, address) + if err != nil { + t.Fatalf("CurrentAddressFor: %v", err) + } + if current != address { + t.Errorf("CurrentAddressFor(%s) = %s, want the address itself: the fake serves it as a Live row nothing carried", address, current) + } + + for _, ev := range f.rec.snapshot() { + statement := strings.ToUpper(strings.TrimSpace(ev)) + for _, verb := range []string{"INSERT", "UPDATE", "DELETE", "REPLACE"} { + if strings.HasPrefix(statement, verb) { + t.Errorf("CurrentAddressFor sent %q: a lookup must not write", ev) + } + } + if strings.Contains(statement, "DOLT_ADD") || strings.Contains(statement, "DOLT_COMMIT") { + t.Errorf("CurrentAddressFor sent %q: a lookup has nothing to publish to Dolt history", ev) + } + } +} + +// epochTablesInWorkingSet lists the R20 epoch tables that still appear in +// dolt_status, staged or not: after a write has been published, neither may. +func epochTablesInWorkingSet(status *DoltStatus) []string { + var dirty []string + for _, entry := range append(slices.Clone(status.Staged), status.Unstaged...) { + if entry.Table == "store_epoch" || entry.Table == "epoch_minted_addresses" { + dirty = append(dirty, fmt.Sprintf("%s (%s)", entry.Table, entry.Status)) + } + } + return dirty +} + +// TestFirstMintLeavesTheEpochTablesCommitted reproduces bee-ghosttrack's probe +// for gastownhall/beads#6664 Major 2 on a real server: Status() after the +// first mint on a fresh store read unstaged=[{store_epoch modified}]. +func TestFirstMintLeavesTheEpochTablesCommitted(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + // A fresh branch: store_epoch has no row, so this mint is what seeds it. + if _, err := store.MintUnderEpoch(ctx, epochPublishStoreID, epochPublishMintedID); err != nil { + t.Fatalf("MintUnderEpoch on a fresh store: %v", err) + } + + status, err := store.Status(ctx) + if err != nil { + t.Fatalf("Status: %v", err) + } + if dirty := epochTablesInWorkingSet(status); len(dirty) > 0 { + t.Errorf("after the first mint the working set still holds %v: the mint's Dolt commit must include the singleton row it seeded", dirty) + } +} + +// TestBumpEpochCarryingLeavesTheEpochTablesCommitted applies the same probe to +// the carrying bump, which writes two tables: the counter, and one carry row per +// live address in epoch_minted_addresses. Publishing only the counter would +// leave those rows modified and uncommitted in the working set. +func TestBumpEpochCarryingLeavesTheEpochTablesCommitted(t *testing.T) { + skipIfNoServer(t) + ctx, cancel := testContext(t) + defer cancel() + store := newMultiConnectionEpochStore(t, ctx, "epoch_carry_published") + + if _, err := store.MintUnderEpoch(ctx, epochPublishStoreID, epochPublishMintedID); err != nil { + t.Fatalf("MintUnderEpoch: %v", err) + } + newEpoch, err := store.BumpEpochCarrying(ctx, epochPublishStoreID, "token-scheme-change") + if err != nil { + t.Fatalf("BumpEpochCarrying: %v", err) + } + + // The carry row is what makes the publication more than the counter: without + // it this test would pass for a bump that wrote nothing but store_epoch. + var carryRows int + if err := store.db.QueryRowContext(ctx, + "SELECT COUNT(*) FROM epoch_minted_addresses WHERE minted_id = ? AND minted_epoch = ?", + epochPublishMintedID, newEpoch, + ).Scan(&carryRows); err != nil { + t.Fatalf("count the carry rows: %v", err) + } + if carryRows != 1 { + t.Fatalf("epoch_minted_addresses holds %d rows for %s at epoch %d after the carrying bump, want exactly 1 carry row", carryRows, epochPublishMintedID, newEpoch) + } + + status, err := store.Status(ctx) + if err != nil { + t.Fatalf("Status: %v", err) + } + if dirty := epochTablesInWorkingSet(status); len(dirty) > 0 { + t.Errorf("after the carrying bump the working set still holds %v: its Dolt commit must include the carry rows", dirty) + } +} + +// TestLoseVersionLeavesTheEpochTablesCommitted applies the probe to a recorded +// loss, which changes only epoch_minted_addresses: the row it marks must be in +// the Dolt commit, not left modified in the working set. +func TestLoseVersionLeavesTheEpochTablesCommitted(t *testing.T) { + skipIfNoServer(t) + ctx, cancel := testContext(t) + defer cancel() + store := newMultiConnectionEpochStore(t, ctx, "epoch_loss_published") + + address, err := store.MintUnderEpoch(ctx, epochPublishStoreID, epochPublishMintedID) + if err != nil { + t.Fatalf("MintUnderEpoch: %v", err) + } + if err := store.LoseVersion(ctx, epochPublishStoreID, address); err != nil { + t.Fatalf("LoseVersion: %v", err) + } + serves, err := store.StillServes(ctx, epochPublishStoreID, address) + if err != nil { + t.Fatalf("StillServes: %v", err) + } + if serves { + t.Fatalf("StillServes(%s) = true after LoseVersion, want false: the loss must have taken effect for the working-set check below to mean anything", address) + } + + status, err := store.Status(ctx) + if err != nil { + t.Fatalf("Status: %v", err) + } + if dirty := epochTablesInWorkingSet(status); len(dirty) > 0 { + t.Errorf("after LoseVersion the working set still holds %v: its Dolt commit must include the row it marked", dirty) + } +} + +// TestCurrentAddressForLeavesAnAbsentStoreEpochRowAbsent is what remains of the +// check that CurrentAddressFor committed a re-seeded singleton row. A store +// whose singleton row is absent is legitimate state — migration 0067 starts the +// table empty and treats "no row" as epoch 1 — and CurrentAddressFor used to +// re-seed it through ensureStoreEpochRow exactly as a first mint does. It only +// reads now: it answers from that default, leaves the singleton absent, and +// leaves nothing in the working set to commit. +func TestCurrentAddressForLeavesAnAbsentStoreEpochRowAbsent(t *testing.T) { + skipIfNoServer(t) + ctx, cancel := testContext(t) + defer cancel() + store := newMultiConnectionEpochStore(t, ctx, "epoch_lookup_absent_row") + + address, err := store.MintUnderEpoch(ctx, epochPublishStoreID, epochPublishMintedID) + if err != nil { + t.Fatalf("MintUnderEpoch: %v", err) + } + // Settle everything so far, then leave the singleton row absent with a + // clean working set. + if _, err := store.CommitAll(ctx, "test: settle the first mint"); err != nil { + t.Fatalf("CommitAll: %v", err) + } + if _, err := store.db.ExecContext(ctx, "DELETE FROM store_epoch"); err != nil { + t.Fatalf("delete the singleton row: %v", err) + } + if _, err := store.CommitAll(ctx, "test: absent singleton row"); err != nil { + t.Fatalf("CommitAll: %v", err) + } + + current, err := store.CurrentAddressFor(ctx, epochPublishStoreID, address) + if err != nil { + t.Fatalf("CurrentAddressFor: %v", err) + } + if current != address { + t.Errorf("CurrentAddressFor(%s) = %s, want the address itself: nothing carried it, and a lookup must not mint a replacement", address, current) + } + + var singletonRows int + if err := store.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM store_epoch").Scan(&singletonRows); err != nil { + t.Fatalf("count the store_epoch rows: %v", err) + } + if singletonRows != 0 { + t.Errorf("store_epoch holds %d rows after CurrentAddressFor, want 0: a lookup must not re-seed the singleton", singletonRows) + } + + status, err := store.Status(ctx) + if err != nil { + t.Fatalf("Status: %v", err) + } + if dirty := epochTablesInWorkingSet(status); len(dirty) > 0 { + t.Errorf("after CurrentAddressFor the working set still holds %v: a lookup must leave nothing to commit", dirty) + } +} + +// newMultiConnectionEpochStore opens a store on its own database with several +// connections to one branch, as in production server mode. setupTestStore pins +// a single connection, which cannot overlap two transactions. A store of its own +// also never queues on the shared test database's schema-init lock, whose +// five-second acquire timeout runs out when many parallel tests set up at once +// on a loaded host, so the publication probes that need a database of their own +// for no other reason use it too. +func newMultiConnectionEpochStore(t *testing.T, ctx context.Context, name string) *DoltStore { + t.Helper() + store, err := New(ctx, &Config{ + Path: t.TempDir(), + ServerHost: "127.0.0.1", + ServerPort: testServerPort, + Database: fmt.Sprintf("%s_%d", name, time.Now().UnixNano()), + MaxOpenConns: 4, + CreateIfMissing: true, + CommitterName: "test", + CommitterEmail: "test@example.com", + }) + if err != nil { + t.Fatalf("create store: %v", err) + } + t.Cleanup(func() { _ = store.Close() }) + return store +} + +// TestMintOverlappingABumpLeavesALiveSurvivor measures, on a real server, an +// outcome bee-ghosttrack marked "Unverified" (gastownhall/beads#6664, review +// 5360880888): a mint whose transaction overlaps a BumpEpoch reads the epoch +// before the bump, does not conflict with it (the two write different tables), +// and commits an address minted under the epoch the bump has just replaced. +// +// That address is a Live survivor, and that is the right answer. A restore or +// destructive-reinit bump writes no address rows: it advances only the counter, +// and an address's status is read from its own row, which says only that it was +// minted in an epoch that has happened. So an address minted before the bump is +// still served after it, however the two interleave, exactly as if the two +// calls had run back to back. (Before survival was recorded, this interleaving +// committed an address that already read GoneReorganization, because the bump +// voided every address nobody minted again; that outcome no longer exists.) +// +// The engine does not prevent the overlap, and this test depends on that. +// Measured: SELECT ... FOR UPDATE and LOCK IN SHARE MODE take no lock, and a +// write that changes no value (UPDATE store_epoch SET epoch = epoch matches 0 +// rows) is not a write to the merge. Only a write that CHANGES a cell the bump +// also changes conflicts (Error 1213), and a mint has no cell of its own to +// change. That is also why the token-scheme-change bump, the one bump that reads +// address rows, needs its caller to hold the workspace gate exclusively, as a +// restore does: a mint cannot be made to conflict with it. +// +// If this test fails because the bump timed out or the mint's commit +// conflicted, the engine now prevents the overlap: revisit that requirement and +// this test together. +func TestMintOverlappingABumpLeavesALiveSurvivor(t *testing.T) { + skipIfNoServer(t) + ctx, cancel := testContext(t) + defer cancel() + + store := newMultiConnectionEpochStore(t, ctx, "epoch_overlap") + + // Seed the singleton so the mint below only reads the epoch. + if _, err := store.MintUnderEpoch(ctx, epochPublishStoreID, "seed"); err != nil { + t.Fatalf("seed mint: %v", err) + } + + mintTx, err := store.db.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("begin mint transaction: %v", err) + } + defer func() { _ = mintTx.Rollback() }() + address, err := storeops.MintUnderEpochInTx(ctx, mintTx, epochPublishStoreID, epochPublishMintedID) + if err != nil { + t.Fatalf("mint inside the open transaction: %v", err) + } + + // The bump runs on another connection while the mint transaction is open. + bumpCtx, cancelBump := context.WithTimeout(ctx, 10*time.Second) + defer cancelBump() + bumpedEpoch, err := store.BumpEpoch(bumpCtx, epochPublishStoreID, "restore") + if err != nil { + t.Fatalf("BumpEpoch beside an open mint transaction: %v: the engine now holds the bump back, so the overlap this test documents is prevented", err) + } + if err := mintTx.Commit(); err != nil { + t.Fatalf("mint commit after a concurrent bump: %v: the engine now conflicts the mint, so the overlap this test documents is prevented", err) + } + + serves, err := store.StillServes(ctx, epochPublishStoreID, address) + if err != nil { + t.Fatalf("StillServes: %v", err) + } + if !serves { + t.Errorf("address %s, minted before a bump that committed first, is no longer served: a restore or reinit bump writes no address rows, so the address must survive it", address) + } + answer, err := store.Resolve(ctx, epochPublishStoreID, address) + if err != nil { + t.Fatalf("Resolve: %v", err) + } + if answer.Restriction != storeops.EpochRestrictionLive { + t.Errorf("Resolve(%s) = %v, want EpochRestrictionLive: the address survived the bump", address, answer.Restriction) + } + if answer.Epoch == nil { + t.Errorf("Resolve(%s).Epoch = nil, want the current epoch %d", address, bumpedEpoch) + } else if *answer.Epoch != bumpedEpoch { + t.Errorf("Resolve(%s).Epoch = %d, want the current epoch %d", address, *answer.Epoch, bumpedEpoch) + } +} + +// TestAMintAfterAConcurrentLossNeverRevivesTheLostAddress choreographs, with no +// timing in it, the interleaving that could bring a lost address back: one +// transaction reads the address Live, another commits the loss of it, and the +// first then mints the same id and commits. +// +// The first transaction read the address before the loss committed, so its +// snapshot holds a Live row and the mint finds nothing to do. A mint never +// writes an existing row (it inserts a root row for a new address and otherwise +// touches nothing), so the first transaction commits no write at all and has +// nothing to conflict with the loss. The loss is the only change to the row, so +// the address ends Gone: no interleaving of a mint and a loss may revive it. +// +// If this test fails because the loss was held back, or because the first +// transaction saw the loss, the engine changed the premise (an open transaction +// reads its own snapshot, and a commit that wrote nothing conflicts with +// nothing) and the interleaving needs another look. +func TestAMintAfterAConcurrentLossNeverRevivesTheLostAddress(t *testing.T) { + skipIfNoServer(t) + ctx, cancel := testContext(t) + defer cancel() + + store := newMultiConnectionEpochStore(t, ctx, "epoch_loss_overlap") + + address, err := store.MintUnderEpoch(ctx, epochPublishStoreID, epochPublishMintedID) + if err != nil { + t.Fatalf("seed mint: %v", err) + } + + // The first transaction reads the address Live and stays open. + mintTx, err := store.db.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("begin the first transaction: %v", err) + } + defer func() { _ = mintTx.Rollback() }() + serves, err := storeops.StillServesInTx(ctx, mintTx, epochPublishStoreID, address) + if err != nil { + t.Fatalf("StillServesInTx inside the first transaction: %v", err) + } + if !serves { + t.Fatalf("address %s reads as not served inside the first transaction before any loss: the seed mint did not leave it Live", address) + } + + // The loss commits on another connection while the first transaction is open. + lossCtx, cancelLoss := context.WithTimeout(ctx, 10*time.Second) + defer cancelLoss() + if err := store.LoseVersion(lossCtx, epochPublishStoreID, address); err != nil { + t.Fatalf("LoseVersion beside an open transaction: %v (if the engine now holds the loss back, the interleaving this test choreographs is prevented)", err) + } + + // The first transaction now mints the same id, which is the same address. + minted, err := storeops.MintUnderEpochInTx(ctx, mintTx, epochPublishStoreID, epochPublishMintedID) + if err != nil { + t.Fatalf("mint inside the first transaction after the loss committed: %v: the transaction should still read the address Live from its own snapshot, and if it now sees the loss this test no longer exercises the interleaving it choreographs", err) + } + if minted != address { + t.Fatalf("mint returned %s, want the same address %s: minting the same id under the same epoch reproduces its address", minted, address) + } + if err := mintTx.Commit(); err != nil { + t.Fatalf("commit of the first transaction: %v: a mint must not write an existing row, so it has nothing to conflict with the loss", err) + } + + serves, err = store.StillServes(ctx, epochPublishStoreID, address) + if err != nil { + t.Fatalf("StillServes: %v", err) + } + if serves { + t.Errorf("address %s is served again after a mint overlapped its loss: no interleaving may revive a lost address", address) + } + answer, err := store.Resolve(ctx, epochPublishStoreID, address) + if err != nil { + t.Fatalf("Resolve: %v", err) + } + if answer.Restriction != storeops.EpochRestrictionGoneReorganization { + t.Errorf("Resolve(%s) = %v, want EpochRestrictionGoneReorganization: the loss is the only change to the row", address, answer.Restriction) + } +} + +// TestAMintRacingTheMintAndLossOfItsAddressConflictsAtCommit measures what +// upsertEpochMintedAddressInTx says about Dolt. One transaction reads before +// the address exists; another mints the address and loses it, and both commit; +// the first then mints the same id. Its snapshot holds no row at the address, +// so it inserts one, and the INSERT does not fail as a duplicate key: Dolt +// checks the key against the transaction's snapshot. The two inserts conflict at +// COMMIT instead (Error 1213), which withRetryTx answers by running the mint +// again, and run again against the committed row, the mint refuses the address +// (ErrEpochAddressNotServed) rather than reviving a lost one. +// +// If the first transaction's INSERT fails as a duplicate key, the engine now +// checks keys past the snapshot, and the re-read in +// upsertEpochMintedAddressInTx answers this race instead of the commit; if the +// commit succeeds, the engine merged the two rows, and the race needs another +// look. +func TestAMintRacingTheMintAndLossOfItsAddressConflictsAtCommit(t *testing.T) { + skipIfNoServer(t) + ctx, cancel := testContext(t) + defer cancel() + + store := newMultiConnectionEpochStore(t, ctx, "epoch_mint_race") + + // Seed the singleton so the first transaction's mint only reads the epoch. + if _, err := store.MintUnderEpoch(ctx, epochPublishStoreID, "seed"); err != nil { + t.Fatalf("seed mint: %v", err) + } + + // The first transaction reads before the address exists and stays open. + mintTx, err := store.db.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("begin the first transaction: %v", err) + } + defer func() { _ = mintTx.Rollback() }() + if _, err := storeops.CurrentEpochInTx(ctx, mintTx, epochPublishStoreID); err != nil { + t.Fatalf("CurrentEpochInTx inside the first transaction: %v", err) + } + + // Another connection mints the address and loses it while the first is open. + otherCtx, cancelOther := context.WithTimeout(ctx, 10*time.Second) + defer cancelOther() + address, err := store.MintUnderEpoch(otherCtx, epochPublishStoreID, epochPublishMintedID) + if err != nil { + t.Fatalf("MintUnderEpoch beside an open transaction: %v", err) + } + if err := store.LoseVersion(otherCtx, epochPublishStoreID, address); err != nil { + t.Fatalf("LoseVersion beside an open transaction: %v", err) + } + + // The first transaction mints the same id, which is the same address. + minted, err := storeops.MintUnderEpochInTx(ctx, mintTx, epochPublishStoreID, epochPublishMintedID) + if err != nil { + t.Fatalf("mint inside the first transaction: %v: the INSERT now sees a row committed after the transaction's snapshot, so the duplicate-key re-read answers this race, not the commit", err) + } + if minted != address { + t.Fatalf("mint returned %s, want the same address %s: minting the same id under the same epoch reproduces its address", minted, address) + } + err = mintTx.Commit() + if err == nil { + t.Fatalf("the first transaction committed its insert of %s over the committed, lost row: the engine merged the two inserts, and a lost address may come back", address) + } + if !isSerializationError(err) { + t.Fatalf("commit of the first transaction = %v, want a serialization failure (Error 1213), the conflict withRetryTx runs the mint again on", err) + } + + // Run again, as withRetryTx runs it, the mint reads the committed row. + if _, err := store.MintUnderEpoch(ctx, epochPublishStoreID, epochPublishMintedID); !errors.Is(err, storeops.ErrEpochAddressNotServed) { + t.Errorf("mint of %s run again after the conflict = %v, want ErrEpochAddressNotServed: the committed row is lost", epochPublishMintedID, err) + } + serves, err := store.StillServes(ctx, epochPublishStoreID, address) + if err != nil { + t.Fatalf("StillServes: %v", err) + } + if serves { + t.Errorf("address %s is served again after a mint raced its mint and loss: no interleaving may revive a lost address", address) + } +} diff --git a/internal/storage/dolt/expected_revision_contract_test.go b/internal/storage/dolt/expected_revision_contract_test.go index ef76db9b1..1ab82264e 100644 --- a/internal/storage/dolt/expected_revision_contract_test.go +++ b/internal/storage/dolt/expected_revision_contract_test.go @@ -8,11 +8,21 @@ import ( ) // TestExpectedRevisionContract wires this leg into the R16/R17 -// expected-revision contract. Phase 0 leaves every hook nil in every -// backend's fixture kit (architecture §12), so each case below skips by -// name; this file exists so TestEveryLegWiresEveryRoleContract counts this -// leg, and so the cases start running for real the moment this leg's fixture -// kit grows a non-nil hook. +// expected-revision contract. Every hook is nil (architecture §12's Phase 0 +// default), so each case below skips by name; this file exists so +// TestEveryLegWiresEveryRoleContract counts this leg, and so the cases +// start running for real the moment this leg grows a real +// CompareAndSetVersion (gastownhall/beads#6358). +// +// This leg previously wired every hook to an honest "not implemented" stub +// instead of leaving them nil, so each case ran for real and +// failed for that documented reason rather than skipping. That premise — +// that a later slice would give this leg its CompareAndSetVersion — went +// stale once review moved the real implementation to +// gastownhall/beads#6358 instead; a stub whose failure message blames a +// bead that will never land it is worse than an honest skip, so this file +// reverts to nil hooks (gastownhall/beads#6664, bee-ghosttrack review +// 5268699223). func TestExpectedRevisionContract(t *testing.T) { ctx := context.Background() fixture := conformance.ExpectedRevisionFixture{IssuePrefix: "erev"} diff --git a/internal/storage/dolt/retention_epoch_contract_test.go b/internal/storage/dolt/retention_epoch_contract_test.go index ce3569454..83e3a35bc 100644 --- a/internal/storage/dolt/retention_epoch_contract_test.go +++ b/internal/storage/dolt/retention_epoch_contract_test.go @@ -5,6 +5,7 @@ import ( "testing" "github.com/steveyegge/beads/backend/conformance" + storeops "github.com/steveyegge/beads/internal/storage/issueops" ) // TestRetentionContract wires this leg into the R20 retention contract. @@ -57,11 +58,20 @@ func TestRetentionContract(t *testing.T) { }) } -// TestEpochContract wires this leg into the R20 epoch contract. Same Phase 0 -// all-nil state as TestRetentionContract above. +// TestEpochContract runs the R20 epoch contract (gastownhall/beads#5898 +// revision 9, #6136) against the server-backed +// store, which reaches internal/storage/issueops's epoch Tx functions +// through this leg's own retrying write transaction or read transaction +// (DoltStore.BumpEpoch/BumpEpochCarrying/MintUnderEpoch/LoseVersion and +// DoltStore.CurrentEpoch/StillServes/Resolve/CurrentAddressFor) — see +// internal/storage/dolt/epoch_cas.go. +// +// All three legs run that one shared body, so this is not an independent +// vote on the design — it is the check on THIS leg's wrapper and this file's +// own conformance-vocabulary translation. func TestEpochContract(t *testing.T) { - ctx := context.Background() - fixture := conformance.EpochFixture{IssuePrefix: "epch"} + fixture, ctx, cleanup := newEpochDoltFixture(t, "epch") + defer cleanup() t.Run("AnEpochBumpIsTriggeredOnlyByRestoreReinitOrSchemeChange", func(t *testing.T) { conformance.RunAnEpochBumpIsTriggeredOnlyByRestoreReinitOrSchemeChange(t, ctx, fixture) @@ -69,4 +79,127 @@ func TestEpochContract(t *testing.T) { t.Run("EpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed", func(t *testing.T) { conformance.RunEpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed(t, ctx, fixture) }) + t.Run("TokenSchemeChangeCarriesLiveAddressesAndOnlyThose", func(t *testing.T) { + conformance.RunTokenSchemeChangeCarriesLiveAddressesAndOnlyThose(t, ctx, fixture) + }) + t.Run("ACarriedAddressKeepsResolvingAcrossLaterBumps", func(t *testing.T) { + conformance.RunACarriedAddressKeepsResolvingAcrossLaterBumps(t, ctx, fixture) + }) + t.Run("ALostAddressStaysGoneAcrossASchemeChange", func(t *testing.T) { + conformance.RunALostAddressStaysGoneAcrossASchemeChange(t, ctx, fixture) + }) +} + +// newEpochDoltFixture wires this leg's *DoltStore into the R20 epoch +// contract. Like newExpectedRevisionDoltFixture, this fixture needs none of +// roleFixtureKit's CreateIssue/CreateWisp/QueryScalar/CountHistory hooks — +// it has no fields for them — so it does not route through +// newDoltRoleFixtureKit at all; IssuePrefix is set directly, the same value +// the kit would have set verbatim. +func newEpochDoltFixture(t *testing.T, prefix string) (conformance.EpochFixture, context.Context, func()) { + t.Helper() + skipIfNoServer(t) + // A database of its own, not setupTestStore's branch on the shared one. The + // shared database's schema-init lock has a five-second acquire timeout, and a + // setup that queues behind another parallel test's runs out of it under load. + // The context is built after the open, so the open cannot drain its deadline. + openCtx, cancelOpen := testContext(t) + store := newMultiConnectionEpochStore(t, openCtx, "epoch_contract") + cancelOpen() + ctx, cancel := testContext(t) + return conformance.EpochFixture{ + IssuePrefix: prefix, + CurrentEpoch: epochDoltCurrentEpoch(store), + BumpEpoch: epochDoltBumpEpoch(store), + MintUnderEpoch: epochDoltMintUnderEpoch(store), + StillServes: epochDoltStillServes(store), + Resolve: epochDoltResolve(store), + CurrentAddressFor: epochDoltCurrentAddressFor(store), + LoseVersion: epochDoltLoseVersion(store), + }, ctx, cancel +} + +func epochDoltCurrentEpoch(store *DoltStore) func(ctx context.Context, storeID string) (int, error) { + return func(ctx context.Context, storeID string) (int, error) { + return store.CurrentEpoch(ctx, storeID) + } +} + +// epochDoltBumpEpoch dispatches on the typed trigger: a token-scheme change is +// the one bump that also carries every live address to the new epoch, and a +// restore or destructive reinit advances only the counter. The dispatch lives +// in this file, the translation boundary, because the shared body has no +// switch on the reason string. +func epochDoltBumpEpoch(store *DoltStore) func(ctx context.Context, storeID string, trigger conformance.EpochBumpTrigger) (int, error) { + return func(ctx context.Context, storeID string, trigger conformance.EpochBumpTrigger) (int, error) { + if trigger == conformance.EpochBumpTriggerTokenSchemeChange { + return store.BumpEpochCarrying(ctx, storeID, trigger.String()) + } + return store.BumpEpoch(ctx, storeID, trigger.String()) + } +} + +func epochDoltMintUnderEpoch(store *DoltStore) func(ctx context.Context, storeID, id string) (conformance.Address, error) { + return func(ctx context.Context, storeID, id string) (conformance.Address, error) { + address, err := store.MintUnderEpoch(ctx, storeID, id) + if err != nil { + return "", err + } + return conformance.Address(address), nil + } +} + +func epochDoltStillServes(store *DoltStore) func(ctx context.Context, storeID string, address conformance.Address) (bool, error) { + return func(ctx context.Context, storeID string, address conformance.Address) (bool, error) { + return store.StillServes(ctx, storeID, string(address)) + } +} + +func epochDoltResolve(store *DoltStore) func(ctx context.Context, storeID string, address conformance.Address) (conformance.RetentionAnswer, error) { + return func(ctx context.Context, storeID string, address conformance.Address) (conformance.RetentionAnswer, error) { + result, err := store.Resolve(ctx, storeID, string(address)) + if err != nil { + return conformance.RetentionAnswer{}, err + } + return epochResolveResultToConformance(result), nil + } +} + +func epochDoltCurrentAddressFor(store *DoltStore) func(ctx context.Context, storeID string, oldAddress conformance.Address) (conformance.Address, error) { + return func(ctx context.Context, storeID string, oldAddress conformance.Address) (conformance.Address, error) { + address, err := store.CurrentAddressFor(ctx, storeID, string(oldAddress)) + if err != nil { + return "", err + } + return conformance.Address(address), nil + } +} + +func epochDoltLoseVersion(store *DoltStore) func(ctx context.Context, storeID string, address conformance.Address) error { + return func(ctx context.Context, storeID string, address conformance.Address) error { + return store.LoseVersion(ctx, storeID, string(address)) + } +} + +// epochRestrictionToConformance translates this leg's local +// storeops.EpochRestriction into backend/conformance's own Restriction +// vocabulary — the contract-test file is the translation boundary, per +// epoch_cas.go's package doc. +func epochRestrictionToConformance(r storeops.EpochRestriction) conformance.Restriction { + switch r { + case storeops.EpochRestrictionLive: + return conformance.RestrictionLive + case storeops.EpochRestrictionGoneReorganization: + return conformance.RestrictionGoneReorganization + default: + return conformance.RestrictionUnknown + } +} + +func epochResolveResultToConformance(r storeops.EpochResolveResult) conformance.RetentionAnswer { + return conformance.RetentionAnswer{ + Restriction: epochRestrictionToConformance(r.Restriction), + ProducingStore: r.ProducingStore, + Epoch: r.Epoch, + } } diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index c7744f5a8..0391b56e9 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -26,6 +26,7 @@ go_library( "dependency_editor.go", "detection.go", "edgereader.go", + "epoch_cas.go", "federation.go", "get_issue.go", "graph_counter.go", @@ -150,6 +151,8 @@ go_test( "dualwrite_history_contract_test.go", "dualwrite_participation_generation_contract_test.go", "edge_reader_contract_test.go", + "epoch_cas_publish_internal_test.go", + "epoch_cas_status_test.go", "events_since_test.go", "expected_revision_contract_test.go", "external_dependencies_test.go", diff --git a/internal/storage/embeddeddolt/epoch_cas.go b/internal/storage/embeddeddolt/epoch_cas.go new file mode 100644 index 000000000..abc551087 --- /dev/null +++ b/internal/storage/embeddeddolt/epoch_cas.go @@ -0,0 +1,197 @@ +//go:build cgo + +package embeddeddolt + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "github.com/steveyegge/beads/internal/storage" + storeops "github.com/steveyegge/beads/internal/storage/issueops" +) + +// CurrentEpoch, BumpEpoch, BumpEpochCarrying, MintUnderEpoch, LoseVersion, +// StillServes, Resolve and CurrentAddressFor give this leg R20's +// epoch-transition enforcement +// (gastownhall/beads#5898 revision 9, #6136), +// backed by store_epoch (migration 0067) and epoch_minted_addresses +// (migration 0072). See internal/storage/dolt/epoch_cas.go's matching +// header comment for why these are direct methods on *EmbeddedDoltStore +// rather than a wrapper struct or root-storage-package types. +// +// The writers publish after their SQL transaction commits, and a failed Dolt +// commit is returned (runTransactionWithMessage), so all four meet the +// ordering the server leg is held to. They also return what the server leg +// returns once the SQL commit has landed or may have: runTransactionWithMessage +// wraps every such error in storage.ErrCommitIndeterminate, and a bump hands +// back the epoch its body set (failedBump) and a mint its address, never the +// zero value that would read as nothing applied. Unlike the server leg's mint, +// which logs a failed Dolt commit, this mint returns it, as every write on this +// leg does. A mint must still stage store_epoch (storeops.EpochMintDirtyTables): +// its first use inserts the singleton row. + +// CurrentEpoch reports storeID's current epoch generation. +// +// The epoch counter is per database, not per store. storeID appears only in +// error text; one bump advances the epoch seen through every storeID that +// shares this database. +func (s *EmbeddedDoltStore) CurrentEpoch(ctx context.Context, storeID string) (int, error) { + var epoch int + if err := s.withConn(ctx, false, func(tx *sql.Tx) error { + var err error + epoch, err = storeops.CurrentEpochInTx(ctx, tx, storeID) + return err + }); err != nil { + return 0, err + } + return epoch, nil +} + +// BumpEpoch advances storeID's epoch generation by one for reason. An error +// wrapping storage.ErrCommitIndeterminate comes back with the epoch the bump +// committed, or may have: the caller must not bump again to retry. +// +// The epoch counter is per database, not per store. storeID appears only in +// error text; one bump advances the epoch seen through every storeID that +// shares this database. +func (s *EmbeddedDoltStore) BumpEpoch(ctx context.Context, storeID, reason string) (int, error) { + var epoch int + if err := s.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) { + var err error + epoch, err = storeops.BumpEpochInTx(ctx, tx, storeID, reason) + if err != nil { + return nil, "", err + } + return storeops.ChangedTables{"store_epoch": true}, + fmt.Sprintf("bd: bump epoch for %s (%s)", storeID, reason), nil + }); err != nil { + return failedBump(storeID, epoch, err) + } + return epoch, nil +} + +// failedBump is what a bump returns when it fails. An error wrapping +// storage.ErrCommitIndeterminate comes from a body that ran to completion and a +// SQL commit that landed or may have (runTransactionWithMessage), so the epoch +// that body set is returned with it, and the caller reads CurrentEpoch to learn +// which. Any other error means nothing landed. +func failedBump(storeID string, epoch int, err error) (int, error) { + if errors.Is(err, storage.ErrCommitIndeterminate) { + return epoch, fmt.Errorf("epoch bump for %s may have applied (epoch %d): %w", storeID, epoch, err) + } + return 0, err +} + +// BumpEpochCarrying is the token-scheme-change bump: BumpEpoch, and also the +// carry of every live address in the database to the new epoch +// (storeops.BumpEpochCarryingInTx). It writes store_epoch and +// epoch_minted_addresses, so it publishes both. +// +// The epoch counter is per database, not per store. storeID appears only in +// error text; one bump advances the epoch seen through every storeID that +// shares this database. +// +// It fails as BumpEpoch does (failedBump). +// +// CALLERS MUST HOLD THE WORKSPACE GATE (internal/workspacegate) EXCLUSIVELY FOR +// THE BUMP, as a restore does: see storeops.BumpEpochCarryingInTx. +func (s *EmbeddedDoltStore) BumpEpochCarrying(ctx context.Context, storeID, reason string) (int, error) { + var epoch int + if err := s.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) { + var err error + epoch, err = storeops.BumpEpochCarryingInTx(ctx, tx, storeID, reason) + if err != nil { + return nil, "", err + } + return storeops.ChangedTables{"store_epoch": true, "epoch_minted_addresses": true}, + fmt.Sprintf("bd: carry addresses to epoch %d for %s (%s)", epoch, storeID, reason), nil + }); err != nil { + return failedBump(storeID, epoch, err) + } + return epoch, nil +} + +// MintUnderEpoch mints id's address under storeID's current epoch. An error +// wrapping storage.ErrCommitIndeterminate comes back with the address the mint +// committed, or may have; StillServes says whether it did. +// +// Embedded transactions serialize (see recheckBlockedAfterCommit), so a mint +// cannot overlap a BumpEpoch here the way it can on the server leg (see +// TestMintOverlappingABumpLeavesALiveSurvivor in the dolt package). +func (s *EmbeddedDoltStore) MintUnderEpoch(ctx context.Context, storeID, id string) (string, error) { + var address string + if err := s.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) { + var err error + address, err = storeops.MintUnderEpochInTx(ctx, tx, storeID, id) + if err != nil { + return nil, "", err + } + return storeops.EpochMintDirtyTables(), + fmt.Sprintf("bd: mint %s under epoch for %s", id, storeID), nil + }); err != nil { + if errors.Is(err, storage.ErrCommitIndeterminate) { + return address, err + } + return "", err + } + return address, nil +} + +// StillServes reports whether address is still served under storeID's +// current epoch. +func (s *EmbeddedDoltStore) StillServes(ctx context.Context, storeID, address string) (bool, error) { + var serves bool + if err := s.withConn(ctx, false, func(tx *sql.Tx) error { + var err error + serves, err = storeops.StillServesInTx(ctx, tx, storeID, address) + return err + }); err != nil { + return false, err + } + return serves, nil +} + +// Resolve answers R20's epoch-only restriction for address. +func (s *EmbeddedDoltStore) Resolve(ctx context.Context, storeID, address string) (storeops.EpochResolveResult, error) { + var result storeops.EpochResolveResult + if err := s.withConn(ctx, false, func(tx *sql.Tx) error { + var err error + result, err = storeops.ResolveEpochInTx(ctx, tx, storeID, address) + return err + }); err != nil { + return storeops.EpochResolveResult{}, err + } + return result, nil +} + +// CurrentAddressFor reports the address oldAddress now resolves to: the newest +// address of its lineage, or oldAddress itself when nothing carried it. It only +// reads, so it runs on a read-only connection and publishes nothing. +func (s *EmbeddedDoltStore) CurrentAddressFor(ctx context.Context, storeID, oldAddress string) (string, error) { + var address string + if err := s.withConn(ctx, false, func(tx *sql.Tx) error { + var err error + address, err = storeops.CurrentAddressForInTx(ctx, tx, storeID, oldAddress) + return err + }); err != nil { + return "", err + } + return address, nil +} + +// LoseVersion records that storeID no longer serves the Version named by +// address (storeops.LoseVersionInTx). It writes epoch_minted_addresses, so it +// publishes that table, and a failed Dolt commit is returned, wrapping +// storage.ErrCommitIndeterminate: a loss its caller believes recorded must not +// be left unpublished. +func (s *EmbeddedDoltStore) LoseVersion(ctx context.Context, storeID, address string) error { + return s.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) { + if err := storeops.LoseVersionInTx(ctx, tx, storeID, address); err != nil { + return nil, "", err + } + return storeops.ChangedTables{"epoch_minted_addresses": true}, + fmt.Sprintf("bd: lose version %s for %s", address, storeID), nil + }) +} diff --git a/internal/storage/embeddeddolt/epoch_cas_publish_internal_test.go b/internal/storage/embeddeddolt/epoch_cas_publish_internal_test.go new file mode 100644 index 000000000..96623da36 --- /dev/null +++ b/internal/storage/embeddeddolt/epoch_cas_publish_internal_test.go @@ -0,0 +1,141 @@ +//go:build cgo + +package embeddeddolt + +import ( + "context" + "errors" + "path/filepath" + "testing" + + "github.com/steveyegge/beads/internal/storage" +) + +// These tests are this leg's half of the publication-failure rule for the R20 +// epoch writers (gastownhall/beads#6664); the server leg's half is +// internal/storage/dolt/epoch_cas_publish_test.go. A write whose SQL commit has +// landed and whose Dolt commit then fails must wrap +// storage.ErrCommitIndeterminate and hand back what it applied, so that nothing +// replays a bump that has already advanced the counter. + +// withFailedPublication runs write with every Dolt commit after its SQL commit +// refused: afterSQLCommit marks store read-only in that window, so the rows have +// landed when withMutatingDBConn refuses to stage and commit them. +func withFailedPublication(t *testing.T, store *EmbeddedDoltStore, write func()) { + t.Helper() + saved := afterSQLCommit + afterSQLCommit = func() { store.readOnly = true } + defer func() { + afterSQLCommit = saved + store.readOnly = false + }() + write() +} + +func TestEpochWritesReportAnUnpublishedCommitAsIndeterminate(t *testing.T) { + skipUnlessEmbeddedDolt(t) + ctx := context.Background() + store, err := Open(ctx, filepath.Join(t.TempDir(), ".beads"), "epochpub", "main") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + const storeID = "epochpub-store" + + requireUnpublished := func(t *testing.T, name string, err error) { + t.Helper() + if !errors.Is(err, ErrReadOnly) { + t.Fatalf("%s = %v, want the refused Dolt commit (ErrReadOnly)", name, err) + } + if !errors.Is(err, storage.ErrCommitIndeterminate) { + t.Fatalf("%s = %v, want it to wrap storage.ErrCommitIndeterminate: the write has applied, and a caller that takes the error for a write that did not would replay it", name, err) + } + } + currentEpoch := func(t *testing.T) int { + t.Helper() + epoch, err := store.CurrentEpoch(ctx, storeID) + if err != nil { + t.Fatalf("CurrentEpoch: %v", err) + } + return epoch + } + stillServes := func(t *testing.T, address string) bool { + t.Helper() + serves, err := store.StillServes(ctx, storeID, address) + if err != nil { + t.Fatalf("StillServes(%s): %v", address, err) + } + return serves + } + + for _, tc := range []struct { + name string + bump func(ctx context.Context, storeID, reason string) (int, error) + }{ + {name: "BumpEpoch", bump: store.BumpEpoch}, + {name: "BumpEpochCarrying", bump: store.BumpEpochCarrying}, + } { + t.Run(tc.name, func(t *testing.T) { + before := currentEpoch(t) + var epoch int + var err error + withFailedPublication(t, store, func() { + epoch, err = tc.bump(ctx, storeID, "publication fails") + }) + requireUnpublished(t, tc.name, err) + if epoch != before+1 { + t.Errorf("%s returned epoch %d with the publication error, want %d: the bump applied, the error only says its Dolt commit did not", tc.name, epoch, before+1) + } + if got := currentEpoch(t); got != before+1 { + t.Errorf("CurrentEpoch after an unpublished %s = %d, want %d", tc.name, got, before+1) + } + }) + } + + t.Run("MintUnderEpoch", func(t *testing.T) { + var address string + var err error + withFailedPublication(t, store, func() { + address, err = store.MintUnderEpoch(ctx, storeID, "record-a") + }) + requireUnpublished(t, "MintUnderEpoch", err) + if address == "" { + t.Fatal("MintUnderEpoch returned no address with the publication error, want the address it minted") + } + if !stillServes(t, address) { + t.Errorf("StillServes(%s) = false after an unpublished mint, want the minted address served", address) + } + }) + + t.Run("LoseVersion", func(t *testing.T) { + address, err := store.MintUnderEpoch(ctx, storeID, "record-b") + if err != nil { + t.Fatalf("MintUnderEpoch: %v", err) + } + withFailedPublication(t, store, func() { + err = store.LoseVersion(ctx, storeID, address) + }) + requireUnpublished(t, "LoseVersion", err) + if stillServes(t, address) { + t.Errorf("StillServes(%s) = true after an unpublished loss, want the loss applied", address) + } + }) + + // A write refused before its SQL commit applied nothing, so its error must + // not claim it may have. + t.Run("a bump refused before its SQL commit", func(t *testing.T) { + before := currentEpoch(t) + store.readOnly = true + epoch, err := store.BumpEpoch(ctx, storeID, "refused") + store.readOnly = false + if !errors.Is(err, ErrReadOnly) || errors.Is(err, storage.ErrCommitIndeterminate) { + t.Fatalf("BumpEpoch on a read-only store = %v, want ErrReadOnly, not marked storage.ErrCommitIndeterminate", err) + } + if epoch != 0 { + t.Errorf("BumpEpoch on a read-only store returned epoch %d, want 0", epoch) + } + if got := currentEpoch(t); got != before { + t.Errorf("CurrentEpoch after a refused bump = %d, want %d", got, before) + } + }) +} diff --git a/internal/storage/embeddeddolt/epoch_cas_status_test.go b/internal/storage/embeddeddolt/epoch_cas_status_test.go new file mode 100644 index 000000000..ac28e12a5 --- /dev/null +++ b/internal/storage/embeddeddolt/epoch_cas_status_test.go @@ -0,0 +1,562 @@ +//go:build cgo + +package embeddeddolt_test + +import ( + "context" + "database/sql" + "fmt" + "slices" + "testing" + + "github.com/steveyegge/beads/internal/storage" + storeops "github.com/steveyegge/beads/internal/storage/issueops" +) + +// These tests are the embedded-leg half of gastownhall/beads#6664 Major 2 +// (bee-ghosttrack, review 5360880888). +// MintUnderEpoch and CurrentAddressFor staged only epoch_minted_addresses, +// while ensureStoreEpochRow lazily inserts store_epoch's singleton row on +// first use, so that row stayed modified and uncommitted. The embedded leg +// never published inside the transaction as the server leg did; it is not free +// of this omission. +// +// The tests after those run the rest of the epoch write surface on the real +// engine: the token-scheme-change carry (BumpEpochCarrying), the explicit loss +// (LoseVersion), and CurrentAddressFor, which only reads. They check what the +// statement-level unit tests cannot: set semantics across rows, a bump that +// rolls back, and what the working set holds once each write is published. + +// epochTablesInWorkingSet lists the R20 epoch tables that still appear in +// dolt_status, staged or not: after a write has been published, neither may. +func epochTablesInWorkingSet(status *storage.Status) []string { + var dirty []string + for _, entry := range append(slices.Clone(status.Staged), status.Unstaged...) { + if entry.Table == "store_epoch" || entry.Table == "epoch_minted_addresses" { + dirty = append(dirty, fmt.Sprintf("%s (%s)", entry.Table, entry.Status)) + } + } + return dirty +} + +// TestFirstMintLeavesTheEpochTablesCommitted reproduces bee-ghosttrack's probe +// (Status() after the first mint on a fresh store read +// unstaged=[{store_epoch modified}]) on the embedded leg. +func TestFirstMintLeavesTheEpochTablesCommitted(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + + // A pristine store: store_epoch has no row, so this mint is what seeds it. + if _, err := te.store.MintUnderEpoch(ctx, "epoch-store", "record-a"); err != nil { + t.Fatalf("MintUnderEpoch on a fresh store: %v", err) + } + + status, err := te.store.Status(ctx) + if err != nil { + t.Fatalf("Status: %v", err) + } + if dirty := epochTablesInWorkingSet(status); len(dirty) > 0 { + t.Errorf("after the first mint the working set still holds %v: the mint's Dolt commit must include the singleton row it seeded", dirty) + } +} + +// TestCurrentAddressForNeverReseedsAnAbsentStoreEpochRow is the successor of the +// test that pinned CurrentAddressFor committing a store_epoch row it re-seeded. +// CurrentAddressFor no longer writes anything (gastownhall/beads#6664, +// bee-ghosttrack review 5360880888, Major 1: a lookup must not mint), so the +// same setup now pins the opposite. A store whose singleton row is absent is +// legitimate state — migration 0067 starts the table empty and treats "no row" +// as epoch 1 — and CurrentAddressFor answers from that default: it returns the +// address it was given, leaves the working set clean, and does not create the +// row. +func TestCurrentAddressForNeverReseedsAnAbsentStoreEpochRow(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + + address, err := te.store.MintUnderEpoch(ctx, "epoch-store", "record-a") + if err != nil { + t.Fatalf("MintUnderEpoch: %v", err) + } + // Settle everything so far, then leave the singleton row absent with a + // clean working set. + if _, err := te.store.CommitAll(ctx, "test: settle the first mint"); err != nil { + t.Fatalf("CommitAll: %v", err) + } + te.exec(t, ctx, "DELETE FROM store_epoch") + if _, err := te.store.CommitAll(ctx, "test: absent singleton row"); err != nil { + t.Fatalf("CommitAll: %v", err) + } + + got, err := te.store.CurrentAddressFor(ctx, "epoch-store", address) + if err != nil { + t.Fatalf("CurrentAddressFor: %v", err) + } + if got != address { + t.Errorf("CurrentAddressFor = %s, want the address it was given (%s): nothing carried the Version, so it resolves to itself", got, address) + } + + var rows int + te.queryScalar(t, ctx, "SELECT COUNT(*) FROM store_epoch", nil, &rows) + if rows != 0 { + t.Errorf("store_epoch holds %d rows after CurrentAddressFor, want 0: a lookup must not re-create the singleton row", rows) + } + + status, err := te.store.Status(ctx) + if err != nil { + t.Fatalf("Status: %v", err) + } + if dirty := epochTablesInWorkingSet(status); len(dirty) > 0 { + t.Errorf("after CurrentAddressFor the working set holds %v: a lookup changes nothing, so there is nothing to commit", dirty) + } +} + +// TestBumpEpochCarryingLeavesTheEpochTablesCommitted is the publication rule +// for the token-scheme-change bump. It writes store_epoch (the counter) and +// epoch_minted_addresses (the carry rows), so its Dolt commit has to stage +// both: a bump that staged only store_epoch would leave the carry rows +// modified and uncommitted. +func TestBumpEpochCarryingLeavesTheEpochTablesCommitted(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + const store = "epoch-store" + + epochMint(t, ctx, te, store, "record-a") + newEpoch := epochCarry(t, ctx, te, store) + + // The carry has to have happened for the check below to mean anything. + if got := epochRowsAt(t, ctx, te, store, newEpoch); got != 1 { + t.Fatalf("after the carrying bump to epoch %d the store holds %d rows minted at that epoch, want 1 carry row", newEpoch, got) + } + status, err := te.store.Status(ctx) + if err != nil { + t.Fatalf("Status: %v", err) + } + if dirty := epochTablesInWorkingSet(status); len(dirty) > 0 { + t.Errorf("after the carrying bump the working set still holds %v: its Dolt commit must include the carry rows as well as the counter", dirty) + } +} + +// TestLoseVersionLeavesTheEpochTablesCommitted is the same rule for an explicit +// loss, which writes epoch_minted_addresses alone: a loss the caller believes +// recorded must not be left modified and uncommitted. +func TestLoseVersionLeavesTheEpochTablesCommitted(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + const store = "epoch-store" + + address := epochMint(t, ctx, te, store, "record-a") + if err := te.store.LoseVersion(ctx, store, address); err != nil { + t.Fatalf("LoseVersion: %v", err) + } + + // The loss has to have been recorded for the check below to mean anything. + epoch, err := te.store.CurrentEpoch(ctx, store) + if err != nil { + t.Fatalf("CurrentEpoch: %v", err) + } + epochRequireGone(t, ctx, te, store, address, epoch, "LoseVersion recorded the loss of this address") + + status, err := te.store.Status(ctx) + if err != nil { + t.Fatalf("Status: %v", err) + } + if dirty := epochTablesInWorkingSet(status); len(dirty) > 0 { + t.Errorf("after LoseVersion the working set still holds %v: the loss must be published with the row it changed", dirty) + } +} + +// TestBumpEpochCarryingWritesOneCarryRowPerLiveIDNamingTheLineageRoot pins the +// write surface of the token-scheme-change bump on the real engine +// (gastownhall/beads#6664, bee-ghosttrack review 5360880888, Major 1): the bump +// records the carry itself, one row per live id at the new epoch, instead of +// leaving it to be inferred from a later mint. Each carry row names the lineage +// ROOT in carried_from, so a second carry still names the first root, lineages +// stay flat, and every address of a lineage is one lookup from the newest. +func TestBumpEpochCarryingWritesOneCarryRowPerLiveIDNamingTheLineageRoot(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + const store = "epoch-store" + + rootA := epochMint(t, ctx, te, store, "record-a") + rootB := epochMint(t, ctx, te, store, "record-b") + + firstEpoch := epochCarry(t, ctx, te, store) + if got := epochRowsAt(t, ctx, te, store, firstEpoch); got != 2 { + t.Fatalf("after the carrying bump to epoch %d the store holds %d rows minted at that epoch, want 2: one carry row per live id", firstEpoch, got) + } + carryA := epochRowAddress(t, ctx, te, store, "record-a", firstEpoch) + carryB := epochRowAddress(t, ctx, te, store, "record-b", firstEpoch) + for _, c := range []struct{ id, carry, root string }{ + {"record-a", carryA, rootA}, + {"record-b", carryB, rootB}, + } { + if carriedFrom := epochCarriedFrom(t, ctx, te, c.carry); !carriedFrom.Valid || carriedFrom.String != c.root { + t.Errorf("the carry row of %s names carried_from %v, want its root %s", c.id, carriedFrom, c.root) + } + if goneAt := epochGoneAt(t, ctx, te, c.carry); goneAt.Valid { + t.Errorf("the carry row of %s is already marked gone at epoch %d, want it live", c.id, goneAt.Int64) + } + } + for _, c := range []struct{ old, carry string }{{rootA, carryA}, {rootB, carryB}} { + if got, err := te.store.CurrentAddressFor(ctx, store, c.old); err != nil || got != c.carry { + t.Errorf("CurrentAddressFor(%s) = %q, %v, want the carry address %s", c.old, got, err, c.carry) + } + epochRequireServed(t, ctx, te, store, c.old, "the old address keeps resolving through the carry") + epochRequireServed(t, ctx, te, store, c.carry, "the carry address is served under the new epoch") + } + + // A second carry still names the first ROOT, not the previous carry row. + secondEpoch := epochCarry(t, ctx, te, store) + secondCarryA := epochRowAddress(t, ctx, te, store, "record-a", secondEpoch) + if carriedFrom := epochCarriedFrom(t, ctx, te, secondCarryA); !carriedFrom.Valid || carriedFrom.String != rootA { + t.Errorf("the second carry row of record-a names carried_from %v, want the first root %s: carried_from names the root, never the previous carry", carriedFrom, rootA) + } + for _, member := range []string{rootA, carryA} { + if got, err := te.store.CurrentAddressFor(ctx, store, member); err != nil || got != secondCarryA { + t.Errorf("CurrentAddressFor(%s) = %q, %v, want %s: every address of the lineage points straight at the newest one", member, got, err, secondCarryA) + } + } +} + +// TestBumpEpochCarryingCarriesOnlyTheLineageHoldingTheNewestRow pins the +// collision rule. A mint never retires an address, so one id can have two live +// lineages: here a second root minted after a restore. Both cannot receive an +// address, because an address is a function of the store, the id and the epoch +// and so is one per id per epoch. The lineage holding the newest row is +// carried, and the other is marked gone at the bump's epoch: over-voiding is +// the safe direction, reviving is not. +func TestBumpEpochCarryingCarriesOnlyTheLineageHoldingTheNewestRow(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + const store = "epoch-store" + + first := epochMint(t, ctx, te, store, "record-x") + epochBump(t, ctx, te, store) + second := epochMint(t, ctx, te, store, "record-x") + if first == second { + t.Fatalf("minting record-x again after a bump returned %s, want a second root for the new epoch", first) + } + + newEpoch := epochCarry(t, ctx, te, store) + + carry := epochRowAddress(t, ctx, te, store, "record-x", newEpoch) + if carriedFrom := epochCarriedFrom(t, ctx, te, carry); !carriedFrom.Valid || carriedFrom.String != second { + t.Errorf("the carry row names carried_from %v, want %s: the lineage holding the newest row is the one carried", carriedFrom, second) + } + if goneAt := epochGoneAt(t, ctx, te, first); !goneAt.Valid || goneAt.Int64 != int64(newEpoch) { + t.Errorf("the losing lineage's root has gone_at_epoch %v, want %d: it is marked gone at the epoch of the bump", goneAt, newEpoch) + } + if rows := epochCarryRowsOf(t, ctx, te, first); rows != 0 { + t.Errorf("%d carry rows name the losing lineage's root, want 0: a lineage that lost the collision is not carried", rows) + } + + epochRequireGone(t, ctx, te, store, first, newEpoch, "the lineage that lost the collision is no longer served") + epochRequireServed(t, ctx, te, store, second, "the lineage holding the newest row is still served") + if got, err := te.store.CurrentAddressFor(ctx, store, second); err != nil || got != carry { + t.Errorf("CurrentAddressFor(%s) = %q, %v, want the carry address %s", second, got, err, carry) + } + if got, err := te.store.CurrentAddressFor(ctx, store, first); err == nil { + t.Errorf("CurrentAddressFor(the losing lineage's root) = %s with no error, want an error: a lost lineage has no current address", got) + } +} + +// TestBumpEpochCarryingNeitherCarriesNorServesARowFromAFutureEpoch pins how the +// carry treats an anomaly a rewound counter can leave behind: a row minted in +// an epoch that has not happened yet. It is not carried, and it is not served +// either: a status read fails closed rather than trusting it. The carried live +// row beside it is what makes the check mean anything. +func TestBumpEpochCarryingNeitherCarriesNorServesARowFromAFutureEpoch(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + const store = "epoch-store" + + live := epochMint(t, ctx, te, store, "record-a") + future := epochMint(t, ctx, te, store, "record-future") + te.exec(t, ctx, "UPDATE epoch_minted_addresses SET minted_epoch = minted_epoch + 5 WHERE address = ?", future) + + newEpoch := epochCarry(t, ctx, te, store) + + if got := epochRowsAt(t, ctx, te, store, newEpoch); got != 1 { + t.Fatalf("after the carrying bump to epoch %d the store holds %d rows minted at that epoch, want 1: the live row is carried and the row from a future epoch is not", newEpoch, got) + } + carry := epochRowAddress(t, ctx, te, store, "record-a", newEpoch) + if carriedFrom := epochCarriedFrom(t, ctx, te, carry); !carriedFrom.Valid || carriedFrom.String != live { + t.Errorf("the carry row names carried_from %v, want %s", carriedFrom, live) + } + if rows := epochCarryRowsOf(t, ctx, te, future); rows != 0 { + t.Errorf("%d carry rows name the future-epoch row, want 0: a row from an epoch that has not happened is not carried", rows) + } + epochRequireServed(t, ctx, te, store, live, "the live row is served") + epochRequireGone(t, ctx, te, store, future, newEpoch, "a row minted in an epoch that has not happened yet fails closed") +} + +// TestBumpEpochCarryingFailsAndRollsBackWhenARowAlreadyExistsAtTheNewEpoch pins +// the one place a duplicate key always fails the write. A mint whose insert +// collides with a row reads it and keeps the address when the row is served, +// but a row already sitting at the carry's address means the epoch counter +// rewound. The bump must then fail and roll back whole, including its increment +// of the counter, rather than report a carry it did not make. +func TestBumpEpochCarryingFailsAndRollsBackWhenARowAlreadyExistsAtTheNewEpoch(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + const store = "epoch-store" + + epochMint(t, ctx, te, store, "record-a") + startEpoch, err := te.store.CurrentEpoch(ctx, store) + if err != nil { + t.Fatalf("CurrentEpoch: %v", err) + } + nextEpoch := epochBump(t, ctx, te, store) + epochMint(t, ctx, te, store, "record-a") // a row now exists at nextEpoch + te.exec(t, ctx, "UPDATE store_epoch SET epoch = ? WHERE id = 1", startEpoch) + + if got, err := te.store.BumpEpochCarrying(ctx, store, "token-scheme-change"); err == nil { + t.Errorf("BumpEpochCarrying = epoch %d with no error although a row already exists at epoch %d: the counter rewound, so the carry must fail the bump", got, nextEpoch) + } + epoch, err := te.store.CurrentEpoch(ctx, store) + if err != nil { + t.Fatalf("CurrentEpoch: %v", err) + } + if epoch != startEpoch { + t.Errorf("CurrentEpoch = %d after the failed bump, want %d: a failed carry rolls the counter's increment back with it", epoch, startEpoch) + } +} + +// TestBumpEpochCarryingCarriesTheLiveRowsOfEveryStoreInTheDatabase pins why the +// carry is database-wide. The epoch counter is per database, so one bump moves +// the epoch seen through every storeID; carrying only the rows of the storeID +// that asked would strand every other store's addresses at the scheme boundary. +func TestBumpEpochCarryingCarriesTheLiveRowsOfEveryStoreInTheDatabase(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + const first, second = "epoch-store-one", "epoch-store-two" + + oldFirst := epochMint(t, ctx, te, first, "record-a") + oldSecond := epochMint(t, ctx, te, second, "record-b") + + newEpoch := epochCarry(t, ctx, te, first) // made under the first storeID only + + for _, c := range []struct{ store, id, old string }{ + {first, "record-a", oldFirst}, + {second, "record-b", oldSecond}, + } { + carry := epochRowAddress(t, ctx, te, c.store, c.id, newEpoch) + if carriedFrom := epochCarriedFrom(t, ctx, te, carry); !carriedFrom.Valid || carriedFrom.String != c.old { + t.Errorf("the carry row of %s in %s names carried_from %v, want its root %s", c.id, c.store, carriedFrom, c.old) + } + if got, err := te.store.CurrentAddressFor(ctx, c.store, c.old); err != nil || got != carry { + t.Errorf("CurrentAddressFor(%s) in %s = %q, %v, want the carry address %s", c.old, c.store, got, err, carry) + } + epochRequireServed(t, ctx, te, c.store, c.old, "the old address keeps resolving through the carry") + epochRequireServed(t, ctx, te, c.store, carry, "the carry address is served under the new epoch") + } +} + +// TestBumpEpochCarryingDoesNotCarryALostLineage pins that loss is terminal for +// the carry: a lineage the store lost before the bump gets no carry row and +// stays Gone, while the live lineage beside it is carried. +func TestBumpEpochCarryingDoesNotCarryALostLineage(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + const store = "epoch-store" + + kept := epochMint(t, ctx, te, store, "record-kept") + lost := epochMint(t, ctx, te, store, "record-lost") + if err := te.store.LoseVersion(ctx, store, lost); err != nil { + t.Fatalf("LoseVersion(%s): %v", lost, err) + } + + newEpoch := epochCarry(t, ctx, te, store) + + if got := epochRowsAt(t, ctx, te, store, newEpoch); got != 1 { + t.Fatalf("after the carrying bump to epoch %d the store holds %d rows minted at that epoch, want 1: only the live lineage is carried", newEpoch, got) + } + epochRowAddress(t, ctx, te, store, "record-kept", newEpoch) + if rows := epochCarryRowsOf(t, ctx, te, lost); rows != 0 { + t.Errorf("%d carry rows name the lost lineage's root, want 0: a lost lineage is not carried", rows) + } + epochRequireGone(t, ctx, te, store, lost, newEpoch, "the carry must not bring a lost lineage back") + if got, err := te.store.CurrentAddressFor(ctx, store, lost); err == nil { + t.Errorf("CurrentAddressFor(a lost address) = %s with no error, want an error", got) + } + epochRequireServed(t, ctx, te, store, kept, "the live lineage is served") +} + +// TestABumpUnderOneStoreIDMovesTheEpochSeenThroughEveryStoreID pins the fact the +// bump documentation states: the epoch counter is per database, not per store, +// and storeID appears only in error text. It deliberately pins no guard in the +// shared body, because the conformance suite uses several synthetic storeIDs in +// one database. +func TestABumpUnderOneStoreIDMovesTheEpochSeenThroughEveryStoreID(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + const one, two = "epoch-store-one", "epoch-store-two" + + for _, bump := range []struct { + name string + run func() (int, error) + }{ + {"BumpEpoch", func() (int, error) { return te.store.BumpEpoch(ctx, one, "restore") }}, + {"BumpEpochCarrying", func() (int, error) { return te.store.BumpEpochCarrying(ctx, one, "token-scheme-change") }}, + } { + before, err := te.store.CurrentEpoch(ctx, two) + if err != nil { + t.Fatalf("CurrentEpoch(%s): %v", two, err) + } + after, err := bump.run() + if err != nil { + t.Fatalf("%s(%s): %v", bump.name, one, err) + } + seen, err := te.store.CurrentEpoch(ctx, two) + if err != nil { + t.Fatalf("CurrentEpoch(%s): %v", two, err) + } + if after != before+1 || seen != after { + t.Errorf("%s under %s moved the epoch from %d to %d, and CurrentEpoch under %s reads %d: the counter is per database, so one bump advances the epoch every storeID sees", bump.name, one, before, after, two, seen) + } + } +} + +// epochMint mints id under store, failing the test if the mint fails. +func epochMint(t *testing.T, ctx context.Context, te *testEnv, store, id string) string { + t.Helper() + address, err := te.store.MintUnderEpoch(ctx, store, id) + if err != nil { + t.Fatalf("MintUnderEpoch(%s, %s): %v", store, id, err) + } + return address +} + +// epochBump moves the epoch with the counter-only bump (a restore, which +// writes no address rows) and returns the new epoch. +func epochBump(t *testing.T, ctx context.Context, te *testEnv, store string) int { + t.Helper() + epoch, err := te.store.BumpEpoch(ctx, store, "restore") + if err != nil { + t.Fatalf("BumpEpoch(%s): %v", store, err) + } + return epoch +} + +// epochCarry runs the token-scheme-change bump, which also carries every live +// address to the new epoch, and returns the new epoch. +func epochCarry(t *testing.T, ctx context.Context, te *testEnv, store string) int { + t.Helper() + epoch, err := te.store.BumpEpochCarrying(ctx, store, "token-scheme-change") + if err != nil { + t.Fatalf("BumpEpochCarrying(%s): %v", store, err) + } + return epoch +} + +// epochRowsAt counts the rows of store that were minted at epoch. After a +// carrying bump to epoch that is the number of carry rows it wrote, plus any +// row the test minted there itself. +func epochRowsAt(t *testing.T, ctx context.Context, te *testEnv, store string, epoch int) int { + t.Helper() + var count int + te.queryScalar(t, ctx, "SELECT COUNT(*) FROM epoch_minted_addresses WHERE store_id = ? AND minted_epoch = ?", + []any{store, epoch}, &count) + return count +} + +// epochRowAddress returns the address of the one row of id in store minted at +// epoch, failing the test unless there is exactly one. +func epochRowAddress(t *testing.T, ctx context.Context, te *testEnv, store, id string, epoch int) string { + t.Helper() + var count int + te.queryScalar(t, ctx, "SELECT COUNT(*) FROM epoch_minted_addresses WHERE store_id = ? AND minted_id = ? AND minted_epoch = ?", + []any{store, id, epoch}, &count) + if count != 1 { + t.Fatalf("store %s holds %d rows for %s minted at epoch %d, want exactly 1", store, count, id, epoch) + } + var address string + te.queryScalar(t, ctx, "SELECT address FROM epoch_minted_addresses WHERE store_id = ? AND minted_id = ? AND minted_epoch = ?", + []any{store, id, epoch}, &address) + return address +} + +// epochCarriedFrom reads address's carried_from: not Valid for a row that was +// minted fresh. +func epochCarriedFrom(t *testing.T, ctx context.Context, te *testEnv, address string) sql.NullString { + t.Helper() + var carriedFrom sql.NullString + te.queryScalar(t, ctx, "SELECT carried_from FROM epoch_minted_addresses WHERE address = ?", + []any{address}, &carriedFrom) + return carriedFrom +} + +// epochGoneAt reads address's gone_at_epoch: not Valid while the address is +// live. +func epochGoneAt(t *testing.T, ctx context.Context, te *testEnv, address string) sql.NullInt64 { + t.Helper() + var goneAt sql.NullInt64 + te.queryScalar(t, ctx, "SELECT gone_at_epoch FROM epoch_minted_addresses WHERE address = ?", + []any{address}, &goneAt) + return goneAt +} + +// epochCarryRowsOf counts the carry rows that name root as their lineage root. +func epochCarryRowsOf(t *testing.T, ctx context.Context, te *testEnv, root string) int { + t.Helper() + var count int + te.queryScalar(t, ctx, "SELECT COUNT(*) FROM epoch_minted_addresses WHERE carried_from = ?", + []any{root}, &count) + return count +} + +// epochRequireServed fails unless the store serves address: StillServes is true +// and Resolve answers Live. why says what the assertion checks. +func epochRequireServed(t *testing.T, ctx context.Context, te *testEnv, store, address, why string) { + t.Helper() + serves, err := te.store.StillServes(ctx, store, address) + if err != nil { + t.Fatalf("StillServes(%s): %v", address, err) + } + if !serves { + t.Errorf("StillServes(%s) = false, want true: %s", address, why) + } + result, err := te.store.Resolve(ctx, store, address) + if err != nil { + t.Fatalf("Resolve(%s): %v", address, err) + } + if result.Restriction != storeops.EpochRestrictionLive { + t.Errorf("Resolve(%s).Restriction = %d, want Live (%d): %s", address, result.Restriction, storeops.EpochRestrictionLive, why) + } +} + +// epochRequireGone fails unless address is no longer served at epoch: +// StillServes is false and Resolve answers GoneReorganization naming epoch. +func epochRequireGone(t *testing.T, ctx context.Context, te *testEnv, store, address string, epoch int, why string) { + t.Helper() + serves, err := te.store.StillServes(ctx, store, address) + if err != nil { + t.Fatalf("StillServes(%s): %v", address, err) + } + if serves { + t.Errorf("StillServes(%s) = true, want false: %s", address, why) + } + result, err := te.store.Resolve(ctx, store, address) + if err != nil { + t.Fatalf("Resolve(%s): %v", address, err) + } + if result.Restriction != storeops.EpochRestrictionGoneReorganization { + t.Errorf("Resolve(%s).Restriction = %d, want GoneReorganization (%d): %s", address, result.Restriction, storeops.EpochRestrictionGoneReorganization, why) + } + if result.Epoch == nil { + t.Errorf("Resolve(%s).Epoch = nil, want the current epoch %d: %s", address, epoch, why) + } else if *result.Epoch != epoch { + t.Errorf("Resolve(%s).Epoch = %d, want the current epoch %d: %s", address, *result.Epoch, epoch, why) + } +} diff --git a/internal/storage/embeddeddolt/expected_revision_contract_test.go b/internal/storage/embeddeddolt/expected_revision_contract_test.go index a716384f6..6db4e2f0e 100644 --- a/internal/storage/embeddeddolt/expected_revision_contract_test.go +++ b/internal/storage/embeddeddolt/expected_revision_contract_test.go @@ -10,11 +10,21 @@ import ( ) // TestExpectedRevisionContract wires this leg into the R16/R17 -// expected-revision contract. Phase 0 leaves every hook nil in every -// backend's fixture kit (architecture §12), so each case below skips by -// name; this file exists so TestEveryLegWiresEveryRoleContract counts this -// leg, and so the cases start running for real the moment this leg's fixture -// kit grows a non-nil hook. +// expected-revision contract. Every hook is nil (architecture §12's Phase 0 +// default), so each case below skips by name; this file exists so +// TestEveryLegWiresEveryRoleContract counts this leg, and so the cases +// start running for real the moment this leg grows a real +// CompareAndSetVersion (gastownhall/beads#6358). +// +// This leg previously wired every hook to an honest "not implemented" stub +// instead of leaving them nil, so each case ran for real and +// failed for that documented reason rather than skipping. That premise — +// that a later slice would give this leg its CompareAndSetVersion — went +// stale once review moved the real implementation to +// gastownhall/beads#6358 instead; a stub whose failure message blames a +// bead that will never land it is worse than an honest skip, so this file +// reverts to nil hooks (gastownhall/beads#6664, bee-ghosttrack review +// 5268699223). func TestExpectedRevisionContract(t *testing.T) { ctx := context.Background() fixture := conformance.ExpectedRevisionFixture{IssuePrefix: "erev"} diff --git a/internal/storage/embeddeddolt/retention_epoch_contract_test.go b/internal/storage/embeddeddolt/retention_epoch_contract_test.go index 648341895..6c9a7b52e 100644 --- a/internal/storage/embeddeddolt/retention_epoch_contract_test.go +++ b/internal/storage/embeddeddolt/retention_epoch_contract_test.go @@ -4,9 +4,12 @@ package embeddeddolt_test import ( "context" + "testing" "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + storeops "github.com/steveyegge/beads/internal/storage/issueops" ) // TestRetentionContract wires this leg into the R20 retention contract. @@ -59,11 +62,31 @@ func TestRetentionContract(t *testing.T) { }) } -// TestEpochContract wires this leg into the R20 epoch contract. Same Phase 0 -// all-nil state as TestRetentionContract above. +// TestEpochContract runs the R20 epoch contract (gastownhall/beads#5898 +// revision 9, #6136) against the embedded-Dolt- +// backed store, which reaches internal/storage/issueops's epoch Tx +// functions through this leg's own issue-operation transaction +// (EmbeddedDoltStore.BumpEpoch/BumpEpochCarrying/MintUnderEpoch/LoseVersion) +// or a read-only connection (EmbeddedDoltStore.CurrentEpoch/StillServes/ +// Resolve/CurrentAddressFor) — see internal/storage/embeddeddolt/epoch_cas.go. +// +// All three legs run that one shared body, so this is not an independent +// vote on the design — it is the check on THIS leg's wrapper and this file's +// own conformance-vocabulary translation. func TestEpochContract(t *testing.T) { - ctx := context.Background() - fixture := conformance.EpochFixture{IssuePrefix: "epch"} + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "epch") + ctx := t.Context() + fixture := conformance.EpochFixture{ + IssuePrefix: "epch", + CurrentEpoch: epochEmbeddedCurrentEpoch(te.store), + BumpEpoch: epochEmbeddedBumpEpoch(te.store), + MintUnderEpoch: epochEmbeddedMintUnderEpoch(te.store), + StillServes: epochEmbeddedStillServes(te.store), + Resolve: epochEmbeddedResolve(te.store), + CurrentAddressFor: epochEmbeddedCurrentAddressFor(te.store), + LoseVersion: epochEmbeddedLoseVersion(te.store), + } t.Run("AnEpochBumpIsTriggeredOnlyByRestoreReinitOrSchemeChange", func(t *testing.T) { conformance.RunAnEpochBumpIsTriggeredOnlyByRestoreReinitOrSchemeChange(t, ctx, fixture) @@ -71,4 +94,98 @@ func TestEpochContract(t *testing.T) { t.Run("EpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed", func(t *testing.T) { conformance.RunEpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed(t, ctx, fixture) }) + t.Run("TokenSchemeChangeCarriesLiveAddressesAndOnlyThose", func(t *testing.T) { + conformance.RunTokenSchemeChangeCarriesLiveAddressesAndOnlyThose(t, ctx, fixture) + }) + t.Run("ACarriedAddressKeepsResolvingAcrossLaterBumps", func(t *testing.T) { + conformance.RunACarriedAddressKeepsResolvingAcrossLaterBumps(t, ctx, fixture) + }) + t.Run("ALostAddressStaysGoneAcrossASchemeChange", func(t *testing.T) { + conformance.RunALostAddressStaysGoneAcrossASchemeChange(t, ctx, fixture) + }) +} + +func epochEmbeddedCurrentEpoch(store *embeddeddolt.EmbeddedDoltStore) func(ctx context.Context, storeID string) (int, error) { + return func(ctx context.Context, storeID string) (int, error) { + return store.CurrentEpoch(ctx, storeID) + } +} + +// epochEmbeddedBumpEpoch chooses the bump by the typed trigger: only a +// token-scheme-change bump carries live addresses to the new epoch, and the +// other two triggers move the counter alone. The choice lives here, in the +// contract-test translation layer, because the shared body has no switch on +// the reason string. +func epochEmbeddedBumpEpoch(store *embeddeddolt.EmbeddedDoltStore) func(ctx context.Context, storeID string, trigger conformance.EpochBumpTrigger) (int, error) { + return func(ctx context.Context, storeID string, trigger conformance.EpochBumpTrigger) (int, error) { + if trigger == conformance.EpochBumpTriggerTokenSchemeChange { + return store.BumpEpochCarrying(ctx, storeID, trigger.String()) + } + return store.BumpEpoch(ctx, storeID, trigger.String()) + } +} + +func epochEmbeddedMintUnderEpoch(store *embeddeddolt.EmbeddedDoltStore) func(ctx context.Context, storeID, id string) (conformance.Address, error) { + return func(ctx context.Context, storeID, id string) (conformance.Address, error) { + address, err := store.MintUnderEpoch(ctx, storeID, id) + if err != nil { + return "", err + } + return conformance.Address(address), nil + } +} + +func epochEmbeddedStillServes(store *embeddeddolt.EmbeddedDoltStore) func(ctx context.Context, storeID string, address conformance.Address) (bool, error) { + return func(ctx context.Context, storeID string, address conformance.Address) (bool, error) { + return store.StillServes(ctx, storeID, string(address)) + } +} + +func epochEmbeddedResolve(store *embeddeddolt.EmbeddedDoltStore) func(ctx context.Context, storeID string, address conformance.Address) (conformance.RetentionAnswer, error) { + return func(ctx context.Context, storeID string, address conformance.Address) (conformance.RetentionAnswer, error) { + result, err := store.Resolve(ctx, storeID, string(address)) + if err != nil { + return conformance.RetentionAnswer{}, err + } + return epochResolveResultToConformance(result), nil + } +} + +func epochEmbeddedCurrentAddressFor(store *embeddeddolt.EmbeddedDoltStore) func(ctx context.Context, storeID string, oldAddress conformance.Address) (conformance.Address, error) { + return func(ctx context.Context, storeID string, oldAddress conformance.Address) (conformance.Address, error) { + address, err := store.CurrentAddressFor(ctx, storeID, string(oldAddress)) + if err != nil { + return "", err + } + return conformance.Address(address), nil + } +} + +func epochEmbeddedLoseVersion(store *embeddeddolt.EmbeddedDoltStore) func(ctx context.Context, storeID string, address conformance.Address) error { + return func(ctx context.Context, storeID string, address conformance.Address) error { + return store.LoseVersion(ctx, storeID, string(address)) + } +} + +// epochRestrictionToConformance translates this leg's local +// storeops.EpochRestriction into backend/conformance's own Restriction +// vocabulary — the contract-test file is the translation boundary, per +// epoch_cas.go's package doc. +func epochRestrictionToConformance(r storeops.EpochRestriction) conformance.Restriction { + switch r { + case storeops.EpochRestrictionLive: + return conformance.RestrictionLive + case storeops.EpochRestrictionGoneReorganization: + return conformance.RestrictionGoneReorganization + default: + return conformance.RestrictionUnknown + } +} + +func epochResolveResultToConformance(r storeops.EpochResolveResult) conformance.RetentionAnswer { + return conformance.RetentionAnswer{ + Restriction: epochRestrictionToConformance(r.Restriction), + ProducingStore: r.ProducingStore, + Epoch: r.Epoch, + } } diff --git a/internal/storage/embeddeddolt/transaction.go b/internal/storage/embeddeddolt/transaction.go index ed1d25682..a9eb77f3a 100644 --- a/internal/storage/embeddeddolt/transaction.go +++ b/internal/storage/embeddeddolt/transaction.go @@ -50,6 +50,7 @@ func (s *EmbeddedDoltStore) runTransactionWithMessage(ctx context.Context, fn fu }); err != nil { return err } + afterSQLCommit() // Anything the body minted ran with versioned history scoped on this // store's transaction (withConn), so its version rows must be staged with @@ -67,6 +68,11 @@ func (s *EmbeddedDoltStore) runTransactionWithMessage(ctx context.Context, fn fu return nil } +// afterSQLCommit runs between runTransactionWithMessage's SQL commit and the +// Dolt commit that publishes it. It is a no-op outside tests, which use it to +// fail the publication of a write whose rows have already landed. +var afterSQLCommit = func() {} + func wrapCommitIndeterminate(op string, err error) error { return fmt.Errorf("%s: %w: %w", op, err, storage.ErrCommitIndeterminate) } diff --git a/internal/storage/issueops/BUILD.bazel b/internal/storage/issueops/BUILD.bazel index 138b27d5b..3ade280ae 100644 --- a/internal/storage/issueops/BUILD.bazel +++ b/internal/storage/issueops/BUILD.bazel @@ -53,6 +53,7 @@ go_library( "edge_counts.go", "edges.go", "epic_closure.go", + "epoch_cas.go", "errors.go", "events.go", "execution.go", @@ -164,6 +165,7 @@ go_test( "dependency_tree_test.go", "edge_counts_test.go", "edge_role_requests_test.go", + "epoch_cas_test.go", "execution_test.go", "filters_test.go", "gate_fields_test.go", @@ -212,6 +214,7 @@ go_test( ], data = glob(["testdata/**"]) + [ ":go_srcs", + "//internal/storage/dolt:go_srcs", "//internal/storage/domain/db:go_srcs", "//internal/storage/schema:migrations", "//internal/storage/versioncontrolops:go_srcs", diff --git a/internal/storage/issueops/epoch_cas.go b/internal/storage/issueops/epoch_cas.go new file mode 100644 index 000000000..994334ac3 --- /dev/null +++ b/internal/storage/issueops/epoch_cas.go @@ -0,0 +1,680 @@ +package issueops + +import ( + "context" + "database/sql" + "errors" + "fmt" + "math" + "strings" + "time" + + "github.com/steveyegge/beads/internal/storage/dberrors" +) + +// This file implements R20 epoch-transition enforcement (gastownhall/beads#5898 +// revision 9, #6136): a store-wide epoch generation +// counter (store_epoch, migration 0067) plus a durable record of the +// addresses minted under each generation (epoch_minted_addresses, migration +// 0072), used to answer whether a previously-minted address is still served +// by the store's current epoch. It adds no RetentionFixture/R17 resolve, +// remove, hold, force-remove, erase, or mint logic — R20 epoch reasoning is +// evaluated entirely on its own (out of scope for this slice). +// +// ALL THREE LEGS SHARE THIS BODY, the same way they share +// CompareAndSetMetadataKeyInTx: the two Dolt-backed stores wrap it in their +// own transaction, and the unit-of-work provider reaches it through its own +// leg-specific adapter. Each leg's own contract-test file is the +// translation boundary to and from backend/conformance's +// Address/EpochBumpTrigger/RetentionAnswer vocabulary — this file knows +// nothing about the conformance package. +// +// ADDRESSES ARE NEVER RECOMPUTED FROM store_epoch. Each is a deterministic +// token over (storeID, id, the epoch current at mint time) — see +// epochAddress — persisted once at mint: a later mint of the same id under a +// bumped epoch produces a DIFFERENT address and a new row, and every +// address's row survives to answer StillServes/Resolve after the epoch moves +// on. +// +// SURVIVAL IS RECORDED ON THE ROW, NEVER INFERRED (gastownhall/beads#6664, +// bee-ghosttrack review 5360880888, Major 1). An address is served unless its +// own row says gone_at_epoch, and a row minted in an epoch that has not +// happened yet is not served: the answer is a function of the address's row +// and the current epoch, and nothing reads another row to reach it. Gone is +// terminal. Two writers set gone_at_epoch, both guarded by IS NULL so that +// nothing already lost is touched and nothing is ever cleared: +// LoseVersionInTx, for a Version the store lost, and BumpEpochCarryingInTx, +// for the losing lineage when two live lineages of one id meet at a +// token-scheme change. +// +// A restore or a destructive reinit moves the counter and writes no address +// row (BumpEpochInTx), so every address that was served stays Live at its own +// address with no re-mint. A token-scheme change is the one bump that writes +// address rows (BumpEpochCarryingInTx): it carries every live lineage to the +// new epoch with one carry row whose carried_from names the lineage's ROOT +// address, never the previous carry, so lineages stay flat and any member of +// one reaches its newest address in a single indexed lookup. +// +// THE EPOCH COUNTER IS PER DATABASE, NOT PER STORE (gastownhall/beads#6664, +// bee-ghosttrack review 5360880888, Minor 3). store_epoch has no store_id +// column, and storeID appears only in error text: one bump advances the epoch +// seen through every storeID that shares the database. There is deliberately +// no storeID guard in this body, because the conformance suite uses several +// synthetic storeIDs in one database. +// +// epochAddress'S ENCODING IS LENGTH-PREFIXED, NOT BARE-COLON-JOINED +// (gastownhall/beads#6664, bee-ghosttrack review 5268699223, item B2): a +// colon inside storeID or id can no longer shift the (storeID, id, epoch) +// field boundary onto a different triple the way the old "epch:%s:%s:%d" +// form allowed. MintUnderEpochInTx additionally refuses a storeID or id +// containing ":" outright (validateEpochAddressInputs) so an address stays +// readable by eye without needing to count length prefixes — belt and +// braces, not a correctness dependency of epochAddress itself. Given that, +// upsertEpochMintedAddressInTx's exists branch is a pure no-op: an +// address's row existing now provably means its stored triple already +// matches the incoming one, so there is nothing left to write. + +// EpochRestriction is this file's own local answer vocabulary for R20 — +// deliberately not backend/conformance.Restriction (see package doc above). +// It omits GoneRetention/GoneErasure: those are RetentionFixture/R17 +// outcomes this slice does not produce. +type EpochRestriction int + +const ( + EpochRestrictionLive EpochRestriction = iota + EpochRestrictionGoneReorganization + EpochRestrictionUnknown +) + +// EpochResolveResult is ResolveEpochInTx's answer. Epoch is non-nil exactly +// when the answer required comparing an epoch (Live and GoneReorganization); +// it is nil for Unknown, which never looks store_epoch up at all. +type EpochResolveResult struct { + Restriction EpochRestriction + ProducingStore string + Epoch *int +} + +// ErrEpochAddressNotFound means the address named was never minted by +// storeID — distinct from an ordinary "gone" answer, which requires having +// minted the address in the first place. +var ErrEpochAddressNotFound = errors.New("epoch CAS: address not minted by this store") + +// ErrEpochAddressNotServed means the address exists in this store but the +// store does not serve it: its lineage was lost, or it was minted in an epoch +// that has not happened yet. It is distinct from ErrEpochAddressNotFound so a +// caller can tell an address the store never minted from one it minted and no +// longer serves. +var ErrEpochAddressNotServed = errors.New("epoch CAS: address exists in this store but is not served") + +// ErrEpochAddressSeparator means a storeID or id passed to +// MintUnderEpochInTx contains ":", the character epochAddress's encoding +// uses as a field delimiter (gastownhall/beads#6664, bee-ghosttrack review +// 5268699223, item B2b). epochAddress's length-prefixed encoding does not +// actually depend on this for correctness (see epochAddress's own doc +// comment), but refusing it outright keeps a minted address readable by eye. +var ErrEpochAddressSeparator = errors.New("epoch CAS: storeID or id contains \":\", epochAddress's field separator") + +// ErrEpochAddressTooLong means a storeID and id compose an address that +// epoch_minted_addresses.address, a VARCHAR(255) PRIMARY KEY (migration 0072), +// may not hold. MintUnderEpochInTx refuses such a pair before it reads or +// writes anything, so the caller gets this error instead of a raw driver error +// from the INSERT. The bound is charged at the widest epoch the INT counter can +// reach, not the current one: a token-scheme change re-encodes every live pair +// at the new epoch (BumpEpochCarryingInTx), so a pair that fit at epoch 9 must +// still fit at epoch 10, or the carry, and with it every later scheme change, +// would fail database-wide. +var ErrEpochAddressTooLong = errors.New("epoch CAS: storeID and id compose an address longer than epoch_minted_addresses.address holds") + +// maxEpochAddressLen is the width of epoch_minted_addresses.address, +// VARCHAR(255). The check against it counts bytes, which are never fewer than +// the characters the column counts, so it refuses no later than the column +// would. +const maxEpochAddressLen = 255 + +// epochAddress is a minted address: a deterministic token over storeID, id, +// and the epoch current at mint time, so re-minting the same id under the +// same epoch always reproduces the same address. storeID and id are +// length-prefixed, not just colon-joined (gastownhall/beads#6664, +// bee-ghosttrack review 5268699223, item B2a): reading exactly len(storeID) +// bytes for the first field and len(id) bytes for the second makes the +// (storeID, id) boundary unambiguous no matter what characters either one +// contains, so two different triples can never collide on the same address. +// MintUnderEpochInTx also refuses a storeID or id containing ":" outright +// (validateEpochAddressInputs, item B2b) so an address stays readable by +// eye without relying on that. +func epochAddress(storeID, id string, epoch int) string { + return fmt.Sprintf("epch:%d:%s:%d:%s:%d", len(storeID), storeID, len(id), id, epoch) +} + +// validateEpochAddressInputs rejects a storeID or id containing ":" before +// MintUnderEpochInTx mints an address from them (item B2b above), and a pair +// whose address would not fit the address column at the widest epoch +// (ErrEpochAddressTooLong). +func validateEpochAddressInputs(storeID, id string) error { + if strings.Contains(storeID, ":") { + return fmt.Errorf("%w: storeID %q", ErrEpochAddressSeparator, storeID) + } + if strings.Contains(id, ":") { + return fmt.Errorf("%w: id %q", ErrEpochAddressSeparator, id) + } + if n := len(epochAddress(storeID, id, math.MaxInt32)); n > maxEpochAddressLen { + return fmt.Errorf("%w: %d bytes at the widest epoch, limit %d", ErrEpochAddressTooLong, n, maxEpochAddressLen) + } + return nil +} + +// ensureStoreEpochRow lazily initializes store_epoch's singleton row +// (migration 0067; the table starts empty) to epoch 1 on first use, and +// reports the current epoch either way. Only BumpEpochInTx and +// MintUnderEpochInTx call this: both are about to write regardless, so +// creating the row here is not a surprising extra side effect. Every +// read-only path calls readStoreEpochWithoutSeedingInTx instead +// (gastownhall/beads#6664, bee-ghosttrack review 5268699223, item B4): a write +// from what callers reasonably expect to be a read is surprising at best, and +// a hard failure against a genuinely read-only connection at worst. +// +// The seed follows version_history.go's seed of the same row (gastownhall/ +// beads#6664, bee-ghosttrack review 5360880888): read first, so the INSERT +// happens once per store rather than once per mint; INSERT IGNORE, so a row +// that turns up between the read and the write is not an error; and read the +// row back, so the epoch reported is what the row holds even when the seed was +// ignored, not the 1 it would have written. +func ensureStoreEpochRow(ctx context.Context, tx DBTX) (int, error) { + var epoch int + err := tx.QueryRowContext(ctx, `SELECT epoch FROM store_epoch WHERE id = 1`).Scan(&epoch) + if errors.Is(err, sql.ErrNoRows) { + if _, err := tx.ExecContext(ctx, `INSERT IGNORE INTO store_epoch (id, epoch) VALUES (1, 1)`); err != nil { + return 0, fmt.Errorf("initialize store_epoch: %w", err) + } + err = tx.QueryRowContext(ctx, `SELECT epoch FROM store_epoch WHERE id = 1`).Scan(&epoch) + } + if err != nil { + return 0, fmt.Errorf("read store_epoch: %w", err) + } + return epoch, nil +} + +// EpochMintDirtyTables names the tables a mint (MintUnderEpochInTx) can leave +// modified, for the leg that publishes them to Dolt history. A token-scheme +// bump (BumpEpochCarryingInTx) leaves the same two tables modified. store_epoch is in the set although a mint +// never updates it: ensureStoreEpochRow INSERTs the singleton row on first use, +// so a mint that published only epoch_minted_addresses would leave that row +// modified and uncommitted, and the Dolt commit would hold minted rows without +// the epoch row they depend on (gastownhall/beads#6664 Major 2, bee-ghosttrack +// review 5360880888). Staging a table that turns out to be clean is free. +func EpochMintDirtyTables() ChangedTables { + return ChangedTables{"epoch_minted_addresses": true, "store_epoch": true} +} + +// readStoreEpochWithoutSeedingInTx reports the current epoch without writing: +// store_epoch's singleton row starting absent (migration 0067) and its +// epoch being 1 are the same state, so a read-only caller can answer from +// that default instead of initializing the row the way ensureStoreEpochRow +// does (item B4 above). It is deliberately not version_history.go's +// readStoreEpochInTx, which seeds the row when it is absent. CurrentEpochInTx, +// StillServesInTx, ResolveEpochInTx, CurrentAddressForInTx and +// LoseVersionInTx all read this way; only BumpEpochInTx and +// MintUnderEpochInTx, which are about to write regardless, use +// ensureStoreEpochRow. +func readStoreEpochWithoutSeedingInTx(ctx context.Context, tx DBTX) (int, error) { + var epoch int + err := tx.QueryRowContext(ctx, `SELECT epoch FROM store_epoch WHERE id = 1`).Scan(&epoch) + if errors.Is(err, sql.ErrNoRows) { + return 1, nil + } + if err != nil { + return 0, fmt.Errorf("read store_epoch: %w", err) + } + return epoch, nil +} + +// epochMintedAddress is one row as read from epoch_minted_addresses. +// carriedFrom is NULL for a root row (a fresh mint) and otherwise the address of +// the lineage's root; goneAtEpoch is NULL while the store serves the row and +// otherwise the epoch it was lost in. +type epochMintedAddress struct { + storeID string + mintedID string + mintedEpoch int + carriedFrom sql.NullString + goneAtEpoch sql.NullInt64 +} + +// served reports whether the store serves the row at epoch. A row is served +// unless it was lost, and a row minted in an epoch that has not happened yet +// fails closed rather than being trusted. +func (r epochMintedAddress) served(epoch int) bool { + return !r.goneAtEpoch.Valid && r.mintedEpoch <= epoch +} + +// lineageRoot is the address every member of the row's lineage names: the +// row's carried_from when it is a carry row, and its own address when it is a +// root. +func (r epochMintedAddress) lineageRoot(address string) string { + if r.carriedFrom.Valid { + return r.carriedFrom.String + } + return address +} + +// readEpochMintedAddressInTx reads address's row, if any. found is false +// (with a zero-value row and nil error) when address was never minted — +// distinguishing "not found" from an actual query error is the caller's +// job, the same way readExpectedRevisionRowInTx's callers do it. +func readEpochMintedAddressInTx(ctx context.Context, tx DBTX, address string) (epochMintedAddress, bool, error) { + var row epochMintedAddress + err := tx.QueryRowContext(ctx, + `SELECT store_id, minted_id, minted_epoch, carried_from, gone_at_epoch FROM epoch_minted_addresses WHERE address = ?`, address, + ).Scan(&row.storeID, &row.mintedID, &row.mintedEpoch, &row.carriedFrom, &row.goneAtEpoch) + if errors.Is(err, sql.ErrNoRows) { + return epochMintedAddress{}, false, nil + } + if err != nil { + return epochMintedAddress{}, false, fmt.Errorf("epoch CAS: read minted address %s: %w", address, err) + } + return row, true, nil +} + +// upsertEpochMintedAddressInTx writes address's row. exists (from the +// caller's own prior read) says whether one already does: if so, this is a +// pure no-op (gastownhall/beads#6664, bee-ghosttrack review 5268699223, +// item B2c) — epochAddress is now injective (item B2a), so an existing row +// at this exact address provably already carries this exact (storeID, +// mintedID, mintedEpoch) triple, and the old exists-branch UPDATE (which +// had no store_id/minted_id guard) could only ever have mattered by +// rewriting a DIFFERENT triple's row out from under it on an address +// collision the new encoding no longer allows. +// +// A duplicate-key error from the INSERT is NOT success. It means a row the +// caller's read did not find: another transaction inserted it, and that +// transaction or a later one may already have lost it, so taking the error as +// the benign race of two minters inserting the same row (item B3) could hand +// back an address that reads Gone. The row is read again and trusted only as +// far as this transaction can see it: served at mintedEpoch, the mint stands, +// because the row is the one it would have inserted; present but not served, +// the mint fails with ErrEpochAddressNotServed, as it does when the caller's +// own read finds such a row; not visible at all, the mint fails with the +// duplicate-key error, and a new transaction, which reads the committed row, +// answers it. +// +// On Dolt, the only supported backend, the INSERT does not hit that error for +// a row committed after the transaction's snapshot: it checks the snapshot, +// and the two inserts of one address conflict at COMMIT instead (Error 1213), +// which withRetryTx answers by running the mint again against the committed +// row (TestAMintRacingTheMintAndLossOfItsAddressConflictsAtCommit in the dolt +// package). FOR UPDATE is a parse-only no-op there, so no locking read could +// see further than the re-read does. +func upsertEpochMintedAddressInTx(ctx context.Context, tx DBTX, address, storeID, mintedID string, mintedEpoch int, exists bool) error { + if exists { + return nil + } + _, err := tx.ExecContext(ctx, + `INSERT INTO epoch_minted_addresses (address, store_id, minted_id, minted_epoch, minted_at) VALUES (?, ?, ?, ?, ?)`, + address, storeID, mintedID, mintedEpoch, time.Now().UTC(), + ) + if err == nil { + return nil + } + if !dberrors.IsDuplicateKey(err) { + return fmt.Errorf("epoch CAS: insert minted address %s: %w", address, err) + } + row, found, readErr := readEpochMintedAddressInTx(ctx, tx, address) + switch { + case readErr != nil: + return errors.Join(fmt.Errorf("epoch CAS: insert minted address %s: %w", address, err), readErr) + case !found: + return fmt.Errorf("epoch CAS: insert minted address %s: another transaction holds a row at it that this one cannot read, so whether the store serves it is unknown: %w", address, err) + case !row.served(mintedEpoch): + return fmt.Errorf("epoch CAS: mint %s under epoch for %s: %w: %s", mintedID, storeID, ErrEpochAddressNotServed, address) + } + return nil +} + +// CurrentEpochInTx reports storeID's current epoch generation. store_epoch +// has no store_id column: migration 0067 established it as one physical +// counter per database, matching production reality (one logical store per +// database already) — storeID is accepted here only to satisfy +// EpochFixture's hook signature. +// +// The epoch counter is per database, not per store. storeID appears only in +// error text; one bump advances the epoch seen through every storeID that +// shares this database. +func CurrentEpochInTx(ctx context.Context, tx DBTX, storeID string) (int, error) { + epoch, err := readStoreEpochWithoutSeedingInTx(ctx, tx) + if err != nil { + return 0, fmt.Errorf("epoch CAS: current epoch for %s: %w", storeID, err) + } + return epoch, nil +} + +// BumpEpochInTx advances storeID's epoch generation by one and records why +// (R20-a: a bump is triggered only by restore, destructive-reinit, or +// token-scheme-change — the leg adapter converts the fixture's +// conformance.EpochBumpTrigger to this plain string via trigger.String(), +// keeping that vocabulary out of this file per the package doc above). +// +// It writes store_epoch and nothing else: a restore or a destructive reinit +// leaves every address row alone, so every address that was served stays Live +// at its own address. A token-scheme change, which does write address rows, +// calls BumpEpochCarryingInTx instead. +// +// The epoch counter is per database, not per store. storeID appears only in +// error text; one bump advances the epoch seen through every storeID that +// shares this database. +func BumpEpochInTx(ctx context.Context, tx DBTX, storeID, reason string) (int, error) { + if _, err := ensureStoreEpochRow(ctx, tx); err != nil { + return 0, fmt.Errorf("epoch CAS: bump epoch for %s: %w", storeID, err) + } + if _, err := tx.ExecContext(ctx, + `UPDATE store_epoch SET epoch = epoch + 1, bumped_at = ?, bumped_reason = ? WHERE id = 1`, + time.Now().UTC(), reason, + ); err != nil { + return 0, fmt.Errorf("epoch CAS: bump epoch for %s: %w", storeID, err) + } + var newEpoch int + if err := tx.QueryRowContext(ctx, `SELECT epoch FROM store_epoch WHERE id = 1`).Scan(&newEpoch); err != nil { + return 0, fmt.Errorf("epoch CAS: read epoch after bump for %s: %w", storeID, err) + } + return newEpoch, nil +} + +// BumpEpochCarryingInTx is BumpEpochInTx for a token-scheme change, the one +// trigger that changes how addresses are encoded. It advances the epoch exactly +// as BumpEpochInTx does and also records the carry: every lineage the store +// still serves gets one new row at the new epoch, whose address is the +// deterministic token for (store, id, new epoch) and whose carried_from names +// the lineage's ROOT address. The old addresses keep resolving, and +// CurrentAddressForInTx reports the new one from any member of the lineage. +// +// The carry is DATABASE-WIDE on purpose. The counter is per database, so this +// bump moves the epoch every storeID sees, and carrying only storeID's rows +// would strand every other store's addresses at the scheme boundary. storeID is +// used for error text only, exactly as in BumpEpochInTx. It costs one read of +// every live row and one insert per live id, in one transaction: acceptable +// because a scheme change is a once-per-change migration, not a runtime path. +// +// Two live lineages of one id cannot both receive an address, because an +// address is a function of (store, id, epoch) and so is one per id per epoch. +// The lineage holding the newest row is carried, and every other lineage of +// that id is marked gone at the new epoch (the safe direction: over-void, never +// revive). Lineages already lost are not read, so they are never carried, and a +// row minted in an epoch that has not happened yet is not read either: it stays +// not served. A row already sitting at a carry's address means the counter +// rewound, so that is a failure of the bump: no re-read can settle it the way +// a mint settles a duplicate key (upsertEpochMintedAddressInTx). +// +// CALLERS MUST HOLD THE WORKSPACE GATE (internal/workspacegate) EXCLUSIVELY FOR +// THE BUMP, as a restore does. This is the one epoch operation that needs +// mints excluded by contract: the store cannot make a mint conflict with a bump +// (a mint has no cell of its own to change), so a mint that commits after this +// bump read the live rows would leave a live row that was never carried. +func BumpEpochCarryingInTx(ctx context.Context, tx DBTX, storeID, reason string) (int, error) { + newEpoch, err := BumpEpochInTx(ctx, tx, storeID, reason) + if err != nil { + return 0, err + } + candidates, err := readEpochCarryCandidatesInTx(ctx, tx, newEpoch) + if err != nil { + return 0, fmt.Errorf("epoch CAS: carry addresses to epoch %d for %s: %w", newEpoch, storeID, err) + } + + // The candidates are ordered newest first within each (store, id), so the + // first row of a group holds the lineage that wins it. + type lineageKey struct{ storeID, mintedID string } + var groups []lineageKey + winningRoot := map[lineageKey]string{} + type losingLineage struct{ storeID, root string } + var losers []losingLineage + seenLoser := map[losingLineage]bool{} + for _, c := range candidates { + key := lineageKey{c.storeID, c.mintedID} + winner, seen := winningRoot[key] + if !seen { + winningRoot[key] = c.root() + groups = append(groups, key) + continue + } + if loser := (losingLineage{c.storeID, c.root()}); c.root() != winner && !seenLoser[loser] { + seenLoser[loser] = true + losers = append(losers, loser) + } + } + + for _, loser := range losers { + if err := loseLineageInTx(ctx, tx, loser.storeID, loser.root, newEpoch); err != nil { + return 0, fmt.Errorf("epoch CAS: carry addresses to epoch %d for %s: %w", newEpoch, storeID, err) + } + } + for _, key := range groups { + address := epochAddress(key.storeID, key.mintedID, newEpoch) + if _, err := tx.ExecContext(ctx, + `INSERT INTO epoch_minted_addresses (address, store_id, minted_id, minted_epoch, minted_at, carried_from) VALUES (?, ?, ?, ?, ?, ?)`, + address, key.storeID, key.mintedID, newEpoch, time.Now().UTC(), winningRoot[key], + ); err != nil { + return 0, fmt.Errorf("epoch CAS: carry %s to epoch %d for %s: %w", key.mintedID, newEpoch, storeID, err) + } + } + return newEpoch, nil +} + +// epochCarryCandidate is one live row below the new epoch, as +// BumpEpochCarryingInTx reads it. +type epochCarryCandidate struct { + address string + storeID string + mintedID string + carriedFrom sql.NullString +} + +// root is the address of the candidate's lineage root. +func (c epochCarryCandidate) root() string { + if c.carriedFrom.Valid { + return c.carriedFrom.String + } + return c.address +} + +// readEpochCarryCandidatesInTx reads every row the store still serves below +// newEpoch, newest first within each (store, id). Rows already lost are not +// read, and neither are rows minted at or above newEpoch: those are anomalies +// and stay not served. The result is read to the end before any other statement +// runs on the transaction. +func readEpochCarryCandidatesInTx(ctx context.Context, tx DBTX, newEpoch int) ([]epochCarryCandidate, error) { + rows, err := tx.QueryContext(ctx, + `SELECT address, store_id, minted_id, carried_from FROM epoch_minted_addresses WHERE gone_at_epoch IS NULL AND minted_epoch < ? ORDER BY store_id, minted_id, minted_epoch DESC`, + newEpoch, + ) + if err != nil { + return nil, fmt.Errorf("read live addresses: %w", err) + } + defer func() { _ = rows.Close() }() + var candidates []epochCarryCandidate + for rows.Next() { + var c epochCarryCandidate + if err := rows.Scan(&c.address, &c.storeID, &c.mintedID, &c.carriedFrom); err != nil { + return nil, fmt.Errorf("scan live address: %w", err) + } + candidates = append(candidates, c) + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("read live addresses: %w", err) + } + return candidates, nil +} + +// loseLineageInTx marks the lineage rooted at root gone at epoch: the root's own +// row, then every carry row that names it. Each statement is guarded by +// gone_at_epoch IS NULL, so a row already lost is never touched and nothing is +// ever cleared. It is the only place gone_at_epoch is written, shared by +// LoseVersionInTx and BumpEpochCarryingInTx. +func loseLineageInTx(ctx context.Context, tx DBTX, storeID, root string, epoch int) error { + if _, err := tx.ExecContext(ctx, + `UPDATE epoch_minted_addresses SET gone_at_epoch = ? WHERE address = ? AND store_id = ? AND gone_at_epoch IS NULL`, + epoch, root, storeID, + ); err != nil { + return fmt.Errorf("mark %s lost at epoch %d: %w", root, epoch, err) + } + if _, err := tx.ExecContext(ctx, + `UPDATE epoch_minted_addresses SET gone_at_epoch = ? WHERE carried_from = ? AND store_id = ? AND gone_at_epoch IS NULL`, + epoch, root, storeID, + ); err != nil { + return fmt.Errorf("mark the carries of %s lost at epoch %d: %w", root, epoch, err) + } + return nil +} + +// MintUnderEpochInTx mints id's address under storeID's CURRENT epoch, +// deterministically (epochAddress): minting the same id again under the +// same epoch reproduces the same address and is an idempotent no-op upsert +// of the same row. +// +// A mint only ever INSERTS its own root row. It never updates another row, so +// no mint can retire or revive an address, and it never hands back an address +// the store does not serve: a row already sitting at the computed address is +// only possible when the lineage was minted in this epoch and lost in this same +// epoch, and the mint then returns ErrEpochAddressNotServed instead of an +// address that reads Gone. That includes a row another transaction inserts +// after this one read: on Dolt the two inserts conflict at COMMIT and the mint +// runs again against the committed row, and an INSERT that reports the row as +// a duplicate key reads it again first (upsertEpochMintedAddressInTx). +// +// A storeID or id containing ":" is refused (ErrEpochAddressSeparator), and so +// is a pair whose address could outgrow the address column at some epoch +// (ErrEpochAddressTooLong), before anything is read or written. +func MintUnderEpochInTx(ctx context.Context, tx DBTX, storeID, id string) (string, error) { + if err := validateEpochAddressInputs(storeID, id); err != nil { + return "", fmt.Errorf("epoch CAS: mint %s under epoch for %s: %w", id, storeID, err) + } + epoch, err := ensureStoreEpochRow(ctx, tx) + if err != nil { + return "", fmt.Errorf("epoch CAS: mint %s under epoch for %s: %w", id, storeID, err) + } + address := epochAddress(storeID, id, epoch) + row, found, err := readEpochMintedAddressInTx(ctx, tx, address) + if err != nil { + return "", err + } + if found && !row.served(epoch) { + return "", fmt.Errorf("epoch CAS: mint %s under epoch for %s: %w: %s", id, storeID, ErrEpochAddressNotServed, address) + } + if err := upsertEpochMintedAddressInTx(ctx, tx, address, storeID, id, epoch, found); err != nil { + return "", err + } + return address, nil +} + +// StillServesInTx reports whether address is still served under storeID's +// CURRENT epoch. The answer is read from the address's own row and the epoch and +// from nothing else: the row's gone_at_epoch says whether the store lost it, and +// a row minted in an epoch that has not happened yet is not served. An address +// that survived a restore or a destructive reinit therefore stays served with no +// upkeep, and an address from a different store, or one never minted, is not +// served either. +func StillServesInTx(ctx context.Context, tx DBTX, storeID, address string) (bool, error) { + row, found, err := readEpochMintedAddressInTx(ctx, tx, address) + if err != nil { + return false, err + } + if !found || row.storeID != storeID { + return false, nil + } + epoch, err := readStoreEpochWithoutSeedingInTx(ctx, tx) + if err != nil { + return false, fmt.Errorf("epoch CAS: still serves %s for %s: %w", address, storeID, err) + } + return row.served(epoch), nil +} + +// ResolveEpochInTx answers R20's epoch-only restriction for address: Live while +// the store serves it, GoneReorganization once the store lost it (a +// reorganization, not a retention or erasure outcome; RetentionFixture/R17 +// states are out of scope here), and Unknown for an address this store never +// minted. Like StillServesInTx it reads only the address's own row and the +// epoch. ProducingStore is always storeID: this file has no lineage/replica +// model to attribute a foreign store to (unlike RetentionFixture's cross-store +// answers). +func ResolveEpochInTx(ctx context.Context, tx DBTX, storeID, address string) (EpochResolveResult, error) { + row, found, err := readEpochMintedAddressInTx(ctx, tx, address) + if err != nil { + return EpochResolveResult{}, err + } + if !found || row.storeID != storeID { + return EpochResolveResult{Restriction: EpochRestrictionUnknown, ProducingStore: storeID}, nil + } + epoch, err := readStoreEpochWithoutSeedingInTx(ctx, tx) + if err != nil { + return EpochResolveResult{}, fmt.Errorf("epoch CAS: resolve %s for %s: %w", address, storeID, err) + } + if row.served(epoch) { + return EpochResolveResult{Restriction: EpochRestrictionLive, ProducingStore: storeID, Epoch: &epoch}, nil + } + return EpochResolveResult{Restriction: EpochRestrictionGoneReorganization, ProducingStore: storeID, Epoch: &epoch}, nil +} + +// CurrentAddressForInTx reports the address oldAddress now resolves to: the +// newest address of its lineage, or oldAddress itself when nothing carried the +// Version to a new address. It only reads. It executes no INSERT, UPDATE or +// DELETE, because a lookup that minted an address could make a lost address +// look served again (gastownhall/beads#6664, bee-ghosttrack review 5360880888, +// Major 1), and both Dolt legs run it on a read-write transaction that is +// always rolled back, so a stray write would vanish silently instead of +// failing. oldAddress must be one storeID minted (ErrEpochAddressNotFound +// otherwise) and still serves (ErrEpochAddressNotServed otherwise). +func CurrentAddressForInTx(ctx context.Context, tx DBTX, storeID, oldAddress string) (string, error) { + row, found, err := readEpochMintedAddressInTx(ctx, tx, oldAddress) + if err != nil { + return "", err + } + if !found || row.storeID != storeID { + return "", fmt.Errorf("epoch CAS: current address for %s: %w: %s", storeID, ErrEpochAddressNotFound, oldAddress) + } + epoch, err := readStoreEpochWithoutSeedingInTx(ctx, tx) + if err != nil { + return "", fmt.Errorf("epoch CAS: current address for %s: %w", storeID, err) + } + if !row.served(epoch) { + return "", fmt.Errorf("epoch CAS: current address for %s: %w: %s", storeID, ErrEpochAddressNotServed, oldAddress) + } + var newest string + err = tx.QueryRowContext(ctx, + `SELECT address FROM epoch_minted_addresses WHERE carried_from = ? AND store_id = ? AND gone_at_epoch IS NULL AND minted_epoch <= ? ORDER BY minted_epoch DESC LIMIT 1`, + row.lineageRoot(oldAddress), storeID, epoch, + ).Scan(&newest) + if errors.Is(err, sql.ErrNoRows) { + return oldAddress, nil + } + if err != nil { + return "", fmt.Errorf("epoch CAS: current address for %s: read the newest carry of %s: %w", storeID, oldAddress, err) + } + return newest, nil +} + +// LoseVersionInTx records that storeID no longer serves the Version named by +// address: it marks the lineage ROOT's own row and every carry row of that +// root gone at the current epoch, so every address of the Version resolves +// GoneReorganization from then on. Each statement is guarded by +// gone_at_epoch IS NULL, so nothing already lost is touched and nothing is ever +// cleared (Gone is terminal). Losing a lineage that is already lost matches no +// rows and returns nil, so a retried transaction may replay it. address must be +// one storeID minted, or ErrEpochAddressNotFound is returned and nothing is +// written. A lineage member minted in an epoch that has not happened yet +// violates the table's gone-not-before-mint check, and the resulting error is +// returned rather than papered over. +func LoseVersionInTx(ctx context.Context, tx DBTX, storeID, address string) error { + row, found, err := readEpochMintedAddressInTx(ctx, tx, address) + if err != nil { + return err + } + if !found || row.storeID != storeID { + return fmt.Errorf("epoch CAS: lose version for %s: %w: %s", storeID, ErrEpochAddressNotFound, address) + } + epoch, err := readStoreEpochWithoutSeedingInTx(ctx, tx) + if err != nil { + return fmt.Errorf("epoch CAS: lose version %s for %s: %w", address, storeID, err) + } + if err := loseLineageInTx(ctx, tx, storeID, row.lineageRoot(address), epoch); err != nil { + return fmt.Errorf("epoch CAS: lose version %s for %s: %w", address, storeID, err) + } + return nil +} diff --git a/internal/storage/issueops/epoch_cas_test.go b/internal/storage/issueops/epoch_cas_test.go new file mode 100644 index 000000000..223a40581 --- /dev/null +++ b/internal/storage/issueops/epoch_cas_test.go @@ -0,0 +1,1141 @@ +package issueops + +import ( + "context" + "database/sql/driver" + "errors" + "go/ast" + "go/parser" + "go/token" + "math" + "path/filepath" + "strconv" + "strings" + "testing" + + "github.com/DATA-DOG/go-sqlmock" +) + +// These are sqlmock tests because each one pins a SHAPE: which statements the +// shared epoch body runs, in what order, and above all which statements it does +// not run. sqlmock rejects any statement without a registered expectation, so a +// test that registers no ExpectExec is a test that forbids every write. What a +// statement does to real rows (set semantics, carry, collision) is checked +// against a real store by the embedded leg's tests. +// +// The survival model these tests pin (gastownhall/beads#6664, bee-ghosttrack +// review 5360880888, Major 1): an address's status is a function of its OWN +// row and the current epoch. It is Live unless gone_at_epoch is set, and a row +// minted in an epoch that has not happened yet is not served. Nothing infers +// status from other rows. +const ( + epochSQLRow = `SELECT store_id, minted_id, minted_epoch, carried_from, gone_at_epoch FROM epoch_minted_addresses WHERE address = \?` + epochSQLReadEpoch = `SELECT epoch FROM store_epoch WHERE id = 1` + epochSQLNewest = `SELECT address FROM epoch_minted_addresses WHERE carried_from = \? AND store_id = \? AND gone_at_epoch IS NULL AND minted_epoch <= \? ORDER BY minted_epoch DESC LIMIT 1` + epochSQLLoseRoot = `UPDATE epoch_minted_addresses SET gone_at_epoch = \? WHERE address = \? AND store_id = \? AND gone_at_epoch IS NULL` + epochSQLLoseCarry = `UPDATE epoch_minted_addresses SET gone_at_epoch = \? WHERE carried_from = \? AND store_id = \? AND gone_at_epoch IS NULL` + epochSQLInsertRoot = `INSERT INTO epoch_minted_addresses \(address, store_id, minted_id, minted_epoch, minted_at\) VALUES \(\?, \?, \?, \?, \?\)` +) + +var epochSQLRowColumns = []string{"store_id", "minted_id", "minted_epoch", "carried_from", "gone_at_epoch"} + +// epochTestRow is one epoch_minted_addresses row as the shared body reads it. +type epochTestRow struct { + storeID string + mintedID string + mintedEpoch int + // carriedFrom is nil for a root row, else the lineage root's address. + carriedFrom any + // goneAtEpoch is nil while the lineage is served, else the epoch it was + // lost in. + goneAtEpoch any +} + +// expectEpochRow registers the read of address's row. A nil row is an address +// the store never minted. +func expectEpochRow(mock sqlmock.Sqlmock, address string, row *epochTestRow) { + rows := sqlmock.NewRows(epochSQLRowColumns) + if row != nil { + rows.AddRow(row.storeID, row.mintedID, row.mintedEpoch, row.carriedFrom, row.goneAtEpoch) + } + mock.ExpectQuery(epochSQLRow).WithArgs(address).WillReturnRows(rows) +} + +// expectEpochRead registers the read of store_epoch's singleton. It takes no +// arguments: the counter is per database, so no storeID ever reaches this SQL. +func expectEpochRead(mock sqlmock.Sqlmock, epoch int) { + mock.ExpectQuery(epochSQLReadEpoch). + WithArgs([]driver.Value{}...). + WillReturnRows(sqlmock.NewRows([]string{"epoch"}).AddRow(epoch)) +} + +// expectEpochAbsent registers the read of store_epoch's singleton finding no +// row, the state the table starts in (migration 0067). The other cases register +// a present row, where a reader that seeds the row and one that does not run the +// same SQL; only an absent row tells them apart. +func expectEpochAbsent(mock sqlmock.Sqlmock) { + mock.ExpectQuery(epochSQLReadEpoch). + WithArgs([]driver.Value{}...). + WillReturnRows(sqlmock.NewRows([]string{"epoch"})) +} + +func intPtr(v int) *int { return &v } + +// TestEpochReadPathsNeverSeedAnAbsentStoreEpochRow pins item B4 +// (gastownhall/beads#6664, bee-ghosttrack review 5268699223): every path that +// only reads answers an absent store_epoch row from the epoch-1 default and +// never inserts it. LoseVersionInTx is here too, because the epoch it reads is +// not one it writes. Each case registers no INSERT, so a read path switched to +// the seeding reader (ensureStoreEpochRow) runs an unexpected statement and +// fails; the Dolt legs cannot catch that switch, because their read +// transactions roll back. +func TestEpochReadPathsNeverSeedAnAbsentStoreEpochRow(t *testing.T) { + t.Parallel() + + const storeID = "absent-epoch-store" + root := epochAddress(storeID, "record-a", 1) + row := &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1} + ctx := context.Background() + + t.Run("CurrentEpochInTx", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochAbsent(mock) + got, err := CurrentEpochInTx(ctx, tx, storeID) + if err != nil || got != 1 { + t.Fatalf("CurrentEpochInTx with no store_epoch row = (%d, %v), want (1, nil)", got, err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("CurrentEpochInTx must only read: %v", err) + } + }) + t.Run("StillServesInTx", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, root, row) + expectEpochAbsent(mock) + got, err := StillServesInTx(ctx, tx, storeID, root) + if err != nil || !got { + t.Fatalf("StillServesInTx with no store_epoch row = (%v, %v), want (true, nil): an address minted at epoch 1 is served at the default epoch", got, err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("StillServesInTx must only read: %v", err) + } + }) + t.Run("ResolveEpochInTx", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, root, row) + expectEpochAbsent(mock) + got, err := ResolveEpochInTx(ctx, tx, storeID, root) + if err != nil || got.Restriction != EpochRestrictionLive || got.Epoch == nil || *got.Epoch != 1 { + t.Fatalf("ResolveEpochInTx with no store_epoch row = (%+v, %v), want Live at epoch 1", got, err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("ResolveEpochInTx must only read: %v", err) + } + }) + t.Run("CurrentAddressForInTx", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, root, row) + expectEpochAbsent(mock) + mock.ExpectQuery(epochSQLNewest).WithArgs(root, storeID, int64(1)). + WillReturnRows(sqlmock.NewRows([]string{"address"})) + got, err := CurrentAddressForInTx(ctx, tx, storeID, root) + if err != nil || got != root { + t.Fatalf("CurrentAddressForInTx with no store_epoch row = (%q, %v), want (%q, nil)", got, err, root) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("CurrentAddressForInTx must only read: %v", err) + } + }) + t.Run("LoseVersionInTx", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, root, row) + expectEpochAbsent(mock) + mock.ExpectExec(epochSQLLoseRoot).WithArgs(int64(1), root, storeID).WillReturnResult(sqlmock.NewResult(0, 1)) + mock.ExpectExec(epochSQLLoseCarry).WithArgs(int64(1), root, storeID).WillReturnResult(sqlmock.NewResult(0, 0)) + if err := LoseVersionInTx(ctx, tx, storeID, root); err != nil { + t.Fatalf("LoseVersionInTx with no store_epoch row: %v", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("LoseVersionInTx must write only the loss, at the default epoch 1: %v", err) + } + }) +} + +// TestStillServesAndResolveAnswerFromTheAddressRowAndTheEpochAlone pins the +// whole resolver. Each case registers exactly the statements the answer may +// use, the address's own row and then the epoch, so a third statement (for +// instance a look at the id's other rows, which is what the earlier chain +// check did) is an unexpected call the mock rejects. An address nobody minted, +// or another store's, is answered from the row alone: the epoch is not read. +func TestStillServesAndResolveAnswerFromTheAddressRowAndTheEpochAlone(t *testing.T) { + t.Parallel() + + const storeID = "status-store" + const currentEpoch = 5 + root := epochAddress(storeID, "record-a", 1) + + cases := []struct { + name string + row *epochTestRow + wantServes bool + wantRestriction EpochRestriction + readsEpoch bool + }{ + { + name: "a row minted in an earlier epoch and never lost is served", + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1}, + wantServes: true, + wantRestriction: EpochRestrictionLive, + readsEpoch: true, + }, + { + name: "a row minted in the current epoch is served", + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: currentEpoch}, + wantServes: true, + wantRestriction: EpochRestrictionLive, + readsEpoch: true, + }, + { + name: "a carry row is served on its own row, with no look at the root", + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 3, carriedFrom: root}, + wantServes: true, + wantRestriction: EpochRestrictionLive, + readsEpoch: true, + }, + { + name: "a lost row is not served", + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1, goneAtEpoch: 3}, + wantServes: false, + wantRestriction: EpochRestrictionGoneReorganization, + readsEpoch: true, + }, + { + name: "a row lost in the epoch it was minted in is not served", + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: currentEpoch, goneAtEpoch: currentEpoch}, + wantServes: false, + wantRestriction: EpochRestrictionGoneReorganization, + readsEpoch: true, + }, + { + name: "a row minted in an epoch that has not happened yet fails closed", + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: currentEpoch + 2}, + wantServes: false, + wantRestriction: EpochRestrictionGoneReorganization, + readsEpoch: true, + }, + { + name: "an address the store never minted is unknown", + row: nil, + wantServes: false, + wantRestriction: EpochRestrictionUnknown, + }, + { + name: "an address another store minted is unknown", + row: &epochTestRow{storeID: "other-store", mintedID: "record-a", mintedEpoch: 1}, + wantServes: false, + wantRestriction: EpochRestrictionUnknown, + }, + } + + for _, tc := range cases { + t.Run(tc.name+" (StillServes)", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, root, tc.row) + if tc.readsEpoch { + expectEpochRead(mock, currentEpoch) + } + + got, err := StillServesInTx(context.Background(), tx, storeID, root) + if err != nil { + t.Fatalf("StillServesInTx: %v", err) + } + if got != tc.wantServes { + t.Errorf("StillServesInTx = %v, want %v", got, tc.wantServes) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("StillServesInTx ran a different set of statements than the row read and the epoch read: %v", err) + } + }) + t.Run(tc.name+" (Resolve)", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, root, tc.row) + if tc.readsEpoch { + expectEpochRead(mock, currentEpoch) + } + + got, err := ResolveEpochInTx(context.Background(), tx, storeID, root) + if err != nil { + t.Fatalf("ResolveEpochInTx: %v", err) + } + if got.Restriction != tc.wantRestriction { + t.Errorf("ResolveEpochInTx.Restriction = %v, want %v", got.Restriction, tc.wantRestriction) + } + if got.ProducingStore != storeID { + t.Errorf("ResolveEpochInTx.ProducingStore = %q, want %q", got.ProducingStore, storeID) + } + var wantEpoch *int + if tc.readsEpoch { + wantEpoch = intPtr(currentEpoch) + } + switch { + case wantEpoch == nil && got.Epoch != nil: + t.Errorf("ResolveEpochInTx.Epoch = %d, want nil: an unknown address never looks the epoch up", *got.Epoch) + case wantEpoch != nil && got.Epoch == nil: + t.Errorf("ResolveEpochInTx.Epoch = nil, want the current epoch %d", *wantEpoch) + case wantEpoch != nil && *got.Epoch != *wantEpoch: + t.Errorf("ResolveEpochInTx.Epoch = %d, want the current epoch %d", *got.Epoch, *wantEpoch) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("ResolveEpochInTx ran a different set of statements than the row read and the epoch read: %v", err) + } + }) + } +} + +// TestCurrentAddressForInTxReadsAndNeverWrites pins that CurrentAddressFor is +// a pure read (bee-ghosttrack, review 5360880888, Major 1). The earlier form +// minted a fresh address under every read, which made a lost address look +// served again. Both Dolt legs open a read-write transaction for their read +// path and always roll it back, so a stray write would vanish with no error: +// the transaction type cannot enforce this. These cases register no ExpectExec, +// so any INSERT, UPDATE or DELETE is an unexpected call. +func TestCurrentAddressForInTxReadsAndNeverWrites(t *testing.T) { + t.Parallel() + + const storeID = "current-address-store" + const currentEpoch = 5 + root := epochAddress(storeID, "record-a", 1) + firstCarry := epochAddress(storeID, "record-a", 3) + newestCarry := epochAddress(storeID, "record-a", 5) + + cases := []struct { + name string + // asked is the address the caller holds; row is its row. + asked string + row *epochTestRow + // newest is the lineage's newest carry row, "" when it has none. + newest string + want string + }{ + { + name: "a survivor nothing carried returns its own address", + asked: root, + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1}, + want: root, + }, + { + name: "the root of a carried lineage returns the newest carry", + asked: root, + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1}, + newest: newestCarry, + want: newestCarry, + }, + { + name: "an older carry resolves through the lineage root, one lookup from any member", + asked: firstCarry, + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 3, carriedFrom: root}, + newest: newestCarry, + want: newestCarry, + }, + { + name: "the newest carry returns itself", + asked: newestCarry, + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 5, carriedFrom: root}, + newest: newestCarry, + want: newestCarry, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, tc.asked, tc.row) + expectEpochRead(mock, currentEpoch) + lookup := mock.ExpectQuery(epochSQLNewest).WithArgs(root, storeID, int64(currentEpoch)) + if tc.newest == "" { + lookup.WillReturnRows(sqlmock.NewRows([]string{"address"})) + } else { + lookup.WillReturnRows(sqlmock.NewRows([]string{"address"}).AddRow(tc.newest)) + } + + got, err := CurrentAddressForInTx(context.Background(), tx, storeID, tc.asked) + if err != nil { + t.Fatalf("CurrentAddressForInTx: %v", err) + } + if got != tc.want { + t.Errorf("CurrentAddressForInTx = %q, want %q", got, tc.want) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("CurrentAddressForInTx ran a different set of statements than reads: %v", err) + } + }) + } +} + +// TestCurrentAddressForInTxReturnsTypedErrorsForAddressesItCannotAnswer pins the +// two errors a caller can tell apart: an address the store never minted, and +// one it minted but no longer serves (lost, or from an epoch that has not +// happened). Neither runs the lineage lookup, and neither writes. +func TestCurrentAddressForInTxReturnsTypedErrorsForAddressesItCannotAnswer(t *testing.T) { + t.Parallel() + + const storeID = "typed-error-store" + const currentEpoch = 5 + address := epochAddress(storeID, "record-a", 1) + + cases := []struct { + name string + row *epochTestRow + readsEpoch bool + want error + notWant error + }{ + { + name: "a lost address is not served", + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1, goneAtEpoch: 3}, + readsEpoch: true, + want: ErrEpochAddressNotServed, + notWant: ErrEpochAddressNotFound, + }, + { + name: "an address from an epoch that has not happened is not served", + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: currentEpoch + 1}, + readsEpoch: true, + want: ErrEpochAddressNotServed, + notWant: ErrEpochAddressNotFound, + }, + { + name: "an address the store never minted is not found", + row: nil, + want: ErrEpochAddressNotFound, + notWant: ErrEpochAddressNotServed, + }, + { + name: "an address another store minted is not found", + row: &epochTestRow{storeID: "other-store", mintedID: "record-a", mintedEpoch: 1}, + want: ErrEpochAddressNotFound, + notWant: ErrEpochAddressNotServed, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, address, tc.row) + if tc.readsEpoch { + expectEpochRead(mock, currentEpoch) + } + + got, err := CurrentAddressForInTx(context.Background(), tx, storeID, address) + if err == nil { + t.Fatalf("CurrentAddressForInTx = %q with no error, want %v", got, tc.want) + } + if !errors.Is(err, tc.want) { + t.Errorf("CurrentAddressForInTx error = %v, want it to wrap %v", err, tc.want) + } + if errors.Is(err, tc.notWant) { + t.Errorf("CurrentAddressForInTx error = %v, must not wrap %v: callers tell the two apart", err, tc.notWant) + } + if got != "" { + t.Errorf("CurrentAddressForInTx returned %q alongside an error, want an empty address", got) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + } +} + +// TestLoseVersionInTxMarksTheWholeLineageOnceAndOnlyWhereStillLive pins loss: +// it reads the address's row and the epoch (never creating the singleton), then +// marks the lineage ROOT's own row and every carry row of that root, each +// statement guarded by gone_at_epoch IS NULL so that nothing already lost is +// touched and nothing is ever cleared. Losing a lineage that is already lost +// matches no rows and is not an error, so a retried transaction may replay it. +func TestLoseVersionInTxMarksTheWholeLineageOnceAndOnlyWhereStillLive(t *testing.T) { + t.Parallel() + + const storeID = "lose-store" + const currentEpoch = 4 + root := epochAddress(storeID, "record-a", 1) + carry := epochAddress(storeID, "record-a", 3) + + cases := []struct { + name string + asked string + row *epochTestRow + rootAffected int64 + carryRows int64 + }{ + { + name: "losing the root marks the root row and its carry rows", + asked: root, + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1}, + rootAffected: 1, + carryRows: 1, + }, + { + name: "losing a carry address marks the same lineage through its root", + asked: carry, + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 3, carriedFrom: root}, + rootAffected: 1, + carryRows: 1, + }, + { + name: "a lineage that was never carried still runs both guarded statements", + asked: root, + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1}, + rootAffected: 1, + carryRows: 0, + }, + { + name: "a lineage that is already lost matches no rows and is not an error", + asked: root, + row: &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1, goneAtEpoch: 3}, + rootAffected: 0, + carryRows: 0, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, tc.asked, tc.row) + expectEpochRead(mock, currentEpoch) + mock.ExpectExec(epochSQLLoseRoot). + WithArgs(int64(currentEpoch), root, storeID). + WillReturnResult(sqlmock.NewResult(0, tc.rootAffected)) + mock.ExpectExec(epochSQLLoseCarry). + WithArgs(int64(currentEpoch), root, storeID). + WillReturnResult(sqlmock.NewResult(0, tc.carryRows)) + + if err := LoseVersionInTx(context.Background(), tx, storeID, tc.asked); err != nil { + t.Fatalf("LoseVersionInTx: %v", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("LoseVersionInTx did not run the root update and then the carry update, each guarded: %v", err) + } + }) + } +} + +// TestLoseVersionInTxReportsAnAddressTheStoreNeverMinted pins that a loss +// naming an address this store never minted is an error and writes nothing. +func TestLoseVersionInTxReportsAnAddressTheStoreNeverMinted(t *testing.T) { + t.Parallel() + + const storeID = "lose-absent-store" + address := epochAddress(storeID, "record-a", 1) + + cases := []struct { + name string + row *epochTestRow + }{ + {name: "an address nobody minted", row: nil}, + {name: "an address another store minted", row: &epochTestRow{storeID: "other-store", mintedID: "record-a", mintedEpoch: 1}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, address, tc.row) + + err := LoseVersionInTx(context.Background(), tx, storeID, address) + if !errors.Is(err, ErrEpochAddressNotFound) { + t.Fatalf("LoseVersionInTx = %v, want it to wrap ErrEpochAddressNotFound", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("a loss that cannot be recorded must run the row read and nothing else: %v", err) + } + }) + } +} + +// TestLoseVersionInTxSurfacesAStatementFailure pins that a failed update is +// returned, not swallowed. A lineage member minted in an epoch that has not +// happened yet violates the table's gone-not-before-mint check, and the loss +// must report that anomaly rather than paper over it. +func TestLoseVersionInTxSurfacesAStatementFailure(t *testing.T) { + t.Parallel() + + const storeID = "lose-failure-store" + address := epochAddress(storeID, "record-a", 1) + checkViolation := errors.New("check constraint violated") + + _, mock, tx := beginMockTx(t) + expectEpochRow(mock, address, &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: 1}) + expectEpochRead(mock, 2) + mock.ExpectExec(epochSQLLoseRoot). + WithArgs(int64(2), address, storeID). + WillReturnResult(sqlmock.NewResult(0, 1)) + mock.ExpectExec(epochSQLLoseCarry). + WithArgs(int64(2), address, storeID). + WillReturnError(checkViolation) + + err := LoseVersionInTx(context.Background(), tx, storeID, address) + if !errors.Is(err, checkViolation) { + t.Fatalf("LoseVersionInTx = %v, want it to wrap the failed statement's error", err) + } +} + +// TestBumpEpochInTxWritesOnlyStoreEpoch pins that a plain bump (restore and +// destructive reinit) touches the counter and nothing else. Restore replaces +// the database and reinit keeps what survives, so neither may write an address +// row: an address row a bump wrote would be an address that was not minted. +// Only BumpEpochCarryingInTx, for a token-scheme change, writes address rows. +func TestBumpEpochInTxWritesOnlyStoreEpoch(t *testing.T) { + t.Parallel() + + _, mock, tx := beginMockTx(t) + expectEpochRead(mock, 3) + mock.ExpectExec(`UPDATE store_epoch SET epoch = epoch \+ 1, bumped_at = \?, bumped_reason = \? WHERE id = 1`). + WithArgs(sqlmock.AnyArg(), "restore"). + WillReturnResult(sqlmock.NewResult(0, 1)) + expectEpochRead(mock, 4) + + got, err := BumpEpochInTx(context.Background(), tx, "bump-store", "restore") + if err != nil { + t.Fatalf("BumpEpochInTx: %v", err) + } + if got != 4 { + t.Errorf("BumpEpochInTx = %d, want 4", got) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("BumpEpochInTx ran statements beyond the store_epoch read, update and read-back: %v", err) + } +} + +// TestABumpUnderOneStoreIDMovesTheEpochSeenUnderEveryOther pins the sentence +// the epoch functions' docs now carry (bee-ghosttrack, review 5360880888, +// Minor 3): the counter is per database, not per store, and storeID appears +// only in error text. No SQL here takes the storeID as an argument, so a bump +// made under one storeID is seen under every storeID that shares the database. +// There is deliberately no storeID guard: the conformance suite uses several +// synthetic storeIDs in one database. +func TestABumpUnderOneStoreIDMovesTheEpochSeenUnderEveryOther(t *testing.T) { + t.Parallel() + + _, mock, tx := beginMockTx(t) + expectEpochRead(mock, 2) + mock.ExpectExec(`UPDATE store_epoch SET epoch = epoch \+ 1, bumped_at = \?, bumped_reason = \? WHERE id = 1`). + WithArgs(sqlmock.AnyArg(), "restore"). + WillReturnResult(sqlmock.NewResult(0, 1)) + expectEpochRead(mock, 3) + expectEpochRead(mock, 3) + + bumped, err := BumpEpochInTx(context.Background(), tx, "store-one", "restore") + if err != nil { + t.Fatalf("BumpEpochInTx under store-one: %v", err) + } + seen, err := CurrentEpochInTx(context.Background(), tx, "store-two") + if err != nil { + t.Fatalf("CurrentEpochInTx under store-two: %v", err) + } + if seen != bumped { + t.Errorf("CurrentEpochInTx(store-two) = %d after a bump under store-one reported %d, want the same epoch: the counter is per database", seen, bumped) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } +} + +// TestMintUnderEpochInTxNeverHandsBackAnAddressTheStoreDoesNotServe pins the +// mint's refusal. A mint is deterministic in (storeID, id, epoch), so it can +// find a row already sitting at the address it computes; that row is only +// possible when the lineage was minted in this epoch and lost in this same +// epoch. Handing that address back would give the caller an address that +// reads Gone, so the mint returns ErrEpochAddressNotServed instead, and it +// does not touch the row. +func TestMintUnderEpochInTxNeverHandsBackAnAddressTheStoreDoesNotServe(t *testing.T) { + t.Parallel() + + const storeID = "mint-store" + const currentEpoch = 5 + address := epochAddress(storeID, "record-a", currentEpoch) + + _, mock, tx := beginMockTx(t) + expectEpochRead(mock, currentEpoch) + expectEpochRow(mock, address, &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: currentEpoch, goneAtEpoch: currentEpoch}) + + got, err := MintUnderEpochInTx(context.Background(), tx, storeID, "record-a") + if !errors.Is(err, ErrEpochAddressNotServed) { + t.Fatalf("MintUnderEpochInTx = (%q, %v), want it to wrap ErrEpochAddressNotServed", got, err) + } + if got != "" { + t.Errorf("MintUnderEpochInTx returned %q alongside the refusal, want an empty address", got) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("a refused mint must read and write nothing else: %v", err) + } +} + +// TestMintUnderEpochInTxInsertsOnlyAFreshRootRow pins that the only write a +// mint ever makes is the INSERT of its own root row. A mint that finds its +// address already served returns it with no write at all, so it can never +// modify, retire or revive another row. +func TestMintUnderEpochInTxInsertsOnlyAFreshRootRow(t *testing.T) { + t.Parallel() + + const storeID = "mint-store" + const currentEpoch = 5 + address := epochAddress(storeID, "record-a", currentEpoch) + + t.Run("an absent row is inserted as a root row", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRead(mock, currentEpoch) + expectEpochRow(mock, address, nil) + mock.ExpectExec(epochSQLInsertRoot). + WithArgs(address, storeID, "record-a", int64(currentEpoch), sqlmock.AnyArg()). + WillReturnResult(sqlmock.NewResult(0, 1)) + + got, err := MintUnderEpochInTx(context.Background(), tx, storeID, "record-a") + if err != nil { + t.Fatalf("MintUnderEpochInTx: %v", err) + } + if got != address { + t.Errorf("MintUnderEpochInTx = %q, want %q", got, address) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("the mint's only write must be the root INSERT: %v", err) + } + }) + + t.Run("a served row at the address is returned with no write", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectEpochRead(mock, currentEpoch) + expectEpochRow(mock, address, &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: currentEpoch}) + + got, err := MintUnderEpochInTx(context.Background(), tx, storeID, "record-a") + if err != nil { + t.Fatalf("MintUnderEpochInTx: %v", err) + } + if got != address { + t.Errorf("MintUnderEpochInTx = %q, want %q", got, address) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("minting an address that is already served must write nothing: %v", err) + } + }) +} + +// TestMintUnderEpochInTxTrustsADuplicateKeyOnlyAsFarAsItCanReadTheRow pins +// what a mint does when its INSERT reports a duplicate key for an address its +// own read found absent: another transaction inserted the row, and may have +// lost it since. The mint reads the row again and returns the address only +// when that row is served; it refuses a row that is not served, and a row it +// cannot read at all is a failure, never the success item B3 once assumed. +func TestMintUnderEpochInTxTrustsADuplicateKeyOnlyAsFarAsItCanReadTheRow(t *testing.T) { + t.Parallel() + + const storeID = "mint-race-store" + const currentEpoch = 5 + address := epochAddress(storeID, "record-a", currentEpoch) + duplicate := errors.New("Error 1062: Duplicate entry for key 'PRIMARY'") + + // expectLostInsert registers a mint whose read finds no row and whose + // INSERT then hits the row another transaction wrote. + expectLostInsert := func(mock sqlmock.Sqlmock) { + expectEpochRead(mock, currentEpoch) + expectEpochRow(mock, address, nil) + mock.ExpectExec(epochSQLInsertRoot). + WithArgs(address, storeID, "record-a", int64(currentEpoch), sqlmock.AnyArg()). + WillReturnError(duplicate) + } + + t.Run("a served row is the one the mint would have inserted", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectLostInsert(mock) + expectEpochRow(mock, address, &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: currentEpoch}) + + got, err := MintUnderEpochInTx(context.Background(), tx, storeID, "record-a") + if err != nil { + t.Fatalf("MintUnderEpochInTx after losing the insert to a row it then reads served: %v", err) + } + if got != address { + t.Errorf("MintUnderEpochInTx = %q, want %q", got, address) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + t.Run("a row lost since it was inserted is refused", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectLostInsert(mock) + expectEpochRow(mock, address, &epochTestRow{storeID: storeID, mintedID: "record-a", mintedEpoch: currentEpoch, goneAtEpoch: currentEpoch}) + + got, err := MintUnderEpochInTx(context.Background(), tx, storeID, "record-a") + if !errors.Is(err, ErrEpochAddressNotServed) { + t.Fatalf("MintUnderEpochInTx = (%q, %v), want it to wrap ErrEpochAddressNotServed: the row at the address reads Gone", got, err) + } + if got != "" { + t.Errorf("MintUnderEpochInTx returned %q alongside the refusal, want an empty address", got) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + t.Run("a row the transaction cannot read is not success", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectLostInsert(mock) + expectEpochRow(mock, address, nil) + + got, err := MintUnderEpochInTx(context.Background(), tx, storeID, "record-a") + if err == nil { + t.Fatalf("MintUnderEpochInTx = %q, nil, want an error: nothing this transaction can read says the store serves the row another transaction inserted", got) + } + if !errors.Is(err, duplicate) { + t.Errorf("MintUnderEpochInTx error = %v, want it to wrap the duplicate-key error, so a caller can tell a race from a refusal", err) + } + if errors.Is(err, ErrEpochAddressNotServed) { + t.Errorf("MintUnderEpochInTx error = %v wraps ErrEpochAddressNotServed, want not: the row's status is unknown, not Gone", err) + } + if got != "" { + t.Errorf("MintUnderEpochInTx returned %q alongside the error, want an empty address", got) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + t.Run("a failed re-read reports both errors", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + expectLostInsert(mock) + readFailure := errors.New("read failed") + mock.ExpectQuery(epochSQLRow).WithArgs(address).WillReturnError(readFailure) + + got, err := MintUnderEpochInTx(context.Background(), tx, storeID, "record-a") + if !errors.Is(err, duplicate) || !errors.Is(err, readFailure) { + t.Fatalf("MintUnderEpochInTx = (%q, %v), want an error wrapping both the duplicate key and the failed re-read", got, err) + } + if got != "" { + t.Errorf("MintUnderEpochInTx returned %q alongside the error, want an empty address", got) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) +} + +// TestMintUnderEpochInTxRefusesAPairWhoseAddressCouldOutgrowItsColumn pins the +// bound on epoch_minted_addresses.address, a VARCHAR(255) PRIMARY KEY: a pair +// one byte too long is refused with ErrEpochAddressTooLong before any +// statement runs, and the widest pair that fits is minted. The bound is the +// address at the widest epoch the INT counter holds, so a pair that mints +// today still fits after any number of bumps. +func TestMintUnderEpochInTxRefusesAPairWhoseAddressCouldOutgrowItsColumn(t *testing.T) { + t.Parallel() + + storeID := strings.Repeat("s", 115) + widest := strings.Repeat("i", 115) + if n := len(epochAddress(storeID, widest, math.MaxInt32)); n != maxEpochAddressLen { + t.Fatalf("the widest pair composes to %d bytes at the widest epoch, want exactly %d: the boundary below is not at the limit", n, maxEpochAddressLen) + } + + t.Run("one byte over is refused before any statement", func(t *testing.T) { + t.Parallel() + _, mock, tx := beginMockTx(t) + + got, err := MintUnderEpochInTx(context.Background(), tx, storeID, widest+"i") + if !errors.Is(err, ErrEpochAddressTooLong) { + t.Fatalf("MintUnderEpochInTx(one byte over) = (%q, %v), want it to wrap ErrEpochAddressTooLong", got, err) + } + if got != "" { + t.Errorf("MintUnderEpochInTx returned %q alongside the refusal, want an empty address", got) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("a refused pair must run no statement: %v", err) + } + }) + + t.Run("the widest pair that fits is minted", func(t *testing.T) { + t.Parallel() + // Epoch 1 is the epoch whose address is SHORTEST, so this pair passes + // only because the bound is charged at the widest epoch and met there. + address := epochAddress(storeID, widest, 1) + _, mock, tx := beginMockTx(t) + expectEpochRead(mock, 1) + expectEpochRow(mock, address, nil) + mock.ExpectExec(epochSQLInsertRoot). + WithArgs(address, storeID, widest, int64(1), sqlmock.AnyArg()). + WillReturnResult(sqlmock.NewResult(0, 1)) + + got, err := MintUnderEpochInTx(context.Background(), tx, storeID, widest) + if err != nil { + t.Fatalf("MintUnderEpochInTx(the widest pair): %v", err) + } + if got != address { + t.Errorf("MintUnderEpochInTx = %q, want %q", got, address) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) +} + +// TestBumpEpochCarryingInTxCarriesTheWidestMintablePairToTheWidestEpoch is the +// carry half of the address bound. A token-scheme change re-encodes every live +// pair at the new epoch, so the widest pair a mint accepts, carried to the +// widest epoch the counter holds, must produce an address the column holds; +// the mint's bound is what guarantees it. +func TestBumpEpochCarryingInTxCarriesTheWidestMintablePairToTheWidestEpoch(t *testing.T) { + t.Parallel() + + storeID := strings.Repeat("s", 115) + widest := strings.Repeat("i", 115) + if err := validateEpochAddressInputs(storeID, widest); err != nil { + t.Fatalf("validateEpochAddressInputs(the widest pair) = %v, want nil: a mint accepts it", err) + } + root := epochAddress(storeID, widest, 1) + carried := epochAddress(storeID, widest, math.MaxInt32) + if len(carried) > maxEpochAddressLen { + t.Fatalf("the widest mintable pair carried to epoch %d is %d bytes, over the %d the column holds", math.MaxInt32, len(carried), maxEpochAddressLen) + } + + _, mock, tx := beginMockTx(t) + expectEpochRead(mock, math.MaxInt32-1) + mock.ExpectExec(`UPDATE store_epoch SET epoch = epoch \+ 1, bumped_at = \?, bumped_reason = \? WHERE id = 1`). + WithArgs(sqlmock.AnyArg(), "token-scheme-change"). + WillReturnResult(sqlmock.NewResult(0, 1)) + expectEpochRead(mock, math.MaxInt32) + mock.ExpectQuery(`SELECT address, store_id, minted_id, carried_from FROM epoch_minted_addresses WHERE gone_at_epoch IS NULL AND minted_epoch < \? ORDER BY store_id, minted_id, minted_epoch DESC`). + WithArgs(int64(math.MaxInt32)). + WillReturnRows(sqlmock.NewRows([]string{"address", "store_id", "minted_id", "carried_from"}).AddRow(root, storeID, widest, nil)) + mock.ExpectExec(`INSERT INTO epoch_minted_addresses \(address, store_id, minted_id, minted_epoch, minted_at, carried_from\) VALUES \(\?, \?, \?, \?, \?, \?\)`). + WithArgs(carried, storeID, widest, int64(math.MaxInt32), sqlmock.AnyArg(), root). + WillReturnResult(sqlmock.NewResult(0, 1)) + + got, err := BumpEpochCarryingInTx(context.Background(), tx, storeID, "token-scheme-change") + if err != nil { + t.Fatalf("BumpEpochCarryingInTx to the widest epoch: %v", err) + } + if got != math.MaxInt32 { + t.Errorf("BumpEpochCarryingInTx = %d, want %d", got, math.MaxInt32) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("the carry must insert exactly the %d-byte address of the widest pair at the widest epoch: %v", len(carried), err) + } +} + +// TestEpochAddressNeverCollidesAcrossDifferentStoreIDBoundaries is the +// regression test for gastownhall/beads#6664 (bee-ghosttrack, maintainer +// review 5268699223, item B2, first part): epochAddress's ":"-joined +// encoding is ambiguous — a colon inside storeID or id lets two +// semantically different (storeID, id, epoch) triples collide on the +// identical address string. This directly contradicts the package doc's +// "persisted once at mint and never rewritten in place" promise: a +// colliding second mint would silently steal the first triple's row +// (upsertEpochMintedAddressInTx's exists-branch UPDATE has no store_id/ +// minted_id guard). +func TestEpochAddressNeverCollidesAcrossDifferentStoreIDBoundaries(t *testing.T) { + t.Parallel() + + // Under the old "epch:%s:%s:%d" encoding, these two distinct triples + // both produce "epch:store-a:b:c:1": the colon inside the first + // triple's id ("b:c") lands exactly on the second triple's + // storeID/id boundary ("store-a:b" / "c"). + const storeA, idA = "store-a", "b:c" + const storeB, idB = "store-a:b", "c" + a := epochAddress(storeA, idA, 1) + b := epochAddress(storeB, idB, 1) + if a == b { + t.Fatalf("epochAddress(%q, %q, 1) and epochAddress(%q, %q, 1) both produced %q; a colon inside storeID or id must not let two different triples collide on the same address (B2, gastownhall/beads#6664 review 5268699223)", storeA, idA, storeB, idB, a) + } +} + +// TestUpsertEpochMintedAddressInTxIsANoOpWhenTheAddressAlreadyExists is the +// regression test for gastownhall/beads#6664 (bee-ghosttrack, maintainer +// review 5268699223, item B2, third part): once epochAddress is injective +// (TestEpochAddressNeverCollidesAcrossDifferentStoreIDBoundaries above), a +// row's address already existing provably means its stored (store_id, +// minted_id, minted_epoch) triple already matches the incoming one exactly +// — so the exists branch must perform no write at all, rather than the +// UPDATE it runs today, which (before epochAddress is injective) can steal +// another store's row through a colliding address. +func TestUpsertEpochMintedAddressInTxIsANoOpWhenTheAddressAlreadyExists(t *testing.T) { + t.Parallel() + + _, mock, tx := beginMockTx(t) + address := epochAddress("no-op-store", "record-a", 1) + + // Deliberately no ExpectExec/ExpectQuery at all: any write this branch + // attempts is an unexpected call sqlmock rejects. + err := upsertEpochMintedAddressInTx(context.Background(), tx, address, "no-op-store", "record-a", 1, true) + if err != nil { + t.Fatalf("upsertEpochMintedAddressInTx(exists=true) = %v, want nil: an already-existing address must be a no-op, not attempt a write this test set up no expectation for (B2, gastownhall/beads#6664 review 5268699223)", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet no-op-on-exists SQL expectations: %v", err) + } +} + +// TestEnsureStoreEpochRowSeedsTheSingletonWithInsertIgnore is the regression +// test for the "Unverified" note of gastownhall/beads#6664 (bee-ghosttrack, +// review 5360880888): ensureStoreEpochRow seeded +// store_epoch's singleton with a plain INSERT where version_history.go seeds +// the very same row with INSERT IGNORE. It follows that seed's shape: read +// first, seed only when the row is absent (so the seed happens once per store, +// not once per mint), then read the row back so the epoch it reports is what +// the row actually holds. +func TestEnsureStoreEpochRowSeedsTheSingletonWithInsertIgnore(t *testing.T) { + t.Parallel() + + _, mock, tx := beginMockTx(t) + + // store_epoch starts empty (migration 0067), so the first read finds no row. + mock.ExpectQuery(`SELECT epoch FROM store_epoch WHERE id = 1`). + WillReturnRows(sqlmock.NewRows([]string{"epoch"})) + mock.ExpectExec(`INSERT IGNORE INTO store_epoch \(id, epoch\) VALUES \(1, 1\)`). + WillReturnResult(sqlmock.NewResult(0, 1)) + mock.ExpectQuery(`SELECT epoch FROM store_epoch WHERE id = 1`). + WillReturnRows(sqlmock.NewRows([]string{"epoch"}).AddRow(1)) + + got, err := ensureStoreEpochRow(context.Background(), tx) + if err != nil { + t.Fatalf("ensureStoreEpochRow on an empty store_epoch: %v", err) + } + if got != 1 { + t.Fatalf("ensureStoreEpochRow on an empty store_epoch = epoch %d, want 1 (a freshly seeded singleton starts at epoch 1)", got) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet store_epoch seed SQL expectations: %v", err) + } +} + +// TestEnsureStoreEpochRowReportsTheEpochOfARowItsSeedDidNotWrite pins why the +// seed reads back instead of returning the literal 1 it would have written: +// INSERT IGNORE writes nothing when the singleton already exists, and the row +// it leaves behind may hold a later epoch than the one this call tried to seed. +func TestEnsureStoreEpochRowReportsTheEpochOfARowItsSeedDidNotWrite(t *testing.T) { + t.Parallel() + + _, mock, tx := beginMockTx(t) + + mock.ExpectQuery(`SELECT epoch FROM store_epoch WHERE id = 1`). + WillReturnRows(sqlmock.NewRows([]string{"epoch"})) + // The seed is ignored: another writer's row is already there. + mock.ExpectExec(`INSERT IGNORE INTO store_epoch \(id, epoch\) VALUES \(1, 1\)`). + WillReturnResult(sqlmock.NewResult(0, 0)) + mock.ExpectQuery(`SELECT epoch FROM store_epoch WHERE id = 1`). + WillReturnRows(sqlmock.NewRows([]string{"epoch"}).AddRow(3)) + + got, err := ensureStoreEpochRow(context.Background(), tx) + if err != nil { + t.Fatalf("ensureStoreEpochRow when the seed is ignored: %v", err) + } + if got != 3 { + t.Fatalf("ensureStoreEpochRow when the seed is ignored = epoch %d, want 3 (the epoch the surviving row holds, not the 1 the ignored seed would have written)", got) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet ignored-seed SQL expectations: %v", err) + } +} + +// handWrittenEpochAddresses returns the position of every string literal in src +// that writes an address by hand, meaning one that starts with the address +// encoding's prefix. It reads string literals only, so a comment that merely +// describes an address, or a call to epochAddress, is never reported. +func handWrittenEpochAddresses(filename string, src any) ([]string, error) { + fset := token.NewFileSet() + parsed, err := parser.ParseFile(fset, filename, src, 0) + if err != nil { + return nil, err + } + // The prefix is assembled here so this guard's own source holds no + // literal that starts with it. + addressPrefix := "epch" + ":" + var hits []string + ast.Inspect(parsed, func(n ast.Node) bool { + lit, ok := n.(*ast.BasicLit) + if !ok || lit.Kind != token.STRING { + return true + } + value, err := strconv.Unquote(lit.Value) + if err != nil { + return true + } + if strings.HasPrefix(value, addressPrefix) { + hits = append(hits, fset.Position(lit.Pos()).String()+": "+lit.Value) + } + return true + }) + return hits, nil +} + +// TestEpochCASTestsBuildEveryAddressWithEpochAddress pins the rule that this +// file never writes an address by hand (bee-ghosttrack, review 5360880888, +// Minor 4). The review found colon-form literals of the encoding that +// epochAddress retired, and one length-prefixed literal whose lengths were +// wrong. A hand-written address goes stale the moment the encoding moves, +// while a test that builds it with epochAddress cannot disagree with the code +// under test. +// +// The server leg's publication tests are held to the same rule (#6664 review): +// their fake serves a row at any address, so a stale one would still pass +// there. They cannot call epochAddress from that package, so they mint the +// address instead. +func TestEpochCASTestsBuildEveryAddressWithEpochAddress(t *testing.T) { + t.Parallel() + + for _, filename := range []string{ + "epoch_cas_test.go", + filepath.Join("..", "dolt", "epoch_cas_publish_test.go"), + } { + hits, err := handWrittenEpochAddresses(filename, nil) + if err != nil { + t.Fatalf("parse %s: %v", filename, err) + } + for _, hit := range hits { + t.Errorf("%s writes an address by hand; build it with epochAddress(storeID, id, epoch), or mint it where epochAddress is out of reach", hit) + } + } +} + +// TestHandWrittenAddressGuardHasTeeth proves the guard above can fail: it must +// report a colon-form literal and a length-prefixed literal, and must stay +// quiet for a call to epochAddress and for a comment describing an address. +func TestHandWrittenAddressGuardHasTeeth(t *testing.T) { + t.Parallel() + + prefix := "epch" + ":" + cases := []struct { + name string + src string + want int + }{ + {name: "a colon-form literal", src: "package p\nvar a = \"" + prefix + "store:id:1\"\n", want: 1}, + {name: "a length-prefixed literal", src: "package p\nvar a = \"" + prefix + "5:store:2:id:1\"\n", want: 1}, + {name: "two literals", src: "package p\nvar a, b = \"" + prefix + "x\", \"" + prefix + "y\"\n", want: 2}, + {name: "a call to epochAddress", src: "package p\nvar a = epochAddress(\"store\", \"id\", 1)\n", want: 0}, + {name: "a comment describing an address", src: "package p\n// the old form was " + prefix + "store:id:1\nvar a = 1\n", want: 0}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + hits, err := handWrittenEpochAddresses("sample.go", tc.src) + if err != nil { + t.Fatalf("parse sample: %v", err) + } + if len(hits) != tc.want { + t.Errorf("the guard reported %d hand-written addresses (%v), want %d", len(hits), hits, tc.want) + } + }) + } +} diff --git a/internal/storage/leg_contract_wiring_test.go b/internal/storage/leg_contract_wiring_test.go index 00fdcdd46..960efc16b 100644 --- a/internal/storage/leg_contract_wiring_test.go +++ b/internal/storage/leg_contract_wiring_test.go @@ -106,8 +106,10 @@ func init() { "RunImporterWiresTheCrossPlaneEdgeBetweenItsRows": importerOneAccessorWaiverReason, "RunImporterReportsTheCycleEdgeItDropped": importerOneAccessorWaiverReason, + "RunACarriedAddressKeepsResolvingAcrossLaterBumps": retentionEpochNoRoleWaiverReason, "RunADestructiveOperationEnumeratesAffectedAddressesFirst": retentionEpochNoRoleWaiverReason, "RunAHoldPreventsRemovalAndReportsInRetainedBounds": retentionEpochNoRoleWaiverReason, + "RunALostAddressStaysGoneAcrossASchemeChange": retentionEpochNoRoleWaiverReason, "RunAStoreThatRemovesStateStillAnswersGoneDurably": retentionEpochNoRoleWaiverReason, "RunAStoreWithNoLineageKnowledgeAnswersUnknownNotGone": retentionEpochNoRoleWaiverReason, "RunAnAddressNeverResolvesToADifferentState": retentionEpochNoRoleWaiverReason, @@ -121,6 +123,7 @@ func init() { "RunRemovalNeverReassignsASurvivingAddress": retentionEpochNoRoleWaiverReason, "RunRemovalReasonIsRetentionErasureOrReorganizationDistinctly": retentionEpochNoRoleWaiverReason, "RunRemovalReportsTheSurvivingRetainedWindow": retentionEpochNoRoleWaiverReason, + "RunTokenSchemeChangeCarriesLiveAddressesAndOnlyThose": retentionEpochNoRoleWaiverReason, "RunAdvisoryLeaseAloneDoesNotPreventTheInvariantViolation": crossRecordInvariantNoRoleWaiverReason, "RunCrossRecordInvariantSurvivesTwoPassingPerRecordGuards": crossRecordInvariantNoRoleWaiverReason, @@ -185,30 +188,44 @@ const cycleTracksWireGapWaiverReason = "issueops.DetectCyclesRequest.IncludeTrac "\"L-cycles-tracks\" — rather than silently answering the narrower walk); no slice through S13 adds a wire " + "parameter for it, so closing this is a new, not-yet-numbered slice rather than a missing test line" -// retentionEpochNoRoleWaiverReason covers the fifteen RetentionFixture and -// EpochFixture cases: PERMANENTLY unservable over HTTP, not merely unwired -// yet. +// retentionEpochNoRoleWaiverReason covers the eighteen RetentionFixture and +// EpochFixture cases (backend/conformance/retention_epoch_contract.go). +// All but four of the participation-generation cases run on this leg because +// their subject is a create or an update the client can send +// (internal/httpclient/served_participation_generation_test.go; those four +// stay waived under participationGenerationWireGapWaiverReason). These +// cases are different: every hook of these two fixtures IS the operation +// under test, and none has a form on the wire: +// - EpochFixture's hooks (CurrentEpoch, BumpEpoch, MintUnderEpoch, +// StillServes, Resolve, CurrentAddressFor, LoseVersion) are the R20 epoch +// operations themselves: methods on the concrete *DoltStore and +// *EmbeddedDoltStore over issueops' *InTx bodies, on no storage +// interface, no publicops role and no wire operation. Every call path +// into them starts in a test: the methods are called only from each +// store leg's own _test.go files (its EpochFixture, which the +// backend/conformance contracts drive, and its epoch tests), and the +// *InTx bodies only from those methods, from each other and from tests +// (the uow leg's fixture, through a test-only type assertion, and +// issueops' and the dolt leg's own tests). No command, role or server +// handler reaches them, so no client mutation touches epoch state either. +// - RetentionFixture's hooks (Resolve, Remove, Hold, ForceRemove, Erase, +// Mint) are retention operations no backend implements yet: every +// store leg wires them nil and its cases skip, and the wire has no +// removal, hold, erasure or gone/unknown resolve to bind them to. // -// Both fixtures are RAW STORAGE HOOKS, not a publicops role: RetentionFixture -// and EpochFixture (backend/conformance/retention_epoch_contract.go) carry no -// publicops.XXX field at all, only closures like Resolve, Remove, Hold, -// Erase, Mint, CurrentEpoch, and BumpEpoch that reach past every role -// straight at a backend's own address/epoch/retention bookkeeping — the same -// bookkeeping a storage engine keeps about itself, not a capability it was -// ever going to publish to a caller. dolt, embeddeddolt, and uow all wire -// this contract because all three ARE storage engines with that bookkeeping -// to show; the http leg is a REMOTE CLIENT of one, with no address table, -// epoch counter, or retention window of its own to report — there is nothing -// behind this door for a wire operation to open onto, now or in a later -// slice, unless some future design gives retention/epoch state its own -// publicops role and wire surface. That would be a new role, not a missing -// test line, which is why this is a named waiver rather than a ceiling -// entry. -const retentionEpochNoRoleWaiverReason = "RetentionFixture and EpochFixture carry raw address/epoch/retention " + - "storage hooks (Resolve, Remove, Hold, Erase, Mint, CurrentEpoch, BumpEpoch, ...) with no publicops role " + - "behind them; dolt, embeddeddolt and uow wire this contract because they ARE the storage engine this " + - "bookkeeping belongs to, and the http leg is a remote client with no such bookkeeping of its own to report " + - "— there is no role or wire surface for this to ever route through unless one is designed from scratch" +// Binding the hooks to the reference store behind the served env would run +// the epoch cases green, but as a property of that store, never of the +// client; the store legs already pin it. The waiver stands until retention or +// epoch state gets a publicops role and a wire operation of its own, which +// would be a new role, not a missing test line. +const retentionEpochNoRoleWaiverReason = "every RetentionFixture/EpochFixture hook is the operation under test and " + + "none has a form on the http wire: the R20 epoch operations (CurrentEpoch, BumpEpoch, MintUnderEpoch, " + + "StillServes, Resolve, CurrentAddressFor, LoseVersion) are concrete-store methods on no storage interface, " + + "publicops role or wire operation, reached only from tests (each store leg's fixture and epoch tests call the " + + "methods, the uow fixture calls the issueops bodies through a test-only type assertion), and the retention " + + "hooks (Resolve, Remove, Hold, ForceRemove, Erase, Mint) are implemented by no backend and have no wire " + + "removal, hold, erasure or resolve; binding them to the served env's reference store would test that store, " + + "not the client" // crossRecordInvariantNoRoleWaiverReason covers the seven CrossRecordInvariantFixture // cases, for the same shape of reason as retentionEpochNoRoleWaiverReason. diff --git a/internal/storage/schema/BUILD.bazel b/internal/storage/schema/BUILD.bazel index f81aca9bc..68e8b9b75 100644 --- a/internal/storage/schema/BUILD.bazel +++ b/internal/storage/schema/BUILD.bazel @@ -121,6 +121,7 @@ go_library( "migrations/ignored/0028_add_wisps_participation_generation.up.sql", "migrations/0070_add_participation_generation.up.sql", "migrations/0071_add_removed_restriction.up.sql", + "migrations/0072_add_epoch_minted_addresses.up.sql", ], importpath = "github.com/steveyegge/beads/internal/storage/schema", visibility = ["//:__subpackages__"], @@ -156,6 +157,7 @@ go_test( "migration_0069_datetime_precision_test.go", "migration_0070_add_participation_generation_test.go", "migration_0071_add_removed_restriction_test.go", + "migration_0072_add_epoch_minted_addresses_test.go", "migration_body_drain_test.go", "migration_freeze_test.go", "migration_watchdog_test.go", diff --git a/internal/storage/schema/migration_0071_add_removed_restriction_test.go b/internal/storage/schema/migration_0071_add_removed_restriction_test.go index 2b4da39ac..5ca6598d5 100644 --- a/internal/storage/schema/migration_0071_add_removed_restriction_test.go +++ b/internal/storage/schema/migration_0071_add_removed_restriction_test.go @@ -22,19 +22,11 @@ import ( const migration0071Up = "0071_add_removed_restriction.up.sql" const migration0071Down = "0071_add_removed_restriction.down.sql" -// TestLatestVersionIncludesMigration0071 pins the real next free slot this -// phase claims, superseding 0070's own version of this test -// (migration_0070_add_participation_generation_test.go). Deliberately a -// hardcoded literal for the same reason 0067's, 0068's, 0069's and 0070's -// were: LatestVersion() drifting to 71 for the wrong reason should still be -// caught by this test failing to explain why 71 is removed-restriction-shaped, -// which the CLI test below checks. -func TestLatestVersionIncludesMigration0071(t *testing.T) { - const want = 71 - if got := LatestVersion(); got != want { - t.Fatalf("LatestVersion() = %d, want %d (the migration slot claimed for issue_versions.removed_restriction)", got, want) - } -} +// TestLatestVersionIncludesMigration0071 (pinning LatestVersion() == 71) is +// superseded by TestLatestVersionIncludesMigration0072 +// (migration_0072_add_epoch_minted_addresses_test.go) now that 0072 claims +// the next free slot -- only one such pin lives at a time, matching how this +// test itself already superseded 0070's own version. // TestMigration0071AddsRemovedRestriction is a pure-Go, DB-independent check // of the frozen migration bytes themselves — it runs even where no `dolt` diff --git a/internal/storage/schema/migration_0072_add_epoch_minted_addresses_test.go b/internal/storage/schema/migration_0072_add_epoch_minted_addresses_test.go new file mode 100644 index 000000000..67389a0e0 --- /dev/null +++ b/internal/storage/schema/migration_0072_add_epoch_minted_addresses_test.go @@ -0,0 +1,184 @@ +package schema + +import ( + "os" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/testutil" +) + +// R20's epoch-transition enforcement (gastownhall/beads#5898 revision 9, +// #6136) adds one brand-new table, +// epoch_minted_addresses -- see +// migrations/0072_add_epoch_minted_addresses.up.sql for the full rationale. +// This migration is a plain, unguarded +// CREATE TABLE IF NOT EXISTS with no ALTER and no PREPARE, so it needs none +// of 0067/0068's CLI-hazard-override tests (already covered, for every +// migration including this one, by TestBundleMigrationsWithPreparedALTERAreOverriddenOrJustified +// and TestAllMigrationsSQLUsesDirectDDLForKnownCLIIncompatibilities). + +// TestLatestVersionIncludesMigration0072 pins the real next free slot this +// phase claims, superseding 0071's own version of this test +// (migration_0071_add_removed_restriction_test.go; only one such pin lives at +// a time, the same way 0071's superseded 0070's). +// +// This slice originally claimed 0069 too. It was renumbered because +// R7.1's removed_restriction migration -- a PARALLEL SIBLING of this branch, +// not an ancestor -- had also claimed 0069, and the two are now linearized: +// R7.1 goes first (it is published beneath gastownhall/beads#6661), this one +// takes the next contiguous slot. Both then moved up one more slot when +// upstream's 0069_widen_issue_versions_datetime_precision +// (gastownhall/beads#6675) landed first, and once more when +// gastownhall/beads#6943's 0070_add_participation_generation (the +// versioned-history write fence) was ordered ahead of them: migrations are +// numbered by landing order, so R7.1 is 0071 and this is 0072. +// Two files sharing a +// version number panic checkNoDuplicateVersions at store open, before any +// command's RunE, which bricks every bd invocation rather than merely +// reddening a package -- see the same hazard on #6358 vs #6008. +// +// Deliberately a hardcoded literal for the same reason 0070's and 0071's +// were: LatestVersion() drifting to 72 for the wrong reason (an unrelated +// migration landing first) should still be caught by this test failing to +// explain why 72 is epoch-minted-addresses-shaped. +func TestLatestVersionIncludesMigration0072(t *testing.T) { + const want = 72 + if got := LatestVersion(); got != want { + t.Fatalf("LatestVersion() = %d, want %d (epoch_minted_addresses migration slot claimed by gastownhall/beads#6664)", got, want) + } +} + +const migration0072Up = "0072_add_epoch_minted_addresses.up.sql" +const migration0072Down = "0072_add_epoch_minted_addresses.down.sql" + +// TestMigration0072RecordsSurvivalInTheAddressTable is a pure-Go, +// DB-independent check of the frozen migration bytes: it runs even where no +// `dolt` binary is available. +// +// An address's survival is RECORDED on its own row, not inferred from other +// rows (gastownhall/beads#6664, bee-ghosttrack review 5360880888, Major 1): +// carried_from names the root address of the lineage a token-scheme change +// carried the row from, and gone_at_epoch is the epoch the store lost it in. +// Both are NULL for a fresh mint. The index serves the two lookups that +// follow a lineage (the newest carry of a root, and every carry of a root to +// lose), and the CHECK is the database-level tripwire against a stale or +// rewound epoch being written into a loss marker. It is >=, not >, because a +// carry row is born at the new epoch and can be lost in that same epoch. +// +// The migration is edited IN PLACE and gains no ALTER and no second slot: +// 0072 has never been applied to a store, and an ALTER over an already-applied +// copy of the earlier shape would be a migration slot nothing may claim. +func TestMigration0072RecordsSurvivalInTheAddressTable(t *testing.T) { + upSQL, err := MigrationSQL(migration0072Up) + if err != nil { + t.Fatalf("MigrationSQL(%s) error = %v, want the migration file to exist", migration0072Up, err) + } + for _, want := range []string{ + "CREATE TABLE IF NOT EXISTS epoch_minted_addresses", + "address VARCHAR(255) NOT NULL,", + "store_id VARCHAR(255) NOT NULL,", + "minted_id VARCHAR(255) NOT NULL,", + "minted_epoch INT NOT NULL,", + "minted_at DATETIME NOT NULL,", + "carried_from VARCHAR(255) NULL,", + "gone_at_epoch INT NULL,", + "PRIMARY KEY (address),", + "INDEX idx_epoch_minted_addresses_carried_from (carried_from),", + "CONSTRAINT ck_epoch_minted_addresses_gone_not_before_mint CHECK (gone_at_epoch IS NULL OR gone_at_epoch >= minted_epoch)", + } { + if !strings.Contains(upSQL, want) { + t.Errorf("0072 up migration missing %q\nfull SQL:\n%s", want, upSQL) + } + } + + // Only the statements count here: the header comments describe the + // migration in words that would otherwise trip these checks. + statements := strings.ToUpper(stripSQLLineComments(upSQL)) + if strings.Contains(statements, "ALTER TABLE") { + t.Error("0072 up migration must not ALTER anything: the survival columns belong to its own CREATE TABLE, because the migration is edited in place while no store has applied it") + } + if strings.Contains(statements, "PREPARE") { + t.Error("0072 up migration must stay plain DDL: CREATE TABLE IF NOT EXISTS needs no guarded PREPARE, and adding one would put it on the pre-2.3 CLI hazard path") + } + if got := strings.Count(statements, "CREATE TABLE"); got != 1 { + t.Errorf("0072 up migration has %d CREATE TABLE statements, want exactly 1", got) + } + + // Only migrations/*.up.sql is embedded, so the down side reads from disk by + // package-relative path, as migration 0067's test does. + downBytes, err := os.ReadFile("migrations/" + migration0072Down) + if err != nil { + t.Fatalf("read %s: %v, want the migration file to exist", migration0072Down, err) + } + if !strings.Contains(string(downBytes), "DROP TABLE IF EXISTS epoch_minted_addresses") { + t.Errorf("0072 down migration must drop epoch_minted_addresses, got:\n%s", downBytes) + } +} + +// TestMigration0072RecordsSurvivalInTheAddressTableThroughDoltCLI applies the +// full migration bundle through a real `dolt` binary (skipped without one, see +// testutil.RequireDoltBinary) and checks what a pure-Go SQL-text check cannot: +// the column types and nullability Dolt reports, the index it actually built, +// and that the CHECK is enforced on both INSERT and UPDATE, with the boundary +// it is meant to have. +func TestMigration0072RecordsSurvivalInTheAddressTableThroughDoltCLI(t *testing.T) { + testutil.RequireDoltBinary(t) + + dir := t.TempDir() + runDoltCommand(t, dir, "init", "--name", "test", "--email", "test@example.com") + runDoltSQL(t, dir, AllMigrationsSQL()) + + requireDoltColumnShape(t, dir, "epoch_minted_addresses", "carried_from", "varchar(255)", "YES") + requireDoltDataType(t, dir, "epoch_minted_addresses", "gone_at_epoch", "int", "YES") + // The five columns the mint writes keep their shape, so the mint's + // five-column INSERT is unchanged. + requireDoltColumnShape(t, dir, "epoch_minted_addresses", "address", "varchar(255)", "NO") + requireDoltColumnShape(t, dir, "epoch_minted_addresses", "minted_epoch", "int", "NO") + requireDoltNoRows(t, dir, "SELECT address FROM epoch_minted_addresses", "epoch_minted_addresses") + requireDoltCount(t, dir, + `SELECT COUNT(*) AS c FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = DATABASE() AND TABLE_NAME = 'epoch_minted_addresses' AND INDEX_NAME = 'idx_epoch_minted_addresses_carried_from' AND COLUMN_NAME = 'carried_from'`, + "1") + + // A fresh mint writes the five original columns and is a root row: both + // survival columns come out NULL with no change to the INSERT. + runDoltSQL(t, dir, `INSERT INTO epoch_minted_addresses (address, store_id, minted_id, minted_epoch, minted_at) VALUES ('root-a', 'store', 'record-a', 3, '2026-09-01 00:00:00')`) + requireDoltCount(t, dir, `SELECT COUNT(*) AS c FROM epoch_minted_addresses WHERE address = 'root-a' AND carried_from IS NULL AND gone_at_epoch IS NULL`, "1") + + // A carry row names the lineage root. + runDoltSQL(t, dir, `INSERT INTO epoch_minted_addresses (address, store_id, minted_id, minted_epoch, minted_at, carried_from) VALUES ('carry-a', 'store', 'record-a', 5, '2026-09-01 00:00:00', 'root-a')`) + requireDoltCount(t, dir, `SELECT COUNT(*) AS c FROM epoch_minted_addresses WHERE carried_from = 'root-a'`, "1") + + // The CHECK: an epoch lost in may not precede the epoch minted in. + if err := runDoltSQLExpectingError(t, dir, `UPDATE epoch_minted_addresses SET gone_at_epoch = 2 WHERE address = 'root-a'`); err == nil { + t.Error("UPDATE gone_at_epoch below minted_epoch succeeded, want the CHECK constraint to reject it") + } + if err := runDoltSQLExpectingError(t, dir, `INSERT INTO epoch_minted_addresses (address, store_id, minted_id, minted_epoch, minted_at, gone_at_epoch) VALUES ('bad-b', 'store', 'record-b', 4, '2026-09-01 00:00:00', 3)`); err == nil { + t.Error("INSERT with gone_at_epoch below minted_epoch succeeded, want the CHECK constraint to reject it") + } + // Equal is allowed: a carry row is born at the new epoch and can be lost in + // that same epoch. + runDoltSQL(t, dir, `UPDATE epoch_minted_addresses SET gone_at_epoch = 3 WHERE address = 'root-a'`) + requireDoltCount(t, dir, `SELECT COUNT(*) AS c FROM epoch_minted_addresses WHERE address = 'root-a' AND gone_at_epoch = 3`, "1") + // A later epoch is allowed too. + runDoltSQL(t, dir, `UPDATE epoch_minted_addresses SET gone_at_epoch = 9 WHERE address = 'carry-a'`) + requireDoltCount(t, dir, `SELECT COUNT(*) AS c FROM epoch_minted_addresses WHERE address = 'carry-a' AND gone_at_epoch = 9`, "1") + + // The address stays the primary key. + if err := runDoltSQLExpectingError(t, dir, `INSERT INTO epoch_minted_addresses (address, store_id, minted_id, minted_epoch, minted_at) VALUES ('root-a', 'store', 'record-a', 3, '2026-09-01 00:00:00')`); err == nil { + t.Error("duplicate address insert succeeded, want primary key violation") + } +} + +// stripSQLLineComments drops "--" comment lines so a check over a migration's +// statements is not tripped by the prose in its header. +func stripSQLLineComments(sqlText string) string { + var kept []string + for _, line := range strings.Split(sqlText, "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "--") { + continue + } + kept = append(kept, line) + } + return strings.Join(kept, "\n") +} diff --git a/internal/storage/schema/migrations/0072_add_epoch_minted_addresses.down.sql b/internal/storage/schema/migrations/0072_add_epoch_minted_addresses.down.sql new file mode 100644 index 000000000..af8d83eea --- /dev/null +++ b/internal/storage/schema/migrations/0072_add_epoch_minted_addresses.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS epoch_minted_addresses; diff --git a/internal/storage/schema/migrations/0072_add_epoch_minted_addresses.up.sql b/internal/storage/schema/migrations/0072_add_epoch_minted_addresses.up.sql new file mode 100644 index 000000000..ec946cf8e --- /dev/null +++ b/internal/storage/schema/migrations/0072_add_epoch_minted_addresses.up.sql @@ -0,0 +1,64 @@ +-- epoch_minted_addresses: durable storage for R20 epoch-transition +-- enforcement (gastownhall/beads#5898 revision 9, +-- #6136). Independent of store_epoch (0067, #6135) -- that table holds only +-- the store-wide singleton epoch counter (the current generation number); +-- this table tracks the individual addresses minted under each generation, +-- which store_epoch has no room to hold. It is also independent of +-- expected_revision_records (R16, #6133) and any RetentionFixture/R17 +-- table: R20 epoch reasoning is evaluated on its own, without consulting +-- retention/erasure state, and this migration adds no R17 resolve, remove, +-- hold, force-remove, erase, or mint logic. +-- +-- One row per minted address, not one row per id: MintUnderEpoch is +-- deterministic in (store_id, id, epoch) -- see epochAddress() in +-- internal/storage/issueops/epoch_cas.go -- so minting the same id again +-- under the same epoch reproduces the same address and finds the same +-- row, while a later mint of that id under a bumped epoch produces a +-- DIFFERENT address and adds a new row alongside the old one. Old rows are +-- kept: StillServes and Resolve answer for any address from its own row, +-- whatever epoch it was minted in. +-- +-- Survival is RECORDED on the row, never inferred from other rows +-- (gastownhall/beads#6664, bee-ghosttrack review 5360880888, Major 1). +-- gone_at_epoch is NULL while the store serves the address and otherwise the +-- epoch the store lost it in; it is set once and never cleared, so a lost +-- address stays lost. carried_from is NULL for a fresh mint, and for a row a +-- token-scheme change carried to a new epoch it is the address of the +-- lineage's ROOT (never the previous carry), so every member of a lineage is +-- one indexed lookup from the newest. The index on carried_from serves exactly +-- the two lookups that follow a lineage: the newest carry of a root, and every +-- carry of a root when the lineage is lost. The CHECK is a database-level +-- tripwire against a stale or rewound epoch being written into a loss marker; +-- it is >=, not >, because a carry row is born at the new epoch and can be lost +-- in that same epoch. +-- +-- address is the PRIMARY KEY: it is the deterministic token the fixture's +-- StillServes/Resolve/CurrentAddressFor hooks look addresses up by, so it +-- must be unique and indexed; it is never recomputed from the other columns +-- at read time; it is written once, whatever the epoch was at mint time. +-- +-- store_id and minted_id are VARCHAR(255) for parity with this schema's +-- other identifier columns. minted_epoch is INT to match store_epoch.epoch's +-- own type. minted_at is DATETIME, not a nanosecond-precision integer: unlike +-- expected_revision_records.change_at_nanos (R16, 0069 on that slice), no +-- R20 contract case compares minted_at for exact round-tripping -- it exists +-- for observability only, so it follows store_epoch.bumped_at's own +-- DATETIME precedent rather than needing BIGINT. +-- +-- Plain, unguarded CREATE TABLE IF NOT EXISTS: a brand-new main-plane table +-- with no ALTER, no PREPARE, and no dolt-ignored (clone-local) table +-- involved, so none of cli_migrations.go's CLI-bundle overrides, an +-- ignored/ twin, or a nondeterminism-allowlist entry apply +-- (scripts/check-migration-hygiene.sh checks B-E). +CREATE TABLE IF NOT EXISTS epoch_minted_addresses ( + address VARCHAR(255) NOT NULL, + store_id VARCHAR(255) NOT NULL, + minted_id VARCHAR(255) NOT NULL, + minted_epoch INT NOT NULL, + minted_at DATETIME NOT NULL, + carried_from VARCHAR(255) NULL, + gone_at_epoch INT NULL, + PRIMARY KEY (address), + INDEX idx_epoch_minted_addresses_carried_from (carried_from), + CONSTRAINT ck_epoch_minted_addresses_gone_not_before_mint CHECK (gone_at_epoch IS NULL OR gone_at_epoch >= minted_epoch) +); diff --git a/internal/storage/uow/expected_revision_contract_test.go b/internal/storage/uow/expected_revision_contract_test.go index 53f17947d..a1ccfa1b4 100644 --- a/internal/storage/uow/expected_revision_contract_test.go +++ b/internal/storage/uow/expected_revision_contract_test.go @@ -8,11 +8,21 @@ import ( ) // TestExpectedRevisionContract wires this leg into the R16/R17 -// expected-revision contract. Phase 0 leaves every hook nil in every -// backend's fixture kit (architecture §12), so each case below skips by -// name; this file exists so TestEveryLegWiresEveryRoleContract counts this -// leg, and so the cases start running for real the moment this leg's fixture -// kit grows a non-nil hook. +// expected-revision contract. Every hook is nil (architecture §12's Phase 0 +// default), so each case below skips by name; this file exists so +// TestEveryLegWiresEveryRoleContract counts this leg, and so the cases +// start running for real the moment this leg grows a real +// CompareAndSetVersion (gastownhall/beads#6358). +// +// This leg previously wired every hook to an honest "not implemented" stub +// instead of leaving them nil, so each case ran for real and +// failed for that documented reason rather than skipping. That premise — +// that a later slice would give this leg its CompareAndSetVersion — went +// stale once review moved the real implementation to +// gastownhall/beads#6358 instead; a stub whose failure message blames a +// bead that will never land it is worse than an honest skip, so this file +// reverts to nil hooks (gastownhall/beads#6664, bee-ghosttrack review +// 5268699223). func TestExpectedRevisionContract(t *testing.T) { ctx := context.Background() fixture := conformance.ExpectedRevisionFixture{IssuePrefix: "erev"} diff --git a/internal/storage/uow/retention_epoch_contract_test.go b/internal/storage/uow/retention_epoch_contract_test.go index a2d8bcf00..91d3a5324 100644 --- a/internal/storage/uow/retention_epoch_contract_test.go +++ b/internal/storage/uow/retention_epoch_contract_test.go @@ -2,9 +2,11 @@ package uow import ( "context" + "fmt" "testing" "github.com/steveyegge/beads/backend/conformance" + storeops "github.com/steveyegge/beads/internal/storage/issueops" ) // TestRetentionContract wires this leg into the R20 retention contract. @@ -57,11 +59,28 @@ func TestRetentionContract(t *testing.T) { }) } -// TestEpochContract wires this leg into the R20 epoch contract. Same Phase 0 -// all-nil state as TestRetentionContract above. +// TestEpochContract runs the R20 epoch contract (gastownhall/beads#5898 +// revision 9, #6136) against the unit-of-work +// provider, which reaches the same internal/storage/issueops's epoch Tx +// functions the two store backends wrap — directly against the unit of +// work's own runner (the *baseUOW/base.tx.Runner() pattern +// expected_revision_contract_test.go's expectedRevisionUOWMutateOutside and +// commenter_contract_test.go's SeedCommentAt both use), NOT through the +// domain issue repository: EpochFixture's seven hooks are store-singleton/ +// administrative operations with no natural correspondence to +// IssueUseCase's per-issue CRUD/notification semantics, so unlike +// CompareAndSetVersion there is no domain/use-case method to route through +// in the first place — every hook below needs the escape hatch, not just +// one. +// +// One provider for the whole suite (each newUOWRoleFixtureProvider boots a +// real Dolt sql-server) and NO t.Parallel: this backend has no per-test +// copy-on-write branch, so store_epoch/epoch_minted_addresses are +// database-global and a parallel subtest would corrupt another subtest's +// row — same reasoning as TestExpectedRevisionContract's. func TestEpochContract(t *testing.T) { ctx := context.Background() - fixture := conformance.EpochFixture{IssuePrefix: "epch"} + fixture := newUOWEpochFixture(t, ctx, "epch") t.Run("AnEpochBumpIsTriggeredOnlyByRestoreReinitOrSchemeChange", func(t *testing.T) { conformance.RunAnEpochBumpIsTriggeredOnlyByRestoreReinitOrSchemeChange(t, ctx, fixture) @@ -69,4 +88,184 @@ func TestEpochContract(t *testing.T) { t.Run("EpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed", func(t *testing.T) { conformance.RunEpochBumpVoidsOnlyAddressesOfVersionsNoLongerServed(t, ctx, fixture) }) + t.Run("TokenSchemeChangeCarriesLiveAddressesAndOnlyThose", func(t *testing.T) { + conformance.RunTokenSchemeChangeCarriesLiveAddressesAndOnlyThose(t, ctx, fixture) + }) + t.Run("ACarriedAddressKeepsResolvingAcrossLaterBumps", func(t *testing.T) { + conformance.RunACarriedAddressKeepsResolvingAcrossLaterBumps(t, ctx, fixture) + }) + t.Run("ALostAddressStaysGoneAcrossASchemeChange", func(t *testing.T) { + conformance.RunALostAddressStaysGoneAcrossASchemeChange(t, ctx, fixture) + }) +} + +// newUOWEpochFixture wires this leg's UnitOfWorkProvider into the R20 epoch +// contract. Like newUOWExpectedRevisionFixture, this fixture needs none of +// roleFixtureKit's CreateIssue/CreateWisp/QueryScalar/CountHistory hooks — +// it has no fields for them — so it does not route through +// newUOWRoleFixtureKit at all. +func newUOWEpochFixture(t *testing.T, ctx context.Context, prefix string) conformance.EpochFixture { + t.Helper() + provider := newUOWRoleFixtureProvider(t, ctx, prefix) + return conformance.EpochFixture{ + IssuePrefix: prefix, + CurrentEpoch: epochUOWCurrentEpoch(provider), + BumpEpoch: epochUOWBumpEpoch(provider), + MintUnderEpoch: epochUOWMintUnderEpoch(provider), + StillServes: epochUOWStillServes(provider), + Resolve: epochUOWResolve(provider), + CurrentAddressFor: epochUOWCurrentAddressFor(provider), + LoseVersion: epochUOWLoseVersion(provider), + } +} + +// epochUOWRunner recovers the *baseUOW a given attempt's UnitOfWork actually +// is, the same type assertion expectedRevisionUOWMutateOutside makes: Tx. +// Runner() is unexported outside this package and none of the seven epoch +// operations has a domain/use-case method to reach it through instead. +func epochUOWRunner(uw UnitOfWork) (storeops.DBTX, error) { + base, ok := uw.(*baseUOW) + if !ok { + return nil, fmt.Errorf("epoch CAS: unit of work %T does not expose the runner the epoch Tx functions need", uw) + } + return base.tx.Runner(), nil +} + +func epochUOWCurrentEpoch(provider UnitOfWorkProvider) func(ctx context.Context, storeID string) (int, error) { + return func(ctx context.Context, storeID string) (int, error) { + return RunTxRead(ctx, provider, func(ctx context.Context, uw UnitOfWork) (int, error) { + runner, err := epochUOWRunner(uw) + if err != nil { + return 0, err + } + return storeops.CurrentEpochInTx(ctx, runner, storeID) + }) + } +} + +// epochUOWBumpEpoch chooses the bump by the typed trigger: only a +// token-scheme-change bump carries live addresses to the new epoch, and the +// other two triggers move the counter alone. The choice lives here, in the +// contract-test translation layer, because the shared body has no switch on +// the reason string. +func epochUOWBumpEpoch(provider UnitOfWorkProvider) func(ctx context.Context, storeID string, trigger conformance.EpochBumpTrigger) (int, error) { + return func(ctx context.Context, storeID string, trigger conformance.EpochBumpTrigger) (int, error) { + return RunTxResult(ctx, provider, func(ctx context.Context, uw UnitOfWork) (int, string, error) { + runner, err := epochUOWRunner(uw) + if err != nil { + return 0, "", err + } + bump := storeops.BumpEpochInTx + if trigger == conformance.EpochBumpTriggerTokenSchemeChange { + bump = storeops.BumpEpochCarryingInTx + } + epoch, err := bump(ctx, runner, storeID, trigger.String()) + if err != nil { + return 0, "", err + } + return epoch, fmt.Sprintf("bd: bump epoch for %s (%s)", storeID, trigger.String()), nil + }) + } +} + +func epochUOWMintUnderEpoch(provider UnitOfWorkProvider) func(ctx context.Context, storeID, id string) (conformance.Address, error) { + return func(ctx context.Context, storeID, id string) (conformance.Address, error) { + address, err := RunTxResult(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, string, error) { + runner, err := epochUOWRunner(uw) + if err != nil { + return "", "", err + } + address, err := storeops.MintUnderEpochInTx(ctx, runner, storeID, id) + if err != nil { + return "", "", err + } + return address, fmt.Sprintf("bd: mint %s under epoch for %s", id, storeID), nil + }) + if err != nil { + return "", err + } + return conformance.Address(address), nil + } +} + +func epochUOWStillServes(provider UnitOfWorkProvider) func(ctx context.Context, storeID string, address conformance.Address) (bool, error) { + return func(ctx context.Context, storeID string, address conformance.Address) (bool, error) { + return RunTxRead(ctx, provider, func(ctx context.Context, uw UnitOfWork) (bool, error) { + runner, err := epochUOWRunner(uw) + if err != nil { + return false, err + } + return storeops.StillServesInTx(ctx, runner, storeID, string(address)) + }) + } +} + +func epochUOWResolve(provider UnitOfWorkProvider) func(ctx context.Context, storeID string, address conformance.Address) (conformance.RetentionAnswer, error) { + return func(ctx context.Context, storeID string, address conformance.Address) (conformance.RetentionAnswer, error) { + result, err := RunTxRead(ctx, provider, func(ctx context.Context, uw UnitOfWork) (storeops.EpochResolveResult, error) { + runner, err := epochUOWRunner(uw) + if err != nil { + return storeops.EpochResolveResult{}, err + } + return storeops.ResolveEpochInTx(ctx, runner, storeID, string(address)) + }) + if err != nil { + return conformance.RetentionAnswer{}, err + } + return epochResolveResultToConformance(result), nil + } +} + +func epochUOWCurrentAddressFor(provider UnitOfWorkProvider) func(ctx context.Context, storeID string, oldAddress conformance.Address) (conformance.Address, error) { + return func(ctx context.Context, storeID string, oldAddress conformance.Address) (conformance.Address, error) { + address, err := RunTxRead(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) { + runner, err := epochUOWRunner(uw) + if err != nil { + return "", err + } + return storeops.CurrentAddressForInTx(ctx, runner, storeID, string(oldAddress)) + }) + if err != nil { + return "", err + } + return conformance.Address(address), nil + } +} + +func epochUOWLoseVersion(provider UnitOfWorkProvider) func(ctx context.Context, storeID string, address conformance.Address) error { + return func(ctx context.Context, storeID string, address conformance.Address) error { + return RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) { + runner, err := epochUOWRunner(uw) + if err != nil { + return "", err + } + if err := storeops.LoseVersionInTx(ctx, runner, storeID, string(address)); err != nil { + return "", err + } + return fmt.Sprintf("bd: lose version %s for %s", address, storeID), nil + }) + } +} + +// epochRestrictionToConformance translates this leg's local +// storeops.EpochRestriction into backend/conformance's own Restriction +// vocabulary — the contract-test file is the translation boundary, per +// epoch_cas.go's package doc. +func epochRestrictionToConformance(r storeops.EpochRestriction) conformance.Restriction { + switch r { + case storeops.EpochRestrictionLive: + return conformance.RestrictionLive + case storeops.EpochRestrictionGoneReorganization: + return conformance.RestrictionGoneReorganization + default: + return conformance.RestrictionUnknown + } +} + +func epochResolveResultToConformance(r storeops.EpochResolveResult) conformance.RetentionAnswer { + return conformance.RetentionAnswer{ + Restriction: epochRestrictionToConformance(r.Restriction), + ProducingStore: r.ProducingStore, + Epoch: r.Epoch, + } } From 3488dc4536ee4438c424a8d65031f379744ce67f Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 19 Oct 2026 03:11:14 +0000 Subject: [PATCH 03/41] fix(init): `bd init --server --external` starts no server; a failed `bd init --server` can be retried (#7674) * fix(init): admit a retried `bd init --server` over an empty .beads/dolt A first `bd init --server` that cannot reach its server fails after creating the local Dolt root, leaving an empty .beads/dolt, no .local_version witness, and no config.yaml/metadata.json recording the server choice. The legacy-upgrade guard runs before init writes that choice, so it saw an embedded-looking workspace with a local Dolt root and refused every retry as a "legacy Dolt workspace"; no bd command could repair it. guardLegacyUpgradeWorkspaceForInit(beadsDir, initSelectsServer) carries init's already-resolved server intent into the guard. It is consulted for exactly one decision: an empty, witness-less .beads/dolt is admitted, the same admission a workspace that records server mode already gets (#6935). It is deliberately not folded into the workspace's own server mode, so a non-empty root with an unreadable witness and an embedded repository beside a non-empty root keep their current outcomes. guardLegacyUpgradeWorkspace is now the initSelectsServer=false wrapper, and only init's main guard call passes the flag; the proxied-server init path keeps the plain wrapper. Ported from the versioned-beads fork (donnabox/codex/preview3-source- catchup-46aa), commits bc8d72182a (red) and 5bdfbfe5e2 (green). The fork's graph_mode refusal in the same function is graph-preview-only and is not carried here. Tests: TestLegacyUpgradeGuardForInitAdmitsEmptyDoltRoot and TestLegacyUpgradeGuardForInitAdmissionStaysNarrow pin the admission and its narrowness at the guard; TestInitServerFlagAdmitsEmptyDoltRootWithoutConfig and TestInitServerFlagRetryAfterFailedFirstAttempt run the real binary against the test Dolt server and fail on main with the legacy refusal. Co-authored-by: Jim Wordelman Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * fix(init): `bd init --server --external` must not auto-start a server `--external` declares the Dolt server managed outside bd ("skip server startup", per its help), but init builds its dolt.Config by hand with `AutoStart: initServerMode && BEADS_DOLT_AUTO_START != "0"`, so the flag never reached the auto-start decision. A `bd init --server --external` that named no port therefore ran dolt and tried to start a repo-local sql-server. A named port was already declared external by rule 2c (#5934), but its failure still advised `bd dolt start`, the one thing --external says bd must not do, and each route (host:port, socket, shared server, auto-start pinned off) worded the failure differently. - AutoStart is now `initServerMode && !externalServer && ...`. - externalServerUnreachableMessage rewrites only a failed dial (*net.OpError with Op "dial") under --external: it names the endpoint that was dialed and says bd did not start a server. A dial to port 0 means no port was named, so that message asks for --server-port or BEADS_DOLT_SERVER_PORT and offers no default (GH#2098, GH#2372). Any other open failure keeps its own text. Ported from the versioned-beads fork (donnabox/codex/preview3-source- catchup-46aa), commits 234ca3d502 and a601a80e77 (red/green), 03a64a1aba (socket helper rename), and a6ca0536c9 and 397fc4c69e (red/green: no-port arm and wording). The fork's graph-preview dispatch at the top of init and its --skip-hooks help text change are graph-only and not carried. The TestInitExternalProvisionerShape doc comment is corrected: on main every provisioner cell is a control (a named port is already external), not a reproduction. Tests: TestInitServerExternalDeadPortStartsNoServer, TestInitServerExternalNoPortStartsNoServer and TestInitExternalUnreachableMessage run the real binary against a dolt shim that records every launch and fail on main; TestInitSharedServerExternalDeadPortStillFails, TestInitServerWithoutNamedPortStillStartsServer and TestInitExternalProvisionerShape are controls; and TestExternalServerUnreachableMessage pins which errors are rewritten. Co-authored-by: Jim Wordelman Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Jim Wordelman Co-authored-by: Claude Opus 5.5 (1M context) --- CHANGELOG.md | 18 + cmd/bd/BUILD.bazel | 4 + cmd/bd/init.go | 49 +- cmd/bd/init_external_autostart_test.go | 430 ++++++++++++++++++ cmd/bd/init_external_message_test.go | 97 ++++ cmd/bd/legacy_upgrade_guard.go | 22 + ...legacy_upgrade_guard_init_flag_e2e_test.go | 87 ++++ cmd/bd/legacy_upgrade_guard_init_flag_test.go | 143 ++++++ 8 files changed, 847 insertions(+), 3 deletions(-) create mode 100644 cmd/bd/init_external_autostart_test.go create mode 100644 cmd/bd/init_external_message_test.go create mode 100644 cmd/bd/legacy_upgrade_guard_init_flag_e2e_test.go create mode 100644 cmd/bd/legacy_upgrade_guard_init_flag_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 60d4d31e9..4dbfca8b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -763,6 +763,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **`bd init --server --external` no longer starts a Dolt server of its own.** + Init builds its store config by hand and turned auto-start on for every + server init, so an `--external` init that named no port launched a + repo-local `dolt sql-server`, although `--external` says the server is + managed outside bd. `--external` now turns auto-start off. When the + external server cannot be reached, init gives one message on every route + (host and port, socket, shared server, auto-start pinned off): it names the + endpoint it dialed, says that bd did not start a server, and no longer + advises `bd dolt start`. With no port named at all, it asks for one + (`--server-port` or `BEADS_DOLT_SERVER_PORT`) instead of offering a default. +- **A failed `bd init --server` can be retried.** A first attempt that could + not reach its server left an empty `.beads/dolt` and nothing on disk that + records the server choice, so the legacy-upgrade guard refused every retry + as a "legacy Dolt workspace" and no bd command could repair it. When init + itself selects server mode, the guard now admits exactly that root: empty, + with no `.local_version` witness. A pre-1.0 witness, or a `.beads/dolt` that + holds anything, is still refused, and commands other than `bd init` are + decided as before. - **Time-bound filters and the keyset cursor compare the same instant on every engine, whatever zone they are given in.** The storage filter builder binds every time bound (`created_*`, `updated_*`, `closed_*`, `started_*`, diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 6eb4de6e7..4a4e51b9d 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -891,6 +891,8 @@ go_test( "init_commit_test.go", "init_embedded_test.go", "init_explicit_beads_dir_embedded_test.go", + "init_external_autostart_test.go", + "init_external_message_test.go", "init_gateway_test.go", "init_git_bootstrap_routing_test.go", "init_guard_force_e2e_test.go", @@ -939,6 +941,8 @@ go_test( "legacy_upgrade_guard_e2e_test.go", "legacy_upgrade_guard_empty_root_test.go", "legacy_upgrade_guard_external_server_test.go", + "legacy_upgrade_guard_init_flag_e2e_test.go", + "legacy_upgrade_guard_init_flag_test.go", "legacy_upgrade_guard_test.go", "legacy_upgrade_guard_witness_test.go", "linear_roundtrip_test.go", diff --git a/cmd/bd/init.go b/cmd/bd/init.go index a5b9cd9c8..bca3e336e 100644 --- a/cmd/bd/init.go +++ b/cmd/bd/init.go @@ -5,6 +5,7 @@ import ( "context" "errors" "fmt" + "net" "os" "os/exec" "os/user" @@ -226,6 +227,40 @@ func shouldInitSharedGlobalDB(sharedServer, sharedServerMode, gateway bool) bool return (sharedServer || sharedServerMode) && !gateway } +// externalServerUnreachableMessage returns the message for an --external init +// whose server could not be dialed, or "" when err is any other failure. +// +// --external declares the server managed outside bd, so init starts none. The +// store's own advice for a dead local server, "bd dolt start", would start the +// very server --external says bd must not, and the store words this failure +// differently per route (shared server, socket, remote host, auto-start pinned +// off). One message serves them all: it names the endpoint that was dialed and +// says that bd did not start a server. +// +// A dial to port 0 is the exception. The store leaves a zero port zero for +// auto-start to fill in, so it means nothing named a port: the address is a +// placeholder, not an endpoint. The message says so and how to name one, and +// offers no default, since on a shared machine whatever listens on a default +// port may belong to another project (GH#2098, GH#2372). +func externalServerUnreachableMessage(err error) string { + var opErr *net.OpError + if !errors.As(err, &opErr) || opErr.Op != "dial" { + return "" + } + if addr, ok := opErr.Addr.(*net.TCPAddr); ok && addr.Port == 0 { + return fmt.Sprintf("Error: failed to open Dolt store: %v\n\n"+ + "--external declares the Dolt server externally managed, so bd did not start one, and no port was named for it.\n"+ + "Name the server's port with --server-port or BEADS_DOLT_SERVER_PORT.\n", opErr) + } + endpoint := "the configured endpoint" + if opErr.Addr != nil { + endpoint = opErr.Addr.String() + } + return fmt.Sprintf("Error: failed to open Dolt store: %v\n\n"+ + "--external declares the Dolt server at %s externally managed, so bd did not start one.\n"+ + "Start that server, or point init at the right endpoint.\n", opErr, endpoint) +} + // warnHalfIdentifiedSubstrate reports a substrate carrying one identity marker // and not the other. // @@ -970,7 +1005,7 @@ Non-interactive mode (--non-interactive or BD_NON_INTERACTIVE=1): // before init's existing-workspace checks so even --force cannot create // or rewrite state beside a source that has not been preserved. if beadsDir := resolveInitBeadsDir(); beadsDir != "" { - if err := guardLegacyUpgradeWorkspace(beadsDir); err != nil { + if err := guardLegacyUpgradeWorkspaceForInit(beadsDir, initServerMode); err != nil { return err } } @@ -1554,7 +1589,9 @@ Non-interactive mode (--non-interactive or BD_NON_INTERACTIVE=1): } // Build config. Beads always uses dolt sql-server. - // AutoStart is always enabled during init — we need a server to initialize the database. + // AutoStart is enabled during init — we need a server to initialize the database — + // except under --external: that server is managed outside bd, so bd must not start + // one of its own on the endpoint the caller named. // // Port resolution for init: use ONLY project-local sources (env var, port file) // to prevent cross-project data leakage (GH#2336). DefaultConfig falls through @@ -1602,7 +1639,7 @@ Non-interactive mode (--non-interactive or BD_NON_INTERACTIVE=1): ProxiedServer: initProxiedServer, CreateIfMissing: true, // bd init is the only path that should create databases OpenedByInit: true, // shapes the identity-mismatch advice (GH#5558) - AutoStart: initServerMode && os.Getenv("BEADS_DOLT_AUTO_START") != "0", + AutoStart: initServerMode && !externalServer && os.Getenv("BEADS_DOLT_AUTO_START") != "0", ServerTLS: initDoltServerTLSFromEnv(), } if serverHost != "" { @@ -1727,6 +1764,12 @@ Non-interactive mode (--non-interactive or BD_NON_INTERACTIVE=1): } return &exitError{Code: 1} } + if externalServer { + if msg := externalServerUnreachableMessage(err); msg != "" { + fmt.Fprint(os.Stderr, msg) + return &exitError{Code: 1} + } + } fmt.Fprintf(os.Stderr, "Error: failed to open Dolt store: %v\n", err) return &exitError{Code: 1} } diff --git a/cmd/bd/init_external_autostart_test.go b/cmd/bd/init_external_autostart_test.go new file mode 100644 index 000000000..b08e6c3be --- /dev/null +++ b/cmd/bd/init_external_autostart_test.go @@ -0,0 +1,430 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "net" + "os" + "os/exec" + "path/filepath" + "runtime" + "strconv" + "strings" + "sync" + "testing" + "time" + + "github.com/steveyegge/beads/internal/configfile" +) + +// `bd init --external` declares that the Dolt server is managed outside bd, so +// init must never start one. These tests own one boundary risk that no lower +// seam can show: init builds its dolt config by hand, so the flag reaches the +// auto-start decision only if init carries it there. They run the real bd +// binary against a dolt shim that records every launch, which makes the +// outcome observable without a real server and leaves nothing running. + +// externalInitFixture is one fresh project plus a dolt shim for `bd init` +// attempts. The shim sits first on PATH, records every invocation and fails. +type externalInitFixture struct { + t *testing.T + repoDir string + home string + shimDir string + launches string +} + +type externalInitResult struct { + output string // combined stdout and stderr + failed bool // bd exited non-zero + launches string // dolt invocations the shim saw, one per line +} + +func newExternalInitFixture(t *testing.T) *externalInitFixture { + t.Helper() + if runtime.GOOS == "windows" { + t.Skip("the dolt shim is a POSIX shell script") + } + f := &externalInitFixture{t: t, repoDir: t.TempDir(), home: t.TempDir(), shimDir: t.TempDir()} + f.launches = filepath.Join(f.shimDir, "invocations") + shim := "#!/bin/sh\nprintf '%s\\n' \"$*\" >> \"$EXTERNAL_INIT_SHIM_LOG\"\nexit 1\n" + if err := os.WriteFile(filepath.Join(f.shimDir, "dolt"), []byte(shim), 0o755); err != nil { + t.Fatal(err) + } + initGitRepo(t, f.repoDir) + return f +} + +// run executes bd in the project. autoStart is the value of +// BEADS_DOLT_AUTO_START, or "" to leave it unset: an unpinned environment is +// what most callers run in, so it is the one that matters here. +func (f *externalInitFixture) run(autoStart string, args ...string) externalInitResult { + f.t.Helper() + env := append(envWithoutBeadsStorageSettings(), + "HOME="+f.home, + "XDG_CONFIG_HOME="+filepath.Join(f.home, ".config"), + "BEADS_TEST_IGNORE_REPO_CONFIG=1", + "BD_DISABLE_METRICS=1", + "PATH="+f.shimDir+string(os.PathListSeparator)+os.Getenv("PATH"), + "EXTERNAL_INIT_SHIM_LOG="+f.launches, + ) + if autoStart != "" { + env = append(env, "BEADS_DOLT_AUTO_START="+autoStart) + } + out, err := runExternalServerBD(f.t, f.repoDir, env, args...) + var exitErr *exec.ExitError + if err != nil && !errors.As(err, &exitErr) { + f.t.Fatalf("bd %s did not run: %v\n%s", strings.Join(args, " "), err, out) + } + launches, _ := os.ReadFile(f.launches) + return externalInitResult{output: out, failed: err != nil, launches: strings.TrimSpace(string(launches))} +} + +// externalEndpoint is the server an --external init is pointed at. +type externalEndpoint struct { + flags []string // --server-host/--server-port, or --server-socket + network string // "tcp" or "unix" + addr string // host:port or the socket path, as bd reports it +} + +func tcpEndpoint(port int) externalEndpoint { + return externalEndpoint{ + flags: []string{"--server-host", "127.0.0.1", "--server-port", strconv.Itoa(port)}, + network: "tcp", + addr: net.JoinHostPort("127.0.0.1", strconv.Itoa(port)), + } +} + +func unixEndpoint(path string) externalEndpoint { + return externalEndpoint{flags: []string{"--server-socket", path}, network: "unix", addr: path} +} + +// externalInitSocketPath returns an unused unix socket path. Socket paths are +// capped near 104 bytes and t.TempDir() nests too deeply under some runners. +func externalInitSocketPath(t *testing.T) string { + t.Helper() + dir, err := os.MkdirTemp("", "bdx") + if err != nil { + t.Fatal(err) + } + if len(filepath.Join(dir, "dolt.sock")) > 100 { + _ = os.RemoveAll(dir) + if dir, err = os.MkdirTemp("/tmp", "bdx"); err != nil { + t.Fatal(err) + } + } + t.Cleanup(func() { _ = os.RemoveAll(dir) }) + return filepath.Join(dir, "dolt.sock") +} + +// serveUnixProxy listens on a fresh unix socket and forwards each connection +// to target, so bd sees a reachable server at a socket path. The returned +// func reports how many connections bd made through it. +func serveUnixProxy(t *testing.T, target string) (string, func() int) { + t.Helper() + path := externalInitSocketPath(t) + ln, err := net.Listen("unix", path) + if err != nil { + t.Fatalf("listen on %s: %v", path, err) + } + t.Cleanup(func() { _ = ln.Close() }) + var mu sync.Mutex + accepted := 0 + go func() { + for { + client, err := ln.Accept() + if err != nil { + return + } + mu.Lock() + accepted++ + mu.Unlock() + go func() { + defer client.Close() + upstream, err := net.Dial("tcp", target) + if err != nil { + return + } + defer upstream.Close() + done := make(chan struct{}, 2) + go func() { _, _ = io.Copy(upstream, client); done <- struct{}{} }() + go func() { _, _ = io.Copy(client, upstream); done <- struct{}{} }() + <-done // either side closing ends the session; the deferred closes release the other copy + }() + } + }() + return path, func() int { mu.Lock(); defer mu.Unlock(); return accepted } +} + +// requireNoLaunch fails when bd tried to run dolt at all: an externally +// managed server is never started by bd, and the shim is how a start shows. +func requireNoLaunch(t *testing.T, res externalInitResult) { + t.Helper() + if res.launches != "" { + t.Errorf("bd launched dolt although the server is externally managed (dolt %s):\n%s", + strings.ReplaceAll(res.launches, "\n", "; dolt "), res.output) + } +} + +// requireExternalUnreachableMessage checks the one wording an --external init +// gives for an endpoint it cannot reach: it names the endpoint, says the server +// is declared external and that bd did not start one, and never advises +// `bd dolt start`, which would start the server --external says bd must not. +func requireExternalUnreachableMessage(t *testing.T, output, endpoint string) { + t.Helper() + for _, want := range []string{endpoint, "externally managed", "did not start"} { + if !strings.Contains(output, want) { + t.Errorf("output does not contain %q:\n%s", want, output) + } + } + if strings.Contains(output, "bd dolt start") { + t.Errorf("output advises `bd dolt start`:\n%s", output) + } +} + +func requireNothingListening(t *testing.T, network, addr string) { + t.Helper() + conn, err := net.DialTimeout(network, addr, time.Second) + if err == nil { + _ = conn.Close() + t.Errorf("something accepts connections on %s %s after init", network, addr) + } +} + +// requireDoltRootEmpty fails when .beads/dolt holds anything: init leaves at +// most an empty marker directory for an external server, so any entry is a +// server root a local dolt created. +func requireDoltRootEmpty(t *testing.T, repoDir string) { + t.Helper() + entries, err := os.ReadDir(filepath.Join(repoDir, ".beads", "dolt")) + if errors.Is(err, os.ErrNotExist) { + return + } + if err != nil { + t.Fatal(err) + } + if len(entries) > 0 { + names := make([]string, 0, len(entries)) + for _, e := range entries { + names = append(names, e.Name()) + } + t.Errorf(".beads/dolt holds %v, which init did not put there", names) + } +} + +// TestInitServerExternalDeadPortStartsNoServer is the end-to-end boundary for +// --external: a dead endpoint must fail loudly instead of making bd launch a +// dolt of its own on the port the caller named. +func TestInitServerExternalDeadPortStartsNoServer(t *testing.T) { + f := newExternalInitFixture(t) + ep := tcpEndpoint(freeLoopbackPort(t)) + + res := f.run("", append(append([]string{"init", "--quiet", "--server", "--external"}, ep.flags...), + "--prefix", "x", "--skip-hooks", "--skip-agents")...) + + if !res.failed { + t.Errorf("bd init --server --external succeeded against a dead endpoint %s:\n%s", ep.addr, res.output) + } + requireNoLaunch(t, res) + requireExternalUnreachableMessage(t, res.output, ep.addr) + requireNothingListening(t, ep.network, ep.addr) + requireDoltRootEmpty(t, f.repoDir) +} + +// TestInitServerExternalNoPortStartsNoServer is the --external boundary that +// rule 2c (#5934) cannot see: the argv and the environment name no port, so +// nothing but the flag declares the server external. A server init in that +// shape launches a repo-local dolt by design (see +// TestInitServerWithoutNamedPortStillStartsServer), so with --external it must +// fail instead, and say that bd did not start one. There is no endpoint to +// name, so the wording is checked without one: the child inherits +// BEADS_TEST_MODE, under which the store dials port 1 rather than leave an +// unnamed port at 0, so the port-0 wording itself is pinned by +// TestExternalServerUnreachableMessage. +func TestInitServerExternalNoPortStartsNoServer(t *testing.T) { + f := newExternalInitFixture(t) + + res := f.run("", "init", "--quiet", "--server", "--external", "--prefix", "x", "--skip-hooks", "--skip-agents") + + if !res.failed { + t.Errorf("bd init --server --external succeeded with no server named:\n%s", res.output) + } + requireNoLaunch(t, res) + requireDoltRootEmpty(t, f.repoDir) + for _, want := range []string{"externally managed", "did not start"} { + if !strings.Contains(res.output, want) { + t.Errorf("output does not contain %q:\n%s", want, res.output) + } + } + if strings.Contains(res.output, "bd dolt start") { + t.Errorf("output advises `bd dolt start`:\n%s", res.output) + } +} + +// TestInitSharedServerExternalDeadPortStillFails pins the sibling route that +// already honors --external: the same dead endpoint under --shared-server +// exits non-zero without launching dolt. Without it the test above could pass +// because every init fails here, not because --server honors the flag. +func TestInitSharedServerExternalDeadPortStillFails(t *testing.T) { + f := newExternalInitFixture(t) + ep := tcpEndpoint(freeLoopbackPort(t)) + + res := f.run("", append(append([]string{"init", "--quiet", "--shared-server", "--external"}, ep.flags...), + "--prefix", "x", "--skip-hooks", "--skip-agents")...) + + if !res.failed { + t.Errorf("bd init --shared-server --external succeeded against a dead endpoint %s:\n%s", ep.addr, res.output) + } + requireNoLaunch(t, res) +} + +// TestInitServerWithoutNamedPortStillStartsServer pins the other half of the +// contract: a --server init that names no port is where bd starts a repo-local +// dolt by design, so the no-launch assertions above cannot pass merely because +// nothing can ever launch one. It names no port, in argv or in the environment +// (envWithoutBeadsStorageSettings drops every BEADS_DOLT_* variable): a named +// port is declared external by rule 2c (#5934) with or without --external, so +// it would launch nothing here either way. +func TestInitServerWithoutNamedPortStillStartsServer(t *testing.T) { + f := newExternalInitFixture(t) + + res := f.run("", "init", "--quiet", "--server", "--prefix", "x", "--skip-hooks", "--skip-agents") + + if res.launches == "" { + t.Errorf("bd init --server naming no port never reached dolt, so auto-start regressed:\n%s", res.output) + } +} + +// TestInitExternalUnreachableMessage covers the wording across the routes an +// --external init can take. Every one of them gives the same message, whether +// auto-start is pinned off by the environment, the server is a shared one, or +// the endpoint is a socket. +func TestInitExternalUnreachableMessage(t *testing.T) { + cases := []struct { + name string + mode string + autoStart string + endpoint func(t *testing.T) externalEndpoint + }{ + {"server host:port with auto-start pinned off", "--server", "0", + func(t *testing.T) externalEndpoint { return tcpEndpoint(freeLoopbackPort(t)) }}, + {"shared-server host:port", "--shared-server", "", + func(t *testing.T) externalEndpoint { return tcpEndpoint(freeLoopbackPort(t)) }}, + {"server socket", "--server", "", + func(t *testing.T) externalEndpoint { return unixEndpoint(externalInitSocketPath(t)) }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + f := newExternalInitFixture(t) + ep := tc.endpoint(t) + + res := f.run(tc.autoStart, append(append([]string{"init", "--quiet", tc.mode, "--external"}, ep.flags...), + "--prefix", "x", "--skip-hooks", "--skip-agents")...) + + if !res.failed { + t.Fatalf("bd init succeeded against a dead endpoint %s:\n%s", ep.addr, res.output) + } + requireExternalUnreachableMessage(t, res.output, ep.addr) + }) + } +} + +// TestInitExternalProvisionerShape runs init with the argv a provisioner of an +// externally managed server uses, in both endpoint forms, with auto-start +// unset and pinned off, against a server that is up and one that is not. Every +// cell is a control: a port named with --server-port is already declared +// external by rule 2c (#5934), and a socket is never auto-started, so none of +// them launched dolt before init honored --external either. They pin that the +// provisioner shape keeps launching nothing, keeps failing against a dead +// endpoint, and still records the endpoint it was given. The reachable cells +// need the Dolt test server and skip without it. +func TestInitExternalProvisionerShape(t *testing.T) { + for _, form := range []string{"host-port", "socket"} { + for _, autoStart := range []string{"unset", "0"} { + for _, reachable := range []bool{false, true} { + state := "unreachable" + if reachable { + state = "reachable" + } + t.Run(fmt.Sprintf("%s/auto-start=%s/%s", form, autoStart, state), func(t *testing.T) { + runProvisionerCell(t, form, autoStart, reachable) + }) + } + } + } +} + +func runProvisionerCell(t *testing.T, form, autoStart string, reachable bool) { + t.Helper() + if reachable { + skipIfNoDolt(t) + } + f := newExternalInitFixture(t) + + var ep externalEndpoint + var socketConnections func() int + switch { + case form == "host-port" && reachable: + ep = tcpEndpoint(testDoltServerPort) + case form == "host-port": + ep = tcpEndpoint(freeLoopbackPort(t)) + case reachable: + var sock string + sock, socketConnections = serveUnixProxy(t, net.JoinHostPort("127.0.0.1", strconv.Itoa(testDoltServerPort))) + ep = unixEndpoint(sock) + default: + ep = unixEndpoint(externalInitSocketPath(t)) + } + + database := "provisioned" + if reachable { + database = uniqueTestDBName(t) + t.Cleanup(func() { dropTestDatabase(database, testDoltServerPort) }) + } + + args := append([]string{"init", "--init-if-missing", "--quiet", "--server", "--external"}, ep.flags...) + args = append(args, "-p", "prov", "--skip-hooks", "--skip-agents", "--database", database, f.repoDir) + pin := "" + if autoStart == "0" { + pin = "0" + } + + res := f.run(pin, args...) + + requireNoLaunch(t, res) + if !reachable { + if !res.failed { + t.Errorf("bd init succeeded against an unreachable endpoint %s:\n%s", ep.addr, res.output) + } + requireNothingListening(t, ep.network, ep.addr) + return + } + if res.failed { + t.Fatalf("bd init against the reachable server %s failed:\n%s", ep.addr, res.output) + } + requireDoltRootEmpty(t, f.repoDir) + data, err := os.ReadFile(filepath.Join(f.repoDir, ".beads", "metadata.json")) + if err != nil { + t.Fatalf("expected metadata.json after init: %v", err) + } + var meta configfile.Config + if err := json.Unmarshal(data, &meta); err != nil { + t.Fatalf("parse metadata.json: %v\n%s", err, data) + } + if form == "socket" { + if meta.DoltServerSocket != ep.addr { + t.Errorf("metadata.json dolt_server_socket = %q, want the socket init was given, %q", meta.DoltServerSocket, ep.addr) + } + if socketConnections() == 0 { + t.Errorf("bd never connected through the socket %s", ep.addr) + } + return + } + if meta.DoltServerHost != "127.0.0.1" || meta.DoltServerPort != testDoltServerPort { + t.Errorf("metadata.json names %s:%d, want the endpoint init was given, %s", meta.DoltServerHost, meta.DoltServerPort, ep.addr) + } +} diff --git a/cmd/bd/init_external_message_test.go b/cmd/bd/init_external_message_test.go new file mode 100644 index 000000000..2c06601df --- /dev/null +++ b/cmd/bd/init_external_message_test.go @@ -0,0 +1,97 @@ +package main + +import ( + "errors" + "fmt" + "net" + "strings" + "testing" +) + +// TestExternalServerUnreachableMessage pins which failures get the --external +// wording. Only a failed dial may: the message claims the server could not be +// reached, so any other open failure (an identity mismatch, a handshake that +// was reset) must keep its own text instead of being reported as "externally +// managed". +func TestExternalServerUnreachableMessage(t *testing.T) { + refused := errors.New("connect: connection refused") + dial := &net.OpError{ + Op: "dial", + Net: "tcp", + Addr: &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 3307}, + Err: refused, + } + + tests := []struct { + name string + err error + wantEndpoint string // "" means the error is not rewritten + }{ + {"dial error", dial, "127.0.0.1:3307"}, + { + // The store wraps the dial error with %w and its own advice. + "dial error wrapped like the store does", + fmt.Errorf("Dolt server unreachable at 127.0.0.1:3307: %w\n\nThe Dolt server may not be running.", dial), + "127.0.0.1:3307", + }, + { + "unix socket dial error", + &net.OpError{Op: "dial", Net: "unix", Addr: &net.UnixAddr{Name: "/run/dolt.sock", Net: "unix"}, Err: refused}, + "/run/dolt.sock", + }, + { + // A name that does not resolve has no address to report. + "dial error without an address", + &net.OpError{Op: "dial", Net: "tcp", Err: errors.New("no such host")}, + "the configured endpoint", + }, + {"read error after connecting", &net.OpError{Op: "read", Net: "tcp", Err: errors.New("connection reset by peer")}, ""}, + {"not a network error", errors.New("project identity mismatch"), ""}, + } + // Every rewritten message says the server is declared external and that bd + // did not start one, and never advises `bd dolt start`. + requireDeclaredExternal := func(t *testing.T, got string, wants ...string) { + t.Helper() + for _, want := range append(wants, "externally managed", "did not start") { + if !strings.Contains(got, want) { + t.Errorf("message does not contain %q:\n%s", want, got) + } + } + if strings.Contains(got, "bd dolt start") { + t.Errorf("message advises `bd dolt start`:\n%s", got) + } + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := externalServerUnreachableMessage(tt.err) + if tt.wantEndpoint == "" { + if got != "" { + t.Fatalf("message = %q, want none for an error that is not a failed dial", got) + } + return + } + requireDeclaredExternal(t, got, tt.wantEndpoint) + }) + } + + // A dial to port 0 is an init that was never told a port: the address is a + // placeholder, not the server's endpoint, so the message must not offer it + // as one. The first line quotes the dialled address, so it is cut out + // before looking. + t.Run("dial error with no port named", func(t *testing.T) { + noPort := &net.OpError{ + Op: "dial", + Net: "tcp", + Addr: &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 0}, + Err: refused, + } + got := externalServerUnreachableMessage(noPort) + if got == "" { + t.Fatal("message = none, want the --external wording for a failed dial") + } + requireDeclaredExternal(t, got) + if strings.Contains(strings.ReplaceAll(got, noPort.Error(), ""), ":0") { + t.Errorf("message offers port 0 as the endpoint:\n%s", got) + } + }) +} diff --git a/cmd/bd/legacy_upgrade_guard.go b/cmd/bd/legacy_upgrade_guard.go index 53d680798..71ced0205 100644 --- a/cmd/bd/legacy_upgrade_guard.go +++ b/cmd/bd/legacy_upgrade_guard.go @@ -24,6 +24,19 @@ import ( // SQLite files, and the bounded local version witness; storage internals stay // behind the driver boundary. func guardLegacyUpgradeWorkspace(beadsDir string) error { + return guardLegacyUpgradeWorkspaceForInit(beadsDir, false) +} + +// guardLegacyUpgradeWorkspaceForInit is guardLegacyUpgradeWorkspace for a caller +// that has already resolved server intent from a source the workspace does not +// yet record: `bd init --server`, whose flag is the only place that choice lives +// until init has written it. The guard cannot see the flag, so initSelectsServer +// carries it in. It is consulted for exactly one thing, the empty-root admission +// below, and is deliberately not folded into the workspace's own server mode: +// that would also move a non-empty root with an unreadable witness onto the +// warn-and-open arm, and an embedded repository beside a non-empty .beads/dolt +// onto the refusing one. +func guardLegacyUpgradeWorkspaceForInit(beadsDir string, initSelectsServer bool) error { if beadsDir == "" { return nil } @@ -75,6 +88,15 @@ func guardLegacyUpgradeWorkspace(beadsDir string) error { // did not. return nil } + if initSelectsServer && !present && isEmptyDirectory(filepath.Join(beadsDir, "dolt")) { + // Init has already chosen a server, so this is the empty root the + // serverMode arm below admits for a workspace that records the choice: it + // holds no Dolt data a legacy release could have left, and bd makes it + // itself on use, so a retry after a first attempt that could not reach a + // server finds exactly this. A present witness, a root holding any entry, + // and every caller that does not pass the flag are decided as before. + return nil + } if serverMode { if !present && isEmptyDirectory(filepath.Join(beadsDir, "dolt")) { // An empty .beads/dolt holds no Dolt data, so no legacy release diff --git a/cmd/bd/legacy_upgrade_guard_init_flag_e2e_test.go b/cmd/bd/legacy_upgrade_guard_init_flag_e2e_test.go new file mode 100644 index 000000000..712842ffc --- /dev/null +++ b/cmd/bd/legacy_upgrade_guard_init_flag_e2e_test.go @@ -0,0 +1,87 @@ +//go:build cgo + +package main + +import ( + "os" + "path/filepath" + "slices" + "strconv" + "testing" +) + +// TestInitServerFlagAdmitsEmptyDoltRootWithoutConfig reproduces the retry shape +// with nothing on disk that records the server choice: an empty .beads/dolt and +// neither config.yaml nor metadata.json, so the --server flag is the only place +// the intent lives. The legacy guard used to refuse that init as a legacy Dolt +// workspace, so the workspace was never created and no bd command could repair +// it. +func TestInitServerFlagAdmitsEmptyDoltRootWithoutConfig(t *testing.T) { + skipIfNoDolt(t) + env := externalServerTestEnv(t) + + repoDir := t.TempDir() + initGitRepo(t, repoDir) + beadsDir := filepath.Join(repoDir, ".beads") + if err := os.MkdirAll(filepath.Join(beadsDir, "dolt"), 0o700); err != nil { + t.Fatal(err) + } + + database := uniqueTestDBName(t) + t.Cleanup(func() { dropTestDatabase(database, testDoltServerPort) }) + + out, err := runExternalServerBD(t, repoDir, env, externalServerInitArgs(database)...) + if err != nil { + t.Fatalf("bd init --server over an empty .beads/dolt failed: %v\n%s", err, out) + } + assertCurrentVersionWitness(t, beadsDir) +} + +// TestInitServerFlagRetryAfterFailedFirstAttempt replays the reported sequence: a +// first `bd init --server` that cannot reach a server fails after creating the +// local Dolt root, leaving an empty .beads/dolt and no witness, and the same +// command run again once the server is up must succeed instead of being refused +// as a legacy workspace over the root the first attempt made itself. +func TestInitServerFlagRetryAfterFailedFirstAttempt(t *testing.T) { + skipIfNoDolt(t) + env := externalServerTestEnv(t) + // The first attempt has to fail without starting a server of its own: one + // that started and then failed would leave a populated .beads/dolt, which + // the guard refuses by design. externalServerTestEnv pins + // BEADS_DOLT_AUTO_START=0 for exactly that, which keeps the attempt + // deterministic even once init's separate --external handling stops trying + // to start a per-project server. Fail here, naming the reason, if a change to + // that helper drops it. + if !slices.Contains(env, "BEADS_DOLT_AUTO_START=0") { + t.Fatal("externalServerTestEnv must set BEADS_DOLT_AUTO_START=0: the failed first attempt must not start a server of its own") + } + + repoDir := t.TempDir() + initGitRepo(t, repoDir) + beadsDir := filepath.Join(repoDir, ".beads") + + database := uniqueTestDBName(t) + t.Cleanup(func() { dropTestDatabase(database, testDoltServerPort) }) + + out, err := runExternalServerBD(t, repoDir, env, + externalServerInitArgs(database, "--server-port", strconv.Itoa(freeLoopbackPort(t)))...) + if err == nil { + t.Fatalf("precondition: bd init --server against an unreachable server succeeded:\n%s", out) + } + // The retry only has something to fix if the failed attempt left exactly + // this. Any other leftover is a finding about init, not a reason to loosen + // what follows. + entries, readErr := os.ReadDir(filepath.Join(beadsDir, "dolt")) + if readErr != nil || len(entries) != 0 { + t.Fatalf("precondition: the failed first attempt should leave an empty .beads/dolt, found %d entries (err %v)", len(entries), readErr) + } + if _, statErr := os.Stat(filepath.Join(beadsDir, localVersionFile)); !os.IsNotExist(statErr) { + t.Fatalf("precondition: the failed first attempt should leave no %s (stat err %v)", localVersionFile, statErr) + } + + out, err = runExternalServerBD(t, repoDir, env, externalServerInitArgs(database)...) + if err != nil { + t.Fatalf("retrying bd init --server over the empty .beads/dolt the first attempt left failed: %v\n%s", err, out) + } + assertCurrentVersionWitness(t, beadsDir) +} diff --git a/cmd/bd/legacy_upgrade_guard_init_flag_test.go b/cmd/bd/legacy_upgrade_guard_init_flag_test.go new file mode 100644 index 000000000..7fc5e9de5 --- /dev/null +++ b/cmd/bd/legacy_upgrade_guard_init_flag_test.go @@ -0,0 +1,143 @@ +package main + +import ( + "fmt" + "os" + "path/filepath" + "testing" +) + +// TestLegacyUpgradeGuardForInitAdmitsEmptyDoltRoot pins the one thing init's own +// intent unlocks. `bd init --server` resolves server mode from the flag, a source +// the workspace does not record until init has written it. A retry over the empty +// .beads/dolt that a failed first attempt leaves behind therefore looks like an +// embedded workspace with a local Dolt root, and the guard refuses it. Told what +// init already knows, the guard gives that root the admission a config-selected +// server workspace gets. Asked without it, in either form, it still refuses, so +// the admission cannot pass vacuously. +func TestLegacyUpgradeGuardForInitAdmitsEmptyDoltRoot(t *testing.T) { + warnings := captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, "", "", "") + + if err := guardLegacyUpgradeWorkspaceForInit(beadsDir, true); err != nil { + t.Fatalf("guardLegacyUpgradeWorkspaceForInit(dir, true) = %v, want nil", err) + } + if warnings.Len() != 0 { + t.Fatalf("guard warned about an absent witness: %q", warnings.String()) + } + if err := guardLegacyUpgradeWorkspaceForInit(beadsDir, false); !isLegacyUpgradeRefusal(err) { + t.Fatalf("guardLegacyUpgradeWorkspaceForInit(dir, false) = %v, want migration refusal", err) + } + if err := guardLegacyUpgradeWorkspace(beadsDir); !isLegacyUpgradeRefusal(err) { + t.Fatalf("guardLegacyUpgradeWorkspace(dir) = %v, want migration refusal", err) + } +} + +// TestLegacyUpgradeGuardForInitAdmissionStaysNarrow proves init's intent relaxes +// nothing beyond an empty, witness-less .beads/dolt. With it set, anything that +// could be legacy data still refuses, and the outcomes a broader reading of the +// flag would move (treating it as if the workspace itself selected server mode) +// stay where they are. +func TestLegacyUpgradeGuardForInitAdmissionStaysNarrow(t *testing.T) { + for _, entry := range []string{"beads", ".dolt", "stray-file", "symlink"} { + t.Run("root holding "+entry, func(t *testing.T) { + captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, "", "", "") + path := filepath.Join(beadsDir, "dolt", entry) + var err error + switch entry { + case "stray-file": + err = os.WriteFile(path, nil, 0o600) + case "symlink": + if err = os.Symlink(t.TempDir(), path); err != nil { + t.Skipf("symlink capability unavailable: %v", err) + } + default: + err = os.Mkdir(path, 0o700) + } + if err != nil { + t.Fatal(err) + } + + if err := guardLegacyUpgradeWorkspaceForInit(beadsDir, true); !isLegacyUpgradeRefusal(err) { + t.Fatalf("guardLegacyUpgradeWorkspaceForInit(dir, true) = %v, want migration refusal", err) + } + }) + } + + for _, version := range []string{"0.49.6", "0.62.0"} { + t.Run("pre-1.0 witness "+version, func(t *testing.T) { + captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, "", "", version) + + if err := guardLegacyUpgradeWorkspaceForInit(beadsDir, true); !isLegacyUpgradeRefusal(err) { + t.Fatalf("guardLegacyUpgradeWorkspaceForInit(dir, true) = %v, want migration refusal", err) + } + }) + } + + t.Run("blank witness over a non-empty root", func(t *testing.T) { + warnings := captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, "", "", "") + if err := os.Mkdir(filepath.Join(beadsDir, "dolt", "beads"), 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(beadsDir, localVersionFile), []byte("\n"), 0o600); err != nil { + t.Fatal(err) + } + + // A workspace that selects server mode itself warns and opens over an + // unreadable witness. Init's intent must not move this root onto that arm. + if err := guardLegacyUpgradeWorkspaceForInit(beadsDir, true); !isLegacyUpgradeRefusal(err) { + t.Fatalf("guardLegacyUpgradeWorkspaceForInit(dir, true) = %v, want migration refusal", err) + } + if warnings.Len() != 0 { + t.Fatalf("guard took the unreadable-witness arm: %q", warnings.String()) + } + }) + + t.Run("symlinked dolt root", func(t *testing.T) { + captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, "", "", "") + root := filepath.Join(beadsDir, "dolt") + if err := os.Remove(root); err != nil { + t.Fatal(err) + } + if err := os.Symlink(t.TempDir(), root); err != nil { + t.Skipf("symlink capability unavailable: %v", err) + } + + // hasLegacyDoltRoot does not follow a symlink, so the guard never takes + // such a root for a local Dolt root and it never reaches the empty-root + // admission. Init's intent has to leave its outcome exactly where every + // other caller finds it. + want := fmt.Sprint(guardLegacyUpgradeWorkspace(beadsDir)) + if got := fmt.Sprint(guardLegacyUpgradeWorkspaceForInit(beadsDir, true)); got != want { + t.Fatalf("guardLegacyUpgradeWorkspaceForInit(dir, true) = %s, want %s (unchanged by init's intent)", got, want) + } + }) + + t.Run("embedded repository beside a non-empty root", func(t *testing.T) { + captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, "", "", "") + repo := filepath.Join(beadsDir, "embeddeddolt", "beads", ".dolt") + if err := os.MkdirAll(repo, 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(repo, "repo-entry"), []byte("opaque"), 0o600); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(beadsDir, "dolt", "stray-file"), nil, 0o600); err != nil { + t.Fatal(err) + } + + // An embedded repository admits the workspace before server intent is + // consulted at all. Folding init's intent into the workspace's own mode + // would send this one to the refusing arm. + for _, initSelectsServer := range []bool{false, true} { + if err := guardLegacyUpgradeWorkspaceForInit(beadsDir, initSelectsServer); err != nil { + t.Fatalf("guardLegacyUpgradeWorkspaceForInit(dir, %t) = %v, want nil", initSelectsServer, err) + } + } + }) +} From f6f122894f071fc485c49c9d489ca5710203a6aa Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 19 Oct 2026 03:26:45 +0000 Subject: [PATCH 04/41] test(serve): before-parity pins for the httpapi startup event and ordinary-workspace bd serve (P13, P14) (#7679) * test(httpapi): pin the startup event's capabilities and database source TestStartupLines and TestStartupNamesTheDatabaseSource check the event=startup line by substring, and neither reads its capabilities field. A server could print one capability list and advertise another on GET /v0/beads/context, or a new kind of database source could relabel the line, and both tests would stay green. TestStartupEventPublishesTheAdvertisedCapabilitiesAndSource parses the line back into its fields and pins, for the provider and roles sources with and without the external-dependency policy: the exact field sequence, db=provider|roles, and a capabilities value equal to the handshake's list joined with commas. It is a before-parity pin: a future alternative source on the same server must leave this line unchanged for the Issue API's two sources. Test-only; no behavior change. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(serve): pin ordinary-workspace bd serve under BD_GRAPH_MODE=dependency bd reads no BD_GRAPH_MODE today. "dependency" names the format every existing workspace already has, so any later reader of that variable (a graph-mode serve dispatch, for example) has to keep serving an ordinary workspace through runServe exactly as it does now. These tests pin that before such a reader exists. Each test runs bd serve twice in one workspace, once with the variable unset and once with BD_GRAPH_MODE=dependency. It requires the event=startup and event=limits lines to match byte for byte apart from the ephemeral port, and the GET /v0/beads/context bodies to be equal. Each run is also checked against what runServe builds: the serveResolvedMode label, the db source, and a startup capabilities field equal to the handshake's list. Two runs that were both routed somewhere else therefore cannot pass by agreeing with each other. - TestServeOrdinaryWorkspaceIgnoresGraphModeDependency: in-process through rootCmd (PersistentPreRunE included), on the store source of a registered-backend workspace. Runs in bd_test. - TestServerModeServeIgnoresGraphModeDependency: a real bd on a `bd init --server` workspace, which uses the provider source. Gated on BEADS_TEST_PROXIED_SERVER like the other TestServerMode* tests, and placed in the proxied 34-shard manifest by gen_proxied_shard_manifest.py --write (incremental). The new file is listed in cmd/bd/BUILD.bazel's bd_test srcs. Test-only; no behavior change. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(serve): stop the in-process graph-mode pin hanging when serve fails serveInProcessSnapshot's cleanup decided whether the server was still running by looking at rootCancel. That holds for a clean run, which clears rootCancel in PersistentPostRunE. A run whose RunE returns an error skips PersistentPostRunE, so rootCancel stays set even though the main flow has already read the run's only result from done. The cleanup then called rootCancel and blocked on done for good, and the failure surfaced as a test-binary timeout panic that takes the whole bd_test shard with it instead of a FAIL for this test. The cleanup now keys on whether done has been read. A serve that binds, answers, and then returns an error (one way a future graph-serve dispatch could regress) now fails in about 5s with "bd serve returned , want a clean shutdown". A failure before done is read (a bad GET, say) still cancels the server and waits for it. Test-only; no behavior change. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) --- .github/scripts/proxied-cmd-test-shards.txt | 1 + cmd/bd/BUILD.bazel | 1 + cmd/bd/serve_graph_mode_env_test.go | 278 ++++++++++++++++++++ internal/httpapi/server_test.go | 119 +++++++++ 4 files changed, 399 insertions(+) create mode 100644 cmd/bd/serve_graph_mode_env_test.go diff --git a/.github/scripts/proxied-cmd-test-shards.txt b/.github/scripts/proxied-cmd-test-shards.txt index 7c2e916b9..fca167226 100644 --- a/.github/scripts/proxied-cmd-test-shards.txt +++ b/.github/scripts/proxied-cmd-test-shards.txt @@ -157,6 +157,7 @@ 34 17 TestProxiedServerPrime 34 17 TestProxiedServerSQLDatabaseFlag 34 17 TestProxiedServerServeModeGate +34 17 TestServerModeServeIgnoresGraphModeDependency 34 18 TestProxiedServerClose3 34 18 TestProxiedServerCount diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 4a4e51b9d..8477c2d1a 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1156,6 +1156,7 @@ go_test( "serve_delete_proxied_integration_test.go", "serve_dependencies_proxied_integration_test.go", "serve_edge_counts_proxied_integration_test.go", + "serve_graph_mode_env_test.go", "serve_inprocess_test.go", "serve_list_sort_proxied_integration_test.go", "serve_proxied_integration_test.go", diff --git a/cmd/bd/serve_graph_mode_env_test.go b/cmd/bd/serve_graph_mode_env_test.go new file mode 100644 index 000000000..fd61effd8 --- /dev/null +++ b/cmd/bd/serve_graph_mode_env_test.go @@ -0,0 +1,278 @@ +//go:build cgo + +package main + +import ( + "bytes" + "context" + "io" + "net/http" + "os" + "path/filepath" + "reflect" + "slices" + "strconv" + "strings" + "sync" + "testing" + + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/backends" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" +) + +// `bd serve` in an ordinary workspace is the same server whether or not +// BD_GRAPH_MODE=dependency is exported. +// +// bd reads no BD_GRAPH_MODE today. "dependency" is the value that names the +// format every existing workspace already has, so it is the one value a later +// reader of that variable (a graph-mode serve dispatch, say) must keep treating +// as "serve this workspace the way bd always has". These tests pin that before +// any such reader exists: with the variable exported, the run still has to +// reach runServe and build the same server. +// +// "The same server" is measured where an operator and a client measure it: the +// event=startup and event=limits lines, byte for byte apart from the ephemeral +// port, and the whole GET /v0/beads/context body. Each run is also checked on +// its own against what runServe builds (its mode label, its database source, +// the advertised capabilities), so two runs that were both routed somewhere +// else cannot pass by agreeing with each other. + +const serveGraphModeEnv = "BD_GRAPH_MODE" + +// TestServeOrdinaryWorkspaceIgnoresGraphModeDependency drives the whole root +// command in-process, PersistentPreRunE included, on a workspace served from +// the store source. In-process for the reason the registered-backend serve +// tests are: registration is Go wiring, so a spawned bd could not reach this +// arm. The server-mode twin below covers the provider source with a real bd. +func TestServeOrdinaryWorkspaceIgnoresGraphModeDependency(t *testing.T) { + const name = "serve-graph-mode-env" + + dir := t.TempDir() + initGitRepoAt(t, dir) + beadsDir := filepath.Join(dir, ".beads") + if err := os.MkdirAll(beadsDir, 0o755); err != nil { + t.Fatalf("mkdir .beads: %v", err) + } + seedRegisteredBackendWorkspace(t, beadsDir) + backends.Register(name, backends.Backend{ + Open: func(ctx context.Context, beadsDir string) (storage.DoltStorage, error) { + return embeddeddolt.Open(ctx, beadsDir, serveRegisteredDatabase, "main") + }, + OpenReadOnly: func(ctx context.Context, beadsDir string) (storage.DoltStorage, error) { + return embeddeddolt.OpenReadOnly(ctx, beadsDir, serveRegisteredDatabase, "main") + }, + WorkspaceIsBeadsDir: true, + }) + t.Cleanup(func() { backends.Deregister(name) }) + if err := (&configfile.Config{Backend: name, DoltDatabase: serveRegisteredDatabase}).Save(beadsDir); err != nil { + t.Fatalf("save metadata.json: %v", err) + } + + // Unset rather than empty, so the baseline is a shell that never exported + // it. t.Setenv first so the original value comes back afterwards. + t.Setenv(serveGraphModeEnv, "") + if err := os.Unsetenv(serveGraphModeEnv); err != nil { + t.Fatalf("unset %s: %v", serveGraphModeEnv, err) + } + ordinary := serveInProcessSnapshot(t, dir, beadsDir) + + t.Setenv(serveGraphModeEnv, "dependency") + asserted := serveInProcessSnapshot(t, dir, beadsDir) + + mode := name + " (registered backend)" + assertRunServeBuiltIt(t, serveGraphModeEnv+" unset", ordinary, "roles", mode) + assertRunServeBuiltIt(t, serveGraphModeEnv+"=dependency", asserted, "roles", mode) + if asserted.context["backend"] != name { + t.Errorf("context backend = %v, want %q", asserted.context["backend"], name) + } + assertSameServer(t, ordinary, asserted) +} + +// TestServerModeServeIgnoresGraphModeDependency is the same pin for a +// `bd init --server` workspace, the topology an ordinary deployment serves +// from: the provider source, which serve builds itself. It runs a real bd, +// because that arm fronts the server through a proxy child, and a proxy child +// cannot be spawned from inside the test binary. +func TestServerModeServeIgnoresGraphModeDependency(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + p := newServerModeProject(t, bd, "srvgm") + + ordinaryEnv := envWithout(p.env, serveGraphModeEnv) + assertedEnv := append(slices.Clone(ordinaryEnv), serveGraphModeEnv+"=dependency") + + ordinary := serveProcessSnapshot(t, startServe(t, bd, p.dir, ordinaryEnv)) + asserted := serveProcessSnapshot(t, startServe(t, bd, p.dir, assertedEnv)) + + const mode = "server (external dolt)" + assertRunServeBuiltIt(t, serveGraphModeEnv+" unset", ordinary, "provider", mode) + assertRunServeBuiltIt(t, serveGraphModeEnv+"=dependency", asserted, "provider", mode) + if asserted.context["dolt_mode"] != "server" || asserted.context["database"] != p.database { + t.Errorf("context dolt_mode/database = %v/%v, want server/%s", + asserted.context["dolt_mode"], asserted.context["database"], p.database) + } + assertSameServer(t, ordinary, asserted) +} + +// serveSnapshot is what one bd serve run shows before any operation: its two +// startup events (from "event=" on, without the log prefix) and its handshake. +type serveSnapshot struct { + addr string + startup string + limits string + context map[string]any +} + +// serveInProcessSnapshot runs `bd serve` through rootCmd, reads the handshake, +// and shuts it down the way an operator does. +func serveInProcessSnapshot(t *testing.T, dir, beadsDir string) serveSnapshot { + t.Helper() + stopStderr := captureStderrText(t) + addr, done := startServeInProcess(t, dir, beadsDir) + // Stops a server that a failure below left running. Keyed on whether done + // has been read, not on rootCancel: a run that returns an error skips + // PersistentPostRunE and leaves rootCancel set, and waiting on done again + // after reading its only result would hang the test binary. + drained := false + t.Cleanup(func() { + if !drained && rootCancel != nil { + rootCancel() + <-done + } + }) + body := getJSON(t, "http://"+addr+"/v0/beads/context") + if rootCancel == nil { + t.Fatal("the root command published no cancel function; nothing here can stop the server") + } + rootCancel() + err := <-done + drained = true + if err != nil { + t.Fatalf("bd serve returned %v, want a clean shutdown", err) + } + log := stopStderr() + return serveSnapshot{ + addr: addr, + startup: serveEventLine(t, log, "startup"), + limits: serveEventLine(t, log, "limits"), + context: body, + } +} + +// serveProcessSnapshot reads a spawned serve's startup and handshake, then +// stops it with SIGTERM. +func serveProcessSnapshot(t *testing.T, sp *serveProcess) serveSnapshot { + t.Helper() + startup := sp.awaitLogLine(t, "event=startup") + limits := sp.awaitLogLine(t, "event=limits") + status, body, _ := sp.get(t, "/v0/beads/context") + if status != http.StatusOK { + t.Fatalf("GET /v0/beads/context status = %d, want 200: %v", status, body) + } + sp.shutdown(t) + return serveSnapshot{ + addr: sp.addr, + startup: serveEventLine(t, startup, "startup"), + limits: serveEventLine(t, limits, "limits"), + context: body, + } +} + +// serveEventLine returns the named event's line from "event=" on. +func serveEventLine(t *testing.T, log, name string) string { + t.Helper() + for _, line := range strings.Split(log, "\n") { + if i := strings.Index(line, "event="+name+" "); i >= 0 { + return line[i:] + } + } + t.Fatalf("no event=%s line in:\n%s", name, log) + return "" +} + +// assertRunServeBuiltIt checks one run against the server runServe builds: +// serveResolvedMode's label, the Issue API's database source, and a +// capabilities field equal to the handshake's. +func assertRunServeBuiltIt(t *testing.T, run string, s serveSnapshot, source, mode string) { + t.Helper() + assertServedCapabilities(t, s.context) + if s.context["api_version"] != "v0" { + t.Errorf("%s: context api_version = %v, want v0", run, s.context["api_version"]) + } + caps, _ := s.context["capabilities"].([]any) + tokens := make([]string, 0, len(caps)) + for _, c := range caps { + token, _ := c.(string) + tokens = append(tokens, token) + } + for _, field := range []string{ + "addr=" + s.addr, + "mode=" + strconv.Quote(mode), + "db=" + source, + "capabilities=" + strings.Join(tokens, ","), + } { + if !strings.Contains(s.startup+" ", " "+field+" ") { + t.Errorf("%s: startup line is missing %s:\n%s", run, field, s.startup) + } + } +} + +// assertSameServer compares two runs on everything but the ephemeral port. +func assertSameServer(t *testing.T, ordinary, asserted serveSnapshot) { + t.Helper() + normalize := func(s serveSnapshot) string { + return strings.Replace(s.startup, " addr="+s.addr+" ", " addr= ", 1) + } + if got, want := normalize(asserted), normalize(ordinary); got != want { + t.Errorf("%s=dependency changed the startup line:\n got: %s\nwant: %s", serveGraphModeEnv, got, want) + } + if asserted.limits != ordinary.limits { + t.Errorf("%s=dependency changed the limits line:\n got: %s\nwant: %s", serveGraphModeEnv, asserted.limits, ordinary.limits) + } + if !reflect.DeepEqual(asserted.context, ordinary.context) { + t.Errorf("%s=dependency changed GET /v0/beads/context:\n got: %v\nwant: %v", serveGraphModeEnv, asserted.context, ordinary.context) + } +} + +// captureStderrText redirects os.Stderr until the returned function is called, +// and returns everything written meanwhile. The restore also runs at cleanup, +// so a t.Fatal partway through cannot leak the redirect into later tests; a +// failed test logs what was captured. +func captureStderrText(t *testing.T) func() string { + t.Helper() + r, w, err := os.Pipe() + if err != nil { + t.Fatalf("os.Pipe: %v", err) + } + orig := os.Stderr + os.Stderr = w + + var buf bytes.Buffer + copied := make(chan struct{}) + go func() { + defer close(copied) + _, _ = io.Copy(&buf, r) + }() + + var once sync.Once + var out string + stop := func() string { + once.Do(func() { + os.Stderr = orig + _ = w.Close() + <-copied + _ = r.Close() + out = buf.String() + }) + return out + } + t.Cleanup(func() { + if text := stop(); t.Failed() { + t.Logf("captured stderr:\n%s", text) + } + }) + return stop +} diff --git a/internal/httpapi/server_test.go b/internal/httpapi/server_test.go index ba2105508..54f43e91c 100644 --- a/internal/httpapi/server_test.go +++ b/internal/httpapi/server_test.go @@ -14,6 +14,7 @@ import ( "os" "path/filepath" "slices" + "strconv" "strings" "sync" "testing" @@ -1275,6 +1276,124 @@ func TestStartupLines(t *testing.T) { } } +// TestStartupEventPublishesTheAdvertisedCapabilitiesAndSource pins the whole +// event=startup line of a provider- or roles-backed server: its field sequence, +// its database-source label, and a capabilities field that is the handshake's +// list verbatim. +// +// TestStartupLines and TestStartupNamesTheDatabaseSource check fields by +// substring, and neither reads capabilities at all. So a server could print one +// list and advertise another, or a new kind of database source could relabel +// this line, and both would stay green. The line is how an operator learns +// what a server answers before any client connects, so it is pinned exactly: +// changing it should take a deliberate edit to this test, not happen as a side +// effect of adding a source. +func TestStartupEventPublishesTheAdvertisedCapabilitiesAndSource(t *testing.T) { + workspace := domain.ContextInfo{RepoRoot: "/w/repo", BeadsDir: "/w/repo/.beads", Database: "beads"} + for _, tc := range []struct { + name string + cfg Config + source string + }{ + {"provider", Config{Provider: &tunableProvider{}}, "provider"}, + {"provider with the external-dependency policy", Config{Provider: &tunableProvider{}, ExternalDependencyPolicy: true}, "provider"}, + {"roles", rolesConfig(Config{}), "roles"}, + {"roles with the external-dependency policy", rolesConfig(Config{ExternalDependencyPolicy: true}), "roles"}, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := tc.cfg + cfg.Mode = "server (external dolt)" + cfg.Workspace = workspace + ts := newTestServer(t, cfg) + + var advertised []string + caps, _ := decodeBody(t, ts.get(t, "/v0/beads/context"))["capabilities"].([]any) + for _, c := range caps { + advertised = append(advertised, c.(string)) + } + if len(advertised) == 0 { + t.Fatal("the handshake advertises no capabilities; comparing the startup line against it would pass for an empty one") + } + // The conditional token is in the list exactly when the case asks + // for it, so the policy rows really cover a list that differs. + if got := slices.Contains(advertised, CapExternalDependencies); got != cfg.ExternalDependencyPolicy { + t.Fatalf("handshake advertises %s = %v, want %v", CapExternalDependencies, got, cfg.ExternalDependencyPolicy) + } + + line := findLogLine(t, ts.stderr.String(), "event=startup") + keys, fields := eventFields(t, line) + want := []struct{ key, value string }{ + {"event", "startup"}, + {"addr", ts.Addr()}, + {"mode", cfg.Mode}, + {"db", tc.source}, + {"workspace", workspace.RepoRoot}, + {"beads_dir", workspace.BeadsDir}, + {"database", workspace.Database}, + {"host_allowlist", "127.0.0.1,::1,localhost"}, + {"capabilities", strings.Join(advertised, ",")}, + {"auth", "none"}, + } + var wantKeys []string + for _, w := range want { + wantKeys = append(wantKeys, w.key) + if got := fields[w.key]; got != w.value { + t.Errorf("startup %s = %q, want %q", w.key, got, w.value) + } + } + if !slices.Equal(keys, wantKeys) { + t.Errorf("startup fields = %v, want %v\n%s", keys, wantKeys, line) + } + }) + } +} + +// eventFields splits one event line back into its fields, in order: the +// inverse of Server.event, where every value is a bare token or a Go-quoted +// string (logValue). The log prefix before "event=" is dropped. +func eventFields(t *testing.T, line string) ([]string, map[string]string) { + t.Helper() + start := strings.Index(line, "event=") + if start < 0 { + t.Fatalf("not an event line: %q", line) + } + rest := line[start:] + var keys []string + fields := map[string]string{} + for rest != "" { + key, value, ok := strings.Cut(rest, "=") + if !ok || key == "" || strings.ContainsAny(key, ` "`) { + t.Fatalf("malformed field at %q in %q", rest, line) + } + if strings.HasPrefix(value, `"`) { + quoted, err := strconv.QuotedPrefix(value) + if err != nil { + t.Fatalf("malformed quoted value at %q in %q: %v", value, line, err) + } + unquoted, err := strconv.Unquote(quoted) + if err != nil { + t.Fatalf("unquote %s: %v", quoted, err) + } + rest = value[len(quoted):] + value = unquoted + if rest != "" { + if rest[0] != ' ' { + t.Fatalf("field %s is not followed by a space in %q", key, line) + } + rest = rest[1:] + } + } else { + value, rest, _ = strings.Cut(value, " ") + } + if _, dup := fields[key]; dup { + t.Fatalf("field %s appears twice in %q", key, line) + } + keys = append(keys, key) + fields[key] = value + } + return keys, fields +} + // TestPoolLimitsUnavailableIsAnnounced: running unbounded is a decision, so it // must not be a silent one. func TestPoolLimitsUnavailableIsAnnounced(t *testing.T) { From f142cc2ffdece62ae5830a7f436e4949a48c79fe Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 19 Oct 2026 03:41:08 +0000 Subject: [PATCH 05/41] test(cli): before-parity pins for --format json, ordinary --if-revision, and version/config/history/as-of shapes (P7, P8, P10) (#7676) * test(cli): pin --format json == --json on show, list, ready and memories Before-parity pin P7 (gap analysis 2026-10-17, section 5a). The hidden root --format flag is an alias for --json (GH#2612). It is a root persistent flag, so a subcommand that registers a --format of its own shadows it (the GH#6278 hazard): `bd list` already does and passes `json` through by hand, and a graph-preview fork that added a local --format to `bd show` made `bd show ID --format json` exit nonzero. TestEmbeddedFormatJSONAliasMatchesJSON runs show, list, ready and memories with --json and then with `--format json`, `--format=json` and a leading `bd --format json `, and requires exit 0 and byte-identical stdout. It adds bdRunStreams, a small helper returning stdout, stderr and the exit status separately, which the following pins reuse. The new TestEmbedded* function is listed in the bd_test srcs and placed in the 100-shard embedded cmd manifest by gen_embedded_cmd_shard_manifest.py --write (incremental). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(cli): pin the ordinary --if-revision compare-and-swap contract Before-parity pin P8 (gap analysis 2026-10-17, section 5a). --if-revision is the decimal int64 compare-and-swap merged in #7203, and gascity's bdstore_conditional.go classifier reads both its exit status and its machine body. A second, opaque-token meaning for the same flag in another kind of workspace must leave the ordinary one unchanged. TestEmbeddedIfRevisionOrdinaryCAS pins, through the real binary: - a stale decimal token on update, close, assign and delete (preview and --force) exits 13, prints nothing on stdout, writes nothing, and ends stderr with a precondition_failed body naming the id and the expected and current revisions as exact JSON integers; - a non-decimal token exits 1 with the "invalid --if-revision" refusal in text and --json modes, never 13, and writes nothing; - the current token applies on update, close, assign and delete. It is the ordinary-workspace half of TestGraphPreviewIfRevisionOutsideLinkModeIsUpstreamCAS from the donnabox/codex/preview3-source-catchup-46aa branch (added in 3d16d826be, revised in 1640940fa9, 3ff2fe73ce, 55dbb61156, 653369305a and 97446d1f08), ported onto this package's embedded helpers. The port drops the graph-only forget refusal, extends the stale and non-decimal cases from update/delete to close and assign, and compares revisions as json.Number rather than float64, since they are int64 values past 2^53. It is named TestEmbedded* so the bazel-embedded lane, which discovers `^func TestEmbedded` in *_embedded_test.go, runs it; the existing TestIfRevisionMatchAndMismatch is not discovered there. The file is listed in the bd_test srcs and placed in the 100-shard embedded cmd manifest by gen_embedded_cmd_shard_manifest.py --write (incremental). Co-authored-by: Jim Wordelman Co-authored-by: Donna Box Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(cli): pin version, config set, history and show --as-of shapes Before-parity pin P10 (gap analysis 2026-10-17, section 5a): the store-less and history-adjacent commands a later versioned-history or graph slice could change by accident. - TestEmbeddedVersionOutputShape: `bd version` with no arguments exits 0 and prints one "bd version (...)" line, in a workspace and outside one; --json is one object with version, build and schema_version (plus commit and branch when known) agreeing with the text line, and --format json prints the same document; an argument exits 1 and points at `bd versions ` (#6661), on stderr or as the --json error, through the real binary rather than RunE. - TestEmbeddedConfigSetOtherKeysSkipTheVersionedHistoryScan: in a store holding a row the pre-enable scan refuses over, `bd config set` and `bd config set-many` of another key with the value "true" exit 0, store it, print no refusal and leave the switch off; the switch itself is still refused afterwards, as a control. - TestEmbeddedHistoryAndShowAsOfReadDoltCommits: with versioned history recording, `bd history --json` is still the Dolt-commit history (an array of CommitHash/Committer/CommitDate/Issue entries with full 32-character Dolt hashes, no recorded-version fields, and the same hashes in the text output), and `bd show --as-of ` with a hash from it is the Dolt AS OF read in show's JSON shape, not the R7.1 as-of read over issue_versions; `--as-of main` reads the branch head. The three TestEmbedded* functions are placed in the 100-shard embedded cmd manifest by gen_embedded_cmd_shard_manifest.py --write (incremental). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: Jim Wordelman Co-authored-by: Donna Box --- .github/scripts/embedded-cmd-test-shards.txt | 5 + cmd/bd/BUILD.bazel | 2 + cmd/bd/format_json_alias_embedded_test.go | 102 +++++++++++ cmd/bd/history_embedded_test.go | 127 +++++++++++++ .../if_revision_ordinary_cas_embedded_test.go | 168 ++++++++++++++++++ cmd/bd/version_embedded_test.go | 95 ++++++++++ ...ioned_history_switch_scan_embedded_test.go | 50 ++++++ 7 files changed, 549 insertions(+) create mode 100644 cmd/bd/format_json_alias_embedded_test.go create mode 100644 cmd/bd/if_revision_ordinary_cas_embedded_test.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index 983c57438..e05f241a2 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -123,13 +123,18 @@ 100 49 TestEmbeddedInitAPushConsentAndRemoteBootstrap 100 50 TestEmbeddedDiffConcurrent +100 50 TestEmbeddedVersionOutputShape +100 51 TestEmbeddedHistoryAndShowAsOfReadDoltCommits 100 51 TestEmbeddedInitGitBootstrapRoutingFreshQuiet +100 52 TestEmbeddedConfigSetOtherKeysSkipTheVersionedHistoryScan 100 52 TestEmbeddedSwarmConcurrent +100 53 TestEmbeddedIfRevisionOrdinaryCAS 100 53 TestEmbeddedInitSelectedExcludeRoutingStealthDecoyQuiet +100 54 TestEmbeddedFormatJSONAliasMatchesJSON 100 54 TestEmbeddedSearchConcurrent 100 55 TestEmbeddedUndefer diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 8477c2d1a..f16e65cbb 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -804,6 +804,7 @@ go_test( "find_duplicates_test.go", "flags_test.go", "flatten_embedded_test.go", + "format_json_alias_embedded_test.go", "formula_fixture_proxied_test.go", "formula_help_test.go", "formula_primitives_test.go", @@ -865,6 +866,7 @@ go_test( "identity_preview_refusal_test.go", "if_revision_embedded_test.go", "if_revision_gc_matcher_embedded_test.go", + "if_revision_ordinary_cas_embedded_test.go", "if_revision_proxied_close_delete_test.go", "if_revision_proxied_integration_test.go", "if_revision_reassign_fence_embedded_test.go", diff --git a/cmd/bd/format_json_alias_embedded_test.go b/cmd/bd/format_json_alias_embedded_test.go new file mode 100644 index 000000000..3e163c870 --- /dev/null +++ b/cmd/bd/format_json_alias_embedded_test.go @@ -0,0 +1,102 @@ +//go:build cgo + +package main + +import ( + "bytes" + "encoding/json" + "errors" + "os" + "os/exec" + "slices" + "strings" + "testing" +) + +// bdRunStreams runs one bd process in dir with the hermetic test environment +// and returns stdout, stderr and the exit status separately. A nonzero exit is +// returned, not fatal, so a caller can pin the status itself; only a process +// that could not be started fails the test here. +func bdRunStreams(t *testing.T, bd, dir string, args ...string) (stdout, stderr string, exit int) { + t.Helper() + cmd := exec.Command(bd, args...) + cmd.Dir = dir + cmd.Env = bdEnv(dir) + var outBuf, errBuf bytes.Buffer + cmd.Stdout, cmd.Stderr = &outBuf, &errBuf + if err := cmd.Run(); err != nil { + var ee *exec.ExitError + if !errors.As(err, &ee) { + t.Fatalf("bd %s did not run: %v", strings.Join(args, " "), err) + } + return outBuf.String(), errBuf.String(), ee.ExitCode() + } + return outBuf.String(), errBuf.String(), 0 +} + +// TestEmbeddedFormatJSONAliasMatchesJSON pins the hidden root --format alias +// (GH#2612): on show, list, ready and memories, `--format json` prints exactly +// what `--json` prints, byte for byte, and exits 0 like it. +// +// The alias is a root persistent flag, so any subcommand that registers a +// --format of its own shadows it (the GH#6278 hazard). `bd list` already has +// its own --format and passes `json` through by hand (list_input.go). A +// local --format added to show, ready or memories for another output +// format would turn `bd show ID --format json` into a refusal or into a +// different document. Both are regressions this test catches: it compares +// the whole of stdout, not a parse of it. +// +// Each command is checked in three spellings: `--format json` and +// `--format=json` after the subcommand, and `--format json` before it, where +// the flag is plainly the root's. +func TestEmbeddedFormatJSONAliasMatchesJSON(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "fj") + + id := bdCreateSilent(t, bd, dir, "Format alias parity", "--type", "task", "--description", "read through --format json") + bdRunOK(t, bd, dir, "remember", "the format alias is --json", "--key", "fj-alias") + + for _, tc := range []struct { + name string + args []string + // mention is a value the --json document must contain, so two empty + // documents cannot match each other. + mention string + }{ + {"show", []string{"show", id}, id}, + {"list", []string{"list"}, id}, + {"ready", []string{"ready"}, id}, + {"memories", []string{"memories"}, "fj-alias"}, + } { + t.Run(tc.name, func(t *testing.T) { + wantArgs := slices.Concat(tc.args, []string{"--json"}) + want, wantErr, exit := bdRunStreams(t, bd, dir, wantArgs...) + if exit != 0 { + t.Fatalf("bd %s exited %d\nstdout:\n%s\nstderr:\n%s", strings.Join(wantArgs, " "), exit, want, wantErr) + } + if !json.Valid([]byte(want)) || !strings.Contains(want, tc.mention) { + t.Fatalf("bd %s did not print one JSON document naming %s:\n%s", strings.Join(wantArgs, " "), tc.mention, want) + } + + for _, args := range [][]string{ + slices.Concat(tc.args, []string{"--format", "json"}), + slices.Concat(tc.args, []string{"--format=json"}), + slices.Concat([]string{"--format", "json"}, tc.args), + } { + got, gotErr, exit := bdRunStreams(t, bd, dir, args...) + if exit != 0 { + t.Errorf("bd %s exited %d, want 0 like bd %s\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), exit, strings.Join(wantArgs, " "), got, gotErr) + continue + } + if got != want { + t.Errorf("bd %s printed a different document from bd %s\n--format:\n%s\n--json:\n%s", strings.Join(args, " "), strings.Join(wantArgs, " "), got, want) + } + } + }) + } +} diff --git a/cmd/bd/history_embedded_test.go b/cmd/bd/history_embedded_test.go index 5c7910c20..ab351a1dd 100644 --- a/cmd/bd/history_embedded_test.go +++ b/cmd/bd/history_embedded_test.go @@ -7,6 +7,8 @@ import ( "fmt" "os" "os/exec" + "regexp" + "sort" "strings" "sync" "testing" @@ -403,3 +405,128 @@ func TestEmbeddedHistoryConcurrent(t *testing.T) { } } } + +// doltCommitHashPattern is a full Dolt commit hash: 32 characters of Dolt's +// base32 alphabet. It is what `bd show --as-of` takes, and a prefix is not. +var doltCommitHashPattern = regexp.MustCompile(`^[0-9a-v]{32}$`) + +// issueVersionOnlyKeys are fields of a recorded version (`bd versions --json`, +// storage.IssueVersion) that neither the Dolt-commit history nor the Dolt AS OF +// read has. +var issueVersionOnlyKeys = []string{"local_revision", "epoch", "change_actor", "change_at", "attribution_status", "removed_restriction", "state_bytes"} + +// TestEmbeddedHistoryAndShowAsOfReadDoltCommits pins what `bd history ` and +// `bd show --as-of ` read in an ordinary workspace whose store records +// versioned history, so both planes hold the same issue: +// - `bd history --json` is the Dolt-commit history: an array, newest first, +// of {CommitHash, Committer, CommitDate, Issue} entries, each CommitHash a full +// Dolt commit hash and each Issue the row as that commit left it. No field of a +// recorded version appears, and the text output names the same full hashes. +// - `bd show --as-of `, with a hash from `bd history`, is the Dolt AS OF +// read: the issue as that commit left it, in show's JSON shape (an array of +// issues), not a recorded version. A branch name is a ref too: --as-of main +// reads the branch head. +// +// Recorded versions are a separate plane with its own command (`bd versions`) and +// its own as-of read (R7.1, over issue_versions), which takes an instant or a +// version address and not a Dolt ref. Recording them must change neither answer. +func TestEmbeddedHistoryAndShowAsOfReadDoltCommits(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "hd") + bdConfig(t, bd, dir, "set", "versioned-history.enabled", "true") + + id := bdCreateSilent(t, bd, dir, "Before title", "--type", "task") + bdUpdate(t, bd, dir, id, "--title", "After title") + + // Both planes hold this issue: versions were recorded for it. + var recorded struct { + Recording bool `json:"recording"` + Versions []json.RawMessage `json:"versions"` + } + if out, stderr, exit := bdRunStreams(t, bd, dir, "versions", id, "--json"); exit != 0 || json.Unmarshal([]byte(out), &recorded) != nil || !recorded.Recording || len(recorded.Versions) == 0 { + t.Fatalf("bd versions %s --json (exit %d) did not report recorded versions, so this test would not tell the planes apart\nstdout:\n%s\nstderr:\n%s", id, exit, out, stderr) + } + + historyJSON := bdHistory(t, bd, dir, id, "--json") + var entries []map[string]json.RawMessage + if err := json.Unmarshal([]byte(historyJSON), &entries); err != nil { + t.Fatalf("bd history %s --json is not a JSON array: %v\n%s", id, err, historyJSON) + } + if len(entries) < 2 { + t.Fatalf("bd history %s --json has %d entries, want one for each of the create and the update:\n%s", id, len(entries), historyJSON) + } + type snapshot struct { + ID string `json:"id"` + Title string `json:"title"` + } + hashes := make([]string, len(entries)) + titles := make([]string, len(entries)) + for i, entry := range entries { + keys := make([]string, 0, len(entry)) + for key := range entry { + keys = append(keys, key) + } + sort.Strings(keys) + if got := strings.Join(keys, ","); got != "CommitDate,CommitHash,Committer,Issue" { + t.Errorf("bd history entry %d has keys %s, want the Dolt-commit entry CommitDate,CommitHash,Committer,Issue", i, got) + } + if err := json.Unmarshal(entry["CommitHash"], &hashes[i]); err != nil || !doltCommitHashPattern.MatchString(hashes[i]) { + t.Errorf("bd history entry %d CommitHash %s is not a full Dolt commit hash", i, entry["CommitHash"]) + } + var issue snapshot + if err := json.Unmarshal(entry["Issue"], &issue); err != nil || issue.ID != id { + t.Errorf("bd history entry %d Issue %s is not a snapshot of %s", i, entry["Issue"], id) + } + titles[i] = issue.Title + } + if titles[0] != "After title" || titles[len(titles)-1] != "Before title" { + t.Fatalf("bd history %s titles newest first are %q, want %q first and %q last", id, titles, "After title", "Before title") + } + for _, key := range issueVersionOnlyKeys { + if strings.Contains(historyJSON, `"`+key+`"`) { + t.Errorf("bd history %s --json carries %q, a field of a recorded version:\n%s", id, key, historyJSON) + } + } + text := bdHistory(t, bd, dir, id) + for _, hash := range hashes { + if !strings.Contains(text, hash) { + t.Errorf("bd history %s text output does not name the full commit hash %s:\n%s", id, hash, text) + } + } + + // show --as-of reads the Dolt commit the history named. + first := hashes[len(hashes)-1] + asOfJSON := func(ref string) snapshot { + t.Helper() + out, stderr, exit := bdRunStreams(t, bd, dir, "show", id, "--as-of", ref, "--json") + var shown []map[string]json.RawMessage + if exit != 0 || json.Unmarshal([]byte(out), &shown) != nil || len(shown) != 1 { + t.Fatalf("bd show %s --as-of %s --json exited %d, want an array of one issue\nstdout:\n%s\nstderr:\n%s", id, ref, exit, out, stderr) + } + for _, key := range issueVersionOnlyKeys { + if _, ok := shown[0][key]; ok { + t.Errorf("bd show %s --as-of %s --json carries %q, a field of a recorded version:\n%s", id, ref, key, out) + } + } + var issues []snapshot + if err := json.Unmarshal([]byte(out), &issues); err != nil { + t.Fatalf("bd show %s --as-of %s --json: %v\n%s", id, ref, err, out) + } + return issues[0] + } + if got := asOfJSON(first); got.ID != id || got.Title != "Before title" { + t.Errorf("bd show %s --as-of %s (its first commit) read %+v, want the issue titled %q", id, first, got, "Before title") + } + if got := asOfJSON("main"); got.ID != id || got.Title != "After title" { + t.Errorf("bd show %s --as-of main read %+v, want the branch head's title %q", id, got, "After title") + } + human := bdShowRaw(t, bd, dir, id, "--as-of", first) + if !strings.Contains(human, "as of "+first) || !strings.Contains(human, "Before title") || strings.Contains(human, "After title") { + t.Errorf("bd show %s --as-of %s text output does not show the first commit's issue under that ref:\n%s", id, first, human) + } +} diff --git a/cmd/bd/if_revision_ordinary_cas_embedded_test.go b/cmd/bd/if_revision_ordinary_cas_embedded_test.go new file mode 100644 index 000000000..c51b058fa --- /dev/null +++ b/cmd/bd/if_revision_ordinary_cas_embedded_test.go @@ -0,0 +1,168 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "os" + "slices" + "strconv" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +// ifRevisionStaleExit is a literal on purpose: the contract with gascity's +// bdstore_conditional.go classifier is the number 13, not whatever name the +// constant for it (ExitGuardMismatch) happens to have. +const ifRevisionStaleExit = 13 + +// decodeIfRevisionBodyExact decodes the flat machine body a refused +// --if-revision write puts on the last line of stderr, keeping numbers as +// json.Number. Revisions are int64 values well past 2^53, so comparing them +// as float64 would accept a neighbouring revision; a json.Number also proves +// the field is a JSON number and not a string. +func decodeIfRevisionBodyExact(t *testing.T, line string) map[string]any { + t.Helper() + dec := json.NewDecoder(strings.NewReader(line)) + dec.UseNumber() + var body map[string]any + if err := dec.Decode(&body); err != nil { + t.Fatalf("decode the --if-revision refusal body: %v\nline: %s", err, line) + } + return body +} + +// TestEmbeddedIfRevisionOrdinaryCAS pins the ordinary --if-revision +// compare-and-swap (#7203) end to end, as a gascity-style caller sees it: +// +// - a stale decimal token on update, close, assign and delete (preview and +// --force) exits 13 with a precondition_failed body on the last line of +// stderr naming the id and both revisions as JSON integers, prints +// nothing on stdout and writes nothing; +// - a token that is not a decimal exits 1 with the "invalid --if-revision" +// refusal, never 13, and writes nothing; +// - the current token applies on update, close, assign and delete, and +// each write that leaves the issue in place moves its revision on. +// +// A second meaning for the same flag (an opaque token in some other kind of +// workspace) must leave all of this unchanged in an ordinary one. This is +// the ordinary-workspace half of the fork's +// TestGraphPreviewIfRevisionOutsideLinkModeIsUpstreamCAS, ported onto this +// package's embedded helpers and extended from update/delete to close and +// assign. +func TestEmbeddedIfRevisionOrdinaryCAS(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "ifrev") + + id := bdCreateSilent(t, bd, dir, "Stale guard", "--type", "task", "--priority", "2") + stale := bdShowRevision(t, bd, dir, id) + bdRunOK(t, bd, dir, "update", id, "--notes", "bump", "--json") + current := bdShowRevision(t, bd, dir, id) + if current == stale { + t.Fatal("an unguarded update did not advance the revision") + } + staleToken := strconv.FormatInt(stale, 10) + + for _, tc := range []struct { + name string + args []string + }{ + {"update", []string{"update", id, "--priority", "3"}}, + {"close", []string{"close", id}}, + {"assign", []string{"assign", id, "worker"}}, + {"delete preview", []string{"delete", id}}, + {"delete force", []string{"delete", id, "--force"}}, + } { + args := slices.Concat(tc.args, []string{"--if-revision", staleToken, "--json"}) + stdout, stderr, exit := bdRunStreams(t, bd, dir, args...) + if exit != ifRevisionStaleExit { + t.Fatalf("stale --if-revision %s exited %d, want %d\nstdout:\n%s\nstderr:\n%s", tc.name, exit, ifRevisionStaleExit, stdout, stderr) + } + if stdout != "" { + t.Errorf("stale --if-revision %s printed on stdout, want nothing:\n%s", tc.name, stdout) + } + body := decodeIfRevisionBodyExact(t, lastJSONLine(t, stderr)) + if body["code"] != ifRevisionCodePreconditionFailed || body["id"] != id { + t.Errorf("stale --if-revision %s reported %v, want code %q for %s", tc.name, body, ifRevisionCodePreconditionFailed, id) + } + if got := body["expected_revision"]; got != json.Number(staleToken) { + t.Errorf("stale --if-revision %s reported expected_revision %#v, want the JSON integer %s", tc.name, got, staleToken) + } + if got := body["current_revision"]; got != json.Number(strconv.FormatInt(current, 10)) { + t.Errorf("stale --if-revision %s reported current_revision %#v, want the JSON integer %d", tc.name, got, current) + } + if msg, _ := body["error"].(string); !strings.Contains(msg, "precondition failed") { + t.Errorf("stale --if-revision %s error text %q lacks \"precondition failed\"", tc.name, msg) + } + if got := bdShowRevision(t, bd, dir, id); got != current { + t.Fatalf("stale --if-revision %s wrote: revision %d -> %d", tc.name, current, got) + } + } + + // A token that is not a decimal is a typo refused before any write, with + // exit 1 and not the guard's 13, in both output modes. + for _, base := range [][]string{ + {"update", id, "--priority", "3"}, + {"close", id}, + {"assign", id, "worker"}, + {"delete", id}, + {"delete", id, "--force"}, + } { + for _, mode := range [][]string{nil, {"--json"}} { + args := slices.Concat(base, []string{"--if-revision", "observed"}, mode) + stdout, stderr, exit := bdRunStreams(t, bd, dir, args...) + out := stdout + stderr + // The text refusal quotes the token; the --json one carries the + // same text JSON-escaped. + refused := strings.Contains(out, `invalid --if-revision "observed"`) || + strings.Contains(out, `invalid --if-revision \"observed\"`) + if exit != 1 || !refused { + t.Errorf("bd %s exited %d, want exit 1 refusing the non-decimal token\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), exit, stdout, stderr) + } + if strings.Contains(out, ifRevisionCodePreconditionFailed) { + t.Errorf("bd %s reported a stale revision for a token that is not one:\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), stdout, stderr) + } + } + } + if got := bdShowRevision(t, bd, dir, id); got != current { + t.Fatalf("a refused non-decimal token wrote: revision %d -> %d", current, got) + } + + // The current token applies, on each verb in turn. + revision := current + guarded := func(target string, args ...string) { + t.Helper() + bdRunOK(t, bd, dir, slices.Concat(args, []string{"--if-revision", strconv.FormatInt(revision, 10), "--json"})...) + next := bdShowRevision(t, bd, dir, target) + if next == revision { + t.Fatalf("a matching --if-revision on bd %s did not advance the revision", strings.Join(args, " ")) + } + revision = next + } + guarded(id, "update", id, "--priority", "3") + if got := bdShow(t, bd, dir, id); got.Priority != 3 { + t.Errorf("matching --if-revision update did not apply: priority = %d, want 3", got.Priority) + } + guarded(id, "close", id) + if got := bdShow(t, bd, dir, id); got.Status != types.StatusClosed { + t.Errorf("matching --if-revision close did not apply: status = %s, want closed", got.Status) + } + bdRunOK(t, bd, dir, "delete", id, "--force", "--if-revision", strconv.FormatInt(revision, 10), "--json") + bdShowFail(t, bd, dir, id) + + // assign gets an issue of its own: closing an issue assigned to someone + // other than the actor is refused without --force, whatever the guard. + other := bdCreateSilent(t, bd, dir, "Guarded assign", "--type", "task") + revision = bdShowRevision(t, bd, dir, other) + guarded(other, "assign", other, "worker") + if got := bdShow(t, bd, dir, other); got.Assignee != "worker" { + t.Errorf("matching --if-revision assign did not apply: assignee = %q, want worker", got.Assignee) + } +} diff --git a/cmd/bd/version_embedded_test.go b/cmd/bd/version_embedded_test.go index a2ecfea8d..436f7a5be 100644 --- a/cmd/bd/version_embedded_test.go +++ b/cmd/bd/version_embedded_test.go @@ -3,9 +3,11 @@ package main import ( + "encoding/json" "fmt" "os" "os/exec" + "regexp" "strings" "sync" "testing" @@ -72,3 +74,96 @@ func TestEmbeddedVersionConcurrent(t *testing.T) { } } } + +// versionLinePattern is the whole of `bd version`'s stdout: +// "bd version ()", with ": " or +// ": @" inside the parentheses when the binary knows them. +var versionLinePattern = regexp.MustCompile(`^bd version (\S+) \(([^):]+)(?:: (?:\S+@)?\S+)?\)\n$`) + +// TestEmbeddedVersionOutputShape pins `bd version` the way scripts and +// plugins call it, in an ordinary workspace and in a directory that is not +// one: +// - with no arguments it exits 0 and prints exactly one line on stdout, +// "bd version (...)"; +// - --json prints one object with version, build and schema_version, plus +// commit and branch when known, and no other key; its version and build +// agree with the text line, and --format json prints the same document; +// - an argument is refused with exit 1 and points at `bd versions ` +// (#6661): on stderr in text mode, as the JSON error on stdout with +// --json, and without the version line either way. +// +// TestVersionWithAnArgumentPointsAtVersions pins the refusal on RunE; this +// pins it through the real binary, where the root pre-run and workspace +// discovery run first. +func TestEmbeddedVersionOutputShape(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + workspace, _, _ := bdInit(t, bd, "--prefix", "vo") + + for name, dir := range map[string]string{"workspace": workspace, "no workspace": t.TempDir()} { + t.Run(name, func(t *testing.T) { + text, stderr, exit := bdRunStreams(t, bd, dir, "version") + if exit != 0 { + t.Fatalf("bd version exited %d\nstdout:\n%s\nstderr:\n%s", exit, text, stderr) + } + line := versionLinePattern.FindStringSubmatch(text) + if line == nil { + t.Fatalf("bd version stdout is not one %q line:\n%q", versionLinePattern, text) + } + + doc, stderr, exit := bdRunStreams(t, bd, dir, "version", "--json") + if exit != 0 { + t.Fatalf("bd version --json exited %d\nstdout:\n%s\nstderr:\n%s", exit, doc, stderr) + } + var fields map[string]any + if err := json.Unmarshal([]byte(doc), &fields); err != nil { + t.Fatalf("bd version --json is not one JSON object: %v\n%s", err, doc) + } + for key := range fields { + switch key { + case "version", "build", "schema_version", "commit", "branch": + default: + t.Errorf("bd version --json has an unexpected key %q:\n%s", key, doc) + } + } + if _, ok := fields["schema_version"].(float64); !ok { + t.Errorf("bd version --json has no numeric schema_version:\n%s", doc) + } + if fields["version"] != line[1] || fields["build"] != line[2] { + t.Errorf("bd version --json says version %v build %v, the text line says %s and %s:\n%s", fields["version"], fields["build"], line[1], line[2], doc) + } + for _, key := range []string{"commit", "branch"} { + if v, ok := fields[key]; ok { + if s, isString := v.(string); !isString || s == "" { + t.Errorf("bd version --json %s = %#v, want a non-empty string when present", key, v) + } + } + } + if alias, _, exit := bdRunStreams(t, bd, dir, "version", "--format", "json"); exit != 0 || alias != doc { + t.Errorf("bd version --format json exited %d and printed a different document from --json:\n%s\nwant:\n%s", exit, alias, doc) + } + + stdout, stderr, exit := bdRunStreams(t, bd, dir, "version", "vo-123") + if exit != 1 || stdout != "" { + t.Errorf("bd version vo-123 exited %d with stdout %q, want exit 1 and nothing on stdout", exit, stdout) + } + for _, want := range []string{"Did you mean", "bd versions vo-123"} { + if !strings.Contains(stderr, want) { + t.Errorf("bd version vo-123 stderr lacks %q:\n%s", want, stderr) + } + } + + stdout, stderr, exit = bdRunStreams(t, bd, dir, "version", "vo-123", "--json") + var refusal struct { + Error string `json:"error"` + } + if exit != 1 || json.Unmarshal([]byte(stdout), &refusal) != nil || !strings.Contains(refusal.Error, "bd versions vo-123") { + t.Errorf("bd version vo-123 --json exited %d, want exit 1 and a JSON error naming `bd versions vo-123`\nstdout:\n%s\nstderr:\n%s", exit, stdout, stderr) + } + }) + } +} diff --git a/cmd/bd/versioned_history_switch_scan_embedded_test.go b/cmd/bd/versioned_history_switch_scan_embedded_test.go index 6c4c1a286..acf34632f 100644 --- a/cmd/bd/versioned_history_switch_scan_embedded_test.go +++ b/cmd/bd/versioned_history_switch_scan_embedded_test.go @@ -434,3 +434,53 @@ func TestEmbeddedVersionedHistorySwitchRefusesAnUnrecordableGateTimeout(t *testi t.Fatalf("after applying the remedies the switch did not turn on: config get says %q", got) } } + +// TestEmbeddedConfigSetOtherKeysSkipTheVersionedHistoryScan pins that the pre-enable scan belongs +// to versioned-history.enabled alone. In a store that holds a row the scan refuses over, `bd config +// set` and `bd config set-many` of any other key -- given the value "true", which is what turns +// the switch on -- exit 0, store the value, print no refusal and leave the switch off. The switch +// itself is still refused afterwards, which proves the store held a row the scan would have named. +// +// TestEmbeddedVersionedHistorySwitchRefusesUnversionableMetadata pins that turning the switch OFF +// never scans; this pins the other half of "any other key ... passes without reading a row" +// (checkVersionedHistoryCanBeEnabled's doc comment) through the real commands. +func TestEmbeddedConfigSetOtherKeysSkipTheVersionedHistoryScan(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + // The offender below can only be written while nothing records versions. No t.Parallel: + // t.Setenv is incompatible. + t.Setenv(vhEnvVar, "") + + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "vso") + + offender := bdCreateSilent(t, bd, dir, "holds an integer past 2^53", "--metadata", `{"n":9007199254740993}`) + + for _, args := range [][]string{ + {"config", "set", "custom.vso-marker", "true"}, + {"config", "set-many", "custom.vso-batch=true"}, + } { + stdout, stderr, exit := bdRunStreams(t, bd, dir, args...) + if exit != 0 { + t.Fatalf("bd %s exited %d in a store holding %s; the scan must not run for this key\nstdout:\n%s\nstderr:\n%s", strings.Join(args, " "), exit, offender, stdout, stderr) + } + if out := stdout + stderr; strings.Contains(out, "cannot turn versioned history on") || strings.Contains(out, offender) { + t.Errorf("bd %s printed the versioned-history refusal for a key that is not the switch:\n%s", strings.Join(args, " "), out) + } + } + for key, want := range map[string]string{"custom.vso-marker": "true", "custom.vso-batch": "true"} { + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", key)); got != want { + t.Errorf("config get %s = %q, want %q", key, got, want) + } + } + if got := strings.TrimSpace(bdConfig(t, bd, dir, "get", "versioned-history.enabled")); strings.HasPrefix(got, "true") { + t.Fatalf("setting other keys turned versioned history on: config get says %q", got) + } + + // The control: the scan does see the offender when the switch itself is set. + out, _ := bdRunFailCode(t, bd, dir, "config", "set", "versioned-history.enabled", "true") + if !strings.Contains(out, offender) { + t.Fatalf("the refused switch does not name %s, so the pin above proves nothing:\n%s", offender, out) + } +} From 05fffb1e11b4a60a298d884a694a1bd4e8da8152 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 19 Oct 2026 03:47:00 +0000 Subject: [PATCH 06/41] test(storage): versioned-history flag-off differential and statement pins (#6132, #6135) (#7681) * test(domain/db): pin the statements the history-off upsert Insert issues The participation_generation write fence (#6943) gave domain/db Insert's upsert path (CreateOnly unset: the quick, todo, gate and batch proxied servers' create) an existence probe ahead of the upsert, skipped while versioned history is off. Pin that skip with a sqlmock statement-sequence test: with history off, the upsert into issues and into wisps issues exactly the three statements the pre-stack code (4a67d0d028) issues -- the upsert, the created event's same-content scan and its insert -- and nothing else. The mock is reached through internal/storage/sqlcount, and the pin also requires exactly three statements to reach the driver. sqlmock alone answers an unscripted statement with an error and fails only if the caller returns it, so a best-effort read whose error Insert dropped would pass its checks; the count catches it. The history-on run of the same script is the control: the probe (and the mint after it) must make the pin refuse it, which shows the script is exhaustive and that Insert reads the transaction's scope. Passes on 4a67d0d028 and on main; main issues no extra read with history off. Before-parity pin P2 (Insert half) from the fork gap analysis, for gastownhall/beads#6132 / #6135. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(issueops): pin the statements the history-off defer-wake sweep issues Port the fork's sqlmock pin of WakeExpiredDefersInTx with versioned history off (donnabox/codex/preview3-source-catchup-46aa 11c74cb575): a one-row sweep issues exactly the expired-row SELECT, the UPDATE, the event's same-content SELECT and INSERT, and the empty wisps pass, so any read added before or after waking a row while history is off fails the pin. It passes on main unchanged and guards the fork's wake-skip change, which checks each expired row against the mint before waking it and must keep that check free with history off. Adapted for main: the doc no longer describes the fork's pre-wake check as existing; a history-on run of the same script is added as the control (the mint's read must make the pin refuse it), so the pin cannot pass vacuously; and the mock is reached through internal/storage/sqlcount, so the pin also requires exactly five statements to reach the driver. That catches a read whose error the sweep swallowed, which sqlmock alone does not see. Passes on 4a67d0d028 and on main. Before-parity pin P2 (wake half), for gastownhall/beads#6132 / #6135. Co-authored-by: Jim Wordelman Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(storage): versioned-history flag-off differential against pre-stack goldens gastownhall/beads#6132 and #6135 require a differential test proving that with versioned history off (the default) storage behaves byte-identically to before the feature. Existing coverage stops short of that: TestDualWriteContractFlagOff only counts version rows, the participation-generation contracts check one column, and createbatchequiv runs its batch with history on over a column subset. Add internal/storage/historyoffequiv. Each leg (embedded Dolt, Dolt server, unit of work) runs one scripted mutation set through its own front doors on a store whose history was never turned on: - on the issues plane, every write role: creates, batch create, apply batch, updates, the no-op update, reparent, dependency add/remove, comments, claim/heartbeat/release, lease reclaim, metadata CAS, close/reopen, batch close with a refused item, defer and wake, delete; - the persistence moves in both directions; - on the wisps plane, the roles again: create, update and the no-op update, dependency add/remove, comment, metadata CAS, close/reopen, apply batch, batch close, release, defer and wake (the sweep's wisps pass), delete, a forced close of a no-history wisp, and the refusals the lease roles' contracts give a wisp (Claim, Heartbeat; Reclaim finds nothing); - each leg's own surface: the store legs' DoltStorage methods (import upsert, promotion, RunInTransaction, ...) and the uow leg's CreateOnly-unset domain upsert and importer. Neither reclaim sweep finds an expired lease, since no role can age one. The harness then dumps every column of every base table, plus the scenario's Dolt commit messages and the final dolt_status, and compares per-table SHA-256 digests with goldens recorded by running the same harness against origin/main 4a67d0d028, the commit before the stack (#6943, #6661, #6664). The only tolerated differences are listed in schemaAdditions: the stack's new columns (issues/wisps.participation_generation, issue_versions.removed_restriction) must be NULL in every row and are then left out of the digest; its new table (epoch_minted_addresses) must be empty; its own migration bookkeeping rows (0070-0072, ignored/0028) may each be there once. Recording and checking apply that rule the same way, so a golden re-recorded from main is byte-identical to the pre-stack one (verified on all three legs). The package doc says how the goldens are kept: a later migration whose schema flag-off leaves untouched adds its own schemaAdditions entry; a deliberate flag-off change, a scenario change or an engine rendering change re-records them and shows the dump diff. Wall-clock instants, wall-clock-derived ids and random tokens are masked by explicit, documented rules found by diffing two runs; instants the scenario supplies stay verbatim, so a write that starts overwriting one still shows. All three legs pass on main. Before-parity pin P1 from the fork gap analysis; the new embedded test is placed in the 40-shard storage block. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: Jim Wordelman --- .../scripts/embedded-storage-test-shards.txt | 1 + BUILD.bazel | 4 + internal/storage/BUILD.bazel | 1 + internal/storage/dolt/BUILD.bazel | 2 + .../dolt/history_off_differential_test.go | 35 ++ internal/storage/domain/db/BUILD.bazel | 2 + .../db/issue_insert_history_off_test.go | 143 +++++ internal/storage/embeddeddolt/BUILD.bazel | 2 + .../history_off_differential_test.go | 39 ++ internal/storage/historyoffequiv/BUILD.bazel | 96 ++++ internal/storage/historyoffequiv/dump.go | 254 +++++++++ .../historyoffequiv/golden/embedded.json | 539 ++++++++++++++++++ .../historyoffequiv/golden/server.json | 539 ++++++++++++++++++ .../storage/historyoffequiv/golden/uow.json | 531 +++++++++++++++++ .../historyoffequiv/historyoffequiv.go | 528 +++++++++++++++++ internal/storage/historyoffequiv/scenario.go | 539 ++++++++++++++++++ internal/storage/issueops/BUILD.bazel | 2 + internal/storage/issueops/wake_defers_test.go | 127 +++++ internal/storage/uow/BUILD.bazel | 2 + .../uow/history_off_differential_test.go | 92 +++ tools/bazel/BUILD.bazel | 1 + 21 files changed, 3479 insertions(+) create mode 100644 internal/storage/dolt/history_off_differential_test.go create mode 100644 internal/storage/domain/db/issue_insert_history_off_test.go create mode 100644 internal/storage/embeddeddolt/history_off_differential_test.go create mode 100644 internal/storage/historyoffequiv/BUILD.bazel create mode 100644 internal/storage/historyoffequiv/dump.go create mode 100644 internal/storage/historyoffequiv/golden/embedded.json create mode 100644 internal/storage/historyoffequiv/golden/server.json create mode 100644 internal/storage/historyoffequiv/golden/uow.json create mode 100644 internal/storage/historyoffequiv/historyoffequiv.go create mode 100644 internal/storage/historyoffequiv/scenario.go create mode 100644 internal/storage/issueops/wake_defers_test.go create mode 100644 internal/storage/uow/history_off_differential_test.go diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index cef22090a..c3f45aacb 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -237,6 +237,7 @@ 40 10 TestGraphCounterContract 40 10 TestIgnoredCursorHealRespectsOperatorOverride 40 10 TestRunInTransactionPostCommitStageFailureIsIndeterminate +40 10 TestVersionedHistoryOffMatchesPreStackGolden_Embedded 40 10 TestWithPeerAuth_WarnsWhenStoredPeerSuppressesAmbientPasswordAmbient 40 11 TestABumpUnderOneStoreIDMovesTheEpochSeenThroughEveryStoreID diff --git a/BUILD.bazel b/BUILD.bazel index fed381d93..0117bc8b7 100644 --- a/BUILD.bazel +++ b/BUILD.bazel @@ -396,6 +396,7 @@ filegroup( "//internal/storage/externaldeps:repo_doc_files", "//internal/storage/fs:repo_doc_files", "//internal/storage/git:repo_doc_files", + "//internal/storage/historyoffequiv:repo_doc_files", "//internal/storage/issueops:repo_doc_files", "//internal/storage/journalscan:repo_doc_files", "//internal/storage/kvkeys:repo_doc_files", @@ -631,6 +632,7 @@ filegroup( "//internal/storage/externaldeps:repo_go_srcs", "//internal/storage/fs:repo_go_srcs", "//internal/storage/git:repo_go_srcs", + "//internal/storage/historyoffequiv:repo_go_srcs", "//internal/storage/issueops:repo_go_srcs", "//internal/storage/journalscan:repo_go_srcs", "//internal/storage/kvkeys:repo_go_srcs", @@ -851,6 +853,7 @@ filegroup( "//internal/storage/externaldeps:repo_go_test_srcs", "//internal/storage/fs:repo_go_test_srcs", "//internal/storage/git:repo_go_test_srcs", + "//internal/storage/historyoffequiv:repo_go_test_srcs", "//internal/storage/issueops:repo_go_test_srcs", "//internal/storage/journalscan:repo_go_test_srcs", "//internal/storage/kvkeys:repo_go_test_srcs", @@ -1080,6 +1083,7 @@ filegroup( "//internal/storage/externaldeps:repo_other_files", "//internal/storage/fs:repo_other_files", "//internal/storage/git:repo_other_files", + "//internal/storage/historyoffequiv:repo_other_files", "//internal/storage/issueops:repo_other_files", "//internal/storage/journalscan:repo_other_files", "//internal/storage/kvkeys:repo_other_files", diff --git a/internal/storage/BUILD.bazel b/internal/storage/BUILD.bazel index 73737e702..23a5a075f 100644 --- a/internal/storage/BUILD.bazel +++ b/internal/storage/BUILD.bazel @@ -170,6 +170,7 @@ filegroup( "//internal/storage/externaldeps:go_srcs", "//internal/storage/fs:go_srcs", "//internal/storage/git:go_srcs", + "//internal/storage/historyoffequiv:go_srcs", "//internal/storage/issueops:go_srcs", "//internal/storage/journalscan:go_srcs", "//internal/storage/kvkeys:go_srcs", diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 773a53d6b..0ab26a75e 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -244,6 +244,7 @@ go_test( "gm2g3g5r_leak_repro_test.go", "graph_counter_contract_test.go", "heartbeater_contract_test.go", + "history_off_differential_test.go", "history_presence_contract_test.go", "history_test.go", "identity_mismatch_message_test.go", @@ -414,6 +415,7 @@ go_test( "//internal/storage/depid", "//internal/storage/doltutil", "//internal/storage/domain", + "//internal/storage/historyoffequiv", "//internal/storage/issueops", "//internal/storage/journalscan", "//internal/storage/rowid", diff --git a/internal/storage/dolt/history_off_differential_test.go b/internal/storage/dolt/history_off_differential_test.go new file mode 100644 index 000000000..f4bcefae9 --- /dev/null +++ b/internal/storage/dolt/history_off_differential_test.go @@ -0,0 +1,35 @@ +package dolt + +import ( + "context" + "database/sql" + "testing" + + "github.com/steveyegge/beads/internal/storage/historyoffequiv" +) + +// TestVersionedHistoryOffMatchesPreStackGolden_Dolt is the server leg's +// versioned-history flag-off differential (internal/storage/historyoffequiv): +// on a store whose versioned history was never turned on, the scripted +// mutation set must leave every table as the code before the +// versioned-history stack left it, apart from that stack's new columns and +// table staying NULL and empty. +func TestVersionedHistoryOffMatchesPreStackGolden_Dolt(t *testing.T) { + store, cleanup := setupConcurrentTestStore(t) + t.Cleanup(cleanup) + if err := store.SetConfig(t.Context(), "issue_prefix", historyoffequiv.Prefix); err != nil { + t.Fatalf("set issue_prefix: %v", err) + } + historyoffequiv.Run(t, historyoffequiv.Leg{ + Name: "server", + Roles: store, + Wake: func(ctx context.Context) error { + store.WakeExpiredDefersAdvisory(ctx) + return nil + }, + Extra: historyoffequiv.DoltStorageSteps(store), + Query: historyoffequiv.SQLQuery(func(context.Context) (*sql.DB, func(), error) { + return store.DB(), func() {}, nil + }), + }) +} diff --git a/internal/storage/domain/db/BUILD.bazel b/internal/storage/domain/db/BUILD.bazel index 0b7df5abb..22f93bff7 100644 --- a/internal/storage/domain/db/BUILD.bazel +++ b/internal/storage/domain/db/BUILD.bazel @@ -67,6 +67,7 @@ go_test( "issue_cross_tier_test.go", "issue_descendants_test.go", "issue_force_close_policy_test.go", + "issue_insert_history_off_test.go", "issue_mergeops_test.go", "issue_scan_parity_test.go", "issue_search_counts_missing_table_test.go", @@ -114,6 +115,7 @@ go_test( "//internal/storage/journalscan", "//internal/storage/schema", "//internal/storage/sqlbuild", + "//internal/storage/sqlcount", "//internal/testutil", "//internal/types", "//issueops", diff --git a/internal/storage/domain/db/issue_insert_history_off_test.go b/internal/storage/domain/db/issue_insert_history_off_test.go new file mode 100644 index 000000000..355cf5598 --- /dev/null +++ b/internal/storage/domain/db/issue_insert_history_off_test.go @@ -0,0 +1,143 @@ +package db + +import ( + "context" + "database/sql" + "database/sql/driver" + "fmt" + "sync/atomic" + "testing" + "time" + + sqlmock "github.com/DATA-DOG/go-sqlmock" + + "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/storage/sqlcount" + "github.com/steveyegge/beads/internal/types" +) + +// TestInsertUpsertWithHistoryOffIssuesThePreStackStatements pins the +// statements Insert's upsert (CreateOnly unset: the quick, todo, gate and +// batch proxied servers' create) issues while versioned history is off. +// +// The participation_generation write fence (#6943) gave this path an +// existence probe (Exists) before the upsert, so the mint can tell a new row +// from one the upsert resolved to. With history off neither mint entry reads +// anything, so the probe is skipped and the create costs the round trips it +// did before the fence. Those are exactly the three statements below, the +// same three the code before the versioned-history stack (4a67d0d028) +// issues: the upsert, the created event's same-content scan, and its insert. +// The events journal is off too (the default); it adds its own statements +// whatever history does. +// +// Nothing else is scripted, and the mock is reached through sqlcount, which +// counts every statement Insert sends. So a probe, or any other read added +// ahead of the upsert or after it while history is off, fails the pin even +// when Insert swallows the error sqlmock answers it with: sqlmock's own +// checks see only an error the caller returns and expectations left unmet, +// and the count sees the rest. The history-on run of the same script is the +// control: there Insert reads (the probe, then the mint), so the pin has to +// refuse it, which shows that the script is exhaustive and that Insert reads +// the activation this transaction is scoped with. +func TestInsertUpsertWithHistoryOffIssuesThePreStackStatements(t *testing.T) { + for _, tc := range []struct { + name string + wisp bool + table, events string + }{ + {name: "issues table", table: "issues", events: "events"}, + {name: "wisps table", wisp: true, table: "wisps", events: "wisp_events"}, + } { + t.Run(tc.name, func(t *testing.T) { + if err := insertUpsertAgainstScript(t, tc.wisp, tc.table, tc.events, false); err != nil { + t.Fatalf("history off: %v", err) + } + }) + } + t.Run("history on is refused by the same script", func(t *testing.T) { + err := insertUpsertAgainstScript(t, false, "issues", "events", true) + if err == nil { + t.Fatal("Insert with history on passed the history-off script; " + + "the script no longer sees the reads history adds, so the history-off cases above prove nothing") + } + t.Logf("history on, as expected: %v", err) + }) +} + +// upsertStatements is how many statements the script below holds, and so +// how many the history-off upsert may send. +const upsertStatements = 3 + +// insertUpsertAgainstScript runs one upsert Insert into table on a mocked +// transaction scoped with versioned history as asked, against the script of +// the three statements the history-off upsert issues. It returns why the +// Insert broke the script, or nil if it sent exactly the scripted statements. +func insertUpsertAgainstScript(t *testing.T, wisp bool, table, events string, history bool) error { + t.Helper() + mock, tx, counts := countedMockTx(t) + // Every store and provider scopes each transaction with its own setting; + // a store whose switch is off scopes false. + t.Cleanup(issueops.ScopeVersionedHistoryTransaction(tx, history)) + + mock.ExpectExec(`^INSERT INTO ` + table + ` \( id, content_hash, .* ON DUPLICATE KEY UPDATE .* row_lock = VALUES\(row_lock\)$`). + WillReturnResult(sqlmock.NewResult(0, 1)) + mock.ExpectQuery(`^SELECT id FROM `+events+` WHERE issue_id = \? AND event_type = \? AND actor = \?`). + WithArgs("bd-1", string(types.EventCreated), "creator", sqlmock.AnyArg(), sqlmock.AnyArg(), sqlmock.AnyArg(), sqlmock.AnyArg()). + WillReturnRows(sqlmock.NewRows([]string{"id"})) + mock.ExpectExec(`^INSERT INTO ` + events + ` \(id, issue_id, event_type, actor, old_value, new_value, comment, created_at\)`). + WillReturnResult(sqlmock.NewResult(0, 1)) + + at := time.Date(2001, 2, 3, 4, 5, 6, 0, time.UTC) + if err := NewIssueSQLRepository(tx).Insert(t.Context(), &types.Issue{ + ID: "bd-1", Title: "upserted", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + CreatedAt: at, UpdatedAt: at, + }, "creator", domain.InsertIssueOpts{UseWispsTable: wisp}); err != nil { + return fmt.Errorf("the upsert failed against the script: %w", err) + } + if n := counts.Total(); n != upsertStatements { + return fmt.Errorf("the upsert sent %d statements, want exactly the %d scripted; "+ + "sqlmock refused each extra one with an error Insert dropped", n, upsertStatements) + } + if err := mock.ExpectationsWereMet(); err != nil { + return fmt.Errorf("the upsert did not send the scripted statements: %w", err) + } + return nil +} + +var sqlmockSeq atomic.Int64 + +// countedMockTx begins a transaction on a fresh sqlmock connection that is +// reached through sqlcount, so a pin can count every statement the code +// under test sends, including one whose error it swallowed. +func countedMockTx(t *testing.T) (sqlmock.Sqlmock, *sql.Tx, *sqlcount.Counts) { + t.Helper() + dsn := fmt.Sprintf("domain_db_counted_%d", sqlmockSeq.Add(1)) + mockDB, mock, err := sqlmock.NewWithDSN(dsn) + if err != nil { + t.Fatalf("sqlmock.NewWithDSN: %v", err) + } + counts := &sqlcount.Counts{} + db := sql.OpenDB(sqlcount.WrapConnector(sqlmockConnector{dsn: dsn, driver: mockDB.Driver()}, counts)) + t.Cleanup(func() { + _ = db.Close() + _ = mockDB.Close() + }) + mock.ExpectBegin() + tx, err := db.BeginTx(t.Context(), nil) + if err != nil { + t.Fatalf("BeginTx: %v", err) + } + t.Cleanup(func() { _ = tx.Rollback() }) + return mock, tx, counts +} + +// sqlmockConnector opens the sqlmock connection registered under dsn, for +// sql.OpenDB through a wrapping connector. +type sqlmockConnector struct { + dsn string + driver driver.Driver +} + +func (c sqlmockConnector) Connect(context.Context) (driver.Conn, error) { return c.driver.Open(c.dsn) } +func (c sqlmockConnector) Driver() driver.Driver { return c.driver } diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index 0391b56e9..5be835710 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -162,6 +162,7 @@ go_test( "graph_counter_contract_test.go", "graph_walk_plan_test.go", "heartbeater_contract_test.go", + "history_off_differential_test.go", "history_presence_contract_test.go", "history_test.go", "is_blocked_batch_test.go", @@ -267,6 +268,7 @@ go_test( "//internal/storage/doltutil", "//internal/storage/domain/db", "//internal/storage/externaldeps", + "//internal/storage/historyoffequiv", "//internal/storage/issueops", "//internal/storage/schema", "//internal/storage/sqlcount", diff --git a/internal/storage/embeddeddolt/history_off_differential_test.go b/internal/storage/embeddeddolt/history_off_differential_test.go new file mode 100644 index 000000000..505620270 --- /dev/null +++ b/internal/storage/embeddeddolt/history_off_differential_test.go @@ -0,0 +1,39 @@ +//go:build cgo + +package embeddeddolt_test + +import ( + "context" + "database/sql" + "testing" + + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/storage/historyoffequiv" +) + +// TestVersionedHistoryOffMatchesPreStackGolden_Embedded is the embedded +// leg's versioned-history flag-off differential (internal/storage/historyoffequiv): +// on a store whose versioned history was never turned on, the scripted +// mutation set must leave every table as the code before the +// versioned-history stack left it, apart from that stack's new columns and +// table staying NULL and empty. +func TestVersionedHistoryOffMatchesPreStackGolden_Embedded(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, historyoffequiv.Prefix) + historyoffequiv.Run(t, historyoffequiv.Leg{ + Name: "embedded", + Roles: te.store, + Wake: func(ctx context.Context) error { + te.store.WakeExpiredDefersAdvisory(ctx) + return nil + }, + Extra: historyoffequiv.DoltStorageSteps(te.store), + Query: historyoffequiv.SQLQuery(func(ctx context.Context) (*sql.DB, func(), error) { + db, cleanup, err := embeddeddolt.OpenSQL(ctx, te.dataDir, te.database, "main") + if err != nil { + return nil, nil, err + } + return db, func() { _ = cleanup() }, nil + }), + }) +} diff --git a/internal/storage/historyoffequiv/BUILD.bazel b/internal/storage/historyoffequiv/BUILD.bazel new file mode 100644 index 000000000..c9e4b9a8a --- /dev/null +++ b/internal/storage/historyoffequiv/BUILD.bazel @@ -0,0 +1,96 @@ +load("@rules_go//go:def.bzl", "go_library") + +go_library( + name = "historyoffequiv", + srcs = [ + "dump.go", + "historyoffequiv.go", + "scenario.go", + ], + embedsrcs = [ + "golden/embedded.json", + "golden/server.json", + "golden/uow.json", + ], + importpath = "github.com/steveyegge/beads/internal/storage/historyoffequiv", + visibility = ["//:__subpackages__"], + deps = [ + "//internal/storage", + "//internal/types", + "//issueops", + ], +) + +# --- begin go_srcs (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + ), + visibility = ["//:__subpackages__"], +) + +# --- end go_srcs --- + +# --- begin repo_files (managed by tools/bazel/go_srcs.py; run `make bazel-sync`) --- + +filegroup( + name = "repo_doc_files", + srcs = glob( + [ + "**/*.md", + "**/*.mdx", + ], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_srcs", + srcs = glob( + ["**/*.go"], + allow_empty = True, + exclude = [ + "**/*_test.go", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_go_test_srcs", + srcs = glob( + ["**/*_test.go"], + allow_empty = True, + exclude = ["**/node_modules/**"], + ), + visibility = ["//:__subpackages__"], +) + +filegroup( + name = "repo_other_files", + srcs = glob( + ["**"], + exclude = [ + "**/*.db", + "**/*.exe", + "**/*.go", + "**/*.md", + "**/*.mdx", + "**/*.out", + "**/*.prof", + "**/*.pyc", + "**/*.test", + "**/__pycache__/**", + "**/node_modules/**", + ], + ), + visibility = ["//:__subpackages__"], +) + +# --- end repo_files --- diff --git a/internal/storage/historyoffequiv/dump.go b/internal/storage/historyoffequiv/dump.go new file mode 100644 index 000000000..b5dbb36e9 --- /dev/null +++ b/internal/storage/historyoffequiv/dump.go @@ -0,0 +1,254 @@ +package historyoffequiv + +import ( + "context" + "encoding/json" + "fmt" + "regexp" + "strconv" + "strings" + "testing" + "time" +) + +// Every column of every base table is dumped: there is no list of tables to +// keep up to date, so a table a later change starts writing is compared the +// moment it exists. Each column is cast to text in SQL by its declared type +// (datetimes to a fixed format with microseconds, JSON to its text, binary to +// hex), and the text is then normalized only where a value is not a function +// of the scenario: +// +// - A datetime inside the run's window (the instants the scenario's writes +// stamp from the wall clock) reads "", and so does an RFC 3339 +// instant inside the window embedded in text (an event's snapshot of the +// row it changed). NULL stays NULL, and an instant the scenario supplied +// (fixedAt, decades outside the window) stays verbatim, so a write that +// starts or stops stamping a column, or that overwrites a supplied +// instant, still changes the digest. +// - A JSON value is re-serialized with sorted keys, so two engines' (or two +// versions') spacing and key order do not read as a difference. +// - The columns in volatileColumns hold values minted from randomness or +// derived from a wall-clock instant. Each reads "" under its +// rule, which keeps NULL, the empty string and the "none" value apart +// from a minted one. +// +// Rows are compared as a sorted multiset per table. + +type volatility int + +const ( + // volatileAlways masks every non-empty value. + volatileAlways volatility = iota + // volatileNonZero masks every value but "0", the column's "none". + volatileNonZero + // volatileUUID masks a UUID-shaped value and keeps any other verbatim. + volatileUUID +) + +// volatileColumns are the columns whose values differ between two runs of the +// same scenario on the same code, each with why. They were found by running +// the scenario twice on each leg and diffing the dumps. +var volatileColumns = map[string]map[string]volatility{ + // An event's id is a digest over its content and its wall-clock + // created_at (issueops.InsertDerivedEvent). + "events": {"id": volatileAlways}, + "wisp_events": {"id": volatileAlways}, + // A live comment's id is the same kind of digest; an imported comment + // keeps the id it arrived with, which stays verbatim. + "comments": {"id": volatileUUID}, + "wisp_comments": {"id": volatileUUID}, + // row_lock is a fresh random CAS token on every write; 0 means none. + "issues": {"row_lock": volatileNonZero}, + "wisps": {"row_lock": volatileNonZero}, + // Per-session coordination tokens (dependency-coordination/v1/..., + // issue-create/v1/...) are random; their keys are digests of the ids + // involved and stay verbatim. + "local_metadata": {"value": volatileAlways}, +} + +var ( + uuidRe = regexp.MustCompile(`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`) + rfc3339Re = regexp.MustCompile(`\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})`) +) + +func maskVolatile(table, column string, v string) (string, bool) { + rule, ok := volatileColumns[table][column] + if !ok || v == "" { + return v, false + } + switch rule { + case volatileNonZero: + if v == "0" { + return v, false + } + case volatileUUID: + if !uuidRe.MatchString(v) { + return v, false + } + } + return "", true +} + +// Two pseudo-tables carry the Dolt commit graph, compared like any table: the +// messages of the commits the scenario made, in order (a change to what a +// write commits, or whether it commits at all, shows up there), and the +// working set it left behind (a change to what a write stages shows up +// there). +const ( + doltLogTable = "@dolt_log" + doltStatusTable = "@dolt_status" +) + +func doltLogCount(t *testing.T, ctx context.Context, q Query) int { + t.Helper() + rows, err := q(ctx, "SELECT CAST(COUNT(*) AS CHAR) FROM dolt_log") + if err != nil { + t.Fatalf("count dolt_log: %v", err) + } + n, err := strconv.Atoi(*rows[0][0]) + if err != nil { + t.Fatalf("count dolt_log: %v", err) + } + return n +} + +func dumpAll(t *testing.T, ctx context.Context, q Query, windowStart, windowEnd time.Time, logBefore int) dump { + t.Helper() + tableRows, err := q(ctx, `SELECT table_name FROM information_schema.tables + WHERE table_schema = DATABASE() AND table_type = 'BASE TABLE' ORDER BY table_name`) + if err != nil { + t.Fatalf("list tables: %v", err) + } + colRows, err := q(ctx, `SELECT table_name, column_name, data_type FROM information_schema.columns + WHERE table_schema = DATABASE() ORDER BY table_name, ordinal_position`) + if err != nil { + t.Fatalf("list columns: %v", err) + } + type column struct{ name, dataType string } + columns := map[string][]column{} + for _, r := range colRows { + columns[*r[0]] = append(columns[*r[0]], column{name: *r[1], dataType: strings.ToLower(*r[2])}) + } + out := dump{Tables: map[string]tableDump{}} + for _, r := range tableRows { + table := *r[0] + if strings.HasPrefix(table, "dolt_") { + continue // Dolt's own system tables; the commit graph is read below. + } + cols := columns[table] + if len(cols) == 0 { + t.Fatalf("table %s has no columns in information_schema.columns", table) + } + exprs := make([]string, len(cols)) + names := make([]string, len(cols)) + for i, c := range cols { + names[i] = c.name + exprs[i] = textExpr(c.name, c.dataType) + } + rows, err := q(ctx, fmt.Sprintf("SELECT %s FROM `%s`", strings.Join(exprs, ", "), table)) + if err != nil { + t.Fatalf("dump %s: %v", table, err) + } + for _, row := range rows { + for i, c := range cols { + row[i] = normalize(t, table, c.name, c.dataType, row[i], windowStart, windowEnd) + } + } + sortRows(rows) + out.Tables[table] = tableDump{Columns: names, Rows: rows} + } + + // The scenario's commits, newest first as dolt_log lists them, numbered so + // the sorted comparison keeps their order. The committer identity comes + // from the environment (CI sets its own), so only the message is kept. + logRows, err := q(ctx, "SELECT message FROM dolt_log") + if err != nil { + t.Fatalf("read dolt_log: %v", err) + } + made := len(logRows) - logBefore + if made < 0 { + t.Fatalf("dolt_log shrank from %d to %d commits during the scenario", logBefore, len(logRows)) + } + var log [][]*string + for i, r := range logRows[:made] { + seq := fmt.Sprintf("%04d", made-i) + log = append(log, []*string{&seq, r[0]}) + } + sortRows(log) + out.Tables[doltLogTable] = tableDump{Columns: []string{"seq", "message"}, Rows: log} + + status, err := q(ctx, "SELECT table_name, CAST(staged AS CHAR), status FROM dolt_status") + if err != nil { + t.Fatalf("read dolt_status: %v", err) + } + sortRows(status) + out.Tables[doltStatusTable] = tableDump{Columns: []string{"table_name", "staged", "status"}, Rows: status} + return out +} + +func textExpr(name, dataType string) string { + c := "`" + name + "`" + switch dataType { + case "datetime", "timestamp": + return fmt.Sprintf("DATE_FORMAT(%s, '%%Y-%%m-%%d %%H:%%i:%%s.%%f')", c) + case "date": + return fmt.Sprintf("DATE_FORMAT(%s, '%%Y-%%m-%%d')", c) + case "blob", "tinyblob", "mediumblob", "longblob", "binary", "varbinary": + return fmt.Sprintf("HEX(%s)", c) + default: + return fmt.Sprintf("CAST(%s AS CHAR)", c) + } +} + +const datetimeLayout = "2006-01-02 15:04:05.000000" + +func normalize(t *testing.T, table, column, dataType string, v *string, windowStart, windowEnd time.Time) *string { + t.Helper() + if v == nil { + return nil + } + if masked, ok := maskVolatile(table, column, *v); ok { + return &masked + } + switch dataType { + case "datetime", "timestamp": + at, err := time.Parse(datetimeLayout, *v) + if err != nil { + t.Fatalf("%s.%s: datetime %q: %v", table, column, *v, err) + } + if !at.Before(windowStart) && !at.After(windowEnd) { + s := "" + return &s + } + case "json": + var parsed any + dec := json.NewDecoder(strings.NewReader(*v)) + dec.UseNumber() // keep integers past 2^53 exact + if err := dec.Decode(&parsed); err != nil { + return v + } + b, err := json.Marshal(parsed) + if err != nil { + t.Fatalf("%s.%s: re-marshal JSON: %v", table, column, err) + } + s := string(b) + return maskInstants(&s, windowStart, windowEnd) + } + return maskInstants(v, windowStart, windowEnd) +} + +// maskInstants replaces each RFC 3339 instant inside the run's window that v +// embeds with "". +func maskInstants(v *string, windowStart, windowEnd time.Time) *string { + if !rfc3339Re.MatchString(*v) { + return v + } + s := rfc3339Re.ReplaceAllStringFunc(*v, func(m string) string { + at, err := time.Parse(time.RFC3339Nano, m) + if err != nil || at.Before(windowStart) || at.After(windowEnd) { + return m + } + return "" + }) + return &s +} diff --git a/internal/storage/historyoffequiv/golden/embedded.json b/internal/storage/historyoffequiv/golden/embedded.json new file mode 100644 index 000000000..765849833 --- /dev/null +++ b/internal/storage/historyoffequiv/golden/embedded.json @@ -0,0 +1,539 @@ +{ + "recorded_from": "4a67d0d028", + "steps": [ + "create a", + "create b blocked by a", + "create epic", + "create epic.1 under epic", + "create wisp w1", + "create no-history nh", + "create c", + "create d", + "create e", + "update a fields labels metadata", + "update a no-op", + "update w1 to persistent", + "update c to ephemeral", + "update d parent epic", + "add dependencies", + "remove dependency b-\u003ea", + "comment on a and nh", + "claim heartbeat release b", + "claim e and reclaim no lease", + "metadata compare-and-set on a", + "close reopen close epic.1", + "close batch b and refused epic", + "force close blocked no-history nh", + "create batch", + "apply batch", + "defer f1 into the past and wake it", + "delete f2", + "create wisps x y z", + "update wisp x fields labels metadata", + "update wisp x no-op", + "claim and heartbeat refuse wisps", + "reclaim sweep scoped to in-progress wisp x", + "release wisp x", + "add and remove wisp dependencies", + "comment on wisp x", + "metadata compare-and-set on wisp x", + "close reopen wisp x", + "apply batch on the wisps plane", + "close batch wisps x and nh", + "defer wisp z into the past and wake it", + "delete wisp y", + "store create s1", + "store create issues s2 s3", + "store import upsert s1 and new s4", + "store update s2", + "store update s2 no-op", + "store labels", + "store dependencies", + "store comment", + "store claim s3", + "store close and reopen s2", + "store promote wisp sw", + "store transaction", + "store delete s3" + ], + "tables": { + "@dolt_log": { + "columns": [ + "seq", + "message" + ], + "rows": 28, + "sha256": "df2b8cf059f646557d21be3a0eca54f8561525f3d7e7ee714cfb65962d31012f" + }, + "@dolt_status": { + "columns": [ + "table_name", + "staged", + "status" + ], + "rows": 3, + "sha256": "ee3e710041f54216c2ae7e45bc7774cfcaa42aff3f4d96f5996fc4a7cc419f31" + }, + "bd_events_journal": { + "columns": [ + "seq", + "ts", + "op", + "issue_id", + "issue_json", + "dep_json", + "comment_json", + "actor" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "bd_events_seq": { + "columns": [ + "id", + "next_seq" + ], + "rows": 1, + "sha256": "13751b39fe07c3673f53b4f2c5ef7f2a28b26c0daed72eb7332c93c1fe520618" + }, + "child_counters": { + "columns": [ + "parent_id", + "last_child" + ], + "rows": 1, + "sha256": "753b8feea7bdc0fc74468faa597a32984edc501e9873e9f4ee1df7b59d56d0a6" + }, + "comments": { + "columns": [ + "id", + "issue_id", + "author", + "text", + "created_at" + ], + "rows": 2, + "sha256": "d5870471fe72b8e50fe898978f24a80a23e07750ab538d47353acfabfb0e9aed" + }, + "compaction_snapshots": { + "columns": [ + "id", + "issue_id", + "compaction_level", + "snapshot_json", + "created_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "config": { + "columns": [ + "key", + "value" + ], + "rows": 10, + "sha256": "671b33231eb47e4ad897051f4e8d86d3cca88685eeaa9ced537fe24c927fdea9" + }, + "custom_statuses": { + "columns": [ + "name", + "category" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "custom_types": { + "columns": [ + "name" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "dependencies": { + "columns": [ + "id", + "issue_id", + "type", + "created_at", + "created_by", + "metadata", + "thread_id", + "depends_on_issue_id", + "depends_on_wisp_id", + "depends_on_external" + ], + "rows": 6, + "sha256": "90f7dcd9895000df82d3b062d71a4c8ca81e7d5c9912f60d76515cb3eeee40d6" + }, + "events": { + "columns": [ + "issue_id", + "event_type", + "actor", + "old_value", + "new_value", + "comment", + "created_at", + "id" + ], + "rows": 51, + "sha256": "898f31694ce12dcd693f374634aea358b68239f6954fdcaae3b6342c1242f2bd" + }, + "federation_peers": { + "columns": [ + "name", + "remote_url", + "username", + "password_encrypted", + "sovereignty", + "last_sync", + "created_at", + "updated_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "ignored_schema_migrations": { + "columns": [ + "version", + "applied_at", + "content_hash" + ], + "rows": 27, + "sha256": "7164db2c129cbaafbd9c426d27e809d043cafc77e8aef1f68d1760905a685acb" + }, + "interactions": { + "columns": [ + "id", + "kind", + "created_at", + "actor", + "issue_id", + "model", + "prompt", + "response", + "error", + "tool_name", + "exit_code", + "parent_id", + "label", + "reason", + "extra" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issue_counter": { + "columns": [ + "prefix", + "last_id" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issue_snapshots": { + "columns": [ + "id", + "issue_id", + "snapshot_time", + "compaction_level", + "original_size", + "compressed_size", + "original_content", + "archived_events" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issue_versions": { + "columns": [ + "issue_id", + "revision", + "epoch", + "durable_state", + "change_actor", + "change_agent", + "change_message", + "change_at", + "removed_at", + "removed_reason", + "attribution_status" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issues": { + "columns": [ + "id", + "content_hash", + "title", + "description", + "design", + "acceptance_criteria", + "notes", + "status", + "priority", + "issue_type", + "assignee", + "estimated_minutes", + "created_at", + "created_by", + "owner", + "updated_at", + "closed_at", + "closed_by_session", + "external_ref", + "spec_id", + "compaction_level", + "compacted_at", + "compacted_at_commit", + "original_size", + "sender", + "ephemeral", + "wisp_type", + "pinned", + "is_template", + "mol_type", + "work_type", + "source_system", + "metadata", + "source_repo", + "close_reason", + "event_kind", + "actor", + "target", + "payload", + "await_type", + "await_id", + "timeout_ns", + "waiters", + "hook_bead", + "role_bead", + "agent_state", + "last_activity", + "role_type", + "rig", + "due_at", + "defer_until", + "no_history", + "started_at", + "is_blocked", + "row_lock", + "storage_class", + "current_revision" + ], + "rows": 15, + "sha256": "1f81df40f4c4d458f20cfb6535ca13468fa03fbd10276c44752977813a4b7e20" + }, + "labels": { + "columns": [ + "issue_id", + "label" + ], + "rows": 8, + "sha256": "0102293e23eec630b67542f93c5d8da6eef5ca627cff8662970e7cb211b013ef" + }, + "leases": { + "columns": [ + "issue_id", + "holder", + "granted_at", + "lease_expires_at", + "heartbeat_at", + "granted_node" + ], + "rows": 1, + "sha256": "1eca894450e8e801850ed2cbf1c746775ad50868acaaf17e033a5da1e4be10aa" + }, + "local_metadata": { + "columns": [ + "key", + "value" + ], + "rows": 31, + "sha256": "c1d1cbc41ca9968c5c7dd380609c28126c4024868dbade26575ce105b8931fb3" + }, + "metadata": { + "columns": [ + "key", + "value" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "provenance_events": { + "columns": [ + "id", + "issue_id", + "kind", + "actor", + "ref", + "ref_kind", + "payload", + "source", + "occurred_at", + "created_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "repo_mtimes": { + "columns": [ + "repo_path", + "jsonl_path", + "mtime_ns", + "last_checked" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "routes": { + "columns": [ + "prefix", + "path", + "created_at", + "updated_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "schema_migrations": { + "columns": [ + "version", + "content_hash" + ], + "rows": 69, + "sha256": "1d7a6ae809f330f8b4c3516af50ce7604c1db3cd4c1177af3e5c2f1991d74e14" + }, + "store_epoch": { + "columns": [ + "id", + "epoch", + "bumped_at", + "bumped_reason" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "wisp_child_counters": { + "columns": [ + "parent_id", + "last_child" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "wisp_comments": { + "columns": [ + "id", + "issue_id", + "author", + "text", + "created_at" + ], + "rows": 2, + "sha256": "3c5f1e3eac9ff32fb1158c4421909ddd49c9131edff756c4a0e184114da93350" + }, + "wisp_dependencies": { + "columns": [ + "id", + "issue_id", + "depends_on_issue_id", + "depends_on_wisp_id", + "depends_on_external", + "type", + "created_at", + "created_by", + "metadata", + "thread_id" + ], + "rows": 2, + "sha256": "80f14f79545a236513ea88f0e873f3dd477f7763aaa8cbfaa8f659a517c2a909" + }, + "wisp_events": { + "columns": [ + "id", + "issue_id", + "event_type", + "actor", + "old_value", + "new_value", + "comment", + "created_at" + ], + "rows": 25, + "sha256": "1ee25446153c51b39c4de60d7f03bb56172a84e72d28df2dd44ff79aa5fc3b23" + }, + "wisp_labels": { + "columns": [ + "issue_id", + "label" + ], + "rows": 4, + "sha256": "f9c2698a90ceb3bf0ee8c4034af2845b54d65cc7bdd7638965a4682ecd104c37" + }, + "wisps": { + "columns": [ + "id", + "content_hash", + "title", + "description", + "design", + "acceptance_criteria", + "notes", + "status", + "priority", + "issue_type", + "assignee", + "estimated_minutes", + "created_at", + "created_by", + "owner", + "updated_at", + "closed_at", + "closed_by_session", + "external_ref", + "spec_id", + "compaction_level", + "compacted_at", + "compacted_at_commit", + "original_size", + "sender", + "ephemeral", + "wisp_type", + "pinned", + "is_template", + "mol_type", + "work_type", + "source_system", + "metadata", + "source_repo", + "close_reason", + "event_kind", + "actor", + "target", + "payload", + "await_type", + "await_id", + "timeout_ns", + "waiters", + "hook_bead", + "role_bead", + "agent_state", + "last_activity", + "role_type", + "rig", + "due_at", + "defer_until", + "no_history", + "started_at", + "row_lock", + "storage_class", + "current_revision", + "is_blocked" + ], + "rows": 6, + "sha256": "f28b157579dedd4373778dc6050b2d1995e1fe30e545e5c70c6b6ccf08ed75bf" + } + } +} diff --git a/internal/storage/historyoffequiv/golden/server.json b/internal/storage/historyoffequiv/golden/server.json new file mode 100644 index 000000000..73c11eed2 --- /dev/null +++ b/internal/storage/historyoffequiv/golden/server.json @@ -0,0 +1,539 @@ +{ + "recorded_from": "4a67d0d028", + "steps": [ + "create a", + "create b blocked by a", + "create epic", + "create epic.1 under epic", + "create wisp w1", + "create no-history nh", + "create c", + "create d", + "create e", + "update a fields labels metadata", + "update a no-op", + "update w1 to persistent", + "update c to ephemeral", + "update d parent epic", + "add dependencies", + "remove dependency b-\u003ea", + "comment on a and nh", + "claim heartbeat release b", + "claim e and reclaim no lease", + "metadata compare-and-set on a", + "close reopen close epic.1", + "close batch b and refused epic", + "force close blocked no-history nh", + "create batch", + "apply batch", + "defer f1 into the past and wake it", + "delete f2", + "create wisps x y z", + "update wisp x fields labels metadata", + "update wisp x no-op", + "claim and heartbeat refuse wisps", + "reclaim sweep scoped to in-progress wisp x", + "release wisp x", + "add and remove wisp dependencies", + "comment on wisp x", + "metadata compare-and-set on wisp x", + "close reopen wisp x", + "apply batch on the wisps plane", + "close batch wisps x and nh", + "defer wisp z into the past and wake it", + "delete wisp y", + "store create s1", + "store create issues s2 s3", + "store import upsert s1 and new s4", + "store update s2", + "store update s2 no-op", + "store labels", + "store dependencies", + "store comment", + "store claim s3", + "store close and reopen s2", + "store promote wisp sw", + "store transaction", + "store delete s3" + ], + "tables": { + "@dolt_log": { + "columns": [ + "seq", + "message" + ], + "rows": 42, + "sha256": "c43f56fef8a71aa0775d5c79d962b3bc1565305b2c18c3a670941891e376811f" + }, + "@dolt_status": { + "columns": [ + "table_name", + "staged", + "status" + ], + "rows": 1, + "sha256": "8658f682f824193bcb00dd2c9744be1e86b90a249374d368903303dfb62ac195" + }, + "bd_events_journal": { + "columns": [ + "seq", + "ts", + "op", + "issue_id", + "issue_json", + "dep_json", + "comment_json", + "actor" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "bd_events_seq": { + "columns": [ + "id", + "next_seq" + ], + "rows": 1, + "sha256": "13751b39fe07c3673f53b4f2c5ef7f2a28b26c0daed72eb7332c93c1fe520618" + }, + "child_counters": { + "columns": [ + "parent_id", + "last_child" + ], + "rows": 1, + "sha256": "753b8feea7bdc0fc74468faa597a32984edc501e9873e9f4ee1df7b59d56d0a6" + }, + "comments": { + "columns": [ + "id", + "issue_id", + "author", + "text", + "created_at" + ], + "rows": 2, + "sha256": "d5870471fe72b8e50fe898978f24a80a23e07750ab538d47353acfabfb0e9aed" + }, + "compaction_snapshots": { + "columns": [ + "id", + "issue_id", + "compaction_level", + "snapshot_json", + "created_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "config": { + "columns": [ + "key", + "value" + ], + "rows": 10, + "sha256": "671b33231eb47e4ad897051f4e8d86d3cca88685eeaa9ced537fe24c927fdea9" + }, + "custom_statuses": { + "columns": [ + "name", + "category" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "custom_types": { + "columns": [ + "name" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "dependencies": { + "columns": [ + "id", + "issue_id", + "type", + "created_at", + "created_by", + "metadata", + "thread_id", + "depends_on_issue_id", + "depends_on_wisp_id", + "depends_on_external" + ], + "rows": 6, + "sha256": "90f7dcd9895000df82d3b062d71a4c8ca81e7d5c9912f60d76515cb3eeee40d6" + }, + "events": { + "columns": [ + "issue_id", + "event_type", + "actor", + "old_value", + "new_value", + "comment", + "created_at", + "id" + ], + "rows": 51, + "sha256": "898f31694ce12dcd693f374634aea358b68239f6954fdcaae3b6342c1242f2bd" + }, + "federation_peers": { + "columns": [ + "name", + "remote_url", + "username", + "password_encrypted", + "sovereignty", + "last_sync", + "created_at", + "updated_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "ignored_schema_migrations": { + "columns": [ + "version", + "applied_at", + "content_hash" + ], + "rows": 27, + "sha256": "7164db2c129cbaafbd9c426d27e809d043cafc77e8aef1f68d1760905a685acb" + }, + "interactions": { + "columns": [ + "id", + "kind", + "created_at", + "actor", + "issue_id", + "model", + "prompt", + "response", + "error", + "tool_name", + "exit_code", + "parent_id", + "label", + "reason", + "extra" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issue_counter": { + "columns": [ + "prefix", + "last_id" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issue_snapshots": { + "columns": [ + "id", + "issue_id", + "snapshot_time", + "compaction_level", + "original_size", + "compressed_size", + "original_content", + "archived_events" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issue_versions": { + "columns": [ + "issue_id", + "revision", + "epoch", + "durable_state", + "change_actor", + "change_agent", + "change_message", + "change_at", + "removed_at", + "removed_reason", + "attribution_status" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issues": { + "columns": [ + "id", + "content_hash", + "title", + "description", + "design", + "acceptance_criteria", + "notes", + "status", + "priority", + "issue_type", + "assignee", + "estimated_minutes", + "created_at", + "created_by", + "owner", + "updated_at", + "closed_at", + "closed_by_session", + "external_ref", + "spec_id", + "compaction_level", + "compacted_at", + "compacted_at_commit", + "original_size", + "sender", + "ephemeral", + "wisp_type", + "pinned", + "is_template", + "mol_type", + "work_type", + "source_system", + "metadata", + "source_repo", + "close_reason", + "event_kind", + "actor", + "target", + "payload", + "await_type", + "await_id", + "timeout_ns", + "waiters", + "hook_bead", + "role_bead", + "agent_state", + "last_activity", + "role_type", + "rig", + "due_at", + "defer_until", + "no_history", + "started_at", + "is_blocked", + "row_lock", + "storage_class", + "current_revision" + ], + "rows": 15, + "sha256": "1f81df40f4c4d458f20cfb6535ca13468fa03fbd10276c44752977813a4b7e20" + }, + "labels": { + "columns": [ + "issue_id", + "label" + ], + "rows": 8, + "sha256": "0102293e23eec630b67542f93c5d8da6eef5ca627cff8662970e7cb211b013ef" + }, + "leases": { + "columns": [ + "issue_id", + "holder", + "granted_at", + "lease_expires_at", + "heartbeat_at", + "granted_node" + ], + "rows": 1, + "sha256": "1eca894450e8e801850ed2cbf1c746775ad50868acaaf17e033a5da1e4be10aa" + }, + "local_metadata": { + "columns": [ + "key", + "value" + ], + "rows": 31, + "sha256": "c1d1cbc41ca9968c5c7dd380609c28126c4024868dbade26575ce105b8931fb3" + }, + "metadata": { + "columns": [ + "key", + "value" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "provenance_events": { + "columns": [ + "id", + "issue_id", + "kind", + "actor", + "ref", + "ref_kind", + "payload", + "source", + "occurred_at", + "created_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "repo_mtimes": { + "columns": [ + "repo_path", + "jsonl_path", + "mtime_ns", + "last_checked" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "routes": { + "columns": [ + "prefix", + "path", + "created_at", + "updated_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "schema_migrations": { + "columns": [ + "version", + "content_hash" + ], + "rows": 69, + "sha256": "1d7a6ae809f330f8b4c3516af50ce7604c1db3cd4c1177af3e5c2f1991d74e14" + }, + "store_epoch": { + "columns": [ + "id", + "epoch", + "bumped_at", + "bumped_reason" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "wisp_child_counters": { + "columns": [ + "parent_id", + "last_child" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "wisp_comments": { + "columns": [ + "id", + "issue_id", + "author", + "text", + "created_at" + ], + "rows": 2, + "sha256": "3c5f1e3eac9ff32fb1158c4421909ddd49c9131edff756c4a0e184114da93350" + }, + "wisp_dependencies": { + "columns": [ + "id", + "issue_id", + "depends_on_issue_id", + "depends_on_wisp_id", + "depends_on_external", + "type", + "created_at", + "created_by", + "metadata", + "thread_id" + ], + "rows": 2, + "sha256": "80f14f79545a236513ea88f0e873f3dd477f7763aaa8cbfaa8f659a517c2a909" + }, + "wisp_events": { + "columns": [ + "id", + "issue_id", + "event_type", + "actor", + "old_value", + "new_value", + "comment", + "created_at" + ], + "rows": 25, + "sha256": "1ee25446153c51b39c4de60d7f03bb56172a84e72d28df2dd44ff79aa5fc3b23" + }, + "wisp_labels": { + "columns": [ + "issue_id", + "label" + ], + "rows": 4, + "sha256": "f9c2698a90ceb3bf0ee8c4034af2845b54d65cc7bdd7638965a4682ecd104c37" + }, + "wisps": { + "columns": [ + "id", + "content_hash", + "title", + "description", + "design", + "acceptance_criteria", + "notes", + "status", + "priority", + "issue_type", + "assignee", + "estimated_minutes", + "created_at", + "created_by", + "owner", + "updated_at", + "closed_at", + "closed_by_session", + "external_ref", + "spec_id", + "compaction_level", + "compacted_at", + "compacted_at_commit", + "original_size", + "sender", + "ephemeral", + "wisp_type", + "pinned", + "is_template", + "mol_type", + "work_type", + "source_system", + "metadata", + "source_repo", + "close_reason", + "event_kind", + "actor", + "target", + "payload", + "await_type", + "await_id", + "timeout_ns", + "waiters", + "hook_bead", + "role_bead", + "agent_state", + "last_activity", + "role_type", + "rig", + "due_at", + "defer_until", + "no_history", + "started_at", + "row_lock", + "storage_class", + "current_revision", + "is_blocked" + ], + "rows": 6, + "sha256": "f28b157579dedd4373778dc6050b2d1995e1fe30e545e5c70c6b6ccf08ed75bf" + } + } +} diff --git a/internal/storage/historyoffequiv/golden/uow.json b/internal/storage/historyoffequiv/golden/uow.json new file mode 100644 index 000000000..91b2d1f84 --- /dev/null +++ b/internal/storage/historyoffequiv/golden/uow.json @@ -0,0 +1,531 @@ +{ + "recorded_from": "4a67d0d028", + "steps": [ + "create a", + "create b blocked by a", + "create epic", + "create epic.1 under epic", + "create wisp w1", + "create no-history nh", + "create c", + "create d", + "create e", + "update a fields labels metadata", + "update a no-op", + "update w1 to persistent", + "update c to ephemeral", + "update d parent epic", + "add dependencies", + "remove dependency b-\u003ea", + "comment on a and nh", + "claim heartbeat release b", + "claim e and reclaim no lease", + "metadata compare-and-set on a", + "close reopen close epic.1", + "close batch b and refused epic", + "force close blocked no-history nh", + "create batch", + "apply batch", + "defer f1 into the past and wake it", + "delete f2", + "create wisps x y z", + "update wisp x fields labels metadata", + "update wisp x no-op", + "claim and heartbeat refuse wisps", + "reclaim sweep scoped to in-progress wisp x", + "release wisp x", + "add and remove wisp dependencies", + "comment on wisp x", + "metadata compare-and-set on wisp x", + "close reopen wisp x", + "apply batch on the wisps plane", + "close batch wisps x and nh", + "defer wisp z into the past and wake it", + "delete wisp y", + "domain upsert new u1", + "domain upsert existing u1", + "domain upsert new wisp uw", + "domain upsert existing wisp uw", + "import upsert u1 and new u2" + ], + "tables": { + "@dolt_log": { + "columns": [ + "seq", + "message" + ], + "rows": 30, + "sha256": "5f4e90621c1c115e95223419c9c8c8533a342469e359beeac2ec5a4b4ff7ab1b" + }, + "@dolt_status": { + "columns": [ + "table_name", + "staged", + "status" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "bd_events_journal": { + "columns": [ + "seq", + "ts", + "op", + "issue_id", + "issue_json", + "dep_json", + "comment_json", + "actor" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "bd_events_seq": { + "columns": [ + "id", + "next_seq" + ], + "rows": 1, + "sha256": "13751b39fe07c3673f53b4f2c5ef7f2a28b26c0daed72eb7332c93c1fe520618" + }, + "child_counters": { + "columns": [ + "parent_id", + "last_child" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "comments": { + "columns": [ + "id", + "issue_id", + "author", + "text", + "created_at" + ], + "rows": 2, + "sha256": "5b88b60c5a975432baff9ecfc6d0230e3bdc12275471a2db90c6f36b80ab4561" + }, + "compaction_snapshots": { + "columns": [ + "id", + "issue_id", + "compaction_level", + "snapshot_json", + "created_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "config": { + "columns": [ + "key", + "value" + ], + "rows": 10, + "sha256": "671b33231eb47e4ad897051f4e8d86d3cca88685eeaa9ced537fe24c927fdea9" + }, + "custom_statuses": { + "columns": [ + "name", + "category" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "custom_types": { + "columns": [ + "name" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "dependencies": { + "columns": [ + "id", + "issue_id", + "type", + "created_at", + "created_by", + "metadata", + "thread_id", + "depends_on_issue_id", + "depends_on_wisp_id", + "depends_on_external" + ], + "rows": 6, + "sha256": "10891c2f5540db5ffec57100e87d971863a67023d963cda658c43c56ce368a69" + }, + "events": { + "columns": [ + "issue_id", + "event_type", + "actor", + "old_value", + "new_value", + "comment", + "created_at", + "id" + ], + "rows": 39, + "sha256": "372c6143415ba862a0d951695e2c942670fc0a24a3b828d6755df97dbb34ade1" + }, + "federation_peers": { + "columns": [ + "name", + "remote_url", + "username", + "password_encrypted", + "sovereignty", + "last_sync", + "created_at", + "updated_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "ignored_schema_migrations": { + "columns": [ + "version", + "applied_at", + "content_hash" + ], + "rows": 27, + "sha256": "7164db2c129cbaafbd9c426d27e809d043cafc77e8aef1f68d1760905a685acb" + }, + "interactions": { + "columns": [ + "id", + "kind", + "created_at", + "actor", + "issue_id", + "model", + "prompt", + "response", + "error", + "tool_name", + "exit_code", + "parent_id", + "label", + "reason", + "extra" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issue_counter": { + "columns": [ + "prefix", + "last_id" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issue_snapshots": { + "columns": [ + "id", + "issue_id", + "snapshot_time", + "compaction_level", + "original_size", + "compressed_size", + "original_content", + "archived_events" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issue_versions": { + "columns": [ + "issue_id", + "revision", + "epoch", + "durable_state", + "change_actor", + "change_agent", + "change_message", + "change_at", + "removed_at", + "removed_reason", + "attribution_status" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "issues": { + "columns": [ + "id", + "content_hash", + "title", + "description", + "design", + "acceptance_criteria", + "notes", + "status", + "priority", + "issue_type", + "assignee", + "estimated_minutes", + "created_at", + "created_by", + "owner", + "updated_at", + "closed_at", + "closed_by_session", + "external_ref", + "spec_id", + "compaction_level", + "compacted_at", + "compacted_at_commit", + "original_size", + "sender", + "ephemeral", + "wisp_type", + "pinned", + "is_template", + "mol_type", + "work_type", + "source_system", + "metadata", + "source_repo", + "close_reason", + "event_kind", + "actor", + "target", + "payload", + "await_type", + "await_id", + "timeout_ns", + "waiters", + "hook_bead", + "role_bead", + "agent_state", + "last_activity", + "role_type", + "rig", + "due_at", + "defer_until", + "no_history", + "started_at", + "is_blocked", + "row_lock", + "storage_class", + "current_revision" + ], + "rows": 12, + "sha256": "92e9eb8264175b0a6723941b3e946cd796afe4539cf9927798ce62800cab67ef" + }, + "labels": { + "columns": [ + "issue_id", + "label" + ], + "rows": 6, + "sha256": "d8cb82847c5ed046c6f264c12a51b1cc128f314fdc501aad3be0bd38cd3ec341" + }, + "leases": { + "columns": [ + "issue_id", + "holder", + "granted_at", + "lease_expires_at", + "heartbeat_at", + "granted_node" + ], + "rows": 1, + "sha256": "1eca894450e8e801850ed2cbf1c746775ad50868acaaf17e033a5da1e4be10aa" + }, + "local_metadata": { + "columns": [ + "key", + "value" + ], + "rows": 31, + "sha256": "c1d1cbc41ca9968c5c7dd380609c28126c4024868dbade26575ce105b8931fb3" + }, + "metadata": { + "columns": [ + "key", + "value" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "provenance_events": { + "columns": [ + "id", + "issue_id", + "kind", + "actor", + "ref", + "ref_kind", + "payload", + "source", + "occurred_at", + "created_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "repo_mtimes": { + "columns": [ + "repo_path", + "jsonl_path", + "mtime_ns", + "last_checked" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "routes": { + "columns": [ + "prefix", + "path", + "created_at", + "updated_at" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "schema_migrations": { + "columns": [ + "version", + "content_hash" + ], + "rows": 69, + "sha256": "1d7a6ae809f330f8b4c3516af50ce7604c1db3cd4c1177af3e5c2f1991d74e14" + }, + "store_epoch": { + "columns": [ + "id", + "epoch", + "bumped_at", + "bumped_reason" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "wisp_child_counters": { + "columns": [ + "parent_id", + "last_child" + ], + "rows": 0, + "sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" + }, + "wisp_comments": { + "columns": [ + "id", + "issue_id", + "author", + "text", + "created_at" + ], + "rows": 2, + "sha256": "3c5f1e3eac9ff32fb1158c4421909ddd49c9131edff756c4a0e184114da93350" + }, + "wisp_dependencies": { + "columns": [ + "id", + "issue_id", + "depends_on_issue_id", + "depends_on_wisp_id", + "depends_on_external", + "type", + "created_at", + "created_by", + "metadata", + "thread_id" + ], + "rows": 2, + "sha256": "80f14f79545a236513ea88f0e873f3dd477f7763aaa8cbfaa8f659a517c2a909" + }, + "wisp_events": { + "columns": [ + "id", + "issue_id", + "event_type", + "actor", + "old_value", + "new_value", + "comment", + "created_at" + ], + "rows": 27, + "sha256": "34d262640e93ffdc948862667d4bc5f01a9f655bc22d8e11abdfc777731a5771" + }, + "wisp_labels": { + "columns": [ + "issue_id", + "label" + ], + "rows": 4, + "sha256": "f9c2698a90ceb3bf0ee8c4034af2845b54d65cc7bdd7638965a4682ecd104c37" + }, + "wisps": { + "columns": [ + "id", + "content_hash", + "title", + "description", + "design", + "acceptance_criteria", + "notes", + "status", + "priority", + "issue_type", + "assignee", + "estimated_minutes", + "created_at", + "created_by", + "owner", + "updated_at", + "closed_at", + "closed_by_session", + "external_ref", + "spec_id", + "compaction_level", + "compacted_at", + "compacted_at_commit", + "original_size", + "sender", + "ephemeral", + "wisp_type", + "pinned", + "is_template", + "mol_type", + "work_type", + "source_system", + "metadata", + "source_repo", + "close_reason", + "event_kind", + "actor", + "target", + "payload", + "await_type", + "await_id", + "timeout_ns", + "waiters", + "hook_bead", + "role_bead", + "agent_state", + "last_activity", + "role_type", + "rig", + "due_at", + "defer_until", + "no_history", + "started_at", + "row_lock", + "storage_class", + "current_revision", + "is_blocked" + ], + "rows": 7, + "sha256": "f6885bac56aee7d91b48fdf56c657128ff0b530fa186fabfb52bbf271a3bc09b" + } + } +} diff --git a/internal/storage/historyoffequiv/historyoffequiv.go b/internal/storage/historyoffequiv/historyoffequiv.go new file mode 100644 index 000000000..afcdfc7f2 --- /dev/null +++ b/internal/storage/historyoffequiv/historyoffequiv.go @@ -0,0 +1,528 @@ +// Package historyoffequiv is the versioned-history flag-off differential +// (gastownhall/beads#6132, #6135: "flag-off must remain byte-identical to +// today, proven by differential test"). +// +// Each storage leg (embedded Dolt, Dolt server, unit of work) runs one +// scripted mutation set through its own front doors on a store whose +// versioned history was never turned on, the default. Run then dumps every +// column of every table, digests each table, and compares the digests with +// the leg's golden. The goldens were RECORDED FROM THE CODE BEFORE THE +// VERSIONED-HISTORY STACK (origin/main 4a67d0d028, the parent of #6943). The +// schema added since, listed in schemaAdditions under the rule spelled out +// there, is the only difference the comparison tolerates; any other +// difference in any table is a flag-off behavior change and fails the test. +// +// # Maintaining the goldens +// +// Run records instead of checking when GoldenDirEnv is set. Recording and +// checking prepare a run the same way: each schemaAdditions entry the +// database has is checked under the rule and set aside before any table is +// digested. A golden recorded from any tree whose flag-off storage behavior +// matches the committed goldens is therefore byte-identical to them apart +// from recorded_from, and recording from the current tree is also how a +// failure is read: diff its full dump against one recorded from the base. +// +// - A migration whose schema flag-off leaves untouched (nullable columns +// that stay NULL, tables that stay empty) appends an entry to +// schemaAdditions in its own PR: those columns and tables, and its +// bookkeeping rows. The goldens do not change. +// - A deliberate change to what flag-off storage writes, including a +// migration that writes rows or values while history is off, +// re-records the affected legs from its PR's tree. The PR shows the +// golden diff, and that the full dumps differ from a recording of its +// base only where the change means them to. +// - A change to the scenario re-records every leg. Record it from +// 4a67d0d028 while the harness still builds there (copy this package +// and the three legs' test files into a worktree of that commit), so +// the new steps are anchored to the pre-stack code too; once it no +// longer builds there, from the PR's base. +// - A Dolt upgrade that changes only how the engine renders a stored +// value (a JSON or datetime cast) re-records the legs that run the new +// engine from the PR's tree, and the PR shows that the full dumps +// differ from a recording of its base, made with the old engine, only +// in that rendering. The embedded leg runs the engine go.mod pins; the +// server and unit-of-work legs run the dolt binary the test finds (CI: +// DOLT_VERSION in tools/bazel/dolt.bzl). +// +// recorded_from names the tree a golden was recorded from: a commit, or a +// commit and the PR on top of it. +package historyoffequiv + +import ( + "context" + "crypto/sha256" + "database/sql" + "embed" + "encoding/hex" + "encoding/json" + "fmt" + "os" + "path/filepath" + "slices" + "sort" + "strconv" + "strings" + "testing" + "time" +) + +// Prefix is the issue prefix every scenario id carries. A leg configures its +// store with it before Run. +const Prefix = "ho" + +// Query runs one read-only statement against the leg's database (the one its +// store writes) and returns every row, each value as text and nil for SQL +// NULL. Run casts every column to text in the statement itself, so a leg's +// driver never decides how a value is formatted. +type Query func(ctx context.Context, query string) ([][]*string, error) + +// Step is one mutation of the scenario, named for failure messages and for +// the golden's step list. +type Step struct { + Name string + Run func(ctx context.Context) error +} + +// Leg is what a backend hands Run. Its store is constructed the way the leg's +// tests construct one and is never handed SetVersionedHistoryEnabled, so +// "history off" is the default a store starts with rather than a switch +// turned off (FR-7). +type Leg struct { + // Name selects the golden: golden/.json. + Name string + // Roles is the leg's role-accessor surface. Every leg drives the same + // SharedSteps through it. + Roles Roles + // Wake runs the leg's lazy defer-wake sweep, the one write no role + // exposes. + Wake func(ctx context.Context) error + // Extra are the leg's own steps, run after the shared ones: the store + // legs' storage.DoltStorage methods (DoltStorageSteps), the unit-of-work + // leg's domain upsert. + Extra []Step + // Query reads the leg's database. + Query Query +} + +// GoldenDirEnv switches Run from checking to recording: set to a directory, +// Run writes the leg's golden there instead of comparing, with a full dump +// of the rows it digested beside it (.full.json.txt) for diffing two +// recordings by hand. RecordedFromEnv names the tree being recorded from; +// Run copies it into the golden. When to re-record is under "Maintaining the +// goldens" in the package doc. +const ( + GoldenDirEnv = "HISTORYOFFEQUIV_GOLDEN_DIR" + RecordedFromEnv = "HISTORYOFFEQUIV_RECORDED_FROM" +) + +// schemaAddition is schema added since the pre-stack code the goldens are +// anchored to that flag-off storage leaves untouched, grouped by the change +// that added it. +type schemaAddition struct { + // By names the change, for failure messages. + By string + // Columns are its nullable columns, by table. + Columns map[string][]string + // Tables are its tables. + Tables []string + // Migrations are the migration runner's bookkeeping rows for its own + // migrations, by bookkeeping table (each keyed by its version column). + Migrations map[string][]string +} + +// schemaAdditions is THE ONLY DIFFERENCE from the goldens this test +// tolerates. The rule, applied to every run before any table is digested, +// whether the run records a golden or checks one: +// +// 1. A listed column, where the database has it, must be NULL in every +// row; it is then left out of its table's dump, so the rest of every row +// still has to match the golden byte for byte. A listed column holding +// any value fails: with versioned history off nothing may write it. +// 2. A listed table, where the database has it, must be empty; it is then +// left out. +// 3. A listed bookkeeping row, where the database has it, must be there +// once; it is then left out. Every other row of a bookkeeping table is +// compared like any table's. +// +// After that, checking demands the golden's tables and columns exactly, and +// every table's digest over the golden's columns must equal the golden's. +// Nothing else is excused: not a value, not a row, not a row count. A later +// migration whose schema flag-off leaves untouched appends its own entry +// here (see "Maintaining the goldens" in the package doc). +var schemaAdditions = []schemaAddition{{ + By: "the versioned-history stack (#6943, #6661, #6664)", + Columns: map[string][]string{ + // Migration 0070: the participation_generation write fence (#6943). + // A create with history off leaves it NULL, indistinguishable from + // a legacy row (design §16.2b, FR-7). + "issues": {"participation_generation"}, + // Migration 0070 mirrors it, inert, on the wisps table (FR-8). + "wisps": {"participation_generation"}, + // Migration 0071 (#6661). + "issue_versions": {"removed_restriction"}, + }, + // Migration 0072 (#6664). + Tables: []string{"epoch_minted_addresses"}, + // 0070-0072 on the main series, and ignored/0028 (the wisps half of + // 0070) on the dolt_ignored series. + Migrations: map[string][]string{ + "schema_migrations": {"70", "71", "72"}, + "ignored_schema_migrations": {"28"}, + }, +}} + +// runWindow widens the span of the run on each side to bound the wall-clock +// instants the scenario's writes stamp. It is generous so that no time zone a +// DATETIME may be read back in, and no rounding of a sub-second instant, can +// put such an instant outside it. The scenario's own timestamps are decades +// away (fixedAt), so a column whose value moved from a scenario-supplied +// instant to "now" (or back) still shows up in the digest. +const runWindow = 36 * time.Hour + +// Run applies the shared scenario and the leg's extra steps, then checks +// every table against the leg's golden (or records it). +func Run(t *testing.T, leg Leg) { + t.Helper() + if leg.Name == "" || leg.Roles == nil || leg.Wake == nil || leg.Query == nil { + t.Fatalf("historyoffequiv.Run: leg %q needs a Name, Roles, Wake and Query", leg.Name) + } + ctx := context.Background() + before := doltLogCount(t, ctx, leg.Query) + start := time.Now().UTC() + steps := append(SharedSteps(leg.Roles, leg.Wake), leg.Extra...) + var names []string + for _, step := range steps { + if err := step.Run(ctx); err != nil { + t.Fatalf("step %q: %v", step.Name, err) + } + names = append(names, step.Name) + } + end := time.Now().UTC() + got := setAsideAdditions(t, dumpAll(t, ctx, leg.Query, start.Add(-runWindow), end.Add(runWindow), before)) + got.Steps = names + + if dir := os.Getenv(GoldenDirEnv); dir != "" { + if t.Failed() { + t.Fatalf("not recording golden/%s.json: the run breaks the schemaAdditions rule (above)", leg.Name) + } + writeGolden(t, dir, leg.Name, got) + return + } + checkGolden(t, leg.Name, got) +} + +// dump is every table's normalized rows. +type dump struct { + Steps []string + Tables map[string]tableDump +} + +type tableDump struct { + Columns []string + Rows [][]*string +} + +type golden struct { + RecordedFrom string `json:"recorded_from"` + Steps []string `json:"steps"` + Tables map[string]goldenTable `json:"tables"` +} + +type goldenTable struct { + Columns []string `json:"columns"` + Rows int `json:"rows"` + SHA256 string `json:"sha256"` +} + +//go:embed golden/*.json +var goldenFS embed.FS + +// setAsideAdditions applies the schemaAdditions rule to a run's dump: it +// checks every listed column, table and bookkeeping row the database has, +// reporting each breach, and returns the dump without them. Recording and +// checking both digest only what it returns. +func setAsideAdditions(t *testing.T, d dump) dump { + t.Helper() + columns := map[string]map[string]string{} // table -> column -> By + tables := map[string]string{} + migrations := map[string]map[string]string{} // table -> version -> By + for _, a := range schemaAdditions { + for table, cols := range a.Columns { + for _, c := range cols { + if columns[table] == nil { + columns[table] = map[string]string{} + } + columns[table][c] = a.By + } + } + for _, table := range a.Tables { + tables[table] = a.By + } + for table, versions := range a.Migrations { + for _, v := range versions { + if migrations[table] == nil { + migrations[table] = map[string]string{} + } + migrations[table][v] = a.By + } + } + } + + out := dump{Steps: d.Steps, Tables: map[string]tableDump{}} + for _, table := range sortedKeys(d.Tables) { + td := d.Tables[table] + if by, ok := tables[table]; ok { + if len(td.Rows) != 0 { + t.Errorf("table %s, added by %s, has %d rows with versioned history off, want none:\n%s", + table, by, len(td.Rows), formatRows(td.Columns, td.Rows)) + } + continue + } + var keep []int + for i, c := range td.Columns { + by, listed := columns[table][c] + if !listed { + keep = append(keep, i) + continue + } + for _, row := range td.Rows { + if row[i] != nil { + t.Errorf("%s.%s, added by %s, holds %q with versioned history off, want NULL in every row; row: %s", + table, c, by, *row[i], formatRow(row)) + } + } + } + kept := tableDump{Columns: make([]string, len(keep))} + for j, i := range keep { + kept.Columns[j] = td.Columns[i] + } + version := slices.Index(kept.Columns, "version") + if migrations[table] != nil && version < 0 { + t.Errorf("bookkeeping table %s has no version column: %v", table, td.Columns) + } + seen := map[string]int{} + for _, row := range td.Rows { + projected := make([]*string, len(keep)) + for j, i := range keep { + projected[j] = row[i] + } + if version >= 0 && projected[version] != nil { + if _, listed := migrations[table][*projected[version]]; listed { + seen[*projected[version]]++ + continue + } + } + kept.Rows = append(kept.Rows, projected) + } + for _, v := range sortedKeys(seen) { + if seen[v] != 1 { + t.Errorf("%s has %d rows for migration %s, added by %s, want at most 1", table, seen[v], v, migrations[table][v]) + } + } + sortRows(kept.Rows) + out.Tables[table] = kept + } + return out +} + +// checkGolden compares a set-aside dump with golden/.json. +func checkGolden(t *testing.T, leg string, got dump) { + t.Helper() + b, err := goldenFS.ReadFile("golden/" + leg + ".json") + if err != nil { + t.Fatalf("no golden for leg %q: record one with %s (see the package doc): %v", leg, GoldenDirEnv, err) + } + var want golden + if err := json.Unmarshal(b, &want); err != nil { + t.Fatalf("golden/%s.json: %v", leg, err) + } + from := fmt.Sprintf("golden/%s.json (recorded from %s)", leg, want.RecordedFrom) + if !slices.Equal(got.Steps, want.Steps) { + t.Fatalf("the scenario's steps differ from those %s ran, so its digests no longer describe this scenario; "+ + "re-record it (\"Maintaining the goldens\" in the package doc):\ngot: %q\nwant: %q", + from, got.Steps, want.Steps) + } + unexplained := "; a migration whose schema flag-off leaves untouched lists it in schemaAdditions, " + + "and a change to what flag-off writes re-records the goldens (\"Maintaining the goldens\" in the package doc)" + for _, table := range sortedKeys(got.Tables) { + if _, ok := want.Tables[table]; !ok { + t.Errorf("table %s is not in %s and not in schemaAdditions%s:\n%s", + table, from, unexplained, formatRows(got.Tables[table].Columns, got.Tables[table].Rows)) + } + } + for _, table := range sortedKeys(want.Tables) { + wt := want.Tables[table] + gt, ok := got.Tables[table] + if !ok { + t.Errorf("table %s is in %s but not in this database", table, from) + continue + } + index := map[string]int{} + for i, c := range gt.Columns { + index[c] = i + } + ok = true + for _, c := range wt.Columns { + if _, present := index[c]; !present { + t.Errorf("%s.%s is in %s but not in this database", table, c, from) + ok = false + } + } + for _, c := range gt.Columns { + if !slices.Contains(wt.Columns, c) { + t.Errorf("%s.%s is not in %s and not in schemaAdditions%s", table, c, from, unexplained) + ok = false + } + } + if !ok { + continue + } + projected := make([][]*string, 0, len(gt.Rows)) + for _, row := range gt.Rows { + p := make([]*string, len(wt.Columns)) + for j, c := range wt.Columns { + p[j] = row[index[c]] + } + projected = append(projected, p) + } + sortRows(projected) + if d := digest(projected); d != wt.SHA256 || len(projected) != wt.Rows { + t.Errorf("%s differs from %s: got %d rows %s, want %d rows %s; rows now (columns %v):\n%s", + table, from, len(projected), d, wt.Rows, wt.SHA256, wt.Columns, formatRows(wt.Columns, projected)) + } + } +} + +// writeGolden records a set-aside dump as golden/.json in dir. +func writeGolden(t *testing.T, dir, leg string, d dump) { + t.Helper() + g := golden{RecordedFrom: os.Getenv(RecordedFromEnv), Steps: d.Steps, Tables: map[string]goldenTable{}} + if g.RecordedFrom == "" { + t.Fatalf("%s is set but %s is not: name the tree this golden is recorded from", GoldenDirEnv, RecordedFromEnv) + } + full := map[string][]string{} + for table, td := range d.Tables { + g.Tables[table] = goldenTable{Columns: td.Columns, Rows: len(td.Rows), SHA256: digest(td.Rows)} + full[table] = append(full[table], strings.Join(td.Columns, " | ")) + for _, row := range td.Rows { + full[table] = append(full[table], formatRow(row)) + } + } + b, err := json.MarshalIndent(g, "", " ") + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(dir, leg+".json"), append(b, '\n'), 0o600); err != nil { + t.Fatal(err) + } + fb, _ := json.MarshalIndent(full, "", " ") + _ = os.WriteFile(filepath.Join(dir, leg+".full.json.txt"), fb, 0o600) +} + +func digest(rows [][]*string) string { + h := sha256.New() + for _, row := range rows { + h.Write([]byte(formatRow(row))) + h.Write([]byte{'\n'}) + } + return hex.EncodeToString(h.Sum(nil)) +} + +func formatRow(row []*string) string { + parts := make([]string, len(row)) + for i, v := range row { + if v == nil { + parts[i] = "NULL" + } else { + parts[i] = strconv.Quote(*v) + } + } + return strings.Join(parts, " ") +} + +func formatRows(columns []string, rows [][]*string) string { + lines := []string{" " + strings.Join(columns, " | ")} + for _, row := range rows { + lines = append(lines, " "+formatRow(row)) + } + return strings.Join(lines, "\n") +} + +func sortRows(rows [][]*string) { + sort.SliceStable(rows, func(i, j int) bool { return formatRow(rows[i]) < formatRow(rows[j]) }) +} + +func sortedKeys[V any](m map[string]V) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} + +// SQLQuery adapts a leg that reads through database/sql. open returns a +// handle on the leg's database and a function that releases it. +func SQLQuery(open func(ctx context.Context) (*sql.DB, func(), error)) Query { + return func(ctx context.Context, query string) ([][]*string, error) { + db, release, err := open(ctx) + if err != nil { + return nil, err + } + defer release() + rows, err := db.QueryContext(ctx, query) + if err != nil { + return nil, err + } + defer rows.Close() + cols, err := rows.Columns() + if err != nil { + return nil, err + } + var out [][]*string + for rows.Next() { + raw := make([]sql.NullString, len(cols)) + dest := make([]any, len(cols)) + for i := range raw { + dest[i] = &raw[i] + } + if err := rows.Scan(dest...); err != nil { + return nil, err + } + row := make([]*string, len(cols)) + for i, v := range raw { + if v.Valid { + s := v.String + row[i] = &s + } + } + out = append(out, row) + } + return out, rows.Err() + } +} + +// AnyRows converts rows of driver values (a raw-SQL use case's result) to +// Query's shape. +func AnyRows(rows [][]any) [][]*string { + out := make([][]*string, 0, len(rows)) + for _, r := range rows { + row := make([]*string, len(r)) + for i, v := range r { + switch v := v.(type) { + case nil: + case []byte: + s := string(v) + row[i] = &s + case string: + s := v + row[i] = &s + default: + s := fmt.Sprint(v) + row[i] = &s + } + } + out = append(out, row) + } + return out +} diff --git a/internal/storage/historyoffequiv/scenario.go b/internal/storage/historyoffequiv/scenario.go new file mode 100644 index 000000000..90dadd04b --- /dev/null +++ b/internal/storage/historyoffequiv/scenario.go @@ -0,0 +1,539 @@ +package historyoffequiv + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "time" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// Roles is the role-accessor surface all three legs offer. The shared +// scenario reaches storage only through it (and Leg.Wake), so each leg runs +// the scenario through its own implementations: the embedded store's and the +// server store's issueops-backed roles, and the unit-of-work provider's +// domain-backed ones. +type Roles interface { + IssueLifecycle() (publicops.Lifecycle, error) + BatchCreator() (publicops.BatchCreator, error) + BatchApplier() (publicops.BatchApplier, error) + BatchCloser() (publicops.BatchCloser, error) + DependencyEditor() (publicops.DependencyEditor, error) + Commenter() (publicops.Commenter, error) + IssueClaimer() (publicops.Claimer, error) + Heartbeater() (publicops.Heartbeater, error) + Releaser() (publicops.Releaser, error) + LeaseReclaimer() (publicops.LeaseReclaimer, error) + MetadataCAS() (publicops.MetadataCAS, error) + Deleter() (publicops.Deleter, error) +} + +// fixedAt is every timestamp the scenario supplies. It is decades outside +// any run's window, so a column that keeps it reads it verbatim in the dump +// and a column that loses it to the wall clock reads "". +var fixedAt = time.Date(2001, 2, 3, 4, 5, 6, 0, time.UTC) + +func id(suffix string) string { return Prefix + "-" + suffix } + +func newIssue(suffix, title string, issueType types.IssueType) *types.Issue { + return &types.Issue{ + ID: id(suffix), Title: title, Description: "about " + suffix, Status: types.StatusOpen, + Priority: 2, IssueType: issueType, CreatedAt: fixedAt, UpdatedAt: fixedAt, + } +} + +func set[T any](v T) publicops.Field[T] { return publicops.Field[T]{Set: true, Value: v} } + +func raw(s string) *json.RawMessage { + m := json.RawMessage(s) + return &m +} + +// SharedSteps is the scenario every leg runs: each write role on the issues +// plane, then each again on the wisps plane (an ephemeral wisp, and the +// no-history wisp nh along the way), including the paths the +// versioned-history stack rewired (single and batch creates, the persistence +// moves between planes, the defer wake), refusals that must write nothing (a +// batch close item close policy refuses), and no-op updates that must leave +// every row alone. On the wisps plane the lease roles refuse (Claim, +// Heartbeat) or find nothing (Reclaim), as their contracts say, and those +// steps check the refusal. Neither reclaim sweep finds an expired lease: a +// role cannot age one, so they cover the sweep's reads and commit, not a +// revert. +func SharedSteps(r Roles, wake func(context.Context) error) []Step { + create := func(req publicops.CreateRequest) func(context.Context) error { + req.Actor = "alice" + return func(ctx context.Context) error { + lifecycle, err := r.IssueLifecycle() + if err != nil { + return err + } + _, err = lifecycle.Create(ctx, req) + return err + } + } + update := func(issueID string, patch publicops.IssuePatch) func(context.Context) error { + return func(ctx context.Context) error { + lifecycle, err := r.IssueLifecycle() + if err != nil { + return err + } + _, err = lifecycle.Update(ctx, publicops.UpdateRequest{Actor: "bob", IssueID: issueID, Patch: patch}) + return err + } + } + + a := newIssue("a", "alpha", types.TypeTask) + a.Labels = []string{"one", "two"} + a.Metadata = json.RawMessage(`{"k":1,"s":"x"}`) + a.Notes = "first note" + b := newIssue("b", "bravo", types.TypeBug) + b.Priority = 1 + epic := newIssue("epic", "the epic", types.TypeEpic) + child := newIssue("epic.1", "epic child", types.TypeTask) + w1 := newIssue("w1", "a wisp", types.TypeTask) + w1.Ephemeral = true + w1.Labels = []string{"wl"} + nh := newIssue("nh", "no history", types.TypeTask) + nh.NoHistory = true + c := newIssue("c", "charlie", types.TypeFeature) + c.Labels = []string{"moving"} + d := newIssue("d", "delta", types.TypeTask) + e := newIssue("e", "echo", types.TypeTask) + // The wisps-plane block's wisps: x carries the field, lease, comment, + // metadata and close traffic, y the refused claim and the dependency, + // apply and delete traffic, z the defer wake. + wisp := func(suffix, title string) *types.Issue { + w := newIssue(suffix, title, types.TypeTask) + w.Ephemeral = true + return w + } + x := wisp("x", "x-ray") + x.Labels = []string{"xl", "xm"} + x.Metadata = json.RawMessage(`{"k":1}`) + y, z := wisp("y", "yankee"), wisp("z", "zulu") + + return []Step{ + {"create a", create(publicops.CreateRequest{Issue: a})}, + {"create b blocked by a", create(publicops.CreateRequest{Issue: b, + Dependencies: []publicops.CreateDependency{{TargetID: id("a"), Type: types.DepBlocks}}})}, + {"create epic", create(publicops.CreateRequest{Issue: epic})}, + {"create epic.1 under epic", create(publicops.CreateRequest{Issue: child, ParentID: id("epic")})}, + {"create wisp w1", create(publicops.CreateRequest{Issue: w1})}, + {"create no-history nh", create(publicops.CreateRequest{Issue: nh})}, + {"create c", create(publicops.CreateRequest{Issue: c})}, + {"create d", create(publicops.CreateRequest{Issue: d})}, + {"create e", create(publicops.CreateRequest{Issue: e})}, + + {"update a fields labels metadata", update(id("a"), publicops.IssuePatch{ + Title: set("alpha two"), + Priority: set(3), + Assignee: set("bob"), + Status: set(types.StatusInProgress), + AppendNotes: set("second note"), + Labels: publicops.LabelPatch{Add: []string{"three"}, Remove: []string{"two"}}, + Metadata: publicops.MetadataPatch{Set: map[string]json.RawMessage{"k": json.RawMessage(`2`)}}, + })}, + {"update a no-op", update(id("a"), publicops.IssuePatch{Title: set("alpha two")})}, + {"update w1 to persistent", update(id("w1"), publicops.IssuePatch{Persistence: set(types.PersistenceModePersistent)})}, + {"update c to ephemeral", update(id("c"), publicops.IssuePatch{Persistence: set(types.PersistenceModeEphemeral)})}, + {"update d parent epic", update(id("d"), publicops.IssuePatch{ParentID: set(id("epic"))})}, + + {"add dependencies", func(ctx context.Context) error { + editor, err := r.DependencyEditor() + if err != nil { + return err + } + _, err = editor.AddDependencies(ctx, publicops.AddDependenciesRequest{Actor: "carol", Edges: []publicops.DependencyEdge{ + {IssueID: id("d"), DependsOnID: id("b"), Type: types.DepBlocks}, + {IssueID: id("e"), DependsOnID: id("a"), Type: types.DepRelated}, + {IssueID: id("nh"), DependsOnID: id("d"), Type: types.DepBlocks}, + }}) + return err + }}, + {"remove dependency b->a", func(ctx context.Context) error { + editor, err := r.DependencyEditor() + if err != nil { + return err + } + _, err = editor.RemoveDependency(ctx, publicops.RemoveDependencyRequest{Actor: "carol", IssueID: id("b"), DependsOnID: id("a")}) + return err + }}, + {"comment on a and nh", func(ctx context.Context) error { + commenter, err := r.Commenter() + if err != nil { + return err + } + if _, err := commenter.AddComment(ctx, publicops.AddCommentRequest{Author: "carol", IssueID: id("a"), Text: "hello a"}); err != nil { + return err + } + _, err = commenter.AddComment(ctx, publicops.AddCommentRequest{Author: "carol", IssueID: id("nh"), Text: "hello nh"}) + return err + }}, + {"claim heartbeat release b", func(ctx context.Context) error { + claimer, err := r.IssueClaimer() + if err != nil { + return err + } + if _, err := claimer.Claim(ctx, publicops.ClaimRequest{Actor: "dave", IssueID: id("b")}); err != nil { + return fmt.Errorf("claim: %w", err) + } + heartbeater, err := r.Heartbeater() + if err != nil { + return err + } + if _, err := heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{Actor: "dave", IssueID: id("b")}); err != nil { + return fmt.Errorf("heartbeat: %w", err) + } + releaser, err := r.Releaser() + if err != nil { + return err + } + _, err = releaser.Release(ctx, publicops.ReleaseRequest{Actor: "dave", IssueID: id("b")}) + return err + }}, + {"claim e and reclaim no lease", func(ctx context.Context) error { + claimer, err := r.IssueClaimer() + if err != nil { + return err + } + if _, err := claimer.Claim(ctx, publicops.ClaimRequest{Actor: "erin", IssueID: id("e")}); err != nil { + return fmt.Errorf("claim: %w", err) + } + reclaimer, err := r.LeaseReclaimer() + if err != nil { + return err + } + // e's lease has not expired, so the sweep reverts nothing. + _, err = reclaimer.Reclaim(ctx, publicops.ReclaimRequest{Actor: "reaper", + Filter: publicops.ReclaimFilter{IDs: []string{id("e")}}}) + return err + }}, + {"metadata compare-and-set on a", func(ctx context.Context) error { + cas, err := r.MetadataCAS() + if err != nil { + return err + } + _, err = cas.CompareAndSetKey(ctx, publicops.CompareAndSetKeyRequest{Actor: "frank", IssueID: id("a"), Key: "cas", Value: raw(`"v1"`)}) + return err + }}, + {"close reopen close epic.1", func(ctx context.Context) error { + lifecycle, err := r.IssueLifecycle() + if err != nil { + return err + } + if _, err := lifecycle.Close(ctx, publicops.CloseRequest{Actor: "bob", IssueID: id("epic.1"), Reason: "done"}); err != nil { + return fmt.Errorf("close: %w", err) + } + if _, err := lifecycle.Reopen(ctx, publicops.ReopenRequest{Actor: "bob", IssueID: id("epic.1"), Reason: "not yet"}); err != nil { + return fmt.Errorf("reopen: %w", err) + } + _, err = lifecycle.Close(ctx, publicops.CloseRequest{Actor: "bob", IssueID: id("epic.1"), Reason: "done again"}) + return err + }}, + {"close batch b and refused epic", func(ctx context.Context) error { + closer, err := r.BatchCloser() + if err != nil { + return err + } + // b closes; epic keeps its open child d, so close policy refuses + // that item and the batch keeps its survivor. + _, err = closer.CloseBatch(ctx, publicops.CloseBatchRequest{Actor: "bob", Items: []publicops.BatchCloseItem{ + {IssueID: id("b"), Reason: "b done"}, {IssueID: id("epic"), Reason: "epic done"}, + }}) + return err + }}, + {"force close blocked no-history nh", func(ctx context.Context) error { + lifecycle, err := r.IssueLifecycle() + if err != nil { + return err + } + _, err = lifecycle.Close(ctx, publicops.CloseRequest{Actor: "bob", IssueID: id("nh"), Reason: "nh done", Force: true}) + return err + }}, + {"create batch", func(ctx context.Context) error { + creator, err := r.BatchCreator() + if err != nil { + return err + } + f1 := newIssue("f1", "foxtrot one", types.TypeTask) + f1.Labels = []string{"batch"} + f2 := newIssue("f2", "foxtrot two", types.TypeTask) + fw := newIssue("fw", "foxtrot wisp", types.TypeTask) + fw.Ephemeral = true + _, err = creator.CreateBatch(ctx, publicops.CreateBatchRequest{Actor: "gina", Items: []publicops.BatchCreateItem{ + {Issue: f1}, + {Issue: f2, Dependencies: []publicops.CreateDependency{{TargetID: id("f1"), Type: types.DepBlocks}}}, + {Issue: fw}, + }}) + return err + }}, + {"apply batch", func(ctx context.Context) error { + applier, err := r.BatchApplier() + if err != nil { + return err + } + g1 := newIssue("g1", "golf one", types.TypeTask) + g2 := newIssue("g2", "golf two", types.TypeBug) + g2.Labels = []string{"applied"} + _, err = applier.ApplyBatch(ctx, publicops.ApplyBatchRequest{Actor: "hank", Items: []publicops.ApplyItem{ + {Kind: publicops.ItemCreate, Create: &publicops.CreateItem{Key: "g1", Issue: g1}}, + {Kind: publicops.ItemCreate, Create: &publicops.CreateItem{Key: "g2", Issue: g2}}, + {Kind: publicops.ItemDepAdd, DepAdd: &publicops.DepAddItem{Source: publicops.Ref{Key: "g2"}, Target: publicops.Ref{Key: "g1"}, Type: types.DepBlocks}}, + {Kind: publicops.ItemUpdate, Update: &publicops.UpdateItem{Target: publicops.Ref{ID: id("d")}, Patch: publicops.IssuePatch{Assignee: set("hank")}}}, + {Kind: publicops.ItemClose, Close: &publicops.CloseItem{Target: publicops.Ref{Key: "g1"}, Reason: "applied"}}, + }}) + return err + }}, + {"defer f1 into the past and wake it", func(ctx context.Context) error { + past := fixedAt + if err := update(id("f1"), publicops.IssuePatch{Status: set(types.StatusDeferred), DeferUntil: set(&past)})(ctx); err != nil { + return fmt.Errorf("defer: %w", err) + } + return wake(ctx) + }}, + {"delete f2", func(ctx context.Context) error { + deleter, err := r.Deleter() + if err != nil { + return err + } + _, err = deleter.Delete(ctx, publicops.DeleteRequest{Actor: "ivan", IDs: []string{id("f2")}, Force: true}) + return err + }}, + + // The wisps plane, role by role. + {"create wisps x y z", func(ctx context.Context) error { + for _, w := range []*types.Issue{x, y, z} { + if err := create(publicops.CreateRequest{Issue: w})(ctx); err != nil { + return fmt.Errorf("create %s: %w", w.ID, err) + } + } + return nil + }}, + {"update wisp x fields labels metadata", update(id("x"), publicops.IssuePatch{ + Title: set("x-ray two"), + Priority: set(1), + Assignee: set("bob"), + Status: set(types.StatusInProgress), + AppendNotes: set("wisp note"), + Labels: publicops.LabelPatch{Add: []string{"xn"}, Remove: []string{"xm"}}, + Metadata: publicops.MetadataPatch{Set: map[string]json.RawMessage{"k": json.RawMessage(`2`)}}, + })}, + {"update wisp x no-op", update(id("x"), publicops.IssuePatch{Title: set("x-ray two")})}, + {"claim and heartbeat refuse wisps", func(ctx context.Context) error { + // The lease roles do not lease a wisp: Claim answers ErrNotFound, + // Heartbeat ErrNotClaimable, and neither may write. + claimer, err := r.IssueClaimer() + if err != nil { + return err + } + if _, err := claimer.Claim(ctx, publicops.ClaimRequest{Actor: "dave", IssueID: id("y")}); !errors.Is(err, publicops.ErrNotFound) { + return fmt.Errorf("claim wisp y = %v, want ErrNotFound", err) + } + heartbeater, err := r.Heartbeater() + if err != nil { + return err + } + if _, err := heartbeater.Heartbeat(ctx, publicops.HeartbeatRequest{Actor: "bob", IssueID: id("x")}); !errors.Is(err, publicops.ErrNotClaimable) { + return fmt.Errorf("heartbeat wisp x = %v, want ErrNotClaimable", err) + } + return nil + }}, + {"reclaim sweep scoped to in-progress wisp x", func(ctx context.Context) error { + reclaimer, err := r.LeaseReclaimer() + if err != nil { + return err + } + // The sweep only reads the issues table, so it reverts nothing. + _, err = reclaimer.Reclaim(ctx, publicops.ReclaimRequest{Actor: "reaper", + Filter: publicops.ReclaimFilter{IDs: []string{id("x")}}}) + return err + }}, + {"release wisp x", func(ctx context.Context) error { + releaser, err := r.Releaser() + if err != nil { + return err + } + _, err = releaser.Release(ctx, publicops.ReleaseRequest{Actor: "bob", IssueID: id("x")}) + return err + }}, + {"add and remove wisp dependencies", func(ctx context.Context) error { + editor, err := r.DependencyEditor() + if err != nil { + return err + } + if _, err := editor.AddDependencies(ctx, publicops.AddDependenciesRequest{Actor: "carol", Edges: []publicops.DependencyEdge{ + {IssueID: id("y"), DependsOnID: id("x"), Type: types.DepBlocks}, + {IssueID: id("x"), DependsOnID: id("a"), Type: types.DepRelated}, + }}); err != nil { + return fmt.Errorf("add: %w", err) + } + _, err = editor.RemoveDependency(ctx, publicops.RemoveDependencyRequest{Actor: "carol", IssueID: id("y"), DependsOnID: id("x")}) + return err + }}, + {"comment on wisp x", func(ctx context.Context) error { + commenter, err := r.Commenter() + if err != nil { + return err + } + _, err = commenter.AddComment(ctx, publicops.AddCommentRequest{Author: "carol", IssueID: id("x"), Text: "hello x"}) + return err + }}, + {"metadata compare-and-set on wisp x", func(ctx context.Context) error { + cas, err := r.MetadataCAS() + if err != nil { + return err + } + _, err = cas.CompareAndSetKey(ctx, publicops.CompareAndSetKeyRequest{Actor: "frank", IssueID: id("x"), Key: "cas", Value: raw(`"w1"`)}) + return err + }}, + {"close reopen wisp x", func(ctx context.Context) error { + lifecycle, err := r.IssueLifecycle() + if err != nil { + return err + } + if _, err := lifecycle.Close(ctx, publicops.CloseRequest{Actor: "bob", IssueID: id("x"), Reason: "x done"}); err != nil { + return fmt.Errorf("close: %w", err) + } + _, err = lifecycle.Reopen(ctx, publicops.ReopenRequest{Actor: "bob", IssueID: id("x"), Reason: "x not yet"}) + return err + }}, + {"apply batch on the wisps plane", func(ctx context.Context) error { + applier, err := r.BatchApplier() + if err != nil { + return err + } + h1 := wisp("h1", "hotel wisp") + h1.Labels = []string{"applied"} + _, err = applier.ApplyBatch(ctx, publicops.ApplyBatchRequest{Actor: "hank", Items: []publicops.ApplyItem{ + {Kind: publicops.ItemCreate, Create: &publicops.CreateItem{Key: "h1", Issue: h1}}, + {Kind: publicops.ItemDepAdd, DepAdd: &publicops.DepAddItem{Source: publicops.Ref{ID: id("y")}, Target: publicops.Ref{Key: "h1"}, Type: types.DepBlocks}}, + {Kind: publicops.ItemUpdate, Update: &publicops.UpdateItem{Target: publicops.Ref{ID: id("y")}, Patch: publicops.IssuePatch{Assignee: set("hank")}}}, + {Kind: publicops.ItemClose, Close: &publicops.CloseItem{Target: publicops.Ref{Key: "h1"}, Reason: "applied"}}, + }}) + return err + }}, + {"close batch wisps x and nh", func(ctx context.Context) error { + closer, err := r.BatchCloser() + if err != nil { + return err + } + // x closes; nh is already closed, so the batch closes x alone. + _, err = closer.CloseBatch(ctx, publicops.CloseBatchRequest{Actor: "bob", Items: []publicops.BatchCloseItem{ + {IssueID: id("x"), Reason: "x done again"}, {IssueID: id("nh"), Reason: "nh again"}, + }}) + return err + }}, + {"defer wisp z into the past and wake it", func(ctx context.Context) error { + past := fixedAt + if err := update(id("z"), publicops.IssuePatch{Status: set(types.StatusDeferred), DeferUntil: set(&past)})(ctx); err != nil { + return fmt.Errorf("defer: %w", err) + } + return wake(ctx) + }}, + {"delete wisp y", func(ctx context.Context) error { + deleter, err := r.Deleter() + if err != nil { + return err + } + _, err = deleter.Delete(ctx, publicops.DeleteRequest{Actor: "ivan", IDs: []string{id("y")}, Force: true}) + return err + }}, + } +} + +// DoltStorageSteps are the store legs' extra steps: the storage.DoltStorage +// methods most of bd's commands still write through, including the import +// upsert (CreateIssuesWithFullOptions) whose create-versus-update choice the +// stack rewired, promotion out of the wisps table, and RunInTransaction. +func DoltStorageSteps(s storage.DoltStorage) []Step { + return []Step{ + {"store create s1", func(ctx context.Context) error { + s1 := newIssue("s1", "sierra one", types.TypeTask) + s1.Labels = []string{"store"} + return s.CreateIssue(ctx, s1, "sam") + }}, + {"store create issues s2 s3", func(ctx context.Context) error { + return s.CreateIssues(ctx, []*types.Issue{newIssue("s2", "sierra two", types.TypeTask), newIssue("s3", "sierra three", types.TypeBug)}, "sam") + }}, + {"store import upsert s1 and new s4", func(ctx context.Context) error { + s1 := newIssue("s1", "sierra one imported", types.TypeTask) + s1.Labels = []string{"store", "imported"} + s1.UpdatedAt = fixedAt.Add(time.Hour) + s4 := newIssue("s4", "sierra four", types.TypeTask) + s4.Dependencies = []*types.Dependency{{IssueID: id("s4"), DependsOnID: id("s1"), Type: types.DepBlocks, CreatedAt: fixedAt}} + s4.Comments = []*types.Comment{{ID: "ho-imported-comment", IssueID: id("s4"), Author: "importer", Text: "imported", CreatedAt: fixedAt}} + sw := newIssue("sw", "sierra wisp", types.TypeTask) + sw.Ephemeral = true + return s.CreateIssuesWithFullOptions(ctx, []*types.Issue{s1, s4, sw}, "importer", + storage.BatchCreateOptions{SkipPrefixValidation: true}) + }}, + {"store update s2", func(ctx context.Context) error { + return s.UpdateIssue(ctx, id("s2"), map[string]interface{}{"title": "sierra two updated", "priority": 1, "assignee": "sam"}, "sam") + }}, + {"store update s2 no-op", func(ctx context.Context) error { + return s.UpdateIssue(ctx, id("s2"), map[string]interface{}{"title": "sierra two updated"}, "sam") + }}, + {"store labels", func(ctx context.Context) error { + if err := s.AddLabel(ctx, id("s2"), "added", "sam"); err != nil { + return err + } + return s.RemoveLabel(ctx, id("s1"), "store", "sam") + }}, + {"store dependencies", func(ctx context.Context) error { + if err := s.AddDependency(ctx, &types.Dependency{IssueID: id("s3"), DependsOnID: id("s2"), Type: types.DepBlocks}, "sam"); err != nil { + return err + } + return s.RemoveDependency(ctx, id("s4"), id("s1"), "sam") + }}, + {"store comment", func(ctx context.Context) error { + _, err := s.AddIssueComment(ctx, id("s3"), "sam", "store comment") + return err + }}, + {"store claim s3", func(ctx context.Context) error { + bulk, ok := s.(storage.BulkIssueStore) + if !ok { + return fmt.Errorf("%T is not a storage.BulkIssueStore", s) + } + return bulk.ClaimIssue(ctx, id("s3"), "tess") + }}, + {"store close and reopen s2", func(ctx context.Context) error { + if err := s.CloseIssue(ctx, id("s2"), "closed by store", "sam", "session-1"); err != nil { + return err + } + return s.ReopenIssue(ctx, id("s2"), "reopened by store", "sam") + }}, + {"store promote wisp sw", func(ctx context.Context) error { + bulk, ok := s.(storage.BulkIssueStore) + if !ok { + return fmt.Errorf("%T is not a storage.BulkIssueStore", s) + } + return bulk.PromoteFromEphemeral(ctx, id("sw"), "sam") + }}, + {"store transaction", func(ctx context.Context) error { + return s.RunInTransaction(ctx, "bd: historyoffequiv transaction", func(tx storage.Transaction) error { + t1 := newIssue("t1", "tango one", types.TypeTask) + if err := tx.CreateIssue(ctx, t1, "tx"); err != nil { + return err + } + if err := tx.UpdateIssue(ctx, id("t1"), map[string]interface{}{"description": "changed in tx"}, "tx"); err != nil { + return err + } + if err := tx.AddLabel(ctx, id("t1"), "tx", "tx"); err != nil { + return err + } + if err := tx.AddDependency(ctx, &types.Dependency{IssueID: id("t1"), DependsOnID: id("s1"), Type: types.DepRelated}, "tx"); err != nil { + return err + } + return tx.CloseIssue(ctx, id("s4"), "closed in tx", "tx", "") + }) + }}, + {"store delete s3", func(ctx context.Context) error { + return s.DeleteIssue(ctx, id("s3")) + }}, + } +} diff --git a/internal/storage/issueops/BUILD.bazel b/internal/storage/issueops/BUILD.bazel index 3ade280ae..094dd2847 100644 --- a/internal/storage/issueops/BUILD.bazel +++ b/internal/storage/issueops/BUILD.bazel @@ -210,6 +210,7 @@ go_test( "version_history_number_admission_test.go", "version_history_preenable_scan_test.go", "version_history_test.go", + "wake_defers_test.go", "wisp_routing_test.go", ], data = glob(["testdata/**"]) + [ @@ -228,6 +229,7 @@ go_test( "//internal/storage/journalscan", "//internal/storage/rowid", "//internal/storage/sqlbuild", + "//internal/storage/sqlcount", "//internal/types", "//internal/validation", "//issueops", diff --git a/internal/storage/issueops/wake_defers_test.go b/internal/storage/issueops/wake_defers_test.go new file mode 100644 index 000000000..fae1b596e --- /dev/null +++ b/internal/storage/issueops/wake_defers_test.go @@ -0,0 +1,127 @@ +package issueops + +import ( + "context" + "database/sql" + "database/sql/driver" + "fmt" + "slices" + "sync/atomic" + "testing" + + "github.com/DATA-DOG/go-sqlmock" + + "github.com/steveyegge/beads/internal/storage/sqlcount" +) + +// With versioned history off the mint is a no-op, so the sweep has no reason +// to look at an expired row before it wakes it, nor after. With history on it +// reads the woken row to mint its version; a later change that also checks +// each row against the mint before waking it (so that one unversionable row +// cannot abort the whole sweep) must keep that check free when history is +// off. +// +// The mock scripts exactly the statements a one-row sweep issues: the +// expired-row SELECT, the UPDATE, the event's same-content SELECT and its +// INSERT, and the empty wisps pass. Nothing else is expected, and the mock is +// reached through sqlcount, which counts every statement the sweep sends. So +// a sweep that loaded the row (or read its participation) before or after +// waking it fails the pin even if it swallowed the error sqlmock answers that +// statement with, which sqlmock's own checks would not see. The history-on +// run of the same script is the control: the mint's read must make the pin +// refuse it. +func TestWakeExpiredDefersWithHistoryOffLoadsNothingBeforeWaking(t *testing.T) { + if err := wakeAgainstScript(t, false); err != nil { + t.Fatalf("history off: %v", err) + } + t.Run("history on is refused by the same script", func(t *testing.T) { + err := wakeAgainstScript(t, true) + if err == nil { + t.Fatal("the sweep with history on passed the history-off script; " + + "the script no longer sees the reads history adds, so the history-off run above proves nothing") + } + t.Logf("history on, as expected: %v", err) + }) +} + +// wakeStatements is how many statements the script below holds, and so how +// many the history-off sweep may send. +const wakeStatements = 5 + +// wakeAgainstScript runs a one-row sweep on a mocked transaction scoped with +// versioned history as asked, against the script of the statements the +// history-off sweep issues. It returns why the sweep broke the script, or nil +// if it sent exactly the scripted statements and woke the scripted row. +func wakeAgainstScript(t *testing.T, history bool) error { + t.Helper() + mock, tx, counts := countedMockTx(t) + if history { + t.Cleanup(ScopeVersionedHistoryTransaction(tx, true)) + } + // Otherwise tx is not scoped for versioned history, which is how every + // transaction reads while it is off. + mock.ExpectQuery(`SELECT id FROM issues WHERE status = 'deferred'`). + WillReturnRows(sqlmock.NewRows([]string{"id"}).AddRow("bd-1")) + mock.ExpectExec(`UPDATE issues SET status = 'open', defer_until = NULL`). + WithArgs(sqlmock.AnyArg(), sqlmock.AnyArg(), "bd-1"). + WillReturnResult(sqlmock.NewResult(0, 1)) + mock.ExpectQuery(`SELECT id FROM events WHERE issue_id`). + WillReturnRows(sqlmock.NewRows([]string{"id"})) + mock.ExpectExec(`INSERT INTO events`). + WillReturnResult(sqlmock.NewResult(0, 1)) + mock.ExpectQuery(`SELECT id FROM wisps WHERE status = 'deferred'`). + WillReturnRows(sqlmock.NewRows([]string{"id"})) + + got, err := WakeExpiredDefersInTx(t.Context(), tx) + if err != nil { + return fmt.Errorf("the sweep failed against the script: %w", err) + } + if n := counts.Total(); n != wakeStatements { + return fmt.Errorf("the sweep sent %d statements, want exactly the %d scripted; "+ + "sqlmock refused each extra one with an error the sweep dropped", n, wakeStatements) + } + if err := mock.ExpectationsWereMet(); err != nil { + return fmt.Errorf("the sweep did not send the scripted statements: %w", err) + } + if !slices.Equal(got.Issues, []string{"bd-1"}) || len(got.Wisps) != 0 { + return fmt.Errorf("woken = issues %v, wisps %v, want [bd-1] and none", got.Issues, got.Wisps) + } + return nil +} + +var sqlmockSeq atomic.Int64 + +// countedMockTx begins a transaction on a fresh sqlmock connection that is +// reached through sqlcount, so a pin can count every statement the code +// under test sends, including one whose error it swallowed. +func countedMockTx(t *testing.T) (sqlmock.Sqlmock, *sql.Tx, *sqlcount.Counts) { + t.Helper() + dsn := fmt.Sprintf("issueops_counted_%d", sqlmockSeq.Add(1)) + mockDB, mock, err := sqlmock.NewWithDSN(dsn) + if err != nil { + t.Fatalf("sqlmock.NewWithDSN: %v", err) + } + counts := &sqlcount.Counts{} + db := sql.OpenDB(sqlcount.WrapConnector(sqlmockConnector{dsn: dsn, driver: mockDB.Driver()}, counts)) + t.Cleanup(func() { + _ = db.Close() + _ = mockDB.Close() + }) + mock.ExpectBegin() + tx, err := db.BeginTx(t.Context(), nil) + if err != nil { + t.Fatalf("BeginTx: %v", err) + } + t.Cleanup(func() { _ = tx.Rollback() }) + return mock, tx, counts +} + +// sqlmockConnector opens the sqlmock connection registered under dsn, for +// sql.OpenDB through a wrapping connector. +type sqlmockConnector struct { + dsn string + driver driver.Driver +} + +func (c sqlmockConnector) Connect(context.Context) (driver.Conn, error) { return c.driver.Open(c.dsn) } +func (c sqlmockConnector) Driver() driver.Driver { return c.driver } diff --git a/internal/storage/uow/BUILD.bazel b/internal/storage/uow/BUILD.bazel index 96c3d988c..19101d195 100644 --- a/internal/storage/uow/BUILD.bazel +++ b/internal/storage/uow/BUILD.bazel @@ -125,6 +125,7 @@ go_test( "external_doltserver_provider_test.go", "graph_counter_contract_test.go", "heartbeater_contract_test.go", + "history_off_differential_test.go", "importer_contract_test.go", "importer_uow_test.go", "issue_claimer_test.go", @@ -198,6 +199,7 @@ go_test( "//internal/storage/dbproxy/server", "//internal/storage/domain", "//internal/storage/domain/db", + "//internal/storage/historyoffequiv", "//internal/storage/issueops", "//internal/storage/schema", "//internal/testutil", diff --git a/internal/storage/uow/history_off_differential_test.go b/internal/storage/uow/history_off_differential_test.go new file mode 100644 index 000000000..aabe0eefe --- /dev/null +++ b/internal/storage/uow/history_off_differential_test.go @@ -0,0 +1,92 @@ +package uow + +import ( + "context" + "testing" + "time" + + "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/storage/historyoffequiv" + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// TestVersionedHistoryOffMatchesPreStackGolden_UOW is the unit-of-work leg's +// versioned-history flag-off differential (internal/storage/historyoffequiv): +// on a provider whose versioned history was never turned on, the scripted +// mutation set must leave every table as the code before the +// versioned-history stack left it, apart from that stack's new columns and +// table staying NULL and empty. +// +// Beyond the shared role scenario this leg drives the domain create that +// leaves CreateOnly unset (the quick, todo and batch proxied servers' path): +// it is an upsert, and the stack gave it an existence probe that must not run +// while history is off (see TestInsertUpsertWithHistoryOffIssuesThePreStackStatements +// in domain/db). The importer role's upsert of an existing row runs too. +func TestVersionedHistoryOffMatchesPreStackGolden_UOW(t *testing.T) { + ctx := context.Background() + provider := newUOWRoleFixtureProvider(t, ctx, historyoffequiv.Prefix) + roles, ok := provider.(historyoffequiv.Roles) + if !ok { + t.Fatalf("provider %T does not offer every role the scenario drives", provider) + } + fixedAt := time.Date(2001, 2, 3, 4, 5, 6, 0, time.UTC) + issue := func(id, title string) *types.Issue { + return &types.Issue{ID: id, Title: title, Description: "about " + id, Status: types.StatusOpen, + Priority: 2, IssueType: types.TypeTask, CreatedAt: fixedAt, UpdatedAt: fixedAt} + } + domainCreate := func(wisp bool, is *types.Issue) func(context.Context) error { + return func(ctx context.Context) error { + return RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) { + params := domain.CreateIssueParams{Issue: is, ExplicitID: is.ID} + var err error + if wisp { + _, err = uw.IssueUseCase().CreateWisp(ctx, params, "uma") + } else { + _, err = uw.IssueUseCase().CreateIssue(ctx, params, "uma") + } + return "bd: create " + is.ID, err + }) + } + } + historyoffequiv.Run(t, historyoffequiv.Leg{ + Name: "uow", + Roles: roles, + Wake: func(ctx context.Context) error { + _, err := WakeExpiredDefers(ctx, provider) + return err + }, + Extra: []historyoffequiv.Step{ + {Name: "domain upsert new u1", Run: domainCreate(false, issue("ho-u1", "uniform one"))}, + {Name: "domain upsert existing u1", Run: domainCreate(false, issue("ho-u1", "uniform one again"))}, + {Name: "domain upsert new wisp uw", Run: domainCreate(true, issue("ho-uw", "uniform wisp"))}, + {Name: "domain upsert existing wisp uw", Run: domainCreate(true, issue("ho-uw", "uniform wisp again"))}, + {Name: "import upsert u1 and new u2", Run: func(ctx context.Context) error { + importer, err := provider.(interface { + Importer() (publicops.Importer, error) + }).Importer() + if err != nil { + return err + } + u1 := issue("ho-u1", "uniform one imported") + u1.UpdatedAt = fixedAt.Add(time.Hour) + u2 := issue("ho-u2", "uniform two") + u2.Labels = []string{"imported"} + u2.Dependencies = []*types.Dependency{{IssueID: "ho-u2", DependsOnID: "ho-u1", Type: types.DepBlocks, CreatedAt: fixedAt}} + u2.Comments = []*types.Comment{{ID: "ho-u2-comment", IssueID: "ho-u2", Author: "importer", Text: "imported", CreatedAt: fixedAt}} + _, err = importer.ImportBatch(ctx, publicops.ImportBatchRequest{Actor: "importer", + Issues: []*types.Issue{u1, u2}, SkipPrefixValidation: true}) + return err + }}, + }, + Query: func(ctx context.Context, query string) ([][]*string, error) { + return RunTxRead(ctx, provider, func(ctx context.Context, uw UnitOfWork) ([][]*string, error) { + result, err := uw.RawSQLUseCase().Query(ctx, query) + if err != nil { + return nil, err + } + return historyoffequiv.AnyRows(result.Rows), nil + }) + }, + }) +} diff --git a/tools/bazel/BUILD.bazel b/tools/bazel/BUILD.bazel index 410728ae0..73e966ddc 100644 --- a/tools/bazel/BUILD.bazel +++ b/tools/bazel/BUILD.bazel @@ -259,6 +259,7 @@ release_cross_build( "//internal/storage/externaldeps:externaldeps", "//internal/storage/fs:fs", "//internal/storage/git:git", + "//internal/storage/historyoffequiv:historyoffequiv", "//internal/storage/issueops:issueops", "//internal/storage/journalscan:journalscan", "//internal/storage/kvkeys:kvkeys", From 8fac20a7f1ae0d5941094516bb49b66056755656 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 19 Oct 2026 04:27:28 +0000 Subject: [PATCH 07/41] test(oracle-a): normalize per-run revision token and workspace path so main-vs-main is clean (#7680) * test(oracle-a): normalize the per-write revision token so main-vs-main is clean Since #5535, `bd show --json` publishes `"revision"`, the decimal spelling of the row's row_lock. Every create and status/ownership write remints that token from crypto/rand (issueops.freshRowLock), so it differs between any two runs of the same scenario. The harness did not normalize it, so Oracle A failed against itself: on the curated tier, origin/main vs origin/main scored 31 pass / 13 fail in scope, and each failure was a `show --json` step that differed only in `"revision"`. normalize() now collapses a non-zero decimal `"revision"` string to ``. It keeps everything else literal: the member name, its presence and its string type, "0" (a row that was never stamped), number-typed values, other `*_revision` members, and escaped text inside a JSON string. A new unit test covers each of these cases. Curated tier, REF_REF=origin/main, run against itself: before 31 pass / 13 fail in scope (0/1 out of scope); after 44 pass / 0 fail (1/0). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(oracle-a): normalize the per-run workspace path Each scenario runs in a fresh tempfile::tempdir(), so any bd output that echoes the workspace path differs on every run. When origin/main's bd was captured twice with the deep tier (ORACLE_CATALOG=1, 546 scenarios), 40 catalog scenarios differed only in that path. All 40 start with their own `init --prefix X --non-interactive` step, which runs in a workspace the harness already initialized and fails with "Found existing Dolt database: /tmp/.tmpXXXXXX/.beads/embeddeddolt/X". run_scenario now hands normalize_in_workspace() the exact workspace path and its symlink-resolved spelling, and that function replaces only those strings with , longest first. Other paths stay literal, so no stable value is collapsed. All 40 scenarios are out of scope, so in-scope counts do not change. Deep tier, origin/main's bd against itself, scored by the scoreboard: - base harness: 256 pass / 67 fail in scope, 154 / 69 out of scope. - with both normalizers: 323 / 0 in scope, 219 / 4 out of scope. The out-of-scope remainder changes from run to run and none of it is a field value. It comes from randomized "Tip:" lines in human output and one unordered `list --format dot`. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(oracle-a): document and pin the match scope and the workspace wiring (mc-ro9kg) Review fixes for #7680, iteration 1. revision_regex runs over the raw stdout/stderr text before any JSON parse, so it collapses every "revision" member holding a non-zero decimal string, at any depth and in plain text, not only the bd show --json issue token. That includes a user metadata key named revision, which the in-scope --set-metadata/--metadata flags can set, so a change to its value passes the gate. The regex comment and a new README "What green does NOT prove" bullet now say so, and normalize_collapses_revision_by_member_name_at_any_depth pins the scope: a nested metadata member and plain text are collapsed, while a metadata key with any other name still gates. The old test name said nothing else was collapsed; it is now normalize_collapses_non_zero_revision_strings. run_scenario_collapses_the_workspace_in_both_step_streams runs run_scenario against a stub bd that prints its cwd on stdout and its resolved path on stderr, and expects in both. Before, the two normalize_in_workspace calls in run_scenario could be reverted with every unit test and the curated tier still green. The remint wording in the comment and the README now names every writer (create, update, close, reopen, claim and unclaim) instead of only status/ownership writes. Tests: cargo test --locked --offline (7 passed), clippy clean, no new rustfmt hunks. Mutants, each in its own copy and target dir: reverting either or both normalize_in_workspace calls fails the run_scenario test; skipping a nested metadata member, or matching only after { or ",", fails the scope test. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com> --- tests/oracle-a/README.md | 21 ++- tests/oracle-a/harness/src/differential.rs | 181 ++++++++++++++++++++- tests/oracle-a/run-oracle-a.sh | 6 +- 3 files changed, 195 insertions(+), 13 deletions(-) diff --git a/tests/oracle-a/README.md b/tests/oracle-a/README.md index 9ce8c5748..3eafcb558 100644 --- a/tests/oracle-a/README.md +++ b/tests/oracle-a/README.md @@ -15,9 +15,11 @@ Each scenario is an ordered list of `bd` CLI argv steps run as real processes in a throwaway workspace (`bd init` + steps). For every step the harness compares **exit code**, **stderr**, and **JSON-aware stdout** (object key order ignored; array order compared as a multiset by default, as an ordered sequence for -scenarios flagged `ordered`). Volatile values — timestamps, UUIDs, and the host -actor identity — are normalized to ``/``/``/`` before -comparison. **Tolerated in-scope divergences: zero.** +scenarios flagged `ordered`). Volatile values — timestamps, UUIDs, the host +actor identity, the `bd show --json` `"revision"` token (matched by member name, +see below), and the run's own throwaway workspace path — are normalized to +``/``/``/``/``/`` before comparison. +**Tolerated in-scope divergences: zero.** ## Run it @@ -112,7 +114,18 @@ the loop — every scenario uses embedded Dolt in its own tempdir. - **Anything invisible to stdout/stderr/exit diffing.** Post-commit hook firing, rollback-suppressed side effects, and N-commits-instead-of-one are deliberately NOT asserted here. Timestamp *values* are erased by `` normalization, so a - local-vs-UTC divergence in a timestamp value is also out of reach. + local-vs-UTC divergence in a timestamp value is also out of reach. Likewise a + non-zero `"revision"` token is erased to `` (every create, update, close, + reopen, claim and unclaim remints it from crypto/rand), so whether a write + remints it is not gated; the member's presence, its string type, and a + never-stamped `"0"` still are. +- **`` matches a member name, not a position.** The `"revision"` collapse runs + over the raw stdout/stderr text before any JSON parse, so it erases EVERY + `"revision"` member holding a non-zero decimal string — at any depth, and in + plain (non-JSON) text too — not only the `bd show --json` issue token. That + includes a user metadata key named `revision`, which the in-scope + `--set-metadata`/`--metadata` flags can set: a change to its value passes the + gate. No scenario sets one today; use another key name in any that would. - **The scenario environment is scrubbed, and `BEADS_TEST_MODE=1` is ungated.** Scenarios run with an explicit env whitelist (`PATH`/`HOME`/`TMPDIR` + `BEADS_TEST_MODE=1`) rather than the inherited host env, so a green certifies the diff --git a/tests/oracle-a/harness/src/differential.rs b/tests/oracle-a/harness/src/differential.rs index cf0078190..1ab47cdf7 100644 --- a/tests/oracle-a/harness/src/differential.rs +++ b/tests/oracle-a/harness/src/differential.rs @@ -89,20 +89,72 @@ fn uuid_regex() -> &'static regex::Regex { }) } -/// Replace volatile values (timestamps, host identity, random UUIDs) with stable -/// tokens. UUIDs (e.g. comment ids) are random per run, so byte-diffing them is -/// meaningless — collapse them like timestamps. +fn revision_regex() -> &'static regex::Regex { + static RE: OnceLock = OnceLock::new(); + RE.get_or_init(|| { + // The `"revision": ""` JSON member (types.IssueDetails.Revision, on + // `bd show --json` since #5535). The token is the row's row_lock, which + // every create, update, close, reopen, claim and unclaim remints from + // crypto/rand (issueops.freshRowLock), so its value differs on every run. + // Only a NON-ZERO decimal token is collapsed: "0" (a never-stamped row) + // stays literal, and the member's name, presence and string type are + // untouched. + // + // The match is by member NAME over the raw text, before any JSON parse, + // so it collapses EVERY `"revision"` member holding such a string, at any + // depth and in plain (non-JSON) text too, not only the issue token. That + // includes a user metadata key named `revision`, which the in-scope + // --set-metadata/--metadata flags can set: a change to its value would + // pass the gate. No scenario sets one; use another key name in any that + // would. Pinned by normalize_collapses_revision_by_member_name_at_any_depth. + regex::Regex::new(r#"("revision"\s*:\s*")-?[1-9][0-9]*(")"#).expect("valid revision regex") + }) +} + +/// Replace volatile values (timestamps, host identity, random UUIDs, revision +/// tokens) with stable tokens. UUIDs (e.g. comment ids) and revision tokens are +/// random per run, so byte-diffing them is meaningless — collapse them like +/// timestamps. pub fn normalize(s: &str) -> String { - let s = ts_regex().replace_all(s, "").into_owned(); + let s = revision_regex() + .replace_all(s, "${1}${2}") + .into_owned(); + let s = ts_regex().replace_all(&s, "").into_owned(); let s = uuid_regex().replace_all(&s, "").into_owned(); s.replace(ACTOR_EMAIL, "").replace(ACTOR_NAME, "") } +/// `normalize`, after first collapsing this run's own workspace directory to +/// ``. Each run gets a fresh random tempdir, so any output that +/// echoes it (e.g. the "Found existing Dolt database: /.beads/..." error a +/// scenario's own `init` step hits) differs on every run. Only these exact path +/// strings are replaced, longest first, so a spelling that contains another +/// (`/private/var/...` around `/var/...`) is replaced whole. +pub fn normalize_in_workspace(s: &str, workspace: &[String]) -> String { + let mut paths: Vec<&str> = workspace + .iter() + .map(String::as_str) + .filter(|p| !p.is_empty()) + .collect(); + paths.sort_by_key(|p| std::cmp::Reverse(p.len())); + let mut s = s.to_string(); + for p in paths { + s = s.replace(p, ""); + } + normalize(&s) +} + /// Run a scenario against `bin` in a throwaway workspace, returning the /// normalized trace. Requires the binary to support `init -p --quiet` /// and to honor the workspace cwd (real bd uses embedded Dolt under `.beads/`). pub fn run_scenario(bin: &Path, sc: &Scenario) -> std::io::Result { let work = tempfile::tempdir()?; + // Every spelling of the workspace bd may print: the tempdir as created and + // its symlink-resolved form (macOS tempdirs sit under /var -> /private/var). + let mut workspace = vec![work.path().to_string_lossy().into_owned()]; + if let Ok(real) = work.path().canonicalize() { + workspace.push(real.to_string_lossy().into_owned()); + } let run = |args: &[String]| -> std::io::Result { // Scrub the environment to an explicit whitelist. The upstream harness // inherited the FULL host env, so any host BEADS_*/BD_* var @@ -142,8 +194,8 @@ pub fn run_scenario(bin: &Path, sc: &Scenario) -> std::io::Result { steps.push(StepResult { args: args.clone(), exit: out.status.code().unwrap_or(-1), - stdout: normalize(&String::from_utf8_lossy(&out.stdout)), - stderr: normalize(&String::from_utf8_lossy(&out.stderr)), + stdout: normalize_in_workspace(&String::from_utf8_lossy(&out.stdout), &workspace), + stderr: normalize_in_workspace(&String::from_utf8_lossy(&out.stderr), &workspace), }); } Ok(Trace { @@ -325,6 +377,123 @@ mod tests { assert!(!n.contains("2026")); } + #[test] + fn normalize_collapses_non_zero_revision_strings() { + // Two runs of the same `show --json`: only the minted token differs. + let run1 = + "[\n {\n \"id\": \"g-root\",\n \"revision\": \"-1228122709301203494\"\n }\n]\n"; + let run2 = + "[\n {\n \"id\": \"g-root\",\n \"revision\": \"5267907219149755461\"\n }\n]\n"; + assert_eq!(normalize(run1), normalize(run2)); + assert!(normalize(run1).contains(r#""revision": """#)); + assert!(stdout_equiv(&normalize(run1), &normalize(run2), true)); + assert_eq!(normalize(r#"{"revision":"42"}"#), r#"{"revision":""}"#); + + // Everything that is NOT a minted token keeps its literal value: + // "0" (a never-stamped row) must still diverge from a minted token. + assert_eq!(normalize(r#"{"revision":"0"}"#), r#"{"revision":"0"}"#); + assert!(!stdout_equiv( + &normalize(r#"{"revision":"0"}"#), + &normalize(r#"{"revision":"7"}"#), + false + )); + // a number-typed member (a wire-type change) is not collapsed, + assert_eq!(normalize(r#"{"revision":42}"#), r#"{"revision":42}"#); + // nor are other members whose names merely contain "revision", + assert_eq!( + normalize(r#"{"local_revision":"3"}"#), + r#"{"local_revision":"3"}"# + ); + // nor the same text quoted inside a JSON string value. + let quoted = r#"{"description":"see \"revision\": \"12\""}"#; + assert_eq!(normalize(quoted), quoted); + + // Dropping the member is still a divergence. + assert!(!stdout_equiv( + &normalize(r#"{"id":"a","revision":"9"}"#), + &normalize(r#"{"id":"a"}"#), + false + )); + } + + #[test] + fn normalize_collapses_revision_by_member_name_at_any_depth() { + // The scope documented on revision_regex: the match is by member name over + // the raw text, not by position, so a user metadata key named "revision" + // is collapsed too and a change to its value is NOT a divergence. + let meta3 = r#"[{"id":"g-1","revision":"11","metadata":{"revision":"3"}}]"#; + let meta4 = r#"[{"id":"g-1","revision":"22","metadata":{"revision":"4"}}]"#; + assert_eq!( + normalize(meta3), + r#"[{"id":"g-1","revision":"","metadata":{"revision":""}}]"# + ); + assert!(stdout_equiv(&normalize(meta3), &normalize(meta4), false)); + // Plain (non-JSON) text is matched as well. + assert_eq!( + normalize(r#"config "revision": "12" pinned"#), + r#"config "revision": "" pinned"# + ); + // A metadata key with any other name keeps its value, so it still gates. + assert!(!stdout_equiv( + &normalize(r#"{"metadata":{"rev":"3"}}"#), + &normalize(r#"{"metadata":{"rev":"4"}}"#), + false + )); + } + + #[test] + fn normalize_in_workspace_collapses_only_this_runs_workspace() { + // Listed short-first: the longer spelling must still be replaced whole. + let ws = vec![ + "/var/t/.tmpAbC123".to_string(), + "/private/var/t/.tmpAbC123".to_string(), + ]; + let err = "Found existing Dolt database: /private/var/t/.tmpAbC123/.beads/embeddeddolt/c\n"; + assert_eq!( + normalize_in_workspace(err, &ws), + "Found existing Dolt database: /.beads/embeddeddolt/c\n" + ); + // Two runs that differ only in their random workspace normalize equal. + let other = vec!["/var/t/.tmpXyZ789".to_string()]; + assert_eq!( + normalize_in_workspace("db at /var/t/.tmpAbC123/.beads", &ws), + normalize_in_workspace("db at /var/t/.tmpXyZ789/.beads", &other) + ); + // A path that is not this run's workspace stays literal. + let foreign = "db at /var/t/.tmpXyZ789/.beads"; + assert_eq!(normalize_in_workspace(foreign, &ws), foreign); + // The volatile-value normalizers still apply. + assert_eq!( + normalize_in_workspace(r#"{"revision":"5"}"#, &ws), + r#"{"revision":""}"# + ); + } + + #[cfg(unix)] + #[test] + fn run_scenario_collapses_the_workspace_in_both_step_streams() { + // A stub bd that echoes its cwd on stdout and its symlink-resolved form + // on stderr, like the "Found existing Dolt database" error: both must + // reach the trace as , so the run's random tempdir never does. + use std::os::unix::fs::PermissionsExt; + let dir = tempfile::tempdir().unwrap(); + let bin = dir.path().join("bd"); + std::fs::write( + &bin, + "#!/bin/sh\npwd\necho \"Found existing Dolt database: $(pwd -P)/.beads/x\" >&2\n", + ) + .unwrap(); + std::fs::set_permissions(&bin, std::fs::Permissions::from_mode(0o755)).unwrap(); + let sc = Scenario::new("wiring", "w", &[&["show", "--json"]]); + let trace = run_scenario(&bin, &sc).unwrap(); + assert_eq!(trace.steps.len(), 1); + assert_eq!(trace.steps[0].stdout, "\n"); + assert_eq!( + trace.steps[0].stderr, + "Found existing Dolt database: /.beads/x\n" + ); + } + #[test] fn stdout_equiv_ignores_key_and_array_order_but_catches_set_diffs() { // key order ignored diff --git a/tests/oracle-a/run-oracle-a.sh b/tests/oracle-a/run-oracle-a.sh index b896611bc..738119abc 100755 --- a/tests/oracle-a/run-oracle-a.sh +++ b/tests/oracle-a/run-oracle-a.sh @@ -5,9 +5,9 @@ # REFERENCE bd — built from the merge base of HEAD and origin/main (the "before") # CANDIDATE bd — built from the current working tree (the "after") # and diffs each step (exit code, stderr, JSON-aware stdout) with volatile -# normalization (///). ZERO in-scope divergences is the -# gate: any in-scope FAIL means the change altered a user-visible bd behavior on -# the covered contract surface. +# normalization (/////). ZERO in-scope +# divergences is the gate: any in-scope FAIL means the change altered a +# user-visible bd behavior on the covered contract surface. # # The differential is driven by the Rust conformance harness under harness/ # (see harness/PROVENANCE.md); the harness is pointed at two Go bd binaries and From e24940e0c1f8ac3835ca39b610ddc670541d875f Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 19 Oct 2026 04:31:00 +0000 Subject: [PATCH 08/41] fix(workspacegate): strict --readonly never creates workspace gate files (#7672) * fix(workspacegate): strict --readonly never creates workspace gate files Strict --readonly is mutation-free by policy (effectiveRootStorePolicy), but the command chokepoint took its SHARED workspace gates with O_CREATE|O_RDWR. On a workspace no bd command had gated yet, a strict readonly command therefore left /.beads.gate.lock and the physical-root gate file (.beads/embeddeddolt.gate.lock or .beads/dolt.gate.lock) behind. Add workspacegate.Options.ReadOnly. A Shared acquisition with it opens the gate file O_RDONLY (the idiom ExclusiveHolder already uses) and never creates it. A gate file that does not exist means nothing has ever been gated there, so Acquire succeeds with an empty handle whose Release is a no-op. An existing gate file is held SHARED as before, so maintenance still sees the command and a live exclusive holder still makes it wait and then refuses it. The option is ignored for Exclusive: maintenance has to be able to create the file it locks. ReadOnly composes with the fair, bounded waiting from #7225 without changes to it: the writer-fairness probe (ExclusiveQueued) already opens the intent file read-only and treats a missing one as nothing queued, so a waiting ReadOnly acquisition creates no .intent or .info file either. readonly_wait_test.go pins that seam. The chokepoint passes ReadOnly: readonlyMode in the SHARED branch. Every other command keeps creating the gate files, which is what makes it visible to maintenance. Residual, documented at the chokepoint: on a workspace that has never been gated, a strict readonly command is invisible to a maintenance operation that starts while it runs. Ported from the versioned-beads integration fork (donnabox): 0c86e2b205 (red tests), bac6ff95dc (fix), b7a4f96a5a (fair-wait tests), 2af5a664dc (BUILD.bazel registration), and the #7225 conflict resolution in 9d128f8c51. The fork's `--readonly backup restore` refusal (6ac79d6207) is a separate change and is not included. Co-authored-by: Jim Wordelman Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * fix(cli): release the previous command's workspace gate before acquiring acquireCommandWorkspaceGates stored its new handle over whatever workspaceGateHandle already held. cobra skips PersistentPostRunE when RunE fails, so in a process that runs more than one command (tests, embedders) a command could start with the previous command's handle still set. Two things then went wrong: - on the paths that take no new gate (no .beads on disk, a failed or fail-open acquisition) the stale handle stayed set, so the previous workspace stayed held for the whole command; - on a successful acquisition the stale handle was overwritten without being released, so its flock stayed held on the previous workspace until the garbage collector closed the file, and a maintenance operation there was turned away by a command that had finished. Release any held handle first. In a single-command bd process the handle is always nil at that point, so this is a no-op there. Ported from the versioned-beads integration fork (donnabox): 3a25b42ed1 (cherry-pick of ad87442c92). The test TestAcquireCommandWorkspaceGatesReleasesThePreviousCommandsHold is new here and covers the successful-acquisition case the fork's tests do not. Co-authored-by: Donna Box Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * fix(cli): close a stale store before the leading gate release Review fixups for #7672, apply-fixes iteration 1. The leading releaseWorkspaceGates() that the second commit added to the chokepoint was the only production release without a closeStoreBeforeGateRelease() first. When an in-process command's RunE fails, Cobra skips PersistentPostRunE. The next command in the same process then dropped that command's gates while its store and provider were still open, and later overwrote and leaked them. Close them first, as every other release path does. The current command assigns store/uowProvider only after the chokepoint, so anything set there belongs to the previous command. Only multi-command processes (tests, embedders) are affected; production runs one command per process. A strict --readonly hold on a never-gated workspace locks nothing. The chokepoint still exported this PID in BEADS_GATE_SHARED_HOLDER_PID, so bd children skipped the writer-fairness queue behind a parent that blocks no exclusive acquirer. Add Handle.Empty and MultiHandle.Empty, and advertise only when at least one gate is actually locked. Tests: - A regression test runs the chokepoint over a stub store and provider. It pins that both close while the previous command's gate is still held. - A marker test forces advertiseSharedHoldEnabled on. It pins that a hold of nothing is not advertised, and that the hold is advertised once the gate files exist. - The strict-readonly chokepoint subtests now require a handle that holds nothing. SHARED fail-open leaves no handle, so these subtests now fail the review's W2 mutant, which passed them before. - A package test covers Empty. The posture comment now says the gate follows the strict flag only. Allowlisted reads and --dry-run/--inspect previews still create the gate files. Not changed: `bd --readonly backup restore` still creates gate files on its exclusive acquisition. This PR covers shared gates only. Sibling #7673 refuses a strict-readonly restore before it takes the exclusive gate. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Jim Wordelman Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: Donna Box Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com> --- CHANGELOG.md | 9 + cmd/bd/workspace_gate.go | 26 +- cmd/bd/workspace_gate_test.go | 333 ++++++++++++++++++- internal/workspacegate/BUILD.bazel | 5 +- internal/workspacegate/gate.go | 51 ++- internal/workspacegate/gate_test.go | 178 ++++++++++ internal/workspacegate/readonly_wait_test.go | 183 ++++++++++ 7 files changed, 781 insertions(+), 4 deletions(-) create mode 100644 internal/workspacegate/readonly_wait_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 4dbfca8b0..ea893eb2f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -763,6 +763,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **Strict `--readonly` no longer creates workspace gate files.** A + `bd --readonly` command took its shared workspace gates with + create-if-missing, so on a workspace no bd command had gated yet it left + `/.beads.gate.lock` and the physical-root gate file + (`.beads/embeddeddolt.gate.lock` or `.beads/dolt.gate.lock`) behind, although + strict readonly promises not to mutate the workspace. It now opens an + existing gate file read-only and holds it shared as before, so a maintenance + operation still sees it and a running one still makes it wait; when no gate + file exists it holds nothing. Every other command still creates the files. - **`bd init --server --external` no longer starts a Dolt server of its own.** Init builds its store config by hand and turned auto-start on for every server init, so an `--external` init that named no port launched a diff --git a/cmd/bd/workspace_gate.go b/cmd/bd/workspace_gate.go index e4ed66b22..acb54e39b 100644 --- a/cmd/bd/workspace_gate.go +++ b/cmd/bd/workspace_gate.go @@ -48,6 +48,17 @@ import ( // so would treating an interrupted wait as fail-open. // - EXCLUSIVE failures of any kind are hard errors: maintenance refuses // rather than pretends. +// - Strict --readonly is mutation-free (effectiveRootStorePolicy), so its +// SHARED hold opens the gate files read-only and never creates them (see +// workspacegate.Options.ReadOnly). A gate file that does not exist means +// nothing has ever been gated there, so the command proceeds holding +// nothing; an existing one is held SHARED like any other command's. +// Residual: on a workspace that has never been gated, a strict readonly +// command is invisible to a maintenance operation that starts while it +// runs. Every other command keeps creating the files, which is what makes +// it visible. The gate follows the strict flag only: allowlisted reads +// and --dry-run/--inspect previews also open the store read-only, but +// they keep creating the files, so they stay visible too. // // Known residual (PR-B2 scope, documented honestly): the pre-chokepoint // DISCOVERY code paths (configfile.Load at main.go's early config probe and @@ -389,6 +400,13 @@ func buildWorkspaceGateSet(beadsDir string, extraRoots ...string) ([]workspacega // workspace is selected. It stores the handle in workspaceGateHandle on // success; on the fail-open paths it returns nil with no handle. func acquireCommandWorkspaceGates(ctx context.Context, cmd *cobra.Command, beadsDir string) error { + // A process can execute more than one command in tests and embedded uses. + // Never carry the preceding command's handle into an absent-workspace or + // failed acquisition path, where no new gate is held. Gates outlive the + // store, so close whatever store the preceding command left open before + // its gates drop; this command opens its own only after this point. + closeStoreBeforeGateRelease() + releaseWorkspaceGates() exclusive := commandNeedsExclusiveGate(cmd) // No workspace on disk: nothing to guard, and the store-open path will @@ -423,6 +441,9 @@ func acquireCommandWorkspaceGates(ctx context.Context, cmd *cobra.Command, beads opts = workspacegate.Options{ Wait: sharedWait, IgnoreQueuedExclusive: workspacegate.InheritedSharedHold(), + // readonlyMode is already resolved here. Strict readonly promises not to + // mutate the workspace, so its SHARED hold must not create gate files. + ReadOnly: readonlyMode, } if sharedWait > 0 { opts.OnWait, stopNotice = delayedGateNotice() @@ -472,7 +493,10 @@ func acquireCommandWorkspaceGates(ctx context.Context, cmd *cobra.Command, beads return nil } workspaceGateHandle = h - if !exclusive { + // A strict readonly hold on a never-gated workspace locks nothing, so it + // blocks no exclusive acquirer and gives a child nothing to stand behind: + // advertising it would let the child skip the queue for no reason. + if !exclusive && !h.Empty() { advertiseSharedHold() } return nil diff --git a/cmd/bd/workspace_gate_test.go b/cmd/bd/workspace_gate_test.go index ab85176fe..b55106a3b 100644 --- a/cmd/bd/workspace_gate_test.go +++ b/cmd/bd/workspace_gate_test.go @@ -4,7 +4,9 @@ import ( "context" "errors" "os" + "os/exec" "path/filepath" + "slices" "strconv" "strings" "sync" @@ -13,6 +15,8 @@ import ( "time" "github.com/spf13/cobra" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/uow" "github.com/steveyegge/beads/internal/workspacegate" ) @@ -93,6 +97,10 @@ func TestAcquireCommandWorkspaceGatesAbsentWorkspace(t *testing.T) { t.Cleanup(releaseWorkspaceGates) list := &cobra.Command{Use: "list"} + before := newGateTestWorkspace(t) + if err := acquireCommandWorkspaceGates(context.Background(), list, before); err != nil || workspaceGateHandle == nil { + t.Fatalf("initial command must hold a gate: %v", err) + } missing := filepath.Join(t.TempDir(), "nope", ".beads") if err := acquireCommandWorkspaceGates(context.Background(), list, missing); err != nil { t.Fatalf("absent beadsDir must be silently ungated, got %v", err) @@ -102,6 +110,109 @@ func TestAcquireCommandWorkspaceGatesAbsentWorkspace(t *testing.T) { } } +// A process that runs more than one command (tests, embedders) must not keep +// the previous command's SHARED hold once the next command takes its own +// gates: otherwise the first workspace stays held, and a maintenance +// operation there is turned away by a command that has already finished. +func TestAcquireCommandWorkspaceGatesReleasesThePreviousCommandsHold(t *testing.T) { + resetGateTestEnv(t) + t.Cleanup(releaseWorkspaceGates) + list := &cobra.Command{Use: "list"} + ctx := context.Background() + + first := newGateTestWorkspace(t) + if err := acquireCommandWorkspaceGates(ctx, list, first); err != nil || workspaceGateHandle == nil { + t.Fatalf("first command must hold a gate: %v", err) + } + second := newGateTestWorkspace(t) + if err := acquireCommandWorkspaceGates(ctx, list, second); err != nil || workspaceGateHandle == nil { + t.Fatalf("second command must hold a gate: %v", err) + } + + gate, err := workspacegate.ForWorkspace(first) + if err != nil { + t.Fatal(err) + } + holder, err := gate.Acquire(ctx, workspacegate.Exclusive, workspacegate.Options{Reason: "test maintenance"}) + if err != nil { + t.Fatalf("exclusive acquire on the first command's workspace once the second command holds its own gates: %v, want success", err) + } + _ = holder.Release() +} + +// gateCloseProbeStore and gateCloseProbeProvider stand in for the store and +// provider a previous command left open; Close runs the test's probe. +type gateCloseProbeStore struct { + storage.DoltStorage + onClose func() +} + +func (s *gateCloseProbeStore) Close() error { + s.onClose() + return nil +} + +type gateCloseProbeProvider struct { + uow.UnitOfWorkProvider + onClose func() +} + +func (p *gateCloseProbeProvider) Close(context.Context) error { + p.onClose() + return nil +} + +// When a command's RunE fails, Cobra skips PersistentPostRunE, so the store it +// opened is still open when the same process runs its next command. Before the +// next chokepoint drops the previous command's gates it must close that store +// and provider: gates outlive the store, or an exclusive maintenance operation +// could start against storage that has not quiesced. +func TestAcquireCommandWorkspaceGatesClosesThePreviousCommandsStoreFirst(t *testing.T) { + resetGateTestEnv(t) + t.Cleanup(releaseWorkspaceGates) + originalStore, originalProvider := store, uowProvider + t.Cleanup(func() { store, uowProvider = originalStore, originalProvider }) + list := &cobra.Command{Use: "list"} + ctx := context.Background() + + first := newGateTestWorkspace(t) + if err := acquireCommandWorkspaceGates(ctx, list, first); err != nil || workspaceGateHandle == nil { + t.Fatalf("first command must hold a gate: %v", err) + } + gate, err := workspacegate.ForWorkspace(first) + if err != nil { + t.Fatal(err) + } + // Each close records whether the first command's gate was still held. + var closes []string + probe := func(what string) func() { + return func() { + h, err := gate.Acquire(ctx, workspacegate.Exclusive, workspacegate.Options{}) + switch { + case errors.Is(err, workspacegate.ErrBusy): + closes = append(closes, what+" closed under the gate") + case err == nil: + _ = h.Release() + closes = append(closes, what+" closed after the gate dropped") + default: + t.Errorf("probing the first command's gate when its %s closed: %v", what, err) + } + } + } + store = &gateCloseProbeStore{onClose: probe("store")} + uowProvider = &gateCloseProbeProvider{onClose: probe("provider")} + + if err := acquireCommandWorkspaceGates(ctx, list, newGateTestWorkspace(t)); err != nil { + t.Fatalf("second command: %v", err) + } + if want := []string{"provider closed under the gate", "store closed under the gate"}; !slices.Equal(closes, want) { + t.Fatalf("the previous command's storage: %v, want %v", closes, want) + } + if store != nil || uowProvider != nil { + t.Fatal("the previous command's store and provider must not survive into the next command") + } +} + // holdWorkspaceGateExclusive takes beadsDir's workspace gate exclusively, // as a maintenance operation (init/restore/migrate) would. func holdWorkspaceGateExclusive(t *testing.T, beadsDir string) *workspacegate.Handle { @@ -134,11 +245,15 @@ func countGateNotices(t *testing.T) *atomic.Int32 { func TestAcquireCommandWorkspaceGatesBlockedByExclusiveHolder(t *testing.T) { resetGateTestEnv(t) t.Cleanup(releaseWorkspaceGates) + list := &cobra.Command{Use: "list"} + prior := newGateTestWorkspace(t) + if err := acquireCommandWorkspaceGates(context.Background(), list, prior); err != nil || workspaceGateHandle == nil { + t.Fatalf("initial command must hold a gate: %v", err) + } beadsDir := newGateTestWorkspace(t) t.Setenv(sharedGateWaitEnv, "300ms") holdWorkspaceGateExclusive(t, beadsDir) - list := &cobra.Command{Use: "list"} var err error start := time.Now() stderr := captureStderr(t, func() { @@ -393,6 +508,45 @@ func TestSharedHoldAdvertisedToChildren(t *testing.T) { } } +// A strict readonly command on a never-gated workspace holds nothing, so it +// must not advertise a shared hold: a child trusting the marker would skip the +// writer-fairness queue for a maintenance operation this process does not +// block. Once the gate files exist the strict readonly hold is real, and it is +// advertised like any other. +func TestSharedHoldAdvertisedOnlyWhenAGateIsHeld(t *testing.T) { + resetGateTestEnv(t) + advertiseSharedHoldEnabled = true + t.Cleanup(func() { advertiseSharedHoldEnabled = false }) + t.Cleanup(releaseWorkspaceGates) + beadsDir := newGateTestWorkspace(t) + list := &cobra.Command{Use: "list"} + ctx := context.Background() + + setReadonlyMode(t, true) + if err := acquireCommandWorkspaceGates(ctx, list, beadsDir); err != nil { + t.Fatal(err) + } + if got := os.Getenv(workspacegate.InheritedHoldEnv); got != "" { + t.Fatalf("%s while a strict readonly command holds nothing = %q, want it left empty", workspacegate.InheritedHoldEnv, got) + } + releaseWorkspaceGates() + + // A normal command materializes the gate files. + setReadonlyMode(t, false) + if err := acquireCommandWorkspaceGates(ctx, list, beadsDir); err != nil { + t.Fatal(err) + } + releaseWorkspaceGates() + + setReadonlyMode(t, true) + if err := acquireCommandWorkspaceGates(ctx, list, beadsDir); err != nil { + t.Fatal(err) + } + if got, want := os.Getenv(workspacegate.InheritedHoldEnv), strconv.Itoa(os.Getpid()); got != want { + t.Fatalf("%s while a strict readonly command holds existing gates = %q, want %q", workspacegate.InheritedHoldEnv, got, want) + } +} + func TestSharedGateWaitEnv(t *testing.T) { for _, tc := range []struct { raw, hook string @@ -806,3 +960,180 @@ func TestEnvWithoutSharedHoldMarker(t *testing.T) { t.Fatal("envWithoutSharedHoldMarker dropped unrelated variables") } } + +// setReadonlyMode pins strict --readonly for one test, as PersistentPreRunE +// leaves readonlyMode by the time it acquires the gates. +func setReadonlyMode(t *testing.T, on bool) { + t.Helper() + original := readonlyMode + readonlyMode = on + t.Cleanup(func() { readonlyMode = original }) +} + +// newGateTestServerWorkspace is newGateTestWorkspace for a local-server +// workspace: its physical root is .beads/dolt, not .beads/embeddeddolt. +func newGateTestServerWorkspace(t *testing.T) string { + t.Helper() + beadsDir := filepath.Join(t.TempDir(), ".beads") + if err := os.MkdirAll(filepath.Join(beadsDir, "dolt"), 0o755); err != nil { + t.Fatal(err) + } + meta := `{"backend":"dolt","database":"beads.db","dolt_mode":"server"}` + if err := os.WriteFile(filepath.Join(beadsDir, "metadata.json"), []byte(meta), 0o600); err != nil { + t.Fatal(err) + } + return beadsDir +} + +// dirEntryNames lists dir's immediate entries by name, so a test can prove a +// command left the directory exactly as it found it. +func dirEntryNames(t *testing.T, dir string) []string { + t.Helper() + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatal(err) + } + names := make([]string, 0, len(entries)) + for _, e := range entries { + names = append(names, e.Name()) + } + return names +} + +// Strict --readonly is mutation-free (effectiveRootStorePolicy), so taking the +// command's gates must not create any gate file: not the workspace gate beside +// .beads, and not the physical-root gate inside it (.beads/embeddeddolt for an +// embedded workspace, .beads/dolt for a local server one). +func TestAcquireCommandWorkspaceGatesStrictReadonlyCreatesNoGateFiles(t *testing.T) { + layouts := []struct { + name string + build func(*testing.T) string + }{ + {"embedded", newGateTestWorkspace}, + {"local server", newGateTestServerWorkspace}, + } + for _, tc := range layouts { + t.Run(tc.name, func(t *testing.T) { + resetGateTestEnv(t) + t.Cleanup(releaseWorkspaceGates) + setReadonlyMode(t, true) + beadsDir := tc.build(t) + projectDir := filepath.Dir(beadsDir) + projectBefore, beadsBefore := dirEntryNames(t, projectDir), dirEntryNames(t, beadsDir) + + list := &cobra.Command{Use: "list"} + if err := acquireCommandWorkspaceGates(context.Background(), list, beadsDir); err != nil { + t.Fatalf("strict readonly acquisition on a never-gated workspace: %v", err) + } + // SHARED fail-open also returns nil and creates nothing; it is + // told apart by running with no handle at all. + if workspaceGateHandle == nil { + t.Fatal("strict readonly acquisition on a never-gated workspace ran ungated, want a handle that holds nothing") + } + if !workspaceGateHandle.Empty() { + t.Fatal("strict readonly acquisition on a never-gated workspace locked a gate file") + } + if got := dirEntryNames(t, projectDir); !slices.Equal(got, projectBefore) { + t.Errorf("strict readonly changed %s\n before: %v\n after: %v", projectDir, projectBefore, got) + } + if got := dirEntryNames(t, beadsDir); !slices.Equal(got, beadsBefore) { + t.Errorf("strict readonly changed %s\n before: %v\n after: %v", beadsDir, beadsBefore, got) + } + }) + } +} + +// Not creating a gate file must not mean not honoring one. Once a workspace's +// gate files exist (any normal bd command creates them), a strict readonly +// command still holds them SHARED — so maintenance sees it and refuses to run +// over it — and is still refused while maintenance holds the gate. +func TestAcquireCommandWorkspaceGatesStrictReadonlyStillHonorsExistingGates(t *testing.T) { + resetGateTestEnv(t) + t.Cleanup(releaseWorkspaceGates) + beadsDir := newGateTestWorkspace(t) + list := &cobra.Command{Use: "list"} + ctx := context.Background() + + // A normal command materializes the gate files. + setReadonlyMode(t, false) + if err := acquireCommandWorkspaceGates(ctx, list, beadsDir); err != nil { + t.Fatal(err) + } + releaseWorkspaceGates() + + setReadonlyMode(t, true) + if err := acquireCommandWorkspaceGates(ctx, list, beadsDir); err != nil { + t.Fatalf("strict readonly acquisition against existing gate files: %v", err) + } + gate, err := workspacegate.ForWorkspace(beadsDir) + if err != nil { + t.Fatal(err) + } + if _, err := gate.Acquire(ctx, workspacegate.Exclusive, workspacegate.Options{}); !errors.Is(err, workspacegate.ErrBusy) { + t.Fatalf("exclusive acquire while a strict readonly command is running: err = %v, want ErrBusy", err) + } + releaseWorkspaceGates() + + holder, err := gate.Acquire(ctx, workspacegate.Exclusive, workspacegate.Options{Reason: "test maintenance"}) + if err != nil { + t.Fatalf("exclusive acquire after the strict readonly command released: %v", err) + } + defer func() { _ = holder.Release() }() + // Bound the shared wait: resetGateTestEnv clears it, which means the 30s default. + t.Setenv(sharedGateWaitEnv, "300ms") + if err := acquireCommandWorkspaceGates(ctx, list, beadsDir); err == nil { + t.Fatal("strict readonly acquisition under a foreign exclusive holder must abort, got nil error") + } +} + +// Only STRICT readonly skips creating the gate files. A normal command keeps +// creating them: that is what makes it visible to a maintenance operation that +// starts later (the exclusive acquirer locks the same file), so the first gate +// holder on a workspace cannot go unseen. +func TestAcquireCommandWorkspaceGatesNormalCommandStillCreatesGateFiles(t *testing.T) { + resetGateTestEnv(t) + t.Cleanup(releaseWorkspaceGates) + setReadonlyMode(t, false) + beadsDir := newGateTestWorkspace(t) + + list := &cobra.Command{Use: "list"} + if err := acquireCommandWorkspaceGates(context.Background(), list, beadsDir); err != nil { + t.Fatal(err) + } + for _, gateFile := range []string{ + filepath.Join(filepath.Dir(beadsDir), ".beads.gate.lock"), + filepath.Join(beadsDir, "embeddeddolt.gate.lock"), + } { + if _, err := os.Stat(gateFile); err != nil { + t.Errorf("a normal command must create its gate file: %v", err) + } + } +} + +// End to end: the real `bd --readonly` leaves no gate file behind. The +// in-process tests above pin acquireCommandWorkspaceGates; this one pins that +// the command line reaches it with strict readonly already resolved. +func TestStrictReadonlyCommandCreatesNoGateFiles(t *testing.T) { + bdBin := buildBDForInitTests(t) + beadsDir := newGateTestServerWorkspace(t) + projectDir := filepath.Dir(beadsDir) + + cmd := exec.Command(bdBin, "--readonly", "list") + cmd.Dir = projectDir + cmd.Env = hermeticInitEnv(t.TempDir(), "BEADS_DIR="+beadsDir, "BEADS_DOLT_AUTO_START=0") + // No Dolt server sits behind this workspace, so the command is expected to + // fail when it opens the store; only what it left on disk matters here. + out, _ := cmd.CombinedOutput() + + var gateFiles []string + for _, dir := range []string{projectDir, beadsDir} { + for _, name := range dirEntryNames(t, dir) { + if strings.Contains(name, ".gate.lock") { + gateFiles = append(gateFiles, filepath.Join(dir, name)) + } + } + } + if len(gateFiles) != 0 { + t.Fatalf("bd --readonly created gate files %v\noutput: %s", gateFiles, out) + } +} diff --git a/internal/workspacegate/BUILD.bazel b/internal/workspacegate/BUILD.bazel index 85a5fefdf..4928df002 100644 --- a/internal/workspacegate/BUILD.bazel +++ b/internal/workspacegate/BUILD.bazel @@ -10,7 +10,10 @@ go_library( go_test( name = "workspacegate_test", - srcs = ["gate_test.go"], + srcs = [ + "gate_test.go", + "readonly_wait_test.go", + ], embed = [":workspacegate"], deps = ["//internal/testutil/bazeltest"], ) diff --git a/internal/workspacegate/gate.go b/internal/workspacegate/gate.go index 754c14400..d0d27516c 100644 --- a/internal/workspacegate/gate.go +++ b/internal/workspacegate/gate.go @@ -112,6 +112,15 @@ type Options struct { // effort, from the advisory sidecar). AcquireAll fires it at most // once across the whole gate set. OnWait func(holder string) + // ReadOnly makes a SHARED acquisition free of side effects, for commands + // that promise not to mutate the workspace (strict --readonly). The gate + // file is opened O_RDONLY and never created: a gate file that does not + // exist means nothing has ever been gated here (the reading ExclusiveHolder + // uses), so Acquire succeeds holding nothing. An existing file is held + // SHARED as usual, so maintenance still sees the holder and a live + // exclusive holder still turns it away. Ignored for Exclusive: a + // maintenance operation has to be able to create the file it locks. + ReadOnly bool // IgnoreQueuedExclusive makes a waiting Shared acquisition skip the // writer-fairness check. Set it only when an ANCESTOR process already // holds this gate shared (see InheritedSharedHold): the queued @@ -372,6 +381,11 @@ type Handle struct { // Mode reports how this handle was acquired. func (h *Handle) Mode() Mode { return h.mode } +// Empty reports whether the acquisition locked nothing: a ReadOnly shared +// acquisition of a gate file that does not exist (see Options.ReadOnly). An +// empty handle blocks no acquirer. Release does not change the answer. +func (h *Handle) Empty() bool { return h == nil || h.f == nil } + // Release drops the lock and closes the file handle. For exclusive // handles it also best-effort removes the holder-info sidecar. The gate // file itself is intentionally never removed: deleting a lock file that @@ -382,6 +396,11 @@ func (h *Handle) Release() error { return nil } h.once.Do(func() { + if h.f == nil { + // A ReadOnly acquisition of a gate file that does not exist: + // nothing was locked, so there is nothing to drop. + return + } var errs []error if h.mode == Exclusive { // Remove the sidecar BEFORE unlocking: after the unlock a new @@ -455,6 +474,9 @@ func (h *Handle) Release() error { // Lock order is unchanged in substance: the intent lock of a gate is taken // only while acquiring that gate, so it slots directly before the gate in // AcquireAll's sorted total order and cannot form a cycle. +// +// A Shared acquisition with Options.ReadOnly never creates the gate file: +// see that option for what it holds instead. func (g Gate) Acquire(ctx context.Context, mode Mode, opts Options) (*Handle, error) { if g.path == "" { return nil, errors.New("workspacegate: zero Gate; use ForWorkspace/ForPhysicalRoot") @@ -475,8 +497,20 @@ func (g Gate) Acquire(ctx context.Context, mode Mode, opts Options) (*Handle, er } deadline := time.Now().Add(opts.Wait) - f, err := os.OpenFile(g.path, os.O_CREATE|os.O_RDWR, 0o600) + // An exclusive acquirer always has to create the file it locks, so + // ReadOnly applies to shared acquisition only. + readOnly := opts.ReadOnly && mode == Shared + flag := os.O_CREATE | os.O_RDWR + if readOnly { + flag = os.O_RDONLY + } + f, err := os.OpenFile(g.path, flag, 0o600) if err != nil { + if readOnly && os.IsNotExist(err) { + // No gate file: nothing has ever gated here, so there is no + // holder to see and nothing to hold. + return &Handle{gate: g, mode: mode}, nil + } return nil, fmt.Errorf("workspacegate: open gate %s: %w", g.path, err) } @@ -705,6 +739,21 @@ type MultiHandle struct { handles []*Handle } +// Empty reports whether the acquisition locked no gate at all: every gate in +// the set was a ReadOnly acquisition of a gate file that does not exist (see +// Handle.Empty). +func (m *MultiHandle) Empty() bool { + if m == nil { + return true + } + for _, h := range m.handles { + if !h.Empty() { + return false + } + } + return true +} + // Release drops every held gate in reverse order. Idempotent. func (m *MultiHandle) Release() error { if m == nil { diff --git a/internal/workspacegate/gate_test.go b/internal/workspacegate/gate_test.go index 391a6d50d..eb5ddd100 100644 --- a/internal/workspacegate/gate_test.go +++ b/internal/workspacegate/gate_test.go @@ -322,6 +322,184 @@ func TestExclusiveHolderDoesNotDisturbSharedAcquirers(t *testing.T) { <-done } +// A ReadOnly shared acquisition on a workspace that has never been gated must +// succeed and create nothing: an absent gate file means nothing has ever been +// gated here (the reading ExclusiveHolder already uses), so there is no holder +// to observe and nothing to hold. This is what lets a strict --readonly +// command stay mutation-free. +func TestReadOnlySharedAcquireCreatesNothing(t *testing.T) { + g, dir := testGate(t) + + h, err := g.Acquire(context.Background(), Shared, Options{ReadOnly: true}) + if err != nil { + t.Fatalf("ReadOnly shared Acquire on a never-gated workspace: %v, want success (nothing to hold)", err) + } + for i := 1; i <= 2; i++ { + if err := h.Release(); err != nil { + t.Fatalf("Release #%d of the ReadOnly handle: %v", i, err) + } + } + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatal(err) + } + var created []string + for _, e := range entries { + created = append(created, e.Name()) + } + if len(created) != 0 { + t.Fatalf("ReadOnly Acquire created %v in %s, want nothing", created, dir) + } +} + +// Against a gate file that already exists, a ReadOnly shared acquisition is a +// real shared hold — it coexists with other shared holders, is refused by a +// live exclusive holder, and keeps an exclusive acquirer out until released. +// Dropping the create must not make strict readonly commands invisible to +// maintenance. +func TestReadOnlySharedAcquireHoldsExistingGate(t *testing.T) { + g, _ := testGate(t) + // Materialize the gate file once, as any workspace that has run bd has. + h0 := mustAcquire(t, g, Shared, Options{}) + _ = h0.Release() + + h := mustAcquire(t, g, Shared, Options{ReadOnly: true}) + other := mustAcquire(t, g, Shared, Options{}) + _ = other.Release() + if _, err := g.Acquire(context.Background(), Exclusive, Options{}); !errors.Is(err, ErrBusy) { + t.Fatalf("exclusive under a ReadOnly shared holder: err = %v, want ErrBusy", err) + } + if err := h.Release(); err != nil { + t.Fatalf("Release: %v", err) + } + + hEx := mustAcquire(t, g, Exclusive, Options{Reason: "after the ReadOnly holder left"}) + defer hEx.Release() + if _, err := g.Acquire(context.Background(), Shared, Options{ReadOnly: true}); !errors.Is(err, ErrBusy) { + t.Fatalf("ReadOnly shared under an exclusive holder: err = %v, want ErrBusy", err) + } +} + +// The gate file is opened O_RDONLY, not merely without O_CREATE: a strict +// readonly command in a sandbox whose gate file it cannot write must still be +// able to hold it (flock does not need a writable descriptor). +func TestReadOnlySharedAcquireNeedsNoWriteAccess(t *testing.T) { + if runtime.GOOS == "windows" { + t.Skip("POSIX permission bits do not gate opens on Windows") + } + if os.Geteuid() == 0 { + t.Skip("root ignores file permission bits") + } + g, _ := testGate(t) + h0 := mustAcquire(t, g, Shared, Options{}) + _ = h0.Release() + if err := os.Chmod(g.Path(), 0o400); err != nil { + t.Fatal(err) + } + + h, err := g.Acquire(context.Background(), Shared, Options{ReadOnly: true}) + if err != nil { + t.Fatalf("ReadOnly shared Acquire on a read-only gate file: %v, want success", err) + } + _ = h.Release() +} + +// ReadOnly applies to shared acquisition only: an exclusive acquirer has to be +// able to create the gate file it locks, so the option cannot suppress that. +func TestReadOnlyOptionIgnoredForExclusive(t *testing.T) { + g, _ := testGate(t) + + h, err := g.Acquire(context.Background(), Exclusive, Options{ReadOnly: true, Reason: "readonly is ignored"}) + if err != nil { + t.Fatalf("exclusive Acquire with ReadOnly set: %v", err) + } + defer h.Release() + if _, err := os.Stat(g.Path()); err != nil { + t.Fatalf("exclusive Acquire must create the gate file it locks: %v", err) + } + if _, err := g.Acquire(context.Background(), Shared, Options{}); !errors.Is(err, ErrBusy) { + t.Fatalf("shared under that exclusive holder: err = %v, want ErrBusy", err) + } +} + +// AcquireAll passes the option to every gate: gates whose file exists are held +// shared, gates whose file does not are skipped and stay absent — for the +// workspace gate beside .beads and the physical-root gate inside it alike. +func TestAcquireAllReadOnlySkipsAbsentGates(t *testing.T) { + dir := t.TempDir() + beadsDir := filepath.Join(dir, ".beads") + if err := os.MkdirAll(beadsDir, 0o755); err != nil { + t.Fatal(err) + } + ws, err := ForWorkspace(beadsDir) + if err != nil { + t.Fatal(err) + } + root, err := ForPhysicalRoot(filepath.Join(beadsDir, "embeddeddolt")) + if err != nil { + t.Fatal(err) + } + // Only the workspace gate has ever been used. + h0 := mustAcquire(t, ws, Shared, Options{}) + _ = h0.Release() + + m, err := AcquireAll(context.Background(), Shared, Options{ReadOnly: true}, ws, root) + if err != nil { + t.Fatalf("AcquireAll with ReadOnly over an existing and an absent gate: %v", err) + } + if _, err := ws.Acquire(context.Background(), Exclusive, Options{}); !errors.Is(err, ErrBusy) { + t.Fatalf("exclusive on the existing workspace gate under AcquireAll: err = %v, want ErrBusy", err) + } + if err := m.Release(); err != nil { + t.Fatalf("Release: %v", err) + } + if _, err := os.Lstat(root.Path()); !os.IsNotExist(err) { + t.Fatalf("AcquireAll with ReadOnly created the absent physical-root gate %s (stat err: %v)", root.Path(), err) + } +} + +// Empty separates a ReadOnly acquisition that locked nothing from a real +// hold: callers use it to decide whether there is a hold to tell anyone about. +// One gate file that exists makes the whole set a real hold. +func TestEmptyReportsWhetherAnythingIsLocked(t *testing.T) { + never, _ := testGate(t) // never gated: its file stays absent + used, _ := testGate(t) // gated once, below + readOnly := Options{ReadOnly: true} + + if !(*Handle)(nil).Empty() || !(*MultiHandle)(nil).Empty() { + t.Fatal("a nil handle must report Empty") + } + absent, err := AcquireAll(context.Background(), Shared, readOnly, never, used) + if err != nil { + t.Fatalf("ReadOnly AcquireAll over two absent gates: %v", err) + } + if !absent.Empty() { + t.Fatal("ReadOnly AcquireAll over two absent gates reported a hold, want Empty") + } + _ = absent.Release() + + h0 := mustAcquire(t, used, Shared, Options{}) + if h0.Empty() { + t.Fatal("a plain shared acquisition reported Empty") + } + _ = h0.Release() + if h0.Empty() { + t.Fatal("Release changed Empty; it reports the acquisition, not the current state") + } + + partial, err := AcquireAll(context.Background(), Shared, readOnly, never, used) + if err != nil { + t.Fatalf("ReadOnly AcquireAll over an absent and an existing gate: %v", err) + } + defer partial.Release() + if partial.Empty() { + t.Fatal("ReadOnly AcquireAll holding the existing gate reported Empty") + } + if _, err := os.Lstat(never.Path()); !os.IsNotExist(err) { + t.Fatalf("the absent gate %s must stay absent (stat err: %v)", never.Path(), err) + } +} + // A cancelled context must surface as the context error even in // fail-fast mode (Wait == 0), so callers can tell shutdown from // contention. diff --git a/internal/workspacegate/readonly_wait_test.go b/internal/workspacegate/readonly_wait_test.go new file mode 100644 index 000000000..18b623d06 --- /dev/null +++ b/internal/workspacegate/readonly_wait_test.go @@ -0,0 +1,183 @@ +package workspacegate + +import ( + "context" + "errors" + "os" + "slices" + "testing" + "time" +) + +// A strict-readonly SHARED acquisition (Options.ReadOnly) and a waiting one +// (Options.Wait, with its writer-fairness intent queue) meet in exactly one +// place: a ReadOnly acquisition with Wait > 0. The ReadOnly tests in +// gate_test.go are all wait-less, so the tests below pin that seam: waiting +// must not make a ReadOnly acquisition create anything (no gate file, no +// intent file, no holder-info sidecar), and ReadOnly must not change how long +// a shared acquisition waits for an exclusive holder. +// +// None of them touches maxIntentHold or any other package variable a test +// shortens, and none runs in parallel with anything. + +// gateDirEntries lists dir's entries by name (os.ReadDir sorts them), so a +// test can prove an acquisition left the gate's directory as it found it. +func gateDirEntries(t *testing.T, dir string) []string { + t.Helper() + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatal(err) + } + names := make([]string, 0, len(entries)) + for _, e := range entries { + names = append(names, e.Name()) + } + return names +} + +// (i) The gate file exists and no intent file does: a waiting ReadOnly shared +// acquisition takes the gate on its first attempt (OnWait never fires) and +// creates nothing beside it: no .intent, no .intent.info, no .info. +func TestReadOnlyWaitingSharedAcquireOfExistingGateCreatesNothing(t *testing.T) { + g, dir := testGate(t) + // Materialize the gate file the way any ordinary command does. + _ = mustAcquire(t, g, Shared, Options{}).Release() + before := gateDirEntries(t, dir) + if _, err := os.Stat(g.Path()); err != nil { + t.Fatalf("precondition: the gate file must exist: %v", err) + } + if _, err := os.Stat(g.intentPath()); !os.IsNotExist(err) { + t.Fatalf("precondition: no intent file may exist, stat err = %v", err) + } + + const wait = time.Second + var waited []string + start := time.Now() + h, err := g.Acquire(context.Background(), Shared, Options{ + ReadOnly: true, + Wait: wait, + OnWait: func(holder string) { waited = append(waited, holder) }, + }) + elapsed := time.Since(start) + if err != nil { + t.Fatalf("waiting ReadOnly shared Acquire of a free, existing gate: %v", err) + } + defer func() { _ = h.Release() }() + if len(waited) != 0 { + t.Errorf("the first attempt did not take the free gate: OnWait fired with %q", waited) + } + if elapsed >= wait { + t.Errorf("took %s, want far under its %s budget: nothing was holding the gate", elapsed, wait) + } + if h.f == nil { + t.Fatal("handle holds no file: against an existing gate file a ReadOnly acquisition is a real shared hold") + } + if got := gateDirEntries(t, dir); !slices.Equal(got, before) { + t.Errorf("waiting ReadOnly acquisition changed %s\n before: %v\n after: %v", dir, before, got) + } + // It is a real shared hold: a maintenance operation is turned away. + if _, err := g.Acquire(context.Background(), Exclusive, Options{}); !errors.Is(err, ErrBusy) { + t.Fatalf("exclusive Acquire under the ReadOnly shared holder: err = %v, want ErrBusy", err) + } + if err := h.Release(); err != nil { + t.Fatalf("Release: %v", err) + } + if got := gateDirEntries(t, dir); !slices.Equal(got, before) { + t.Errorf("after release %s\n before: %v\n after: %v", dir, before, got) + } +} + +// (ii) No gate file: nothing has ever gated here. A waiting ReadOnly shared +// acquisition returns at once with an empty handle (it holds no file and has +// nothing to wait for) and leaves the directory exactly as it was: it neither +// creates the gate file nor waits for one to appear. +func TestReadOnlyWaitingSharedAcquireOfMissingGateHoldsNothing(t *testing.T) { + g, dir := testGate(t) + before := gateDirEntries(t, dir) + if len(before) != 0 { + t.Fatalf("precondition: %s must start empty, has %v", dir, before) + } + + const wait = time.Second + var waited []string + start := time.Now() + h, err := g.Acquire(context.Background(), Shared, Options{ + ReadOnly: true, + Wait: wait, + OnWait: func(holder string) { waited = append(waited, holder) }, + }) + elapsed := time.Since(start) + if err != nil { + t.Fatalf("waiting ReadOnly shared Acquire with no gate file: %v, want success holding nothing", err) + } + if h == nil || h.f != nil { + t.Fatalf("handle = %+v, want an empty handle (no gate file was opened)", h) + } + if h.Mode() != Shared { + t.Errorf("handle mode = %s, want %s", h.Mode(), Shared) + } + if len(waited) != 0 { + t.Errorf("OnWait fired with %q: with no gate file there is nothing to wait for", waited) + } + if elapsed >= wait { + t.Errorf("took %s, want far under its %s budget: a missing gate file is answered before any retry", elapsed, wait) + } + if got := gateDirEntries(t, dir); !slices.Equal(got, before) { + t.Errorf("waiting ReadOnly acquisition changed %s\n before: %v\n after: %v", dir, before, got) + } + for i := 1; i <= 2; i++ { + if err := h.Release(); err != nil { + t.Fatalf("Release #%d of the empty handle: %v", i, err) + } + } + if got := gateDirEntries(t, dir); !slices.Equal(got, before) { + t.Errorf("after release %s\n before: %v\n after: %v", dir, before, got) + } +} + +// (iii) The gate file exists and is held EXCLUSIVELY: a waiting ReadOnly +// shared acquisition waits out its own budget, exactly like any shared +// acquisition, and then fails with ErrBusy. ReadOnly is not special-cased: +// it neither fails at once nor waits longer (the shared default is 30s, the +// init budget 60s). It also creates nothing while it waits, which is where +// the fairness probe (ExclusiveQueued) meets ReadOnly: the probe opens the +// intent file read-only and treats a missing one as "nothing queued". +func TestReadOnlyWaitingSharedAcquireUnderExclusiveHolderWaitsItsBudget(t *testing.T) { + g, dir := testGate(t) + holder := mustAcquire(t, g, Exclusive, Options{Reason: "test maintenance"}) + defer func() { _ = holder.Release() }() + before := gateDirEntries(t, dir) + + const wait = 300 * time.Millisecond + waited := 0 + start := time.Now() + h, err := g.Acquire(context.Background(), Shared, Options{ + ReadOnly: true, + Wait: wait, + OnWait: func(string) { waited++ }, + }) + elapsed := time.Since(start) + if !errors.Is(err, ErrBusy) { + if h != nil { + _ = h.Release() + } + t.Fatalf("waiting ReadOnly shared Acquire under an exclusive holder: err = %v, want ErrBusy", err) + } + // Acquire's deadline is taken after start, and it reports ErrBusy only + // once that deadline has passed, so this lower bound is exact. + if elapsed < wait { + t.Errorf("gave up after %s, before its %s budget ran out: ReadOnly must wait like any shared acquisition", elapsed, wait) + } + // The loop overshoots by at most one poll interval (100ms by default). + // The ceiling is far more generous than that so a loaded host cannot make + // this flaky; it still catches any special-cased budget. + if limit := wait + 2*time.Second; elapsed > limit { + t.Errorf("gave up after %s, want about its %s budget (at most %s)", elapsed, wait, limit) + } + if waited != 1 { + t.Errorf("OnWait fired %d times, want once (after the first busy attempt)", waited) + } + if got := gateDirEntries(t, dir); !slices.Equal(got, before) { + t.Errorf("waiting ReadOnly acquisition changed %s\n before: %v\n after: %v", dir, before, got) + } +} From 3a5ade3a76e2d6c70395b93255ce3914d1200c39 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 19 Oct 2026 04:51:18 +0000 Subject: [PATCH 09/41] test(cmd/bd): pin workspace discovery refusal text and exit codes (#7677) * test(cmd/bd): pin workspace discovery refusal text and exit codes Add before-parity pins for what an ordinary bd prints, and with which exit code, when workspace discovery cannot select a usable workspace: - no .beads, an empty .beads, or a .beads with config.yaml but no metadata.json and no database ("no beads database found", and bd where's text and --json refusals); - a corrupt metadata.json, for list, create, where (all refuse) and version (warns, exits 0); - an exported BEADS_DIR naming a missing path, an empty directory, a regular file or a corrupt workspace, run from a corrupt workspace so a fallback to cwd discovery would show; - .beads/.env: BEADS_DIR routing, an exported BEADS_DIR winning, BEADS_DB routing bd where but not a data command, BD_JSON applying only in the selected workspace, and a .env BD_BACKEND not being refused; - .beads/redirect: missing and non-workspace targets (warning, then the source), a corrupt target, a chain, and a redirect behind BEADS_DIR; - --db paths and names, and BEADS_DB / BD_DB, including the two inconsistencies main has today (a data command ignores BD_DB, and lets a discoverable cwd workspace shadow BEADS_DB, while bd where honors both); - BD_BACKEND / BD_DATABASE_BACKEND being refused before any of the above, for store-less commands too. The cases use workspaces with a corrupt metadata.json as sentinels: the refusal names the metadata.json it failed to parse, which shows the directory discovery selected without opening a store, so the store-less pins need neither Dolt nor cgo. One embedded-Dolt pin covers the store-backed .env case: BD_READONLY=true in .beads/.env refuses a write with the read-only text and exit 1, reads still run, and an exported BD_READONLY wins over the file. These pin today's origin/main, not an ideal. A change that reorders admission, discovery or checkBlockedEnvVars in the root PersistentPreRunE now has to change an expectation here and is reviewed as a test diff. Built from the merge base with the graph fork (46aa59cc65), bd passes every store-less pin; the fork tip fails two (a BEADS_DIR naming a regular file exits 5 with graph_not_initialized instead of exit 1, also when BD_BACKEND is set). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(cmd/bd): keep the discovery pins independent of PATH and cgo The version case under TestDiscoveryRefusalPinsCorruptMetadata pinned bd version's full stderr. bd version also appends a "Warning: multiple 'bd' binaries found in PATH" block when the caller's PATH holds more than one distinct bd binary (a release install plus a dev build, or Homebrew plus go install), and the fixture passes the caller's PATH through. Such a machine failed the pin spuriously. normalize now drops that block from stderr, as it already drops the sandbox auto-detection notice, so the rest of the case is still pinned exactly. The file also said the store-less pins need no cgo. They need a cgo bd. A CGO_ENABLED=0 bd is server-only, and its no-database hints add "run 'bd doctor' to diagnose", so a CGO_ENABLED=0 go test of the package failed 16 cases on the hint wording alone. The file now builds only with cgo, and its header says why. CI runs none of these in its CGO_ENABLED=0 lane, whose -run subset is fixed by PURE_CMD_BD_TESTS. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(cmd/bd): pin a working-directory .beads that is a regular file The ExplicitBeadsDir pins cover a BEADS_DIR that names a regular file. This adds the same case reached by the working-directory walk instead: a project whose .beads is a regular file. bd list and bd where both exit 1 with "Error: loading config: reading config: open $ROOT/p/.beads/metadata.json: not a directory". bd built from the fork merge base 46aa59cc65 passes both cases. The fork tip b9eebc31f8 fails both: it exits 5 with "graph_not_initialized: open $ROOT/p/.beads/graph-preview-format: not a directory", the same reorder the BEADS_DIR pin already catches. A .beads that is an unreadable directory (chmod 000) splits main and the fork the same way, but it is not pinned: a test run as root can still read the directory. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(cmd/bd): close the embedded discovery pin's environment Maintainer fixup from the adoption review of #7677. TestEmbeddedDiscoveryPinsEnvFileReadonly compares bd's output exactly, but its run helper started from bdEnv, which drops only BEADS_* and BD_ROUTING_MODE. Every other BD_* the caller exported reached bd: with BD_OTEL_METRICS_URL/BD_OTEL_LOGS_URL set (as on agent hosts) the deprecation warning lands in front of the read-only refusal, with BD_JSON_ENVELOPE=1 the empty list is printed as an envelope, and in a sandbox the "Sandbox detected" notice precedes the refusal; each failed the pin. The run helper now uses the store-less pins' closed environment, factored out of discoveryPinFixture.baseEnv as discoveryPinEnv (HOME stays the directory bd init used), and drops the sandbox notice from stderr with the same discoveryPinSandboxNotice pattern normalize now uses. normalize collapsed every consecutive repeat of a stderr line, which also hid a doubled "Error:" line. It now folds only repeated "Warning: redirect" lines, the ones discovery prints once per re-resolve. blocked-backend-not-checked used a corrupt-metadata sentinel, whose bind-time refusal comes before the legacy guard, so a blocked-variable check added after .env loading would not have changed its output. It now uses a config.yaml-only workspace and pins discovery's no-database refusal. Measured with the PR head's bd and test binaries under env -i: - the embedded pin passes on a clean environment and failed with BD_OTEL_*, BD_JSON_ENVELOPE=1, or kill(2) returning EPERM (strace inject); with this change it passes in all of them, plus an exported XDG_CONFIG_HOME, an existing BD_MIGRATION_FREEZE_FILE and the agent host's BD_*/BEADS_* set; - it still fails when .beads/.env overrides the caller (gotenv.OverLoad) or is not loaded; - the store-less suite passes 54/54 subtests; a bd that prints the legacy-guard error twice now fails 12 of them, and a bd that checks the blocked variables again after the legacy guard now fails blocked-backend-not-checked; both passed the suite before. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com> --- .github/scripts/embedded-cmd-test-shards.txt | 1 + cmd/bd/BUILD.bazel | 2 + .../discovery_refusal_pins_embedded_test.go | 86 ++ cmd/bd/discovery_refusal_pins_test.go | 885 ++++++++++++++++++ 4 files changed, 974 insertions(+) create mode 100644 cmd/bd/discovery_refusal_pins_embedded_test.go create mode 100644 cmd/bd/discovery_refusal_pins_test.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index e05f241a2..95d0ef0f0 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -134,6 +134,7 @@ 100 53 TestEmbeddedIfRevisionOrdinaryCAS 100 53 TestEmbeddedInitSelectedExcludeRoutingStealthDecoyQuiet +100 54 TestEmbeddedDiscoveryPinsEnvFileReadonly 100 54 TestEmbeddedFormatJSONAliasMatchesJSON 100 54 TestEmbeddedSearchConcurrent diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index f16e65cbb..3ac61dc7c 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -714,6 +714,8 @@ go_test( "dependency_hierarchy_embedded_test.go", "diff_embedded_test.go", "directory_flag_test.go", + "discovery_refusal_pins_embedded_test.go", + "discovery_refusal_pins_test.go", "doctor_backend_guard_test.go", "doctor_check_gate_test.go", "doctor_context_test.go", diff --git a/cmd/bd/discovery_refusal_pins_embedded_test.go b/cmd/bd/discovery_refusal_pins_embedded_test.go new file mode 100644 index 000000000..5bfda4176 --- /dev/null +++ b/cmd/bd/discovery_refusal_pins_embedded_test.go @@ -0,0 +1,86 @@ +//go:build cgo + +package main + +import ( + "bytes" + "encoding/json" + "errors" + "os" + "os/exec" + "path/filepath" + "testing" +) + +// TestEmbeddedDiscoveryPinsEnvFileReadonly is the store-backed member of the +// before-parity discovery pins (discovery_refusal_pins_test.go holds the +// store-less ones). It pins how an ordinary workspace applies a BD_READONLY +// supplied by its own .beads/.env: a write is refused with the read-only text +// and exit 1, also under --json, while reads still run; and a BD_READONLY the +// caller exports wins over the file. +func TestEmbeddedDiscoveryPinsEnvFileReadonly(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "pin") + if err := os.WriteFile(filepath.Join(beadsDir, ".env"), []byte("BD_READONLY=true\n"), 0o600); err != nil { + t.Fatalf("write .beads/.env: %v", err) + } + + run := func(extraEnv []string, args ...string) (exit int, stdout, stderr string) { + t.Helper() + cmd := exec.Command(bd, args...) + cmd.Dir = dir + // The store-less pins' closed environment: no BD_*/BEADS_* the caller + // exports (BD_READONLY, BD_JSON_ENVELOPE, BD_OTEL_*, ...) reaches bd. + // HOME stays dir, where bd init ran. + cmd.Env = append(discoveryPinEnv(dir, filepath.Join(dir, "no-such-freeze-marker")), + "BEADS_DOLT_AUTO_START=0", "BEADS_NO_DAEMON=1") + cmd.Env = append(cmd.Env, extraEnv...) + var out, errOut bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &errOut + if err := cmd.Run(); err != nil { + var exitErr *exec.ExitError + if !errors.As(err, &exitErr) { + t.Fatalf("run bd %v: %v", args, err) + } + exit = exitErr.ExitCode() + } + return exit, out.String(), discoveryPinSandboxNotice.ReplaceAllLiteralString(errOut.String(), "") + } + + exit, stdout, stderr := run(nil, "create", "refused by .env", "--json") + if exit != 1 || stdout != "" || stderr != "Error: operation 'create' is not allowed in read-only mode\n" { + t.Fatalf("create under .env BD_READONLY=true: exit=%d stdout=%q stderr=%q; want exit=1, no stdout, the read-only refusal", exit, stdout, stderr) + } + + exit, stdout, stderr = run(nil, "list", "--json") + if exit != 0 || stdout != "[]\n" { + t.Fatalf("list under .env BD_READONLY=true: exit=%d stdout=%q stderr=%q; want exit=0 and an empty list", exit, stdout, stderr) + } + + exit, stdout, stderr = run([]string{"BD_READONLY=false"}, "create", "allowed by the caller", "--json") + if exit != 0 { + t.Fatalf("create with exported BD_READONLY=false: exit=%d stdout=%q stderr=%q; want the caller's setting to win", exit, stdout, stderr) + } + var created struct { + Title string `json:"title"` + } + if err := json.Unmarshal([]byte(stdout), &created); err != nil || created.Title != "allowed by the caller" { + t.Fatalf("create with exported BD_READONLY=false printed %q (decode error %v)", stdout, err) + } + + exit, stdout, stderr = run(nil, "list", "--json") + var listed []struct { + Title string `json:"title"` + } + if exit != 0 { + t.Fatalf("list after the allowed create: exit=%d stderr=%q", exit, stderr) + } + if err := json.Unmarshal([]byte(stdout), &listed); err != nil || len(listed) != 1 || listed[0].Title != "allowed by the caller" { + t.Fatalf("list after the allowed create = %q (decode error %v); want only the allowed issue, the refused create wrote nothing", stdout, err) + } +} diff --git a/cmd/bd/discovery_refusal_pins_test.go b/cmd/bd/discovery_refusal_pins_test.go new file mode 100644 index 000000000..c0d4254f9 --- /dev/null +++ b/cmd/bd/discovery_refusal_pins_test.go @@ -0,0 +1,885 @@ +//go:build cgo + +package main + +// Before-parity pins for workspace discovery refusals. +// +// These tests hold the exact text and exit code an ordinary bd prints when +// workspace discovery cannot select a usable workspace: a missing or corrupt +// metadata.json, an explicit BEADS_DIR that names no workspace, selection +// supplied by .beads/.env, .beads/redirect files, --db / BEADS_DB / BD_DB +// selection, and the blocked backend environment variables. They pin what +// origin/main does today, not what it ideally should do. A change that +// reorders admission, discovery or checkBlockedEnvVars in the root +// PersistentPreRunE has to change an expectation here, so it shows up in +// review as a test diff rather than as an unexplained behavior change. +// +// Every case runs the real bd binary in a fresh fixture and stops before any +// store is opened, so these need no Dolt server or database. Fixture +// workspaces with a corrupt metadata.json double as sentinels: the refusal +// names the metadata.json it failed to parse, so the text shows which +// directory discovery selected without opening a database. +// +// The pinned no-database hints are an embedded-capable bd's. A CGO_ENABLED=0 +// bd is server-only and adds "run 'bd doctor' to diagnose" to them, so this +// file builds only with cgo, and the bd it runs (built by +// buildBDForInitTests, or BEADS_TEST_BD_BINARY) must be a cgo build too. + +import ( + "bytes" + "errors" + "os" + "os/exec" + "path/filepath" + "regexp" + "runtime" + "sort" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/testutil" +) + +// pinNoDatabaseRefusal is what a store-requiring command prints when +// discovery finds no workspace with a database. +const pinNoDatabaseRefusal = "Error: no beads database found\n" + + "Hint: run 'bd where' to inspect the resolved workspace, or 'bd init' to create a new database\n" + + " or set BEADS_DIR to point to your .beads directory\n" + +// pinNoWorkspaceWhereRefusal is bd where's text refusal when no workspace is +// selected. +const pinNoWorkspaceWhereRefusal = "Error: No active beads workspace found.\n" + + "Hint: check BEADS_DIR/worktree setup, or run 'bd init' to create a new database\n" + +// pinNoWorkspaceWhereJSON is bd where --json's refusal. It goes to stdout. +const pinNoWorkspaceWhereJSON = `{ + "error": "no_beads_directory", + "hint": "check BEADS_DIR/worktree setup, or run 'bd init' to create a new database", + "message": "No active beads workspace found.", + "schema_version": 1 +} +` + +// pinCorruptMetadata is the content of every corrupt metadata.json fixture. +const pinCorruptMetadata = "{\n" + +// pinCorruptMetadataWarning is the warning printed while the selected +// workspace is bound, before the refusal itself. +func pinCorruptMetadataWarning(beadsDir string) string { + return "warning: load " + beadsDir + "/metadata.json: parsing config: unexpected end of JSON input; " + + "no storage database was opened or modified (storage mode unknown; data commands refuse to fall back to the embedded store)\n" +} + +// pinCorruptMetadataRefusal is the full stderr of a command refused because +// the selected workspace's metadata.json does not parse. +func pinCorruptMetadataRefusal(beadsDir string) string { + return pinCorruptMetadataWarning(beadsDir) + + "Error: loading config: parsing config: unexpected end of JSON input\n" +} + +// pinBlockedEnvRefusal is checkBlockedEnvVars' refusal for name. +func pinBlockedEnvRefusal(name string) string { + return "Error: " + name + " env var is not supported and has been removed to prevent data fragmentation.\n" + + "Unset " + name + "; storage selection comes from .beads/metadata.json. " + + "To choose a different supported backend, follow 'bd help init-safety'; do not edit metadata.json by hand\n" +} + +// discoveryPinCase is one bd invocation and its exact expected result. The +// arguments and the expected strings write the fixture root as $ROOT. +type discoveryPinCase struct { + name string + // setup builds the fixture and returns the working directory and any + // environment the invocation adds to the hermetic base. + setup func(f *discoveryPinFixture) (cwd string, env []string) + args []string + exit int + stdout string + stderr string + // stderrPrefix compares only the start of stderr. Use it only where the + // rest of the output is not part of the pin. + stderrPrefix bool +} + +func runDiscoveryPinCases(t *testing.T, cases []discoveryPinCase) { + t.Helper() + if runtime.GOOS == "windows" { + t.Skip("the pinned texts embed POSIX paths") + } + t.Parallel() + bd := buildBDForInitTests(t) + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + t.Parallel() + f := newDiscoveryPinFixture(t) + cwd, env := tc.setup(f) + args := make([]string, len(tc.args)) + for i, arg := range tc.args { + args[i] = strings.ReplaceAll(arg, "$ROOT", f.root) + } + exit, stdout, stderr := f.run(bd, cwd, env, args...) + if exit != tc.exit { + t.Errorf("bd %s: exit = %d, want %d", strings.Join(tc.args, " "), exit, tc.exit) + } + if stdout != tc.stdout { + t.Errorf("bd %s: stdout differs\n got: %q\nwant: %q", strings.Join(tc.args, " "), stdout, tc.stdout) + } + gotStderr := stderr + if tc.stderrPrefix && len(gotStderr) > len(tc.stderr) { + gotStderr = gotStderr[:len(tc.stderr)] + } + if gotStderr != tc.stderr { + t.Errorf("bd %s: stderr differs\n got: %q\nwant: %q", strings.Join(tc.args, " "), stderr, tc.stderr) + } + }) + } +} + +type discoveryPinFixture struct { + t *testing.T + root string + forms []string // spellings of root to rewrite as $ROOT, longest first +} + +func newDiscoveryPinFixture(t *testing.T) *discoveryPinFixture { + t.Helper() + root := t.TempDir() + forms := []string{root} + if resolved, err := filepath.EvalSymlinks(root); err == nil && resolved != root { + forms = append(forms, resolved) + } + sort.Slice(forms, func(i, j int) bool { return len(forms[i]) > len(forms[j]) }) + f := &discoveryPinFixture{t: t, root: root, forms: forms} + f.mkdir("home") + return f +} + +// path returns the absolute path of a root-relative slash path. +func (f *discoveryPinFixture) path(rel string) string { + return filepath.Join(f.root, filepath.FromSlash(rel)) +} + +func (f *discoveryPinFixture) mkdir(rel string) string { + f.t.Helper() + dir := f.path(rel) + if err := os.MkdirAll(dir, 0o700); err != nil { + f.t.Fatalf("mkdir %s: %v", dir, err) + } + return dir +} + +func (f *discoveryPinFixture) write(rel, content string) { + f.t.Helper() + if err := os.WriteFile(f.path(rel), []byte(content), 0o600); err != nil { + f.t.Fatalf("write %s: %v", rel, err) + } +} + +// project creates rel as a git repository with no .beads. The repository +// root is the discovery walk's boundary, so nothing above the fixture leaks +// into a case. +func (f *discoveryPinFixture) project(rel string) string { + f.t.Helper() + dir := f.mkdir(rel) + initGitRepoAt(f.t, dir) + return dir +} + +// workspace creates project rel with a .beads directory holding files, each +// named relative to .beads. +func (f *discoveryPinFixture) workspace(rel string, files map[string]string) string { + f.t.Helper() + dir := f.project(rel) + f.mkdir(rel + "/.beads") + for name, content := range files { + f.write(rel+"/.beads/"+name, content) + } + return dir +} + +// corruptWorkspace creates project rel whose metadata.json does not parse, +// plus any extra .beads files. +func (f *discoveryPinFixture) corruptWorkspace(rel string, extra map[string]string) string { + f.t.Helper() + files := map[string]string{"metadata.json": pinCorruptMetadata} + for name, content := range extra { + files[name] = content + } + return f.workspace(rel, files) +} + +// yamlWorkspace creates project rel whose .beads holds only a config.yaml +// (no metadata.json and no database), plus any extra .beads files. +func (f *discoveryPinFixture) yamlWorkspace(rel string, extra map[string]string) string { + f.t.Helper() + files := map[string]string{"config.yaml": "# discovery pin fixture\n"} + for name, content := range extra { + files[name] = content + } + return f.workspace(rel, files) +} + +// discoveryPinEnv is the caller environment without any BEADS_*/BD_* +// selector or setting and without git repository-location variables, with +// HOME and XDG_CONFIG_HOME under home, metrics and event flush off, and the +// migration freeze marker pointed at freezeMarker, a path that does not exist. +func discoveryPinEnv(home, freezeMarker string) []string { + var env []string + for _, kv := range testutil.WithoutGitRepoLocationEnv(os.Environ()) { + key, _, _ := strings.Cut(kv, "=") + if strings.HasPrefix(key, "BEADS_") || strings.HasPrefix(key, "BD_") || + key == "HOME" || key == "XDG_CONFIG_HOME" { + continue + } + env = append(env, kv) + } + return append(env, + "HOME="+home, + "XDG_CONFIG_HOME="+filepath.Join(home, ".config"), + "BD_DISABLE_METRICS=1", + "BD_DISABLE_EVENT_FLUSH=1", + "BD_MIGRATION_FREEZE_FILE="+freezeMarker, + ) +} + +func (f *discoveryPinFixture) run(bd, cwd string, extraEnv []string, args ...string) (exit int, stdout, stderr string) { + f.t.Helper() + cmd := exec.Command(bd, args...) + cmd.Dir = cwd + cmd.Env = append(discoveryPinEnv(f.path("home"), f.path("no-such-freeze-marker")), extraEnv...) + var out, errOut bytes.Buffer + cmd.Stdout, cmd.Stderr = &out, &errOut + if err := cmd.Run(); err != nil { + var exitErr *exec.ExitError + if !errors.As(err, &exitErr) { + f.t.Fatalf("run bd %s: %v", strings.Join(args, " "), err) + } + exit = exitErr.ExitCode() + } + return exit, f.normalize(out.String(), false), f.normalize(errOut.String(), true) +} + +var discoveryPinVersionLine = regexp.MustCompile(`(?m)^bd version .*$`) + +// discoveryPinDuplicateBinaries matches the warning bd version appends when +// the caller's PATH holds more than one distinct bd binary. +var discoveryPinDuplicateBinaries = regexp.MustCompile( + `\nWarning: multiple 'bd' binaries found in PATH:\n(?: [^\n]*\n)*` + + `The first one is being used\. Remove duplicates to avoid confusion\.\n`) + +// discoveryPinSandboxNotice matches the notice bd prints when it auto-detects +// a sandbox, which it does when it is not allowed to signal its own process. +var discoveryPinSandboxNotice = regexp.MustCompile(`(?m)^.*Sandbox detected, using direct mode\n`) + +// normalize rewrites the fixture root as $ROOT and the build's version line +// as "bd version $VERSION". For stderr it also drops the notices that depend +// on the caller's environment rather than on the fixture (sandbox +// auto-detection, and bd version's duplicate binaries in PATH) and collapses +// consecutive repeats of a "Warning: redirect" line: discovery re-resolves +// the same redirect several times per invocation and how many times it warns +// is not part of the pin. Any other repeated line, such as a doubled +// "Error:", is kept. +func (f *discoveryPinFixture) normalize(s string, stderr bool) string { + for _, form := range f.forms { + s = strings.ReplaceAll(s, form, "$ROOT") + } + s = discoveryPinVersionLine.ReplaceAllLiteralString(s, "bd version $VERSION") + if !stderr { + return s + } + s = discoveryPinDuplicateBinaries.ReplaceAllLiteralString(s, "") + s = discoveryPinSandboxNotice.ReplaceAllLiteralString(s, "") + var b strings.Builder + prev := "" + for _, line := range strings.SplitAfter(s, "\n") { + if line == prev && strings.HasPrefix(line, "Warning: redirect ") { + continue + } + b.WriteString(line) + prev = line + } + return b.String() +} + +// TestDiscoveryRefusalPinsMissingMetadata pins discovery when no workspace, +// a workspace without metadata.json and without a database, or a .beads that +// is a regular file is found. +func TestDiscoveryRefusalPinsMissingMetadata(t *testing.T) { + noBeads := func(f *discoveryPinFixture) (string, []string) { return f.project("p"), nil } + regularFileBeads := func(f *discoveryPinFixture) (string, []string) { + cwd := f.project("p") + f.write("p/.beads", "not a workspace\n") + return cwd, nil + } + runDiscoveryPinCases(t, []discoveryPinCase{ + { + name: "no-beads-dir/list", + setup: noBeads, + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + // --json does not turn the refusal into a JSON document. + name: "no-beads-dir/list-json", + setup: noBeads, + args: []string{"list", "--json"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + name: "no-beads-dir/where", + setup: noBeads, + args: []string{"where"}, + exit: 1, + stderr: pinNoWorkspaceWhereRefusal, + }, + { + name: "no-beads-dir/where-json", + setup: noBeads, + args: []string{"where", "--json"}, + exit: 1, + stdout: pinNoWorkspaceWhereJSON, + }, + { + name: "empty-beads-dir/list", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.workspace("p", nil), nil + }, + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + name: "config-yaml-only/list", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.yamlWorkspace("p", nil), nil + }, + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + // The same directory is a workspace to bd where. + name: "config-yaml-only/where", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.yamlWorkspace("p", nil), nil + }, + args: []string{"where"}, + stdout: "$ROOT/p/.beads\n", + }, + { + // The working-directory walk selects a .beads that is a regular + // file, then fails to read its metadata.json (the BEADS_DIR form + // is in TestDiscoveryRefusalPinsExplicitBeadsDir). + name: "beads-is-regular-file/list", + setup: regularFileBeads, + args: []string{"list"}, + exit: 1, + stderr: "Error: loading config: reading config: open $ROOT/p/.beads/metadata.json: not a directory\n", + }, + { + name: "beads-is-regular-file/where", + setup: regularFileBeads, + args: []string{"where"}, + exit: 1, + stderr: "Error: loading config: reading config: open $ROOT/p/.beads/metadata.json: not a directory\n", + }, + }) +} + +// TestDiscoveryRefusalPinsCorruptMetadata pins the refusal for a discovered +// workspace whose metadata.json does not parse. +func TestDiscoveryRefusalPinsCorruptMetadata(t *testing.T) { + corrupt := func(f *discoveryPinFixture) (string, []string) { return f.corruptWorkspace("p", nil), nil } + runDiscoveryPinCases(t, []discoveryPinCase{ + { + name: "list", + setup: corrupt, + args: []string{"list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/p/.beads"), + }, + { + name: "list-json", + setup: corrupt, + args: []string{"list", "--json"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/p/.beads"), + }, + { + name: "create", + setup: corrupt, + args: []string{"create", "must not be written", "--json"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/p/.beads"), + }, + { + // bd where never opens a store, but it still refuses. + name: "where", + setup: corrupt, + args: []string{"where"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/p/.beads"), + }, + { + name: "where-json", + setup: corrupt, + args: []string{"where", "--json"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/p/.beads"), + }, + { + // bd version warns and succeeds. + name: "version", + setup: corrupt, + args: []string{"version"}, + stdout: "bd version $VERSION\n", + stderr: pinCorruptMetadataWarning("$ROOT/p/.beads"), + }, + }) +} + +// TestDiscoveryRefusalPinsExplicitBeadsDir pins an exported BEADS_DIR that +// names no workspace. Each case runs from a corrupt workspace, so falling +// back to discovery from the working directory would print that workspace's +// refusal instead. +func TestDiscoveryRefusalPinsExplicitBeadsDir(t *testing.T) { + from := func(beadsDir func(f *discoveryPinFixture) string) func(f *discoveryPinFixture) (string, []string) { + return func(f *discoveryPinFixture) (string, []string) { + cwd := f.corruptWorkspace("cwd", nil) + return cwd, []string{"BEADS_DIR=" + beadsDir(f)} + } + } + missing := from(func(f *discoveryPinFixture) string { return f.path("missing/.beads") }) + runDiscoveryPinCases(t, []discoveryPinCase{ + { + name: "missing-dir/list", + setup: missing, + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + name: "missing-dir/where", + setup: missing, + args: []string{"where"}, + exit: 1, + stderr: pinNoWorkspaceWhereRefusal, + }, + { + name: "missing-dir/where-json", + setup: missing, + args: []string{"where", "--json"}, + exit: 1, + stdout: pinNoWorkspaceWhereJSON, + }, + { + name: "empty-dir/list", + setup: from(func(f *discoveryPinFixture) string { return f.mkdir("empty") }), + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + name: "regular-file/list", + setup: from(func(f *discoveryPinFixture) string { + f.write("file", "not a workspace\n") + return f.path("file") + }), + args: []string{"list"}, + exit: 1, + stderr: "Error: loading config: reading config: open $ROOT/file/metadata.json: not a directory\n", + }, + { + name: "corrupt-workspace/list", + setup: from(func(f *discoveryPinFixture) string { + f.corruptWorkspace("other", nil) + return f.path("other/.beads") + }), + args: []string{"list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/other/.beads"), + }, + }) +} + +// TestDiscoveryRefusalPinsBeadsEnvFile pins settings supplied by the +// discovered workspace's .beads/.env. +func TestDiscoveryRefusalPinsBeadsEnvFile(t *testing.T) { + runDiscoveryPinCases(t, []discoveryPinCase{ + { + // A .env BEADS_DIR routes discovery to its target. + name: "beads-dir-routes-to-target", + setup: func(f *discoveryPinFixture) (string, []string) { + f.corruptWorkspace("target", nil) + return f.yamlWorkspace("p", map[string]string{".env": "BEADS_DIR=" + f.path("target/.beads") + "\n"}), nil + }, + args: []string{"list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + { + // The routing workspace's own corrupt metadata.json causes no refusal. + name: "beads-dir-route-skips-local-metadata", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.corruptWorkspace("p", map[string]string{".env": "BEADS_DIR=" + f.path("missing/.beads") + "\n"}), nil + }, + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + // An exported BEADS_DIR wins; the .env is never read. + name: "exported-beads-dir-wins", + setup: func(f *discoveryPinFixture) (string, []string) { + f.corruptWorkspace("target", nil) + cwd := f.yamlWorkspace("p", map[string]string{".env": "BEADS_DIR=" + f.path("target/.beads") + "\n"}) + return cwd, []string{"BEADS_DIR=" + f.path("missing/.beads")} + }, + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + // A .env BEADS_DB does not route a data command away from the + // workspace that holds the .env (the shadowing + // TestDiscoveryRefusalPinsDBSelection records for an exported + // BEADS_DB) ... + name: "beads-db-list", + setup: func(f *discoveryPinFixture) (string, []string) { + f.corruptWorkspace("target", nil) + return f.yamlWorkspace("p", map[string]string{".env": "BEADS_DB=" + f.path("target/.beads/dolt") + "\n"}), nil + }, + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + // ... but it does route bd where. + name: "beads-db-where", + setup: func(f *discoveryPinFixture) (string, []string) { + f.corruptWorkspace("target", nil) + return f.yamlWorkspace("p", map[string]string{".env": "BEADS_DB=" + f.path("target/.beads/dolt") + "\n"}), nil + }, + args: []string{"where"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + { + // Settings in the selected workspace's .env apply (BD_JSON here). + name: "setting-applies-in-selected-workspace", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.yamlWorkspace("p", map[string]string{".env": "BD_JSON=true\n"}), nil + }, + args: []string{"where"}, + stdout: "{\n \"path\": \"$ROOT/p/.beads\",\n \"schema_version\": 1\n}\n", + }, + { + // A .env that routes elsewhere contributes only its selector: + // its other settings are not imported, so the refusal is text. + name: "setting-not-imported-from-routing-source", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.yamlWorkspace("p", map[string]string{".env": "BD_JSON=true\nBEADS_DIR=" + f.path("missing/.beads") + "\n"}), nil + }, + args: []string{"where"}, + exit: 1, + stderr: pinNoWorkspaceWhereRefusal, + }, + { + // checkBlockedEnvVars sees only the exported environment, and runs + // before any .env is loaded: a .env BD_BACKEND is not refused, here + // or anywhere before discovery's own no-database refusal. + name: "blocked-backend-not-checked", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.yamlWorkspace("p", map[string]string{".env": "BD_BACKEND=sqlite\n"}), nil + }, + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + }) +} + +// TestDiscoveryRefusalPinsRedirect pins .beads/redirect handling: the +// warnings for unusable targets and which directory is selected afterwards. +func TestDiscoveryRefusalPinsRedirect(t *testing.T) { + runDiscoveryPinCases(t, []discoveryPinCase{ + { + // A missing target is ignored and discovery keeps the source. + name: "missing-target", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.corruptWorkspace("p", map[string]string{"redirect": f.path("missing/.beads") + "\n"}), nil + }, + args: []string{"list"}, + exit: 1, + stderr: "Warning: redirect target does not exist or is not a directory: $ROOT/missing/.beads\n" + + pinCorruptMetadataRefusal("$ROOT/p/.beads"), + }, + { + // A target without project files is ignored the same way. + name: "target-without-workspace", + setup: func(f *discoveryPinFixture) (string, []string) { + f.mkdir("bare/.beads") + return f.corruptWorkspace("p", map[string]string{"redirect": "../bare/.beads\n"}), nil + }, + args: []string{"list"}, + exit: 1, + stderr: "Warning: ignoring redirect from $ROOT/p/.beads to $ROOT/bare/.beads because the target has no database or metadata.json; fix or delete the redirect file\n" + + pinCorruptMetadataRefusal("$ROOT/p/.beads"), + }, + { + name: "corrupt-target", + setup: func(f *discoveryPinFixture) (string, []string) { + f.corruptWorkspace("target", nil) + return f.workspace("p", map[string]string{"redirect": "../target/.beads\n"}), nil + }, + args: []string{"list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + { + // Only one level is followed. + name: "chain", + setup: func(f *discoveryPinFixture) (string, []string) { + f.corruptWorkspace("final", nil) + f.corruptWorkspace("middle", map[string]string{"redirect": "../final/.beads\n"}) + return f.workspace("p", map[string]string{"redirect": "../middle/.beads\n"}), nil + }, + args: []string{"list"}, + exit: 1, + stderr: "Warning: redirect chains not allowed, ignoring redirect in $ROOT/middle/.beads\n" + + pinCorruptMetadataRefusal("$ROOT/middle/.beads"), + }, + { + // An explicit BEADS_DIR is followed through its redirect. + name: "explicit-beads-dir", + setup: func(f *discoveryPinFixture) (string, []string) { + f.corruptWorkspace("target", nil) + f.workspace("source", map[string]string{"redirect": "../target/.beads\n"}) + return f.project("p"), []string{"BEADS_DIR=" + f.path("source/.beads")} + }, + args: []string{"list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + { + name: "where-reports-source", + setup: func(f *discoveryPinFixture) (string, []string) { + f.yamlWorkspace("target", nil) + return f.workspace("p", map[string]string{"redirect": "../target/.beads\n"}), nil + }, + args: []string{"where"}, + stdout: "$ROOT/target/.beads\n (via redirect from $ROOT/p/.beads)\n", + }, + }) +} + +// TestDiscoveryRefusalPinsDBSelection pins database selection by --db and by +// the BEADS_DB and BD_DB environment variables. +// +// Two of these record inconsistencies as they stand rather than endorse +// them: a data command ignores BD_DB, and lets a discoverable workspace in +// the working directory shadow BEADS_DB, while bd where honors both. +func TestDiscoveryRefusalPinsDBSelection(t *testing.T) { + target := func(f *discoveryPinFixture) string { + f.corruptWorkspace("target", nil) + return f.path("target/.beads/dolt") + } + runDiscoveryPinCases(t, []discoveryPinCase{ + { + // A --db path selects the workspace that owns it. + name: "db-path", + setup: func(f *discoveryPinFixture) (string, []string) { + target(f) + return f.project("p"), nil + }, + args: []string{"--db", "$ROOT/target/.beads/dolt", "list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + { + name: "db-path-beats-beads-dir", + setup: func(f *discoveryPinFixture) (string, []string) { + target(f) + f.corruptWorkspace("other", nil) + return f.project("p"), []string{"BEADS_DIR=" + f.path("other/.beads")} + }, + args: []string{"--db", "$ROOT/target/.beads/dolt", "list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + { + name: "db-path-beats-working-directory", + setup: func(f *discoveryPinFixture) (string, []string) { + target(f) + return f.corruptWorkspace("p", nil), nil + }, + args: []string{"--db", "$ROOT/target/.beads/dolt", "list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + { + // A --db value that is a database name rather than an existing + // path leaves discovery to the working directory. + name: "db-name-without-workspace", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.project("p"), nil + }, + args: []string{"--db", "pindb", "list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + name: "db-name-in-workspace", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.corruptWorkspace("p", nil), nil + }, + args: []string{"--db", "pindb", "list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/p/.beads"), + }, + { + name: "beads-db-env", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.project("p"), []string{"BEADS_DB=" + target(f)} + }, + args: []string{"list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + { + // Inconsistency: the working directory's workspace shadows BEADS_DB. + name: "beads-db-env-shadowed-by-working-directory", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.corruptWorkspace("p", nil), []string{"BEADS_DB=" + target(f)} + }, + args: []string{"list"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/p/.beads"), + }, + { + name: "beads-db-env-where", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.corruptWorkspace("p", nil), []string{"BEADS_DB=" + target(f)} + }, + args: []string{"where"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + { + // Inconsistency: a data command ignores BD_DB. + name: "bd-db-env", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.project("p"), []string{"BD_DB=" + target(f)} + }, + args: []string{"list"}, + exit: 1, + stderr: pinNoDatabaseRefusal, + }, + { + name: "bd-db-env-where", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.project("p"), []string{"BD_DB=" + target(f)} + }, + args: []string{"where"}, + exit: 1, + stderr: pinCorruptMetadataRefusal("$ROOT/target/.beads"), + }, + }) +} + +// TestDiscoveryRefusalPinsBlockedEnvVarsRunFirst pins that the blocked +// backend variables are refused before any discovery, for every command. +func TestDiscoveryRefusalPinsBlockedEnvVarsRunFirst(t *testing.T) { + runDiscoveryPinCases(t, []discoveryPinCase{ + { + name: "no-workspace", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.project("p"), []string{"BD_BACKEND=sqlite"} + }, + args: []string{"list"}, + exit: 1, + stderr: pinBlockedEnvRefusal("BD_BACKEND"), + }, + { + name: "corrupt-metadata", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.corruptWorkspace("p", nil), []string{"BD_BACKEND=sqlite"} + }, + args: []string{"list"}, + exit: 1, + stderr: pinBlockedEnvRefusal("BD_BACKEND"), + }, + { + name: "explicit-beads-dir", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.project("p"), []string{"BD_BACKEND=sqlite", "BEADS_DIR=" + f.path("missing/.beads")} + }, + args: []string{"list"}, + exit: 1, + stderr: pinBlockedEnvRefusal("BD_BACKEND"), + }, + { + // A BEADS_DIR that cannot even be read as a directory is still + // reached only after the blocked variable is refused. + name: "unreadable-beads-dir", + setup: func(f *discoveryPinFixture) (string, []string) { + f.write("file", "not a workspace\n") + return f.project("p"), []string{"BD_BACKEND=sqlite", "BEADS_DIR=" + f.path("file")} + }, + args: []string{"list"}, + exit: 1, + stderr: pinBlockedEnvRefusal("BD_BACKEND"), + }, + { + name: "db-path", + setup: func(f *discoveryPinFixture) (string, []string) { + f.corruptWorkspace("target", nil) + return f.project("p"), []string{"BD_BACKEND=sqlite"} + }, + args: []string{"--db", "$ROOT/target/.beads/dolt", "list"}, + exit: 1, + stderr: pinBlockedEnvRefusal("BD_BACKEND"), + }, + { + // --json does not turn the refusal into a JSON document. + name: "database-backend-json", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.corruptWorkspace("p", nil), []string{"BD_DATABASE_BACKEND=dolt"} + }, + args: []string{"list", "--json"}, + exit: 1, + stderr: pinBlockedEnvRefusal("BD_DATABASE_BACKEND"), + }, + { + name: "where-json", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.project("p"), []string{"BD_BACKEND=sqlite"} + }, + args: []string{"where", "--json"}, + exit: 1, + stderr: pinBlockedEnvRefusal("BD_BACKEND"), + }, + { + name: "version-command", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.project("p"), []string{"BD_BACKEND=sqlite"} + }, + args: []string{"version"}, + exit: 1, + stderr: pinBlockedEnvRefusal("BD_BACKEND"), + }, + { + // The root command then prints cobra's error and usage; only the + // refusal itself is pinned. + name: "version-flag", + setup: func(f *discoveryPinFixture) (string, []string) { + return f.project("p"), []string{"BD_BACKEND=sqlite"} + }, + args: []string{"--version"}, + exit: 1, + stderr: pinBlockedEnvRefusal("BD_BACKEND"), + stderrPrefix: true, + }, + }) +} From eb5ab48950d9ab73aa0766e0105933ec2611586c Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 19 Oct 2026 05:19:28 +0000 Subject: [PATCH 10/41] test(setup): pin the Claude agent artifacts bd init and bd setup claude write (#7678) * test(setup): pin the Claude agent artifacts bd init and bd setup claude write Before-parity pin P11. Nothing pinned the agent artifacts byte for byte, so a change to them -- a new hook event in .claude/settings.json, new text in the minimal CLAUDE.md/AGENTS.md template and with it a new marker hash that marks every installed section stale -- could land without anyone seeing the user-visible diff. These goldens make any such change a reviewed diff. - internal/templates/agents: TestManagedSectionGolden pins each managed section (minimal and full profile, with and without a Dolt remote) markers included; TestManagedSectionHashes pins their marker hashes as hand-maintained literals; NoPush is pinned to render the no-remote bytes. - cmd/bd/setup: TestClaudeSetupArtifactsGolden records transcripts of bd setup claude, --check and --remove (project, --stealth, --global, a user's own settings and CLAUDE.md, a stale section, a legacy settings.local.json install, a CLAUDE.md that imports AGENTS.md): each step's result and output, and the settings and instruction files it creates, changes or deletes. TestClaudePluginManifestHookEvents pins the hook events and commands the beads Claude plugin declares. - cmd/bd: TestEmbeddedInitAgentArtifactsGolden runs a real bd init (default and --agents-profile full) and then bd setup claude --check, --remove and a reinstall through the binary, with exit codes. A managed section in a transcript is shown by its marker line and a pointer to the template golden it is checked to match byte for byte, so a template change shows up once in full and as a hash bump wherever it lands. Regenerate with BD_UPDATE_GOLDEN=1. CI companions: the new test files and testdata in BUILD.bazel (bd_test gains its internal/templates/agents dep), and the new TestEmbedded* in the 100-shard embedded cmd manifest (incremental --write). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(setup): drop BD_OTEL_* and split HOME in the agent artifact golden TestEmbeddedInitAgentArtifactsGolden pins bd's stderr byte for byte, but bdEnv passes the caller's BD_OTEL_* variables through, and bd prints a deprecation warning on stderr for a legacy one (BD_OTEL_METRICS_URL, BD_OTEL_LOGS_URL, BD_OTEL_STDOUT=true) before it checks whether telemetry is on. On a host that exports them (agent hosts do; CI does not), both scenarios failed at the first stderr block with no code drift. The test now runs bd init and every bd setup claude step with agentArtifactEnv: bdEnv(dir) without BD_OTEL_* and OTEL_*. It filters the env slice instead of calling clearTelemetryEnv, which uses t.Setenv and so cannot run in these parallel tests. The same helper gives bd a HOME of its own instead of bdEnv's HOME == project. With the two equal, a bd that wrote the hook and CLAUDE.md into the home directory instead of the project passed every pin. The transcript now also records $HOME/.claude/settings.json and $HOME/CLAUDE.md, and output paths under HOME read $HOME. bd init runs the steps bdInit does (git repo, seeded schema) with that env. The test doc now names what it does not pin: a proxied-server bd init (no Codex section, HasRemote from its own remote URL), a bd init with a remote, and the Cursor files. It also says a change to AGENTS.md's wrapper text or Codex section can fail only here. The goldens are unchanged. Validation (env -i, go1.26.7, GOFLAGS=-mod=readonly, GOPROXY=off, BEADS_TEST_EMBEDDED_DOLT=1, -tags gms_pure_go): - the golden passes in a clean env; with BD_OTEL_METRICS_URL=x BD_OTEL_LOGS_URL=y; with every legacy selector plus BD_OTEL_ENABLED and OTEL_EXPORTER_OTLP_ENDPOINT; and under -race; - the previous test with BD_OTEL_METRICS_URL=x BD_OTEL_LOGS_URL=y fails at golden line 132 on the deprecation warning; - swapping homeDir and projectDir in setup's defaultClaudeEnv passes the previous test and fails this one at golden line 5 (the transcript shows created $HOME/.claude/settings.json). Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com> --- .github/scripts/embedded-cmd-test-shards.txt | 1 + cmd/bd/BUILD.bazel | 2 + cmd/bd/agent_artifacts_embedded_test.go | 306 +++++++++++ cmd/bd/setup/BUILD.bazel | 3 +- cmd/bd/setup/claude_artifacts_golden_test.go | 494 ++++++++++++++++++ .../claude-setup-agents-import-stub.golden | 84 +++ .../claude-setup-existing-user-files.golden | 174 ++++++ .../setup/testdata/claude-setup-global.golden | 122 +++++ .../claude-setup-legacy-local-settings.golden | 157 ++++++ .../claude-setup-project-stealth.golden | 122 +++++ .../testdata/claude-setup-project.golden | 142 +++++ .../claude-setup-stale-section.golden | 72 +++ .../init-agents-profile-full.golden | 133 +++++ .../agent_artifacts/init-default.golden | 274 ++++++++++ internal/templates/agents/BUILD.bazel | 2 + internal/templates/agents/golden_test.go | 133 +++++ .../managed-section-full-no-remote.md | 129 +++++ .../agents/testdata/managed-section-full.md | 131 +++++ .../managed-section-minimal-no-remote.md | 54 ++ .../testdata/managed-section-minimal.md | 55 ++ 20 files changed, 2589 insertions(+), 1 deletion(-) create mode 100644 cmd/bd/agent_artifacts_embedded_test.go create mode 100644 cmd/bd/setup/claude_artifacts_golden_test.go create mode 100644 cmd/bd/setup/testdata/claude-setup-agents-import-stub.golden create mode 100644 cmd/bd/setup/testdata/claude-setup-existing-user-files.golden create mode 100644 cmd/bd/setup/testdata/claude-setup-global.golden create mode 100644 cmd/bd/setup/testdata/claude-setup-legacy-local-settings.golden create mode 100644 cmd/bd/setup/testdata/claude-setup-project-stealth.golden create mode 100644 cmd/bd/setup/testdata/claude-setup-project.golden create mode 100644 cmd/bd/setup/testdata/claude-setup-stale-section.golden create mode 100644 cmd/bd/testdata/agent_artifacts/init-agents-profile-full.golden create mode 100644 cmd/bd/testdata/agent_artifacts/init-default.golden create mode 100644 internal/templates/agents/golden_test.go create mode 100644 internal/templates/agents/testdata/managed-section-full-no-remote.md create mode 100644 internal/templates/agents/testdata/managed-section-full.md create mode 100644 internal/templates/agents/testdata/managed-section-minimal-no-remote.md create mode 100644 internal/templates/agents/testdata/managed-section-minimal.md diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index 95d0ef0f0..7b45f7915 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -136,6 +136,7 @@ 100 54 TestEmbeddedDiscoveryPinsEnvFileReadonly 100 54 TestEmbeddedFormatJSONAliasMatchesJSON +100 54 TestEmbeddedInitAgentArtifactsGolden 100 54 TestEmbeddedSearchConcurrent 100 55 TestEmbeddedUndefer diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 3ac61dc7c..680301167 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -547,6 +547,7 @@ go_test( "actor_test.go", "ado_roundtrip_test.go", "ado_test.go", + "agent_artifacts_embedded_test.go", "agent_hook_test.go", "assign_fence_embedded_test.go", "assign_fence_proxied_test.go", @@ -1407,6 +1408,7 @@ go_test( "//internal/storage/uow", "//internal/storage/versioncontrolops", "//internal/telemetry", + "//internal/templates/agents", "//internal/testutil", "//internal/testutil/bazeltest", "//internal/testutil/credentialcmd", diff --git a/cmd/bd/agent_artifacts_embedded_test.go b/cmd/bd/agent_artifacts_embedded_test.go new file mode 100644 index 000000000..9636f9429 --- /dev/null +++ b/cmd/bd/agent_artifacts_embedded_test.go @@ -0,0 +1,306 @@ +//go:build cgo + +package main + +import ( + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/templates/agents" +) + +// TestEmbeddedInitAgentArtifactsGolden pins the Claude agent artifacts a real +// `bd init` writes — .claude/settings.json with its hook events and commands, +// CLAUDE.md and AGENTS.md with their managed sections — and what +// `bd setup claude --check`, `--remove` and a reinstall then report and leave +// behind, through the bd binary. Each scenario is a transcript golden under +// testdata/agent_artifacts/, so a change to any artifact or outcome (a new +// hook event, a changed command, new template text and its hash) shows up as +// a reviewed golden diff. +// +// A managed section is shown by its BEGIN marker line (profile and hash) and a +// pointer to the internal/templates/agents golden that pins its bytes +// (TestManagedSectionGolden), after checking the file holds exactly that +// section; one that matches no rendered variant is shown in full. The +// in-process `bd setup claude` scenarios are pinned by +// TestClaudeSetupArtifactsGolden in cmd/bd/setup. +// +// The scenarios run a direct (embedded) bd init with no Dolt remote. Not +// pinned here: a proxied-server bd init, which installs no Codex section and +// decides HasRemote from its own remote URL; a bd init with a remote, which +// renders the managed section's with-remote variant; and the Cursor files. +// AGENTS.md is shown in full apart from its managed section, so a change to +// its wrapper text or its Codex section can pass the cmd/bd/setup and +// internal/templates/agents tests and fail only here. +// +// bd runs with HOME outside the project, so an artifact written to the home +// directory instead of the project shows up under $HOME. +// +// Regenerate with +// +// BEADS_TEST_EMBEDDED_DOLT=1 BD_UPDATE_GOLDEN=1 go test -tags gms_pure_go ./cmd/bd/ -run TestEmbeddedInitAgentArtifactsGolden +func TestEmbeddedInitAgentArtifactsGolden(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + + scenarios := []struct { + name string + initArgs []string + steps [][]string + }{ + { + name: "init-default", + initArgs: []string{"--prefix", "aa"}, + steps: [][]string{ + {"setup", "claude", "--check"}, + {"setup", "claude"}, + {"setup", "claude", "--remove"}, + {"setup", "claude", "--check"}, + {"setup", "claude"}, + {"setup", "claude", "--check"}, + }, + }, + { + name: "init-agents-profile-full", + initArgs: []string{"--prefix", "ab", "--agents-profile", "full"}, + steps: [][]string{ + {"setup", "claude", "--check"}, + }, + }, + } + + for _, sc := range scenarios { + t.Run(sc.name, func(t *testing.T) { + t.Parallel() + // bdInit, with HOME in a directory of its own. + dir, home := t.TempDir(), t.TempDir() + initGitRepoAt(t, dir) + database, seeded := seededInitDatabase(sc.initArgs) + if seeded { + seedEmbeddedSchema(t, dir, database) + } + initArgs := append([]string{"init", "--quiet"}, sc.initArgs...) + if stdout, stderr, exitCode := runAgentArtifactCommand(t, bd, dir, home, initArgs...); exitCode != 0 { + t.Fatalf("bd %s: exit %d\nstdout:\n%s\nstderr:\n%s", strings.Join(initArgs, " "), exitCode, stdout, stderr) + } + if seeded { + requireSeededEmbeddedInit(t, filepath.Join(dir, ".beads"), database) + } + + var transcript strings.Builder + fmt.Fprintf(&transcript, "# bd init agent artifact transcript: %s\n", sc.name) + fmt.Fprintf(&transcript, "# Regenerate with BEADS_TEST_EMBEDDED_DOLT=1 BD_UPDATE_GOLDEN=1 go test -tags gms_pure_go ./cmd/bd/ -run TestEmbeddedInitAgentArtifactsGolden\n") + fmt.Fprintf(&transcript, "\n=== bd %s\n", strings.Join(initArgs, " ")) + previous := map[string]string{} + writeAgentArtifactChanges(t, &transcript, dir, home, previous) + + for _, args := range sc.steps { + stdout, stderr, exitCode := runAgentArtifactCommand(t, bd, dir, home, args...) + fmt.Fprintf(&transcript, "\n=== bd %s\n", strings.Join(args, " ")) + fmt.Fprintf(&transcript, "exit: %d\n", exitCode) + writeAgentArtifactBlock(&transcript, "stdout", normalizeAgentArtifactOutput(stdout, dir, home)) + writeAgentArtifactBlock(&transcript, "stderr", normalizeAgentArtifactOutput(stderr, dir, home)) + writeAgentArtifactChanges(t, &transcript, dir, home, previous) + } + + assertAgentArtifactGolden(t, filepath.Join("testdata", "agent_artifacts", sc.name+".golden"), transcript.String()) + }) + } +} + +// agentArtifactPaths are the Claude agent artifacts bd init and bd setup +// claude write in a project ($PROJECT), and the two a global install writes +// in the home directory ($HOME) instead. +var agentArtifactPaths = []string{ + "$PROJECT/.claude/settings.json", + "$PROJECT/.claude/settings.local.json", + "$PROJECT/AGENTS.md", + "$PROJECT/CLAUDE.md", + "$HOME/.claude/settings.json", + "$HOME/CLAUDE.md", +} + +// agentArtifactManagedSections maps each managed section bd renders to the +// internal/templates/agents golden file that pins its bytes. +var agentArtifactManagedSections = []struct { + golden string + profile agents.Profile + opts agents.RenderOpts +}{ + {golden: "managed-section-minimal.md", profile: agents.ProfileMinimal, opts: agents.RenderOpts{HasRemote: true}}, + {golden: "managed-section-minimal-no-remote.md", profile: agents.ProfileMinimal, opts: agents.RenderOpts{}}, + {golden: "managed-section-full.md", profile: agents.ProfileFull, opts: agents.RenderOpts{HasRemote: true}}, + {golden: "managed-section-full-no-remote.md", profile: agents.ProfileFull, opts: agents.RenderOpts{}}, +} + +// elideAgentArtifactManagedSection replaces the body of a managed section +// that is byte-identical to a rendered variant with a pointer to that +// variant's golden, keeping the BEGIN marker line (profile and hash) and the +// END marker. +func elideAgentArtifactManagedSection(content string) string { + const endMarker = "\n" + begin := strings.Index(content, "\n" + begin := strings.Index(content, "\nOlder beads guidance.\n\n\n## Notes\n", + }, + steps: []claudeArtifactStep{ + claudeCheckStep(), + claudeInstallStep(false, false), + claudeCheckStep(), + }, + }, + { + // A legacy install in settings.local.json: install migrates + // its hook to settings.json. + name: "legacy-local-settings", + seed: map[string]string{ + "project/.claude/settings.local.json": legacyLocalSettings, + }, + steps: []claudeArtifactStep{ + claudeCheckStep(), + claudeInstallStep(false, false), + claudeCheckStep(), + claudeRemoveStep(false), + }, + }, + { + // CLAUDE.md imports AGENTS.md: the managed section goes to + // AGENTS.md, and --remove leaves that shared section in place. + name: "agents-import-stub", + seed: map[string]string{ + "project/AGENTS.md": "# Agent Instructions\n\nShared by every agent.\n", + "project/CLAUDE.md": "@AGENTS.md\n", + }, + steps: []claudeArtifactStep{ + claudeInstallStep(false, false), + claudeCheckStep(), + claudeRemoveStep(false), + claudeCheckStep(), + }, + }, + } + + for _, sc := range scenarios { + t.Run(sc.name, func(t *testing.T) { + env, _, _ := newClaudeTestEnv(t) + root := filepath.Dir(env.projectDir) + for rel, content := range sc.seed { + path := filepath.Join(root, filepath.FromSlash(rel)) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir for seed %s: %v", rel, err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatalf("seed %s: %v", rel, err) + } + } + + var transcript strings.Builder + fmt.Fprintf(&transcript, "# bd setup claude artifact transcript: %s\n", sc.name) + fmt.Fprintf(&transcript, "# Regenerate with BD_UPDATE_GOLDEN=1 go test ./cmd/bd/setup/ -run TestClaudeSetupArtifactsGolden\n") + previous := map[string]string{} + writeClaudeArtifactChanges(t, &transcript, "seed", root, previous) + for _, step := range sc.steps { + out, errOut, err := runClaudeArtifactStep(t, env, step.run) + fmt.Fprintf(&transcript, "\n=== %s\n", step.title) + fmt.Fprintf(&transcript, "result: %s\n", claudeStepResult(err)) + writeTranscriptBlock(&transcript, "stdout", normalizeArtifactOutput(out, root)) + writeTranscriptBlock(&transcript, "stderr", normalizeArtifactOutput(errOut, root)) + writeClaudeArtifactChanges(t, &transcript, "files", root, previous) + } + + assertArtifactGolden(t, filepath.Join("testdata", "claude-setup-"+sc.name+".golden"), transcript.String()) + }) + } +} + +// TestClaudePluginManifestHookEvents pins the hook events and commands the +// beads Claude Code plugin declares. With the plugin enabled bd setup claude +// writes no hooks of its own (GH#3192), so these are the hooks such a user +// runs. +func TestClaudePluginManifestHookEvents(t *testing.T) { + var manifest struct { + Hooks map[string][]struct { + Matcher string `json:"matcher"` + Hooks []struct { + Type string `json:"type"` + Command string `json:"command"` + } `json:"hooks"` + } `json:"hooks"` + } + readJSONFile(t, filepath.Join("..", "..", "..", "plugins", "beads", ".claude-plugin", "plugin.json"), &manifest) + + var got []string + for event, entries := range manifest.Hooks { + for _, entry := range entries { + for _, hook := range entry.Hooks { + got = append(got, fmt.Sprintf("%s matcher=%q %s: %s", event, entry.Matcher, hook.Type, hook.Command)) + } + } + } + sort.Strings(got) + want := []string{ + `PreCompact matcher="" command: bd prime`, + `SessionStart matcher="" command: bd prime`, + } + if strings.Join(got, "\n") != strings.Join(want, "\n") { + t.Errorf("Claude plugin hooks changed:\ngot:\n %s\nwant:\n %s", strings.Join(got, "\n "), strings.Join(want, "\n ")) + } +} + +// runClaudeArtifactStep runs one step with env.stdout and the process stdout +// sent to the same file, so the transcript keeps the order a user sees: the +// hook helpers (addHookCommand, removeHookCommand) print with fmt.Printf +// rather than to env.stdout. Swapping os.Stdout is why these tests must not +// run in parallel. +func runClaudeArtifactStep(t *testing.T, env claudeEnv, run func(claudeEnv) error) (stdout, stderr string, err error) { + t.Helper() + capture, createErr := os.CreateTemp(t.TempDir(), "stdout-*") + if createErr != nil { + t.Fatalf("create stdout capture: %v", createErr) + } + errBuf := &bytes.Buffer{} + env.stdout, env.stderr = capture, errBuf + + origStdout := os.Stdout + os.Stdout = capture + func() { + defer func() { os.Stdout = origStdout }() + err = run(env) + }() + + if closeErr := capture.Close(); closeErr != nil { + t.Fatalf("close stdout capture: %v", closeErr) + } + data, readErr := os.ReadFile(capture.Name()) + if readErr != nil { + t.Fatalf("read stdout capture: %v", readErr) + } + return string(data), errBuf.String(), err +} + +func claudeStepResult(err error) string { + if err == nil { + return "ok" + } + return "error: " + err.Error() +} + +// writeClaudeArtifactChanges appends every artifact file created, modified or +// deleted since previous (which it then updates), with its full contents. +func writeClaudeArtifactChanges(t *testing.T, b *strings.Builder, label, root string, previous map[string]string) { + t.Helper() + changed := false + for _, rel := range claudeArtifactPaths { + data, err := os.ReadFile(filepath.Join(root, filepath.FromSlash(rel))) + before, existed := previous[rel] + switch { + case os.IsNotExist(err): + if existed { + fmt.Fprintf(b, "--- %s: deleted $ROOT/%s\n", label, rel) + delete(previous, rel) + changed = true + } + case err != nil: + t.Fatalf("read %s: %v", rel, err) + case !existed || before != string(data): + verb := "created" + if existed { + verb = "modified" + } + shown := string(data) + if strings.HasSuffix(rel, ".md") { + shown = elideManagedSection(shown) + } + writeTranscriptBlock(b, fmt.Sprintf("%s: %s $ROOT/%s", label, verb, rel), shown) + previous[rel] = string(data) + changed = true + } + } + switch { + case changed: + case label == "seed": + fmt.Fprintf(b, "--- %s: none\n", label) + default: + fmt.Fprintf(b, "--- %s: unchanged\n", label) + } +} + +// writeTranscriptBlock appends a titled block, marking empty content and a +// missing trailing newline so neither is lost in the golden. +func writeTranscriptBlock(b *strings.Builder, title, content string) { + fmt.Fprintf(b, "--- %s\n", title) + switch { + case content == "": + b.WriteString("(empty)\n") + case !strings.HasSuffix(content, "\n"): + b.WriteString(content) + b.WriteString("\n(no newline at end of file)\n") + default: + b.WriteString(content) + } +} + +// normalizeArtifactOutput replaces the test's temporary root with $ROOT and +// uses forward slashes, so the transcript is the same on every host. +func normalizeArtifactOutput(s, root string) string { + s = strings.ReplaceAll(s, root, "$ROOT") + if filepath.Separator != '/' { + s = strings.ReplaceAll(s, string(filepath.Separator), "/") + } + return s +} + +// assertArtifactGolden compares got with the golden file at path, or rewrites +// the file when BD_UPDATE_GOLDEN=1. +func assertArtifactGolden(t *testing.T, path, got string) { + t.Helper() + if os.Getenv("BD_UPDATE_GOLDEN") == "1" { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir %s: %v", filepath.Dir(path), err) + } + if err := os.WriteFile(path, []byte(got), 0o644); err != nil { + t.Fatalf("write golden %s: %v", path, err) + } + t.Logf("updated golden %s", path) + return + } + want, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read golden %s (regenerate with BD_UPDATE_GOLDEN=1): %v", path, err) + } + if got == string(want) { + return + } + wantLines, gotLines := strings.Split(string(want), "\n"), strings.Split(got, "\n") + line := func(lines []string, i int) string { + if i >= len(lines) { + return "(end of output)" + } + return fmt.Sprintf("%q", lines[i]) + } + i := 0 + for i < len(wantLines) && i < len(gotLines) && wantLines[i] == gotLines[i] { + i++ + } + t.Errorf("Claude agent artifacts drifted from %s (regenerate with BD_UPDATE_GOLDEN=1 if intended)\nfirst difference at line %d:\n want: %s\n got: %s\n--- got ---\n%s", path, i+1, line(wantLines, i), line(gotLines, i), got) +} diff --git a/cmd/bd/setup/testdata/claude-setup-agents-import-stub.golden b/cmd/bd/setup/testdata/claude-setup-agents-import-stub.golden new file mode 100644 index 000000000..9b4337a80 --- /dev/null +++ b/cmd/bd/setup/testdata/claude-setup-agents-import-stub.golden @@ -0,0 +1,84 @@ +# bd setup claude artifact transcript: agents-import-stub +# Regenerate with BD_UPDATE_GOLDEN=1 go test ./cmd/bd/setup/ -run TestClaudeSetupArtifactsGolden +--- seed: created $ROOT/project/AGENTS.md +# Agent Instructions + +Shared by every agent. +--- seed: created $ROOT/project/CLAUDE.md +@AGENTS.md + +=== bd setup claude +result: ok +--- stdout +Installing Claude hooks for this project... +✓ Registered SessionStart hook +Installing Claude Code integration... +✓ Added beads section to existing AGENTS.md + +✓ Claude Code integration installed + File: $ROOT/project/AGENTS.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $ROOT/project/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: created $ROOT/project/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "bd prime --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ] + } +} +--- files: modified $ROOT/project/AGENTS.md +# Agent Instructions + +Shared by every agent. + + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal.md] + + +=== bd setup claude --check +result: ok +--- stdout +✓ Project hooks installed: $ROOT/project/.claude/settings.json +✓ Claude Code integration installed: $ROOT/project/AGENTS.md (current) +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude --remove +result: ok +--- stdout +Removing Claude hooks from project... +✓ Removed SessionStart hook + Leaving shared beads section in AGENTS.md untouched (project-authoritative, not Claude-specific) +✓ Claude hooks removed +--- stderr +(empty) +--- files: modified $ROOT/project/.claude/settings.json +{ + "hooks": {} +} + +=== bd setup claude --check +result: error: claude hooks not installed +--- stdout +✗ No hooks installed + Run: bd setup claude +--- stderr +(empty) +--- files: unchanged diff --git a/cmd/bd/setup/testdata/claude-setup-existing-user-files.golden b/cmd/bd/setup/testdata/claude-setup-existing-user-files.golden new file mode 100644 index 000000000..d81091fb8 --- /dev/null +++ b/cmd/bd/setup/testdata/claude-setup-existing-user-files.golden @@ -0,0 +1,174 @@ +# bd setup claude artifact transcript: existing-user-files +# Regenerate with BD_UPDATE_GOLDEN=1 go test ./cmd/bd/setup/ -run TestClaudeSetupArtifactsGolden +--- seed: created $ROOT/project/.claude/settings.json +{ + "permissions": { + "allow": [ + "Bash(go test:*)" + ] + }, + "hooks": { + "SessionStart": [ + { + "matcher": "", + "hooks": [ + { + "type": "command", + "command": "echo session started" + } + ] + } + ], + "Stop": [ + { + "matcher": "", + "hooks": [ + { + "type": "command", + "command": "notify-send done" + } + ] + } + ] + } +} +--- seed: created $ROOT/project/CLAUDE.md +# My Project + +House rules live here. + +=== bd setup claude +result: ok +--- stdout +Installing Claude hooks for this project... +✓ Registered SessionStart hook +Installing Claude Code integration... +✓ Added beads section to existing CLAUDE.md + +✓ Claude Code integration installed + File: $ROOT/project/CLAUDE.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $ROOT/project/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: modified $ROOT/project/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "echo session started", + "type": "command" + } + ], + "matcher": "" + }, + { + "hooks": [ + { + "command": "bd prime --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ], + "Stop": [ + { + "hooks": [ + { + "command": "notify-send done", + "type": "command" + } + ], + "matcher": "" + } + ] + }, + "permissions": { + "allow": [ + "Bash(go test:*)" + ] + } +} +--- files: modified $ROOT/project/CLAUDE.md +# My Project + +House rules live here. + + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal.md] + + +=== bd setup claude --check +result: ok +--- stdout +✓ Project hooks installed: $ROOT/project/.claude/settings.json +✓ Claude Code integration installed: $ROOT/project/CLAUDE.md (current) +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude --remove +result: ok +--- stdout +Removing Claude hooks from project... +✓ Removed SessionStart hook +Removing Claude Code integration... +✓ Removed beads section from CLAUDE.md +✓ Claude hooks removed +--- stderr +(empty) +--- files: modified $ROOT/project/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "echo session started", + "type": "command" + } + ], + "matcher": "" + } + ], + "Stop": [ + { + "hooks": [ + { + "command": "notify-send done", + "type": "command" + } + ], + "matcher": "" + } + ] + }, + "permissions": { + "allow": [ + "Bash(go test:*)" + ] + } +} +--- files: modified $ROOT/project/CLAUDE.md +# My Project + +House rules live here. + + + +=== bd setup claude --check +result: error: claude hooks not installed +--- stdout +✗ No hooks installed + Run: bd setup claude +--- stderr +(empty) +--- files: unchanged diff --git a/cmd/bd/setup/testdata/claude-setup-global.golden b/cmd/bd/setup/testdata/claude-setup-global.golden new file mode 100644 index 000000000..da63d077c --- /dev/null +++ b/cmd/bd/setup/testdata/claude-setup-global.golden @@ -0,0 +1,122 @@ +# bd setup claude artifact transcript: global +# Regenerate with BD_UPDATE_GOLDEN=1 go test ./cmd/bd/setup/ -run TestClaudeSetupArtifactsGolden +--- seed: none + +=== bd setup claude --global +result: ok +--- stdout +Installing Claude hooks globally... +✓ Registered SessionStart hook +Installing Claude Code integration... +✓ Created new CLAUDE.md with beads integration + +✓ Claude Code integration installed + File: $ROOT/project/CLAUDE.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $ROOT/home/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: created $ROOT/home/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "bd prime --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ] + } +} +--- files: created $ROOT/project/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal.md] + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +result: ok +--- stdout +✓ Global hooks installed: $ROOT/home/.claude/settings.json +✓ Claude Code integration installed: $ROOT/project/CLAUDE.md (current) +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude --remove --global +result: ok +--- stdout +Removing Claude hooks globally... +✓ Removed SessionStart hook +Removing Claude Code integration... +✓ Removed beads section from CLAUDE.md +✓ Claude hooks removed +--- stderr +(empty) +--- files: modified $ROOT/home/.claude/settings.json +{ + "hooks": {} +} +--- files: modified $ROOT/project/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +result: error: claude hooks not installed +--- stdout +✗ No hooks installed + Run: bd setup claude +--- stderr +(empty) +--- files: unchanged diff --git a/cmd/bd/setup/testdata/claude-setup-legacy-local-settings.golden b/cmd/bd/setup/testdata/claude-setup-legacy-local-settings.golden new file mode 100644 index 000000000..1e17ea55a --- /dev/null +++ b/cmd/bd/setup/testdata/claude-setup-legacy-local-settings.golden @@ -0,0 +1,157 @@ +# bd setup claude artifact transcript: legacy-local-settings +# Regenerate with BD_UPDATE_GOLDEN=1 go test ./cmd/bd/setup/ -run TestClaudeSetupArtifactsGolden +--- seed: created $ROOT/project/.claude/settings.local.json +{ + "hooks": { + "SessionStart": [ + { + "matcher": "", + "hooks": [ + { + "type": "command", + "command": "bd prime" + } + ] + } + ], + "PreCompact": [ + { + "matcher": "", + "hooks": [ + { + "type": "command", + "command": "bd prime" + } + ] + } + ] + } +} + +=== bd setup claude --check +result: error: agents file not found +--- stdout +✓ Project hooks installed (legacy): $ROOT/project/.claude/settings.local.json + Consider running 'bd setup claude' to migrate to .claude/settings.json +✗ CLAUDE.md not found + Run: bd setup claude +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude +result: ok +--- stdout +Installing Claude hooks for this project... +✓ Registered SessionStart hook +✓ Removed SessionStart hook +✓ Removed PreCompact hook +✓ Migrated hooks from $ROOT/project/.claude/settings.local.json +Installing Claude Code integration... +✓ Created new CLAUDE.md with beads integration + +✓ Claude Code integration installed + File: $ROOT/project/CLAUDE.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $ROOT/project/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: created $ROOT/project/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "bd prime --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ] + } +} +--- files: modified $ROOT/project/.claude/settings.local.json +{ + "hooks": {} +} +--- files: created $ROOT/project/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal.md] + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +result: ok +--- stdout +✓ Project hooks installed: $ROOT/project/.claude/settings.json +✓ Claude Code integration installed: $ROOT/project/CLAUDE.md (current) +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude --remove +result: ok +--- stdout +Removing Claude hooks from project... +✓ Removed SessionStart hook +Removing Claude Code integration... +✓ Removed beads section from CLAUDE.md +✓ Claude hooks removed +--- stderr +(empty) +--- files: modified $ROOT/project/.claude/settings.json +{ + "hooks": {} +} +--- files: modified $ROOT/project/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ diff --git a/cmd/bd/setup/testdata/claude-setup-project-stealth.golden b/cmd/bd/setup/testdata/claude-setup-project-stealth.golden new file mode 100644 index 000000000..71d62d8e6 --- /dev/null +++ b/cmd/bd/setup/testdata/claude-setup-project-stealth.golden @@ -0,0 +1,122 @@ +# bd setup claude artifact transcript: project-stealth +# Regenerate with BD_UPDATE_GOLDEN=1 go test ./cmd/bd/setup/ -run TestClaudeSetupArtifactsGolden +--- seed: none + +=== bd setup claude --stealth +result: ok +--- stdout +Installing Claude hooks for this project... +✓ Registered SessionStart hook +Installing Claude Code integration... +✓ Created new CLAUDE.md with beads integration + +✓ Claude Code integration installed + File: $ROOT/project/CLAUDE.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $ROOT/project/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: created $ROOT/project/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "bd prime --stealth --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ] + } +} +--- files: created $ROOT/project/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal.md] + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +result: ok +--- stdout +✓ Project hooks installed: $ROOT/project/.claude/settings.json +✓ Claude Code integration installed: $ROOT/project/CLAUDE.md (current) +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude --remove +result: ok +--- stdout +Removing Claude hooks from project... +✓ Removed SessionStart hook +Removing Claude Code integration... +✓ Removed beads section from CLAUDE.md +✓ Claude hooks removed +--- stderr +(empty) +--- files: modified $ROOT/project/.claude/settings.json +{ + "hooks": {} +} +--- files: modified $ROOT/project/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +result: error: claude hooks not installed +--- stdout +✗ No hooks installed + Run: bd setup claude +--- stderr +(empty) +--- files: unchanged diff --git a/cmd/bd/setup/testdata/claude-setup-project.golden b/cmd/bd/setup/testdata/claude-setup-project.golden new file mode 100644 index 000000000..aa307d08f --- /dev/null +++ b/cmd/bd/setup/testdata/claude-setup-project.golden @@ -0,0 +1,142 @@ +# bd setup claude artifact transcript: project +# Regenerate with BD_UPDATE_GOLDEN=1 go test ./cmd/bd/setup/ -run TestClaudeSetupArtifactsGolden +--- seed: none + +=== bd setup claude +result: ok +--- stdout +Installing Claude hooks for this project... +✓ Registered SessionStart hook +Installing Claude Code integration... +✓ Created new CLAUDE.md with beads integration + +✓ Claude Code integration installed + File: $ROOT/project/CLAUDE.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $ROOT/project/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: created $ROOT/project/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "bd prime --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ] + } +} +--- files: created $ROOT/project/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal.md] + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +result: ok +--- stdout +✓ Project hooks installed: $ROOT/project/.claude/settings.json +✓ Claude Code integration installed: $ROOT/project/CLAUDE.md (current) +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude +result: ok +--- stdout +Installing Claude hooks for this project... +✓ Hook already registered: SessionStart +Installing Claude Code integration... +✓ Updated existing beads section in CLAUDE.md + +✓ Claude Code integration installed + File: $ROOT/project/CLAUDE.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $ROOT/project/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude --remove +result: ok +--- stdout +Removing Claude hooks from project... +✓ Removed SessionStart hook +Removing Claude Code integration... +✓ Removed beads section from CLAUDE.md +✓ Claude hooks removed +--- stderr +(empty) +--- files: modified $ROOT/project/.claude/settings.json +{ + "hooks": {} +} +--- files: modified $ROOT/project/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +result: error: claude hooks not installed +--- stdout +✗ No hooks installed + Run: bd setup claude +--- stderr +(empty) +--- files: unchanged diff --git a/cmd/bd/setup/testdata/claude-setup-stale-section.golden b/cmd/bd/setup/testdata/claude-setup-stale-section.golden new file mode 100644 index 000000000..c6be8e135 --- /dev/null +++ b/cmd/bd/setup/testdata/claude-setup-stale-section.golden @@ -0,0 +1,72 @@ +# bd setup claude artifact transcript: stale-section +# Regenerate with BD_UPDATE_GOLDEN=1 go test ./cmd/bd/setup/ -run TestClaudeSetupArtifactsGolden +--- seed: created $ROOT/project/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "bd prime --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ] + } +} +--- seed: created $ROOT/project/CLAUDE.md +# My Project + + +Older beads guidance. + + +## Notes + +=== bd setup claude --check +result: error: beads section is stale +--- stdout +✓ Project hooks installed: $ROOT/project/.claude/settings.json +⚠ Claude Code integration installed but stale: $ROOT/project/CLAUDE.md + Run: bd setup claude (to update) +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude +result: ok +--- stdout +Installing Claude hooks for this project... +✓ Hook already registered: SessionStart +Installing Claude Code integration... +✓ Updated existing beads section in CLAUDE.md + +✓ Claude Code integration installed + File: $ROOT/project/CLAUDE.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $ROOT/project/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: modified $ROOT/project/CLAUDE.md +# My Project + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal.md] + + +## Notes + +=== bd setup claude --check +result: ok +--- stdout +✓ Project hooks installed: $ROOT/project/.claude/settings.json +✓ Claude Code integration installed: $ROOT/project/CLAUDE.md (current) +--- stderr +(empty) +--- files: unchanged diff --git a/cmd/bd/testdata/agent_artifacts/init-agents-profile-full.golden b/cmd/bd/testdata/agent_artifacts/init-agents-profile-full.golden new file mode 100644 index 000000000..e7abe0511 --- /dev/null +++ b/cmd/bd/testdata/agent_artifacts/init-agents-profile-full.golden @@ -0,0 +1,133 @@ +# bd init agent artifact transcript: init-agents-profile-full +# Regenerate with BEADS_TEST_EMBEDDED_DOLT=1 BD_UPDATE_GOLDEN=1 go test -tags gms_pure_go ./cmd/bd/ -run TestEmbeddedInitAgentArtifactsGolden + +=== bd init --quiet --prefix ab --agents-profile full +--- files: created $PROJECT/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "bd prime --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ] + } +} +--- files: created $PROJECT/AGENTS.md +# Agent Instructions + +This project uses **bd** (beads) for issue tracking. Run `bd prime` for full workflow context. + +> **Architecture in one line:** Issues live in a local Dolt database +> (`.beads/dolt/`); cross-machine sync uses `bd dolt push/pull` (a +> git-compatible protocol), stored under `refs/dolt/data` on your git +> remote — separate from `refs/heads/*` where your code lives. +> `.beads/issues.jsonl` is a passive export, not the wire protocol. +> +> See [sync-concepts](https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md) +> for the one-screen overview and anti-patterns (don't treat JSONL as the +> source of truth; don't `bd import` during normal operation; don't +> reach for third-party Dolt hosting before trying the default). + +## Quick Reference + +```bash +bd ready # Find available work +bd show # View issue details +bd update --claim # Claim work atomically +bd close # Complete work +bd dolt push # Push beads data to remote +``` + +## Non-Interactive Shell Commands + +**ALWAYS use non-interactive flags** with file operations to avoid hanging on confirmation prompts. + +Shell commands like `cp`, `mv`, and `rm` may be aliased to include `-i` (interactive) mode on some systems, causing the agent to hang indefinitely waiting for y/n input. + +**Use these forms instead:** +```bash +# Force overwrite without prompting +cp -f source dest # NOT: cp source dest +mv -f source dest # NOT: mv source dest +rm -f file # NOT: rm file + +# For recursive operations +rm -rf directory # NOT: rm -r directory +cp -rf source dest # NOT: cp -r source dest +``` + +**Other commands that may prompt:** +- `scp` - use `-o BatchMode=yes` for non-interactive +- `ssh` - use `-o BatchMode=yes` to fail instead of prompting +- `apt-get` - use `-y` flag +- `brew` - use `HOMEBREW_NO_AUTO_UPDATE=1` env var + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-full-no-remote.md] + + + +## Beads Issue Tracker + +Use Beads (`bd`) for durable task tracking in repositories that include it. Use the `beads` skill at `.agents/skills/beads/SKILL.md` (project install) or `~/.agents/skills/beads/SKILL.md` (global install) for Beads workflow guidance, then use the `bd` CLI for issue operations. + +### Quick Reference + +```bash +bd ready # Find available work +bd show # View issue details +bd update --claim # Claim work +bd close # Complete work +bd prime # Refresh Beads context +``` + +### Rules + +- Use `bd` for all task tracking; do not create markdown TODO lists. +- Run `bd prime` when Beads context is missing or stale. Codex 0.129.0+ can load Beads context automatically through native hooks; use `/hooks` to inspect or toggle them. +- Keep durable project facts in Beads via `bd remember` rather than in per-tool memory files, which fragment across accounts; per-operator preferences belong in your harness's own memory. + +**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md for details and anti-patterns. + +--- files: created $PROJECT/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal-no-remote.md] + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +exit: 0 +--- stdout +✓ Project hooks installed: $PROJECT/.claude/settings.json +✓ Claude Code integration installed: $PROJECT/CLAUDE.md (current) +--- stderr +(empty) +--- files: unchanged diff --git a/cmd/bd/testdata/agent_artifacts/init-default.golden b/cmd/bd/testdata/agent_artifacts/init-default.golden new file mode 100644 index 000000000..6d3929882 --- /dev/null +++ b/cmd/bd/testdata/agent_artifacts/init-default.golden @@ -0,0 +1,274 @@ +# bd init agent artifact transcript: init-default +# Regenerate with BEADS_TEST_EMBEDDED_DOLT=1 BD_UPDATE_GOLDEN=1 go test -tags gms_pure_go ./cmd/bd/ -run TestEmbeddedInitAgentArtifactsGolden + +=== bd init --quiet --prefix aa +--- files: created $PROJECT/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "bd prime --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ] + } +} +--- files: created $PROJECT/AGENTS.md +# Agent Instructions + +This project uses **bd** (beads) for issue tracking. Run `bd prime` for full workflow context. + +> **Architecture in one line:** Issues live in a local Dolt database +> (`.beads/dolt/`); cross-machine sync uses `bd dolt push/pull` (a +> git-compatible protocol), stored under `refs/dolt/data` on your git +> remote — separate from `refs/heads/*` where your code lives. +> `.beads/issues.jsonl` is a passive export, not the wire protocol. +> +> See [sync-concepts](https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md) +> for the one-screen overview and anti-patterns (don't treat JSONL as the +> source of truth; don't `bd import` during normal operation; don't +> reach for third-party Dolt hosting before trying the default). + +## Quick Reference + +```bash +bd ready # Find available work +bd show # View issue details +bd update --claim # Claim work atomically +bd close # Complete work +bd dolt push # Push beads data to remote +``` + +## Non-Interactive Shell Commands + +**ALWAYS use non-interactive flags** with file operations to avoid hanging on confirmation prompts. + +Shell commands like `cp`, `mv`, and `rm` may be aliased to include `-i` (interactive) mode on some systems, causing the agent to hang indefinitely waiting for y/n input. + +**Use these forms instead:** +```bash +# Force overwrite without prompting +cp -f source dest # NOT: cp source dest +mv -f source dest # NOT: mv source dest +rm -f file # NOT: rm file + +# For recursive operations +rm -rf directory # NOT: rm -r directory +cp -rf source dest # NOT: cp -r source dest +``` + +**Other commands that may prompt:** +- `scp` - use `-o BatchMode=yes` for non-interactive +- `ssh` - use `-o BatchMode=yes` to fail instead of prompting +- `apt-get` - use `-y` flag +- `brew` - use `HOMEBREW_NO_AUTO_UPDATE=1` env var + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal-no-remote.md] + + + +## Beads Issue Tracker + +Use Beads (`bd`) for durable task tracking in repositories that include it. Use the `beads` skill at `.agents/skills/beads/SKILL.md` (project install) or `~/.agents/skills/beads/SKILL.md` (global install) for Beads workflow guidance, then use the `bd` CLI for issue operations. + +### Quick Reference + +```bash +bd ready # Find available work +bd show # View issue details +bd update --claim # Claim work +bd close # Complete work +bd prime # Refresh Beads context +``` + +### Rules + +- Use `bd` for all task tracking; do not create markdown TODO lists. +- Run `bd prime` when Beads context is missing or stale. Codex 0.129.0+ can load Beads context automatically through native hooks; use `/hooks` to inspect or toggle them. +- Keep durable project facts in Beads via `bd remember` rather than in per-tool memory files, which fragment across accounts; per-operator preferences belong in your harness's own memory. + +**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md for details and anti-patterns. + +--- files: created $PROJECT/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal-no-remote.md] + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +exit: 0 +--- stdout +✓ Project hooks installed: $PROJECT/.claude/settings.json +✓ Claude Code integration installed: $PROJECT/CLAUDE.md (current) +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude +exit: 0 +--- stdout +Installing Claude hooks for this project... +✓ Hook already registered: SessionStart +Installing Claude Code integration... +✓ Updated existing beads section in CLAUDE.md + +✓ Claude Code integration installed + File: $PROJECT/CLAUDE.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $PROJECT/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude --remove +exit: 0 +--- stdout +Removing Claude hooks from project... +✓ Removed SessionStart hook +Removing Claude Code integration... +✓ Removed beads section from CLAUDE.md +✓ Claude hooks removed +--- stderr +(empty) +--- files: modified $PROJECT/.claude/settings.json +{ + "hooks": {} +} +--- files: modified $PROJECT/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + +=== bd setup claude --check +exit: 1 +--- stdout +✗ No hooks installed + Run: bd setup claude +--- stderr +(empty) +--- files: unchanged + +=== bd setup claude +exit: 0 +--- stdout +Installing Claude hooks for this project... +✓ Registered SessionStart hook +Installing Claude Code integration... +✓ Added beads section to existing CLAUDE.md + +✓ Claude Code integration installed + File: $PROJECT/CLAUDE.md +No additional configuration needed! + +✓ Claude Code integration installed + Settings: $PROJECT/.claude/settings.json + +Restart Claude Code for changes to take effect. +--- stderr +(empty) +--- files: modified $PROJECT/.claude/settings.json +{ + "hooks": { + "SessionStart": [ + { + "hooks": [ + { + "command": "bd prime --hook-json", + "type": "command" + } + ], + "matcher": "" + } + ] + } +} +--- files: modified $PROJECT/CLAUDE.md +# Project Instructions for AI Agents + +This file provides instructions and context for AI coding agents working on this project. + + + +## Build & Test + +_Add your build and test commands here_ + +```bash +# Example: +# npm install +# npm test +``` + +## Architecture Overview + +_Add a brief overview of your project architecture_ + +## Conventions & Patterns + +_Add your project-specific conventions here_ + + + +[managed section body: byte-identical to internal/templates/agents/testdata/managed-section-minimal-no-remote.md] + + +=== bd setup claude --check +exit: 0 +--- stdout +✓ Project hooks installed: $PROJECT/.claude/settings.json +✓ Claude Code integration installed: $PROJECT/CLAUDE.md (current) +--- stderr +(empty) +--- files: unchanged diff --git a/internal/templates/agents/BUILD.bazel b/internal/templates/agents/BUILD.bazel index c51ae9c90..8ad3255df 100644 --- a/internal/templates/agents/BUILD.bazel +++ b/internal/templates/agents/BUILD.bazel @@ -20,8 +20,10 @@ go_test( name = "agents_test", srcs = [ "agents_test.go", + "golden_test.go", "render_test.go", ], + data = glob(["testdata/**"]), embed = [":agents"], ) diff --git a/internal/templates/agents/golden_test.go b/internal/templates/agents/golden_test.go new file mode 100644 index 000000000..e49b0acf8 --- /dev/null +++ b/internal/templates/agents/golden_test.go @@ -0,0 +1,133 @@ +package agents + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +// The tests in this file pin the exact managed sections `bd init` and +// `bd setup ` write between the BEGIN/END BEADS INTEGRATION markers of +// CLAUDE.md, AGENTS.md and the other instruction files. A change to a template +// (new guidance, a reworded rule, an added section) changes the bytes every +// user's instruction file receives and, through the marker hash, marks every +// section already installed as stale: `bd setup claude --check` then fails and +// the next `bd init` or `bd setup` rewrites the user's file. Such a change must +// therefore arrive as a reviewed golden diff, never as a side effect. +// +// Regenerate the golden files with +// +// BD_UPDATE_GOLDEN=1 go test ./internal/templates/agents/ -run TestManagedSectionGolden +// +// and update the literal hashes in TestManagedSectionHashes by hand. + +// managedSectionVariant is one managed section bd can render. +type managedSectionVariant struct { + name string + profile Profile + opts RenderOpts +} + +// managedSectionVariants lists every managed section bd renders: both +// profiles, each with and without the `bd dolt push` lines that depend on a +// configured Dolt remote. NoPush renders the no-remote bytes +// (TestManagedSectionNoPushMatchesNoRemote), so it needs no variant of its own. +var managedSectionVariants = []managedSectionVariant{ + {name: "minimal", profile: ProfileMinimal, opts: RenderOpts{HasRemote: true}}, + {name: "minimal-no-remote", profile: ProfileMinimal, opts: RenderOpts{}}, + {name: "full", profile: ProfileFull, opts: RenderOpts{HasRemote: true}}, + {name: "full-no-remote", profile: ProfileFull, opts: RenderOpts{}}, +} + +// TestManagedSectionGolden pins each managed section byte for byte, markers +// included. +func TestManagedSectionGolden(t *testing.T) { + for _, v := range managedSectionVariants { + t.Run(v.name, func(t *testing.T) { + assertGoldenFile(t, filepath.Join("testdata", "managed-section-"+v.name+".md"), RenderSectionWithOpts(v.profile, v.opts)) + }) + } +} + +// TestManagedSectionHashes pins the marker hash of each managed section. The +// hash is what `bd setup --check` compares against an installed +// section, so changing it marks every existing install stale. It is a literal +// that BD_UPDATE_GOLDEN does not rewrite: a template change has to bump it by +// hand. +func TestManagedSectionHashes(t *testing.T) { + want := map[string]string{ + "minimal": "6ffdf896", + "minimal-no-remote": "cdcea925", + "full": "bacef91e", + "full-no-remote": "9c890b20", + } + for _, v := range managedSectionVariants { + got := CurrentHashWithOpts(v.profile, v.opts) + if got != want[v.name] { + t.Errorf("%s managed section hash = %q, want %q: the template changed, so every installed %s section is now stale", v.name, got, want[v.name], v.profile) + } + marker := "\n" + if section := RenderSectionWithOpts(v.profile, v.opts); !strings.HasPrefix(section, marker) { + first, _, _ := strings.Cut(section, "\n") + t.Errorf("%s managed section marker = %q, want %q", v.name, first, strings.TrimSuffix(marker, "\n")) + } + } +} + +// TestManagedSectionNoPushMatchesNoRemote pins that a local-only workspace +// (no-push: true) gets the same managed section as one with no Dolt remote, +// whatever the remote state. +func TestManagedSectionNoPushMatchesNoRemote(t *testing.T) { + for _, profile := range []Profile{ProfileMinimal, ProfileFull} { + want := RenderSectionWithOpts(profile, RenderOpts{}) + for _, hasRemote := range []bool{false, true} { + if got := RenderSectionWithOpts(profile, RenderOpts{HasRemote: hasRemote, NoPush: true}); got != want { + t.Errorf("%s profile with NoPush (HasRemote=%v) differs from the no-remote section", profile, hasRemote) + } + } + } +} + +// assertGoldenFile compares got with the golden file at path, or rewrites the +// file when BD_UPDATE_GOLDEN=1. +func assertGoldenFile(t *testing.T, path, got string) { + t.Helper() + if os.Getenv("BD_UPDATE_GOLDEN") == "1" { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir %s: %v", filepath.Dir(path), err) + } + if err := os.WriteFile(path, []byte(got), 0o644); err != nil { + t.Fatalf("write golden %s: %v", path, err) + } + t.Logf("updated golden %s", path) + return + } + want, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read golden %s (regenerate with BD_UPDATE_GOLDEN=1): %v", path, err) + } + if got == string(want) { + return + } + t.Errorf("rendered output drifted from %s (regenerate with BD_UPDATE_GOLDEN=1 if intended)\n%s", path, firstGoldenDifference(string(want), got)) +} + +// firstGoldenDifference describes the first line on which want and got differ. +func firstGoldenDifference(want, got string) string { + wantLines := strings.Split(want, "\n") + gotLines := strings.Split(got, "\n") + line := func(lines []string, i int) string { + if i >= len(lines) { + return "(end of output)" + } + return fmt.Sprintf("%q", lines[i]) + } + for i := 0; i < len(wantLines) || i < len(gotLines); i++ { + if i >= len(wantLines) || i >= len(gotLines) || wantLines[i] != gotLines[i] { + return fmt.Sprintf("first difference at line %d:\n want: %s\n got: %s", i+1, line(wantLines, i), line(gotLines, i)) + } + } + return "outputs differ" +} diff --git a/internal/templates/agents/testdata/managed-section-full-no-remote.md b/internal/templates/agents/testdata/managed-section-full-no-remote.md new file mode 100644 index 000000000..526550030 --- /dev/null +++ b/internal/templates/agents/testdata/managed-section-full-no-remote.md @@ -0,0 +1,129 @@ + +## Issue Tracking with bd (beads) + +**IMPORTANT**: This project uses **bd (beads)** for ALL issue tracking. Do NOT use markdown TODOs, task lists, or other tracking methods. + +### Why bd? + +- Dependency-aware: Track blockers and relationships between issues +- Git-friendly: Dolt-powered version control with native sync +- Agent-optimized: JSON output, ready work detection, discovered-from links +- Prevents duplicate tracking systems and confusion + +### Quick Start + +**Check for ready work:** + +```bash +bd ready --json +``` + +**Create new issues:** + +```bash +bd create "Issue title" --description="Detailed context" -t bug|feature|task -p 0-4 --json +bd create "Issue title" --description="What this issue is about" -p 1 --deps discovered-from:bd-123 --json +``` + +**Claim and update:** + +```bash +bd update --claim --json +bd update bd-42 --priority 1 --json +``` + +**Complete work:** + +```bash +bd close bd-42 --reason "Completed" --json +``` + +### Issue Types + +- `bug` - Something broken +- `feature` - New functionality +- `task` - Work item (tests, docs, refactoring) +- `epic` - Large feature with subtasks +- `chore` - Maintenance (dependencies, tooling) + +### Priorities + +- `0` - Critical (security, data loss, broken builds) +- `1` - High (major features, important bugs) +- `2` - Medium (default, nice-to-have) +- `3` - Low (polish, optimization) +- `4` - Backlog (future ideas) + +### Workflow for AI Agents + +1. **Check ready work**: `bd ready` shows unblocked issues +2. **Claim your task atomically**: `bd update --claim` +3. **Work on it**: Implement, test, document +4. **Discover new work?** Create linked issue: + - `bd create "Found bug" --description="Details about what was found" -p 1 --deps discovered-from:` +5. **Complete**: `bd close --reason "Done"` + +### Quality +- Use `--acceptance` and `--design` fields when creating issues +- Use `--validate` to check description completeness + +### Lifecycle +- `bd defer ` / `bd supersede ` for issue management +- `bd stale` / `bd orphans` / `bd lint` for hygiene +- `bd human ` to flag for human decisions +- `bd formula list` / `bd mol pour ` for structured workflows + +### Sync + +bd stores issue history in Dolt: + +- Each write auto-commits to Dolt history +- Do not treat `.beads/issues.jsonl` as the sync protocol + +**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md for details and anti-patterns. + +### Important Rules + +- ✅ Use bd for ALL task tracking +- ✅ Always use `--json` flag for programmatic use +- ✅ Link discovered work with `discovered-from` dependencies +- ✅ Check `bd ready` before asking "what should I work on?" +- ❌ Do NOT create markdown TODO lists +- ❌ Do NOT use external issue trackers +- ❌ Do NOT duplicate tracking systems + +For more details, see README.md and https://github.com/gastownhall/beads/blob/main/docs/getting-started/quickstart.md. + +## Agent Context Profiles + +The managed Beads block is task-tracking guidance, not permission to override repository, user, or orchestrator instructions. + +- **Conservative (default)**: Use `bd` for task tracking. Do not run git commits, git pushes, or Dolt remote sync unless explicitly asked. At handoff, report changed files, validation, and suggested next commands. +- **Minimal**: Keep tool instruction files as pointers to `bd prime`; use the same conservative git policy unless active instructions say otherwise. +- **Team-maintainer**: Only when the repository explicitly opts in, agents may close beads, run quality gates, commit, and push as part of session close. A current "do not commit" or "do not push" instruction still wins. + +## Session Completion + +This protocol applies when ending a Beads implementation workflow. It is subordinate to explicit user, repository, and orchestrator instructions. + +1. **File issues for remaining work** - Create beads for anything that needs follow-up +2. **Run quality gates** (if code changed) - Tests, linters, builds +3. **Update issue status** - Close finished work, update in-progress items +4. **Handle git/sync by active profile**: + ```bash + # Conservative/minimal/default: report status and proposed commands; wait for approval. + git status + + # Team-maintainer opt-in only, unless current instructions forbid it: + git pull --rebase + git push + git status + ``` +5. **Hand off** - Summarize changes, validation, issue status, and any blocked sync/commit/push step + +**Critical rules:** +- Explicit user or orchestrator instructions override this Beads block. +- Do not commit or push without clear authority from the active profile or the current user request. +- If a required sync or push is blocked, stop and report the exact command and error. + + diff --git a/internal/templates/agents/testdata/managed-section-full.md b/internal/templates/agents/testdata/managed-section-full.md new file mode 100644 index 000000000..831c2b736 --- /dev/null +++ b/internal/templates/agents/testdata/managed-section-full.md @@ -0,0 +1,131 @@ + +## Issue Tracking with bd (beads) + +**IMPORTANT**: This project uses **bd (beads)** for ALL issue tracking. Do NOT use markdown TODOs, task lists, or other tracking methods. + +### Why bd? + +- Dependency-aware: Track blockers and relationships between issues +- Git-friendly: Dolt-powered version control with native sync +- Agent-optimized: JSON output, ready work detection, discovered-from links +- Prevents duplicate tracking systems and confusion + +### Quick Start + +**Check for ready work:** + +```bash +bd ready --json +``` + +**Create new issues:** + +```bash +bd create "Issue title" --description="Detailed context" -t bug|feature|task -p 0-4 --json +bd create "Issue title" --description="What this issue is about" -p 1 --deps discovered-from:bd-123 --json +``` + +**Claim and update:** + +```bash +bd update --claim --json +bd update bd-42 --priority 1 --json +``` + +**Complete work:** + +```bash +bd close bd-42 --reason "Completed" --json +``` + +### Issue Types + +- `bug` - Something broken +- `feature` - New functionality +- `task` - Work item (tests, docs, refactoring) +- `epic` - Large feature with subtasks +- `chore` - Maintenance (dependencies, tooling) + +### Priorities + +- `0` - Critical (security, data loss, broken builds) +- `1` - High (major features, important bugs) +- `2` - Medium (default, nice-to-have) +- `3` - Low (polish, optimization) +- `4` - Backlog (future ideas) + +### Workflow for AI Agents + +1. **Check ready work**: `bd ready` shows unblocked issues +2. **Claim your task atomically**: `bd update --claim` +3. **Work on it**: Implement, test, document +4. **Discover new work?** Create linked issue: + - `bd create "Found bug" --description="Details about what was found" -p 1 --deps discovered-from:` +5. **Complete**: `bd close --reason "Done"` + +### Quality +- Use `--acceptance` and `--design` fields when creating issues +- Use `--validate` to check description completeness + +### Lifecycle +- `bd defer ` / `bd supersede ` for issue management +- `bd stale` / `bd orphans` / `bd lint` for hygiene +- `bd human ` to flag for human decisions +- `bd formula list` / `bd mol pour ` for structured workflows + +### Sync + +bd stores issue history in Dolt: + +- Each write auto-commits to Dolt history +- Use `bd dolt push`/`bd dolt pull` for remote sync +- Do not treat `.beads/issues.jsonl` as the sync protocol + +**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md for details and anti-patterns. + +### Important Rules + +- ✅ Use bd for ALL task tracking +- ✅ Always use `--json` flag for programmatic use +- ✅ Link discovered work with `discovered-from` dependencies +- ✅ Check `bd ready` before asking "what should I work on?" +- ❌ Do NOT create markdown TODO lists +- ❌ Do NOT use external issue trackers +- ❌ Do NOT duplicate tracking systems + +For more details, see README.md and https://github.com/gastownhall/beads/blob/main/docs/getting-started/quickstart.md. + +## Agent Context Profiles + +The managed Beads block is task-tracking guidance, not permission to override repository, user, or orchestrator instructions. + +- **Conservative (default)**: Use `bd` for task tracking. Do not run git commits, git pushes, or Dolt remote sync unless explicitly asked. At handoff, report changed files, validation, and suggested next commands. +- **Minimal**: Keep tool instruction files as pointers to `bd prime`; use the same conservative git policy unless active instructions say otherwise. +- **Team-maintainer**: Only when the repository explicitly opts in, agents may close beads, run quality gates, commit, and push as part of session close. A current "do not commit" or "do not push" instruction still wins. + +## Session Completion + +This protocol applies when ending a Beads implementation workflow. It is subordinate to explicit user, repository, and orchestrator instructions. + +1. **File issues for remaining work** - Create beads for anything that needs follow-up +2. **Run quality gates** (if code changed) - Tests, linters, builds +3. **Update issue status** - Close finished work, update in-progress items +4. **Handle git/sync by active profile**: + ```bash + # Conservative/minimal/default: report status and proposed commands; wait for approval. + git status + + # Team-maintainer opt-in only, unless current instructions forbid it: + git pull --rebase + bd dolt push + git push + git status + ``` +5. **Hand off** - Summarize changes, validation, issue status, and any blocked sync/commit/push step + +**Critical rules:** +- Explicit user or orchestrator instructions override this Beads block. +- Do not commit or push without clear authority from the active profile or the current user request. +- If a required sync or push is blocked, stop and report the exact command and error. + + diff --git a/internal/templates/agents/testdata/managed-section-minimal-no-remote.md b/internal/templates/agents/testdata/managed-section-minimal-no-remote.md new file mode 100644 index 000000000..1ab497bff --- /dev/null +++ b/internal/templates/agents/testdata/managed-section-minimal-no-remote.md @@ -0,0 +1,54 @@ + +## Beads Issue Tracker + +This project uses **bd (beads)** for issue tracking. Run `bd prime` to see full workflow context and commands. + +### Quick Reference + +```bash +bd ready # Find available work +bd show # View issue details +bd update --claim # Claim work +bd close # Complete work +``` + +### Rules + +- Use `bd` for ALL task tracking — do NOT use TodoWrite, TaskCreate, or markdown TODO lists +- Run `bd prime` for detailed command reference and session close protocol +- Use `bd remember` for durable project facts — not per-tool memory files, which fragment across accounts; keep per-operator preferences in your harness's own memory + +**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md for details and anti-patterns. + +## Agent Context Profiles + +The managed Beads block is task-tracking guidance, not permission to override repository, user, or orchestrator instructions. + +- **Conservative (default)**: Use `bd` for task tracking. Do not run git commits, git pushes, or Dolt remote sync unless explicitly asked. At handoff, report changed files, validation, and suggested next commands. +- **Minimal**: Keep tool instruction files as pointers to `bd prime`; use the same conservative git policy unless active instructions say otherwise. +- **Team-maintainer**: Only when the repository explicitly opts in, agents may close beads, run quality gates, commit, and push as part of session close. A current "do not commit" or "do not push" instruction still wins. + +## Session Completion + +This protocol applies when ending a Beads implementation workflow. It is subordinate to explicit user, repository, and orchestrator instructions. + +1. **File issues for remaining work** - Create beads for anything that needs follow-up +2. **Run quality gates** (if code changed) - Tests, linters, builds +3. **Update issue status** - Close finished work, update in-progress items +4. **Handle git/sync by active profile**: + ```bash + # Conservative/minimal/default: report status and proposed commands; wait for approval. + git status + + # Team-maintainer opt-in only, unless current instructions forbid it: + git pull --rebase + git push + git status + ``` +5. **Hand off** - Summarize changes, validation, issue status, and any blocked sync/commit/push step + +**Critical rules:** +- Explicit user or orchestrator instructions override this Beads block. +- Do not commit or push without clear authority from the active profile or the current user request. +- If a required sync or push is blocked, stop and report the exact command and error. + diff --git a/internal/templates/agents/testdata/managed-section-minimal.md b/internal/templates/agents/testdata/managed-section-minimal.md new file mode 100644 index 000000000..22c48412a --- /dev/null +++ b/internal/templates/agents/testdata/managed-section-minimal.md @@ -0,0 +1,55 @@ + +## Beads Issue Tracker + +This project uses **bd (beads)** for issue tracking. Run `bd prime` to see full workflow context and commands. + +### Quick Reference + +```bash +bd ready # Find available work +bd show # View issue details +bd update --claim # Claim work +bd close # Complete work +``` + +### Rules + +- Use `bd` for ALL task tracking — do NOT use TodoWrite, TaskCreate, or markdown TODO lists +- Run `bd prime` for detailed command reference and session close protocol +- Use `bd remember` for durable project facts — not per-tool memory files, which fragment across accounts; keep per-operator preferences in your harness's own memory + +**Architecture in one line:** issues live in a local Dolt DB; sync uses `refs/dolt/data` on your git remote; `.beads/issues.jsonl` is a passive export. See https://github.com/gastownhall/beads/blob/main/docs/core-concepts/sync-concepts.md for details and anti-patterns. + +## Agent Context Profiles + +The managed Beads block is task-tracking guidance, not permission to override repository, user, or orchestrator instructions. + +- **Conservative (default)**: Use `bd` for task tracking. Do not run git commits, git pushes, or Dolt remote sync unless explicitly asked. At handoff, report changed files, validation, and suggested next commands. +- **Minimal**: Keep tool instruction files as pointers to `bd prime`; use the same conservative git policy unless active instructions say otherwise. +- **Team-maintainer**: Only when the repository explicitly opts in, agents may close beads, run quality gates, commit, and push as part of session close. A current "do not commit" or "do not push" instruction still wins. + +## Session Completion + +This protocol applies when ending a Beads implementation workflow. It is subordinate to explicit user, repository, and orchestrator instructions. + +1. **File issues for remaining work** - Create beads for anything that needs follow-up +2. **Run quality gates** (if code changed) - Tests, linters, builds +3. **Update issue status** - Close finished work, update in-progress items +4. **Handle git/sync by active profile**: + ```bash + # Conservative/minimal/default: report status and proposed commands; wait for approval. + git status + + # Team-maintainer opt-in only, unless current instructions forbid it: + git pull --rebase + bd dolt push + git push + git status + ``` +5. **Hand off** - Summarize changes, validation, issue status, and any blocked sync/commit/push step + +**Critical rules:** +- Explicit user or orchestrator instructions override this Beads block. +- Do not commit or push without clear authority from the active profile or the current user request. +- If a required sync or push is blocked, stop and report the exact command and error. + From 36f81bab1cd7ae915debb0add694f291203506ca Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 19 Oct 2026 05:56:15 +0000 Subject: [PATCH 11/41] test(cmd/bd): pin the command tree and representative --help output (#7675) * test(cmd/bd): pin the command tree and representative --help output Add two in-process goldens for the bd command surface, so any added, removed, hidden or reworded command or flag fails until the goldens are regenerated and shows up as a reviewed testdata diff: - TestCommandTreeGolden walks the whole cobra tree (hidden commands included) and records, per command, Use, Aliases, Short, Long, Example, group, hidden/deprecated state, runnability and annotations, plus every local and persistent flag it defines (name, shorthand, type, default, usage, hidden, deprecation, annotations). Output is sorted and rendered twice to prove it is deterministic. - TestCommandHelpGolden runs `bd --help` through rootCmd for root, create, show, list, update, close, version, versions, config and serve. --help returns before the root pre-run, so no store is opened; color escapes are stripped. The snapshots are byte-identical to a built bd binary's --help output. Both run from a fresh ordinary (dependency-mode) workspace: a .beads/metadata.json with no graph marker and no database behind it. The tree is primed the way the binary is (main()'s help command and its --all/--doc/--list/--docs-root flags, ExecuteC's completion command), and cobra's history-dependent pieces (the lazily added per-command --help flag, the __complete request commands) are excluded, so the result does not depend on which in-process tests ran first. The help runs reset every flag afterwards so a later in-process command does not inherit --help. docs/cli-docs.pin holds check-doc-flags at the v1.2.2 release, so drift on main is not caught there; these goldens are the live guard. They pin the cgo build (the !cgo build swaps in a federation stub). Regenerate with BD_UPDATE_GOLDEN=1 go test -tags gms_pure_go ./cmd/bd -run '^TestCommand(Tree|Help)Golden$'. Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 (1M context) * test(cmd/bd): keep the command goldens from merging flags into the shared tree TestCommandTreeGolden read each command's flags through LocalNonPersistentFlags, and TestCommandHelpGolden dispatches `bd serve --help`. Both run cobra's mergePersistentFlags, which copies every inherited persistent flag (--json, --db, ...) into the command's own flag set for the rest of the process. TestServeFlags pins serveCmd's exact flag set, so it failed whenever a golden test ran before it in the same test binary: tree then serve flags, help then serve flags, and -test.shuffle seeds 2 and 3, on the PR head and on its merge with main. CI passes only because bd_test's 16 Bazel shards happen to run them in different processes. Of 37 candidate tests, TestServeFlags is the only one affected. - goldenLocalFlags reads the set LocalNonPersistentFlags reports without calling it: c.Flags() minus c's own persistent flags and minus inherited copies. Like cobra, it treats a flag as inherited only if it is the nearest ancestor's own *pflag.Flag, so copies left by an earlier in-process execution are skipped and a flag that shadows an inherited one still renders. - TestCommandTreeGolden fails if rendering changes the number of flags in any command's flag set. - TestCommandHelpGolden calls restoreServeGlobals, which re-registers serveCmd's flags after the `bd serve --help` dispatch. Review minors: - The regeneration command (doc comment, failure-message constant, tree.golden header) sets CGO_ENABLED=1. Without cgo the file does not build, so the command matches no test and exits 0. - enterOrdinaryWorkspaceForGolden calls isolateBeadsDirForTest. - renderCommandHelpGolden's comment says when the ANSI strip matters. - cmd/bd/AGENTS.md lists the goldens under "Adding a command" and "Guard tests". Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com> --- cmd/bd/AGENTS.md | 7 + cmd/bd/BUILD.bazel | 1 + cmd/bd/command_tree_golden_test.go | 444 + cmd/bd/testdata/command_tree/help/bd.golden | 163 + .../command_tree/help/bd_close.golden | 57 + .../command_tree/help/bd_config.golden | 122 + .../command_tree/help/bd_create.golden | 72 + .../testdata/command_tree/help/bd_list.golden | 89 + .../command_tree/help/bd_serve.golden | 119 + .../testdata/command_tree/help/bd_show.golden | 41 + .../command_tree/help/bd_update.golden | 78 + .../command_tree/help/bd_version.golden | 24 + .../command_tree/help/bd_versions.golden | 60 + cmd/bd/testdata/command_tree/tree.golden | 7461 +++++++++++++++++ 14 files changed, 8738 insertions(+) create mode 100644 cmd/bd/command_tree_golden_test.go create mode 100644 cmd/bd/testdata/command_tree/help/bd.golden create mode 100644 cmd/bd/testdata/command_tree/help/bd_close.golden create mode 100644 cmd/bd/testdata/command_tree/help/bd_config.golden create mode 100644 cmd/bd/testdata/command_tree/help/bd_create.golden create mode 100644 cmd/bd/testdata/command_tree/help/bd_list.golden create mode 100644 cmd/bd/testdata/command_tree/help/bd_serve.golden create mode 100644 cmd/bd/testdata/command_tree/help/bd_show.golden create mode 100644 cmd/bd/testdata/command_tree/help/bd_update.golden create mode 100644 cmd/bd/testdata/command_tree/help/bd_version.golden create mode 100644 cmd/bd/testdata/command_tree/help/bd_versions.golden create mode 100644 cmd/bd/testdata/command_tree/tree.golden diff --git a/cmd/bd/AGENTS.md b/cmd/bd/AGENTS.md index 5ddbc95f5..fcaf8bef8 100644 --- a/cmd/bd/AGENTS.md +++ b/cmd/bd/AGENTS.md @@ -38,6 +38,10 @@ Which tier a test belongs in, and which command runs it, is decided by 6. Regenerate the CLI reference with `scripts/generate-cli-docs.sh`; `scripts/check-cli-docs-drift.sh` fails a PR whose command-surface change leaves the generated docs stale. +7. Regenerate the command-surface goldens and review the testdata diff: + `BD_UPDATE_GOLDEN=1 CGO_ENABLED=1 go test -tags gms_pure_go ./cmd/bd -run '^TestCommand(Tree|Help)Golden$'`. + `TestCommandTreeGolden` fails on any added, removed or reworded command or + flag until you do. Prefer a flag on an existing command over a new command, and route recovery/repair operations to `bd doctor --fix`; see the CLI design principles @@ -85,6 +89,9 @@ no linter enforces it). or serialize the call under `stdioMutex`. - `TestCorpusGolden` (`protocol/`) — no unreviewed change to the `--json` wire shapes. +- `TestCommandTreeGolden` and `TestCommandHelpGolden` — no unreviewed change + to the command tree (every command and flag, hidden ones included) or to + the pinned `--help` output in `testdata/command_tree/`. Fresh-workspace command fixtures call `isolateBeadsDirForTest(t)` before setup or dispatch and must not use `t.Parallel()`. diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index 680301167..cfff7fd67 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -605,6 +605,7 @@ go_test( "cobra_flag_reset_test.go", "codex_hook_test.go", "command_telemetry_test.go", + "command_tree_golden_test.go", "comment_promote_embedded_test.go", "comment_proxied_integration_test.go", "comment_reserved_word_test.go", diff --git a/cmd/bd/command_tree_golden_test.go b/cmd/bd/command_tree_golden_test.go new file mode 100644 index 000000000..b22600883 --- /dev/null +++ b/cmd/bd/command_tree_golden_test.go @@ -0,0 +1,444 @@ +//go:build cgo + +// cgo only: the goldens pin the cgo build, which every lane that runs this +// package's full suite compiles. The !cgo build replaces the federation +// subtree with a one-command stub (federation_nocgo.go), so its tree differs +// by design. + +package main + +import ( + "bytes" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "testing" + + "github.com/charmbracelet/x/ansi" + "github.com/spf13/cobra" + "github.com/spf13/pflag" +) + +// The committed command-surface goldens. Regenerate after an intentional CLI +// change, then review the diff like code: +// +// BD_UPDATE_GOLDEN=1 CGO_ENABLED=1 go test -tags gms_pure_go ./cmd/bd -run '^TestCommand(Tree|Help)Golden$' +// +// CGO_ENABLED=1 because this file builds only with cgo: with cgo off the +// command matches no test, exits 0 and leaves the goldens as they were. +const ( + commandTreeGoldenPath = "testdata/command_tree/tree.golden" + commandHelpGoldenDir = "testdata/command_tree/help" + commandGoldenRegen = "BD_UPDATE_GOLDEN=1 CGO_ENABLED=1 go test -tags gms_pure_go ./cmd/bd -run '^TestCommand(Tree|Help)Golden$'" +) + +// commandHelpGoldenArgs is the representative set of commands whose exact +// --help output is pinned, each as the argv after "bd". The tree golden +// covers every command's metadata; these cover the rendering (usage line, +// flag table layout, inherited global flags, root command groups). +var commandHelpGoldenArgs = [][]string{ + {}, + {"create"}, + {"show"}, + {"list"}, + {"update"}, + {"close"}, + {"version"}, + {"versions"}, + {"config"}, + {"serve"}, +} + +// TestCommandTreeGolden pins the whole bd command tree: for every command and +// subcommand, hidden ones included, its Use, Aliases, Short, Long, Example, +// group, hidden/deprecated state, runnability and annotations, and every +// local and persistent flag it defines (name, shorthand, type, default, +// usage, hidden, deprecation, annotations). Any added, removed or reworded +// command or flag fails this test until the golden is regenerated, so the +// change shows up as a reviewed testdata diff. It reads the in-memory tree +// only: no store or subprocess. +// +// docs/cli-docs.pin pins the generated CLI reference (and check-doc-flags) +// to a release, so drift on main is not caught there; this golden is the +// live guard for the command surface. +func TestCommandTreeGolden(t *testing.T) { + pkgDir := enterOrdinaryWorkspaceForGolden(t) + primeCommandTreeForGolden() + flagCounts := commandFlagCounts(rootCmd) + got := renderCommandTreeGolden(rootCmd) + // A renderer that iterated a map would pass on the run that regenerated + // the golden and flake afterwards. + if again := renderCommandTreeGolden(rootCmd); again != got { + t.Fatalf("command tree rendering is not deterministic:\n%s", goldenLineDiff(got, again)) + } + // Nor may rendering change the tree it reads: tests sharing this binary + // pin a command's exact flag set (TestServeFlags). See goldenLocalFlags. + if after := commandFlagCounts(rootCmd); after != flagCounts { + t.Fatalf("rendering the command tree added flags to a command's own flag set:\n%s", goldenLineDiff(flagCounts, after)) + } + checkCommandGolden(t, pkgDir, commandTreeGoldenPath, got) +} + +// TestCommandHelpGolden pins the exact `bd --help` output of a +// representative set of commands, executed in-process through the real root +// command. --help returns before the root pre-run, so no store is opened. +// Color escapes are stripped so the snapshot is independent of terminal +// detection; otherwise the snapshots are byte-identical to what a built bd +// prints. +func TestCommandHelpGolden(t *testing.T) { + pkgDir := enterOrdinaryWorkspaceForGolden(t) + // Parsing `bd serve --help` merges the root's persistent flags into + // serveCmd's own flag set, which TestServeFlags pins; restoreServeGlobals + // re-registers that set afterwards. + restoreServeGlobals(t) + + want := map[string]bool{} + for _, args := range commandHelpGoldenArgs { + name := strings.Join(append([]string{"bd"}, args...), "_") + file := name + ".golden" + want[file] = true + t.Run(name, func(t *testing.T) { + checkCommandGolden(t, pkgDir, filepath.Join(commandHelpGoldenDir, file), renderCommandHelpGolden(t, args)) + }) + } + + // The directory holds exactly the listed snapshots: a command dropped + // from the list must not leave a stale file that still looks pinned. + dir := filepath.Join(pkgDir, commandHelpGoldenDir) + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatalf("read %s: %v", commandHelpGoldenDir, err) + } + for _, e := range entries { + if want[e.Name()] { + continue + } + if os.Getenv("BD_UPDATE_GOLDEN") == "1" { + if err := os.Remove(filepath.Join(dir, e.Name())); err != nil { + t.Fatalf("remove stale help golden %s: %v", e.Name(), err) + } + t.Logf("removed stale help golden %s", e.Name()) + continue + } + t.Errorf("%s/%s is not in commandHelpGoldenArgs; delete it or regenerate with %s", commandHelpGoldenDir, e.Name(), commandGoldenRegen) + } +} + +// enterOrdinaryWorkspaceForGolden moves the test into a fresh ordinary +// (dependency-mode) workspace: a .beads/metadata.json with no graph marker +// and no database behind it, so nothing can be read from or written to a +// store. It returns the package directory, where the goldens live. +func enterOrdinaryWorkspaceForGolden(t *testing.T) string { + t.Helper() + isolateBeadsDirForTest(t) + pkgDir, err := os.Getwd() + if err != nil { + t.Fatal(err) + } + ws := t.TempDir() + beadsDir := filepath.Join(ws, ".beads") + if err := os.Mkdir(beadsDir, 0o750); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(beadsDir, "metadata.json"), []byte(`{"backend":"dolt"}`), 0o600); err != nil { + t.Fatal(err) + } + t.Chdir(ws) + return pkgDir +} + +// primeCommandTreeForGolden completes the tree the way the bd binary does +// before it dispatches: main() adds cobra's help command and its +// --all/--doc/--list/--docs-root flags, and ExecuteC adds the completion +// command. All three are idempotent, so the tree is the same whether or not +// an earlier in-process test already executed rootCmd. +func primeCommandTreeForGolden() { + rootCmd.InitDefaultHelpCmd() + registerHelpAllFlag() + rootCmd.InitDefaultCompletionCmd() +} + +// renderCommandTreeGolden walks cmd's tree depth first, children sorted by +// name, and renders one block per command. +func renderCommandTreeGolden(root *cobra.Command) string { + var b strings.Builder + b.WriteString("# bd command tree (cgo build), pinned by TestCommandTreeGolden.\n") + b.WriteString("# Regenerate after an intentional CLI change:\n") + b.WriteString("# " + commandGoldenRegen + "\n") + + var walk func(*cobra.Command) + walk = func(c *cobra.Command) { + writeCommandGoldenBlock(&b, c) + children := append([]*cobra.Command(nil), c.Commands()...) + sort.Slice(children, func(i, j int) bool { return children[i].Name() < children[j].Name() }) + for _, sub := range children { + // cobra adds the hidden shell-completion request commands only + // for an execution that invokes them, and then never removes + // them: whether they are present depends on test history, not + // on the bd surface. + if sub.Name() == cobra.ShellCompRequestCmd || sub.Name() == cobra.ShellCompNoDescRequestCmd { + continue + } + walk(sub) + } + } + walk(root) + return b.String() +} + +func writeCommandGoldenBlock(b *strings.Builder, c *cobra.Command) { + fmt.Fprintf(b, "\n%s\n", c.CommandPath()) + fmt.Fprintf(b, " use: %q\n", c.Use) + if len(c.Aliases) > 0 { + fmt.Fprintf(b, " aliases: %q\n", c.Aliases) + } + fmt.Fprintf(b, " short: %q\n", c.Short) + if c.GroupID != "" { + fmt.Fprintf(b, " group: %q\n", c.GroupID) + } + fmt.Fprintf(b, " hidden: %t\n", c.Hidden) + if c.Deprecated != "" { + fmt.Fprintf(b, " deprecated: %q\n", c.Deprecated) + } + fmt.Fprintf(b, " runnable: %t\n", c.Runnable()) + if c.DisableFlagParsing { + b.WriteString(" disable-flag-parsing: true\n") + } + if len(c.ValidArgs) > 0 { + fmt.Fprintf(b, " valid-args: %q\n", c.ValidArgs) + } + if len(c.Annotations) > 0 { + fmt.Fprintf(b, " annotations: %s\n", formatGoldenAnnotations(c.Annotations)) + } + writeGoldenTextBlock(b, "long", c.Long) + writeGoldenTextBlock(b, "example", c.Example) + for _, line := range commandGoldenFlagLines(c.PersistentFlags()) { + fmt.Fprintf(b, " persistent-flag %s\n", line) + } + for _, line := range commandGoldenFlagLines(goldenLocalFlags(c)) { + fmt.Fprintf(b, " flag %s\n", line) + } +} + +// goldenLocalFlags returns the non-persistent flags c registers itself, +// including any that shadow an inherited flag: the set +// c.LocalNonPersistentFlags() reports, read without calling it. That +// accessor, like LocalFlags and InheritedFlags, runs cobra's +// mergePersistentFlags, which copies every inherited persistent flag into +// c.Flags() for the rest of the process. Over the whole tree that would leave +// `bd serve` carrying the root's --json, --db and the rest, and break +// TestServeFlags (see TestNoSubcommandShadowsRootJSONFlag). +// +// c.Flags() may already hold such copies from an earlier in-process +// execution. They are recognized by identity, the way cobra itself tells +// them apart: an inherited flag is the nearest ancestor's own *pflag.Flag of +// that name, and a shadowing flag is a different one. +func goldenLocalFlags(c *cobra.Command) *pflag.FlagSet { + local := pflag.NewFlagSet(c.Name(), pflag.ContinueOnError) + c.Flags().VisitAll(func(f *pflag.Flag) { + if c.PersistentFlags().Lookup(f.Name) != nil { + return // rendered as a persistent-flag line + } + for p := c.Parent(); p != nil; p = p.Parent() { + if inherited := p.PersistentFlags().Lookup(f.Name); inherited != nil { + if inherited == f { + return + } + break + } + } + local.AddFlag(f) + }) + return local +} + +// commandFlagCounts renders how many flags each command's own flag set holds, +// one line per command, read without merging anything into it. +func commandFlagCounts(root *cobra.Command) string { + var b strings.Builder + var walk func(*cobra.Command) + walk = func(c *cobra.Command) { + n := 0 + c.Flags().VisitAll(func(*pflag.Flag) { n++ }) + fmt.Fprintf(&b, "%s %d\n", c.CommandPath(), n) + for _, sub := range c.Commands() { + walk(sub) + } + } + walk(root) + return b.String() +} + +// writeGoldenTextBlock writes a multi-line field one source line per golden +// line, behind a "|" gutter so leading whitespace and blank lines survive. +func writeGoldenTextBlock(b *strings.Builder, name, text string) { + if text == "" { + return + } + fmt.Fprintf(b, " %s:\n", name) + for _, line := range strings.Split(text, "\n") { + if line == "" { + b.WriteString(" |\n") + continue + } + fmt.Fprintf(b, " | %s\n", line) + } +} + +// commandGoldenFlagLines renders each flag of fs on one line, sorted by name. +// cobra's own lazily added --help flag (FlagSetByCobraAnnotation) is skipped: +// it exists only on commands an earlier in-process execution reached. +func commandGoldenFlagLines(fs *pflag.FlagSet) []string { + var flags []*pflag.Flag + fs.VisitAll(func(f *pflag.Flag) { + if _, byCobra := f.Annotations[cobra.FlagSetByCobraAnnotation]; byCobra { + return + } + flags = append(flags, f) + }) + sort.Slice(flags, func(i, j int) bool { return flags[i].Name < flags[j].Name }) + + lines := make([]string, 0, len(flags)) + for _, f := range flags { + var b strings.Builder + fmt.Fprintf(&b, "--%s", f.Name) + if f.Shorthand != "" { + fmt.Fprintf(&b, " -%s", f.Shorthand) + } + fmt.Fprintf(&b, " type=%s default=%q", f.Value.Type(), f.DefValue) + if f.NoOptDefVal != "" && !implicitNoOptDefault(f) { + fmt.Fprintf(&b, " no-opt-default=%q", f.NoOptDefVal) + } + if f.Hidden { + b.WriteString(" hidden") + } + if f.Deprecated != "" { + fmt.Fprintf(&b, " deprecated=%q", f.Deprecated) + } + if f.ShorthandDeprecated != "" { + fmt.Fprintf(&b, " shorthand-deprecated=%q", f.ShorthandDeprecated) + } + if len(f.Annotations) > 0 { + fmt.Fprintf(&b, " annotations=%s", formatGoldenAnnotations(f.Annotations)) + } + fmt.Fprintf(&b, " usage=%q", f.Usage) + lines = append(lines, b.String()) + } + return lines +} + +// implicitNoOptDefault reports whether f's NoOptDefVal is the one pflag gives +// every flag of its type (a bare --bool means true, a bare --count adds one), +// which would only repeat the type on each line. +func implicitNoOptDefault(f *pflag.Flag) bool { + switch f.Value.Type() { + case "bool": + return f.NoOptDefVal == "true" + case "count": + return f.NoOptDefVal == "+1" + } + return false +} + +// formatGoldenAnnotations renders a command's or a flag's annotations sorted +// by key. +func formatGoldenAnnotations[V string | []string](m map[string]V) string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + parts := make([]string, 0, len(keys)) + for _, k := range keys { + parts = append(parts, fmt.Sprintf("%s=%q", k, m[k])) + } + return "{" + strings.Join(parts, " ") + "}" +} + +// renderCommandHelpGolden runs `bd --help` through rootCmd in-process +// and returns its stdout with color escapes stripped. cobra keeps the parsed +// --help value on the shared tree, so every flag is put back afterwards: a +// later in-process `bd create ...` must not print help instead. +// +// internal/ui decides color once, at init, from CLICOLOR_FORCE and whether the +// test binary's stdout is a terminal. CI never gives it one, so there the +// strip removes nothing; it is what keeps a run from a terminal matching the +// goldens. +func renderCommandHelpGolden(t *testing.T, args []string) string { + t.Helper() + argv := append(append([]string(nil), args...), "--help") + primeCommandTreeForGolden() + var execErr error + out := captureStdout(t, func() error { + rootCmd.SetArgs(argv) + defer rootCmd.SetArgs(nil) + defer resetCommandFlags(rootCmd) + _, execErr = rootCmd.ExecuteC() + return nil + }) + if execErr != nil { + t.Fatalf("bd %s: %v", strings.Join(argv, " "), execErr) + } + return ansi.Strip(out) +} + +// checkCommandGolden compares got with the golden at rel under pkgDir, or +// rewrites the golden when BD_UPDATE_GOLDEN=1 (the package's shared golden +// switch). +func checkCommandGolden(t *testing.T, pkgDir, rel, got string) { + t.Helper() + path := filepath.Join(pkgDir, rel) + if os.Getenv("BD_UPDATE_GOLDEN") == "1" { + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir %s: %v", filepath.Dir(rel), err) + } + if err := os.WriteFile(path, []byte(got), 0o644); err != nil { + t.Fatalf("write golden %s: %v", rel, err) + } + t.Logf("updated golden %s", rel) + return + } + want, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read golden %s (regenerate with %s): %v", rel, commandGoldenRegen, err) + } + if bytes.Equal([]byte(got), want) { + return + } + t.Errorf("bd command surface drifted from %s.\n%s\nIf the change is intended, regenerate with\n %s\nand review the testdata diff.", + rel, goldenLineDiff(string(want), got), commandGoldenRegen) +} + +// goldenLineDiff reports the region where want and got differ: the common +// leading and trailing lines are trimmed and the rest is shown as -/+ lines, +// capped so a large drift stays readable (the reviewed artifact is the +// regenerated testdata diff, not this message). +func goldenLineDiff(want, got string) string { + const maxLines = 40 + w := strings.Split(want, "\n") + g := strings.Split(got, "\n") + pre := 0 + for pre < len(w) && pre < len(g) && w[pre] == g[pre] { + pre++ + } + suf := 0 + for suf < len(w)-pre && suf < len(g)-pre && w[len(w)-1-suf] == g[len(g)-1-suf] { + suf++ + } + var b strings.Builder + fmt.Fprintf(&b, "first difference at line %d:\n", pre+1) + write := func(sign string, lines []string) { + for i, line := range lines { + if i == maxLines { + fmt.Fprintf(&b, "%s ... (%d more lines)\n", sign, len(lines)-maxLines) + return + } + fmt.Fprintf(&b, "%s %s\n", sign, line) + } + } + write("-", w[pre:len(w)-suf]) + write("+", g[pre:len(g)-suf]) + return b.String() +} diff --git a/cmd/bd/testdata/command_tree/help/bd.golden b/cmd/bd/testdata/command_tree/help/bd.golden new file mode 100644 index 000000000..619e70296 --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd.golden @@ -0,0 +1,163 @@ +Issues chained together like beads. A lightweight issue tracker with first-class dependency support. + +Usage: + bd [flags] + bd [command] + +Working With Issues: + assign Assign an issue to someone + children List child beads of a parent + close Close one or more issues + comment Add a comment to an issue + comments View or manage comments on an issue + create Create a new issue (or batch from markdown/graph JSON) + create-form Create a new issue using an interactive form + delete Delete one or more issues and clean up references + edit Edit an issue field in $EDITOR + gate Manage async coordination gates + heartbeat Refresh the lease on an issue you hold in_progress + label Manage issue labels + link Link two issues with a dependency + list List issues + merge-slot Manage merge-slot gates for serialized conflict resolution + note Append a note to an issue + priority Set the priority of an issue + promote Promote a wisp to a permanent bead + provenance Append-only provenance event log + q Quick capture: create issue and output only ID + query Query issues using a simple query language + reclaim Revert stale-lease in_progress issues back to ready (dead-worker recovery) + reopen Reopen one or more closed issues + search Search issues by text query + set-state Set operational state (creates event + updates label) + show Show issue details + state Query the current value of a state dimension + tag Add a label to an issue + todo Manage TODO items (convenience wrapper for task issues) + unclaim Release a claimed issue + update Update one or more issues + +Views & Reports: + count Count issues matching filters + diff Show changes between two commits or branches + find-duplicates Find semantically similar issues using text analysis or AI + history Show version history for an issue + lint Check issues for missing template sections + stale Show stale issues (not updated recently) + status Show issue database overview and statistics + statuses List valid issue statuses + types List valid issue types + versions List the recorded versions of a bead + +Dependencies & Structure: + dep Manage dependencies + duplicate Mark an issue as a duplicate of another + duplicates Find and optionally merge duplicate issues + epic Epic management commands + graph Display issue dependency graph + supersede Mark an issue as superseded by a replacement issue + swarm Swarm management for structured epics + +Sync & Data: + backup Back up your beads database + branch List or create branches + conflicts Inspect and resolve live merge conflicts + export Export issues to JSONL format + federation Manage peer-to-peer federation with other workspaces + import Import issues from a JSONL file or stdin into the database + restore Restore the pre-compaction content of a compacted issue + sync Pull, check for conflicts, repair is_blocked, and push (the federation loop) + vc Version control operations + +Setup & Configuration: + bootstrap Non-destructive database setup for fresh clones and recovery + config Manage configuration settings + connect Attach this workspace to a remote bd serve over HTTP + context Show effective backend identity and repository context + dolt Configure Dolt database settings + forget Remove a persistent memory + hooks Manage git hooks for beads integration + human Show essential commands for human users + info Show database information + init Initialize bd in the current directory + kv Key-value store commands + memories List or search persistent memories + migrate-personal Move personal planning issues from the project database to your planning repo + onboard Display minimal snippet for agent instructions file + prime Output AI-optimized workflow context + quickstart Quick start guide for bd + recall Retrieve a specific memory + remember Store a persistent memory + setup Setup integration with AI editors + where Show active beads location + +Maintenance: + batch Run multiple write operations in a single database transaction + compact Squash old Dolt commits to reduce history size + doctor Check and fix beads installation health (start here) + events Read and manage the durable events journal + flatten Squash all Dolt history into a single commit + gc Garbage collect: decay old issues, compact Dolt commits, run Dolt GC + migrate Database migration commands + ping Check database connectivity + preflight Show PR readiness checklist + prune Delete old closed beads to reclaim space and shrink exports + purge Delete closed ephemeral beads to reclaim space + recompute-blocked Recompute is_blocked for all issues (repairs stale flags after a pull) + rename-prefix Rename the issue prefix for all issues in the database + rules Audit and compact Claude rules + sql Execute raw SQL against the beads database + upgrade Check and manage bd version upgrades + worktree Manage git worktrees for parallel development + +Integrations & Advanced: + admin Administrative commands for database maintenance + jira Jira integration commands + linear Linear integration commands + repo Manage multiple repository configuration + +Additional Commands: + ado Azure DevOps integration commands + audit Record and label agent interactions (append-only JSONL) + blocked Show blocked issues + completion Generate the autocompletion script for the specified shell + cook Compile a formula into a proto (ephemeral by default) + defer Defer one or more issues for later + formula Manage workflow formulas + github GitHub integration commands + gitlab GitLab integration commands + help Help about any command + init-safety Explain bd init flag semantics and the destroy-token format + mail Delegate to mail provider (e.g., gt mail) + metrics Show or change anonymous usage-metrics settings + mol Molecule commands (work templates) + notion Notion integration commands + orphans Identify orphaned issues (referenced in commits but still open) + ready Show ready work (open, no active blockers) + rename Rename an issue ID + schema Print the JSON Schema for bd's --json / export output + serve Serve the beads HTTP API over loopback + ship Publish a capability for cross-project dependencies + undefer Undefer one or more issues (restore to open) + version Print version information + +Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + -h, --help help for bd + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output + -V, --version Print version information + +Use "bd [command] --help" for more information about a command. diff --git a/cmd/bd/testdata/command_tree/help/bd_close.golden b/cmd/bd/testdata/command_tree/help/bd_close.golden new file mode 100644 index 000000000..eefb0db30 --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd_close.golden @@ -0,0 +1,57 @@ +Close one or more issues. + +If no issue ID is provided, closes the last touched issue (from most recent +create, update, show, or close operation). This fallback only applies in +interactive sessions (stdin is a terminal); in scripts and agent sessions a +missing ID is an error, so a command built from an empty variable cannot +silently close an unrelated issue. Set BD_LAST_TOUCHED_FALLBACK=1 to allow +the fallback anywhere, or =0 to disable it entirely. + +When closing multiple issues, provide one --reason for all IDs or repeat +--reason once per ID. Reasons map positionally: the first --reason applies +to the first ID, the second --reason to the second ID, regardless of where +the flags appear in the command line. + +Closing a molecule's last open step also closes the molecule's root, with +the reason "all steps complete", when the root is a poured molecule, a wisp, +or an epic labeled "template"; any other epic stays open as close-eligible +work. A root that would refuse its own close (an open blocker, a pin, another +assignee) stays open with a warning, and the step's close stands. Under +--json, the step's entry names the closed root (auto_closed_molecule) or the +refusal (molecule_auto_close_refusal). --continue claims the next ready step +in the molecule for you; add --no-auto to only show it. + +Usage: + bd close [id...] [flags] + +Aliases: + close, done + +Flags: + --claim-next Automatically claim the next highest priority available issue + --continue Auto-advance to next step in molecule + -f, --force Force close pinned issues or unsatisfied gates + -h, --help help for close + --if-revision bd show --json Apply the write only if the issue's current revision equals this value (the decimal int64 from bd show --json's "revision" field). One id only: refused before any write when combined with more than one issue id. A mismatch writes nothing and exits 13 (vs 1 for other failures). + --no-auto With --continue, show next step but don't claim it + -r, --reason string Reason for closing + --reason-file string Read close reason from file (use - for stdin) + --session string Claude Code session ID (or set CLAUDE_SESSION_ID env var) + --suggest-next Show newly unblocked issues after closing + +Global Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output diff --git a/cmd/bd/testdata/command_tree/help/bd_config.golden b/cmd/bd/testdata/command_tree/help/bd_config.golden new file mode 100644 index 000000000..a5bed0d60 --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd_config.golden @@ -0,0 +1,122 @@ +Manage configuration settings for external integrations and preferences. + +Configuration is stored per-project in the beads database and is version-control-friendly. + +Common namespaces: + - export.* Auto-export settings (stored in config.yaml) + - import.* JSONL import settings (stored in config.yaml) + - jira.* Jira integration settings + - linear.* Linear integration settings + - github.* GitHub integration settings + - gitlab.* GitLab integration settings + - ado.* Azure DevOps integration settings + - notion.* Notion integration settings + - custom.* Custom integration settings + - status.* Issue status configuration + - claim.* Claim arbitration settings (pool-aware claiming) + - doctor.suppress.* Suppress specific bd doctor warnings (GH#1095) + - lint.* Additional per-type lint sections (stored in config.yaml) + +Auto-Export (config.yaml): + Optional JSONL export to .beads/issues.jsonl after write commands (throttled). + Useful for viewers (bv), interchange, and issue-level migration; not a backup. + It is not cross-machine sync; use bd dolt push/pull with a Dolt remote. + Disabled by default. Enable only for integrations that need fresh JSONL. + Auto-staging is separate and disabled by default. + + Keys: + export.auto Enable/disable auto-export (default: false) + export.path Output filename relative to .beads/ (default: issues.jsonl) + export.interval Minimum time between exports (default: 60s) + export.git-add Auto-stage the export file (default: false) + +Auto-Import (config.yaml): + Reads .beads/issues.jsonl by default when a JSONL import path is implied. + Use a relative filename/path so the import stays within the project .beads/ + directory and remains portable across machines. + + Keys: + import.path Input filename relative to .beads/ (default: issues.jsonl) + +Custom Status States: + You can define custom status states for multi-step pipelines using the + status.custom config key. Statuses should be comma-separated. + + Example: + bd config set status.custom "awaiting_review,awaiting_testing,awaiting_docs" + + This enables issues to use statuses like 'awaiting_review' in addition to + the built-in statuses (open, in_progress, blocked, deferred, closed). + +Claim Pools: + A dispatcher can pre-assign issues to a pool pseudo-assignee (e.g. + "fable-crew") and let any actor take them with --claim. List the pool + aliases in the claim.pools config key, comma-separated: + + bd config set claim.pools "fable-crew,night-crew" + + Issues assigned to a real actor (or to an alias not in the list) keep + their anti-steal protection. Pool takes carry the normal lease; note + that if a taker's lease expires, bd reclaim returns the issue to the + unassigned pool, not to the pool alias it was dispatched to. + +Suppressing Doctor Warnings: + Suppress specific bd doctor warnings by check name slug: + bd config set doctor.suppress.pending-migrations true + bd config set doctor.suppress.git-hooks true + Check names are converted to slugs: "Git Hooks" → "git-hooks". + Only warnings are suppressed (errors and passing checks always show). + To unsuppress: bd config unset doctor.suppress. + +Examples: + bd config set export.auto true # Enable auto-export for viewer integrations + bd config set export.path "beads.jsonl" # Custom export filename + bd config set import.path "beads.jsonl" # Custom import filename + bd config set export.git-add true # Also stage the export file + bd config set jira.url "https://company.atlassian.net" + bd config set jira.project "PROJ" + bd config set status.custom "awaiting_review,awaiting_testing" + bd config set claim.pools "fable-crew,night-crew" # Pool aliases claimable by any actor + bd config set doctor.suppress.pending-migrations true + bd config set dolt.debug true # Enable Dolt sql-server debug mode (loglevel=debug, --prof cpu) + bd config set dolt.local-only true # Skip wiring a Dolt sync remote during bd init + bd config set lint.sections.epic "Standards scorecard, Cost" # Extra sections bd lint requires for epics + bd config get export.auto + bd config list + bd config unset jira.url + +Usage: + bd config [command] + +Available Commands: + apply Reconcile system state to match configuration + drift Detect config-vs-reality inconsistencies + get Get a configuration value + list List all configuration + set Set a configuration value + set-many Set multiple configuration values in one operation + show Show all effective configuration with provenance + unset Delete a configuration value + validate Validate sync-related configuration + +Flags: + -h, --help help for config + +Global Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output + +Use "bd config [command] --help" for more information about a command. diff --git a/cmd/bd/testdata/command_tree/help/bd_create.golden b/cmd/bd/testdata/command_tree/help/bd_create.golden new file mode 100644 index 000000000..adc90d852 --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd_create.golden @@ -0,0 +1,72 @@ +Create a new issue (or batch from markdown/graph JSON) + +Usage: + bd create [title] [flags] + +Aliases: + create, new + +Flags: + --acceptance string Acceptance criteria + --allow-empty-description Allow empty description input from stdin or file + --append-notes string Append to existing notes (with newline separator) + -a, --assignee string Assignee + --body-file string Read description from file (use - for stdin) + --context string Additional context for the issue + --defer string Defer until date (issue hidden from bd ready until then). Same formats as --due + --deps strings Dependencies as 'type:id' or bare 'id'. Bare 'id', 'depends-on:id', and 'blocked-by:id' all make THIS issue depend on id; 'blocks:id' reverses direction (id depends on this issue). E.g. 'blocked-by:bd-20,discovered-from:bd-15' + -d, --description string Issue description + --design string Design notes + --design-file string Read design from file (use - for stdin) + --dry-run Preview what would be created without actually creating + --due string Due date/time. Formats: +6h, +1d, +2w, tomorrow, next monday, 2025-01-15 + --ephemeral Create as ephemeral (short-lived, subject to TTL compaction) + -e, --estimate int Time estimate in minutes (e.g., 60 for 1 hour) + --event-actor string Entity URI who caused this event (requires --type=event) + --event-category string Event category (e.g., patrol.muted, agent.started) (requires --type=event) + --event-payload string Event-specific JSON data (requires --type=event) + --event-target string Entity URI or bead ID affected (requires --type=event) + --external-ref string External reference (e.g., 'gh-9', 'jira-ABC', Linear URL) + -f, --file string Create one issue per ## heading (### fields attach to that issue); for title plus description-from-file, use --body-file + --force Force creation even if prefix doesn't match database prefix + --graph string Create a graph of issues with dependencies from JSON plan file + -h, --help help for create + --id string Explicit issue ID (e.g., 'bd-42' for partitioning) + -l, --labels strings Labels (comma-separated) + --metadata string Set custom metadata (JSON object, or @file.json to read from file) + --mol-type string Molecule type: swarm (multi-agent), patrol (recurring ops), work (default) + --no-history Skip Dolt commit history without making GC-eligible (for permanent agent beads) + --no-inherit-labels Don't inherit labels from parent issue + --notes string Notes + --parent string Parent issue ID for hierarchical child (e.g., 'bd-a3f8e9') + -p, --priority string Priority (0-4 or P0-P4, 0=highest) (default "2") + --repo string Target repository for issue (overrides auto-routing) + --silent Output only the issue ID (for scripting) + --skills string Required skills for this issue + --spec-id string Link to specification document + -s, --status string Initial status + --stdin Read description from stdin (alias for --body-file -) + --storage-class string Storage class: versioned, unversioned, or ephemeral (default: storage-class. config, else versioned) + --title string Issue title (alternative to positional argument) + -t, --type string Issue type (bug|feature|task|epic|chore|decision|spike|story|milestone); custom types require types.custom config; aliases: enhancement/feat→feature, dec/adr→decision (default "task") + --validate Validate description contains required sections for issue type + --waits-for string Spawner issue ID to wait for (creates waits-for dependency for fanout gate) + --waits-for-gate string Gate type: all-children (wait for all) or any-children (wait for first) (default "all-children") + --wisp-type string Wisp type for TTL-based compaction: heartbeat, ping, patrol, gc_report, recovery, error, escalation + +Global Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output diff --git a/cmd/bd/testdata/command_tree/help/bd_list.golden b/cmd/bd/testdata/command_tree/help/bd_list.golden new file mode 100644 index 000000000..d9c582c3a --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd_list.golden @@ -0,0 +1,89 @@ +List issues + +Usage: + bd list [flags] + +Flags: + --all Show all issues including closed (overrides default filter) + -a, --assignee string Filter by assignee + --brief Omit the free-form text (description, design, acceptance criteria, notes, payload, waiters) from each row. Filters that read those fields, such as --desc-contains, still select on them. An omitted field is indistinguishable from an empty one in --json; fetch a whole issue with bd show. + --closed-after string Filter issues closed after date (YYYY-MM-DD or RFC3339) + --closed-before string Filter issues closed before date (YYYY-MM-DD or RFC3339) + --created-after string Filter issues created after date (YYYY-MM-DD or RFC3339) + --created-before string Filter issues created before date (YYYY-MM-DD or RFC3339) + --defer-after string Filter issues deferred after date (supports relative: +6h, tomorrow) + --defer-before string Filter issues deferred before date (supports relative: +6h, tomorrow) + --deferred Show only issues with defer_until set + --deps string[="scheduling"] Annotate tree with dependency edges and order siblings by them: 'scheduling' (bare --deps) or 'all' + --desc-contains string Filter by description substring (case-insensitive) + --due-after string Filter issues due after date (supports relative: +6h, tomorrow) + --due-before string Filter issues due before date (supports relative: +6h, tomorrow) + --empty-description Filter issues with empty or missing description + --exclude-label strings Exclude issues that have ANY of these labels + --exclude-type strings Exclude issue types from results (comma-separated or repeatable, e.g., --exclude-type=convoy,epic) + --external-contains string Filter by external ref substring (case-insensitive) + --external-ref string Filter by exact external_ref value + --flat Disable tree format and use legacy flat list output + --format string Export the dependency graph of the listed issues: 'digraph' (edge list for golang.org/x/tools/cmd/digraph), 'dot' (Graphviz), or a Go template rendered once per edge over .IssueID, .DependsOnID, .Type, .Issue, .Dependency. 'json' is the exception and does not export the graph: it is equivalent to --json, which also wins when both flags are given. 'digraph' and templates print nothing when no listed issue depends on another; 'dot' still prints the nodes + --has-metadata-key string Filter issues that have this metadata key set + -h, --help help for list + --id string Filter by specific issue IDs (comma-separated or repeated, e.g., bd-1,bd-5,bd-10) + --include-comments Populate each row's comments field with full comment bodies in JSON output (--json only; costs one query per row that has comments). Without it, a row with comments carries comments_omitted=true, so an absent comments field is never mistaken for having none. Text output is unaffected either way. + --include-ephemeral Include ephemeral wisp-plane rows in output (normally hidden) + --include-gates Include gate issues in output (normally hidden) + --include-infra Include infrastructure beads (agent/role/message) in output + --include-templates Include template molecules in output + -l, --label strings Filter by labels (AND: must have ALL). Can combine with --label-any + --label-any strings Filter by labels (OR: must have AT LEAST ONE). Can combine with --label + --label-pattern string Filter by label glob pattern (e.g., 'tech-*' matches tech-debt, tech-legacy) + --label-regex string Filter by label regex pattern (e.g., 'tech-(debt|legacy)') + -n, --limit int Limit results (an explicit --limit always wins, 0 meaning unlimited; otherwise --all is unlimited; otherwise a configured list.limit applies; otherwise unlimited when piped, else 20 in agent mode at a terminal, else 50) (default 50) + --long Show detailed multi-line output for each issue + --max-rows int Hard upper bound on rows returned. Returns a non-zero exit (code 2) and an error to stderr if exceeded. 0 disables (the default). Overrides BEADS_MAX_ROWS for this invocation. Useful in CI/agent rigs that want a circuit breaker against pathological queries. Honored on both the direct and the --proxied-server route. + --metadata-field stringArray Filter by metadata field (key=value, repeatable) + --mol-type string Filter by molecule type: swarm, patrol, or work + --no-assignee Filter issues with no assignee + --no-labels Filter issues with no labels + --no-pager Disable pager output + --no-parent Exclude child issues (show only top-level issues) + --no-pinned Exclude pinned issues + --notes-contains string Filter by notes substring (case-insensitive) + --offset int Skip the first N matching results (0-based). Only supported under --proxied-server. + --overdue Show only issues with due_at in the past (not closed) + --parent string Filter by parent issue ID (shows children of specified issue) + --pinned Show only pinned issues + --pretty Display issues in a tree format with status/priority symbols + -p, --priority string Priority (0-4 or P0-P4, 0=highest) + --priority-max string Filter by maximum priority (inclusive, 0-4 or P0-P4) + --priority-min string Filter by minimum priority (inclusive, 0-4 or P0-P4) + --ready Show only ready issues (no active blockers, same semantics as bd ready) + -r, --reverse Invert the sort field's default direction (created/updated/closed default to newest-first, so --sort updated --reverse is oldest-first) + --skip-labels Skip label hydration. The labels field in output will be empty regardless of actual labels. Use only when the caller does not depend on label data. Cannot combine with --label, --label-any, --label-pattern, --label-regex, --exclude-label, or --no-labels. + --sort string Sort by field: priority, created, updated, closed, status, id, title, type, assignee + --spec string Filter by spec_id prefix + -s, --status string Filter by stored status (open, in_progress, blocked, deferred, closed). Comma-separated or repeated for multiple: --status open,in_progress + --title string Filter by title text (case-insensitive substring match) + --title-contains string Filter by title substring (case-insensitive) + --tree Hierarchical tree format (default: true; use --flat to disable) (default true) + -t, --type string Filter by type (bug, feature, task, epic, chore, decision, merge-request, molecule, gate, convoy). Aliases: mr→merge-request, feat→feature, mol→molecule, dec/adr→decision + --updated-after string Filter issues updated after date (YYYY-MM-DD or RFC3339) + --updated-before string Filter issues updated before date (YYYY-MM-DD or RFC3339) + -w, --watch Watch for changes and auto-update display (implies --pretty) + --wisp-type string Filter by wisp type: heartbeat, ping, patrol, gc_report, recovery, error, escalation + +Global Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output diff --git a/cmd/bd/testdata/command_tree/help/bd_serve.golden b/cmd/bd/testdata/command_tree/help/bd_serve.golden new file mode 100644 index 000000000..848d00820 --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd_serve.golden @@ -0,0 +1,119 @@ +Serve the beads HTTP API — the same work surface the CLI answers, for +automation clients that would otherwise fork a bd subprocess per call. + +The wire contract is described by an OpenAPI document (/v0); GET +/v0/beads/context reports which operations this build actually implements. + +DEPLOYMENT + + Pass an explicit port. The default 127.0.0.1:0 takes an ephemeral one, which + is right for ad-hoc and test use — where the bound address printed on stdout + is read immediately — but carries no mutual exclusion: two serves against one + workspace then run side by side on different ports with no way to enumerate + them. On a fixed port the second one fails to bind, which is the intended + behavior. Concurrent serves are data-safe either way; claims are arbitrated + in the SQL server. + + Run it under a supervisor. bd shuts down gracefully on SIGHUP as well as + SIGINT and SIGTERM, so closing the terminal of a foreground bd serve stops it. + +PROBES + + GET /healthz is LIVENESS only: it answers from the process and never touches + the database, so it stays green while the database is unreachable. For + readiness use GET /v0/beads/ready?limit=1 — a real query, where 200 means + ready and 503 means live but not ready. + +AUTHENTICATION + + Optional, and off by default on loopback — where the trust model is the + loopback boundary itself, the same one the database behind it already relies + on. --auth-token-file turns it on: every operation except GET /healthz then + requires an "Authorization: Bearer " header, GET /v0/beads/context + included, because it reports the repo root, beads directory and database name. + + The file holds ONE TOKEN PER LINE and every line is accepted. That is the + rotation mechanism: write the new token alongside the old, roll the clients + over, then delete the old line. The file is re-read while the server runs, so + both the addition and the removal take effect within about a second and + neither needs a restart. Write it atomically (a temp file plus rename; a + Kubernetes secret mount already does this). + + There is deliberately no --auth-token flag. A credential passed as an + argument is readable by every local user in the process listing. + + --allow-non-loopback REQUIRES a token file. Beyond loopback, reaching the + address would otherwise be the whole authorization: any peer could read every + issue and claim work as any actor. --insecure-no-auth is the explicit, + auditable way to say you meant that anyway. + + The Host allowlist is what a service deployment usually trips over first. The + DNS-rebinding check answers only to loopback spellings and the bind address, + so a client dialing a service name gets 400 on every request; enumerate the + names it dials with --allowed-host (repeatable). Matching is exact — no + wildcards — and the startup log line prints the effective allowlist. + +WHAT THIS DOES NOT DO + + No TLS. Even with a token, the credential and every issue body travel in + plaintext, so a deployment beyond loopback has to supply confidentiality + itself — a service mesh, or a network boundary you already trust. + + Hooks do not fire. A hook is a user-controlled subprocess per mutation: in a + concurrent server that is an unbounded latency multiplier and an orphaned + child at shutdown, and its working-directory-derived hook lookup is + meaningless in a server process. A CLI claim runs on_update; an HTTP claim + does not. + + The per-command auto-commit machinery does not run. Durability is per request: + a successful claim commits inside its own transaction, exactly as a proxied + CLI claim does today. + + An actor on an HTTP request is caller-asserted provenance for the audit trail, + not authenticated identity — the same thing it has always been on the CLI, + where any local process can pass any --actor. + + It does not run under --readonly, and refuses to start rather than binding. + Every server it binds publishes the issue-claim operation, and the capability + set it advertises is a property of the build rather than of the flags on the + process that started it — so a read-only server would advertise a write it + could never land. + +DESTRUCTIVE OPERATIONS + + POST /v0/beads/issues:sweep deletes closed beads in bulk — the operation + behind bd purge and bd prune — and nothing it deletes comes back. It shares + the library surface those commands call, so it inherits their guards: pinned + beads are never swept, and a durable sweep with neither a cutoff nor an id + pattern is refused rather than clearing every closed bead in the workspace. + Combined with the trust model above, that means anyone who can reach this + address can erase closed work; bind it accordingly. + +Usage: + bd serve [flags] + +Flags: + --addr string Address to bind as IP:PORT; the host must be a numeric IP literal, and port 0 takes an ephemeral port (default "127.0.0.1:0") + --allow-non-loopback Permit a bind beyond loopback. Requires --auth-token-file, since reaching the address would otherwise be the whole authorization + --allowed-host stringArray Additional Host header value to answer to, e.g. a service DNS name. Repeatable; matched exactly, with no wildcards + --auth-token-file string Require an Authorization: Bearer token from this file, one token per line, all accepted. Re-read while running, so rewriting it rotates tokens without a restart (env BEADS_SERVE_TOKEN_FILE) + -h, --help help for serve + --insecure-no-auth Serve a non-loopback bind with NO authentication. Every peer that can reach the address gets full read and claim access + --large-apply-ceiling duration Whole-run budget for a POST issues:batchApply request over 100 items (a ceiling, not a target; requests at or under 100 items are unaffected). Must be positive. The orchestrator's stop grace must be at least this long plus 5s, or an external SIGKILL can cut off an in-flight large apply a graceful drain would otherwise have waited out; see engdocs/SERVE_RUNBOOK.md (default 5m0s) + +Global Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output diff --git a/cmd/bd/testdata/command_tree/help/bd_show.golden b/cmd/bd/testdata/command_tree/help/bd_show.golden new file mode 100644 index 000000000..595a7dd74 --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd_show.golden @@ -0,0 +1,41 @@ +Show issue details + +Usage: + bd show [id...] [--id=...] [--current] [flags] + +Aliases: + show, view + +Flags: + --as-of string Show issue as it existed at a specific commit hash or branch (requires Dolt; commit hashes must be the full 32-character hash from 'bd history', not a prefix) + --brief-deps Reduce each dependency to its identity fields in JSON output (--json only; drops description, design, notes and acceptance criteria) + --children Show only the children of this issue + --comments-tail int Render only the last N comments in text output, preceded by one elision line naming how many older ones were hidden (text output only; 0, the default, renders every comment unchanged; N must be >= 0) + --current Show the currently active issue (in-progress, hooked, or last touched) + -h, --help help for show + --id stringArray Issue ID (use for IDs that look like flags, e.g., --id=gt--xyz) + --include-comments Stream full comment bodies in JSON output (--json only; may be slow on issues with many comments). Without it the comments field is absent entirely, not truncated: an issue with comments carries comment_count and comments_omitted=true, so check those before reading an absent comments field as none. + --include-dependents Stream full dependent issues in JSON output (--json only; may be slow on hub beads) + --local-time Show timestamps in local time instead of UTC (date-only fields are always local) + --long Show all available fields (extended metadata, agent identity, gate fields, etc.) + --refs Show issues that reference this issue (reverse lookup) + --short Show compact one-line output per issue + --thread Show full conversation thread (for messages) + -w, --watch Watch for changes and auto-refresh display + +Global Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output diff --git a/cmd/bd/testdata/command_tree/help/bd_update.golden b/cmd/bd/testdata/command_tree/help/bd_update.golden new file mode 100644 index 000000000..37451cdc6 --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd_update.golden @@ -0,0 +1,78 @@ +Update one or more issues. + +If no issue ID is provided, updates the last touched issue (from most recent +create, update, show, or close operation). This fallback only applies in +interactive sessions (stdin is a terminal); in scripts and agent sessions a +missing ID is an error, so a command built from an empty variable cannot +silently mutate an unrelated issue. Set BD_LAST_TOUCHED_FALLBACK=1 to allow +the fallback anywhere, or =0 to disable it entirely. + +Updates are applied per issue ID, not atomically across IDs: when some IDs +fail, the remaining issues are still updated, every failed ID is reported on +stderr, and the command exits nonzero. + +Exit codes: 1 for general failures; 13 when every failure is a stale +--if-assignee/--if-status guard (the precondition no longer held, nothing was +written — another actor won the race, so retrying the same guard is +pointless). + +Usage: + bd update [id...] [flags] + +Flags: + --acceptance string Acceptance criteria + -l, --add-label strings Add labels (repeatable); -l matches bd create -l + --allow-empty-description Allow empty description replacement when reading from stdin or file + --append-notes string Append to existing notes (with newline separator) + -a, --assignee string Assignee + --await-id string Set gate await_id (e.g., GitHub run ID for gh:run gates) + --body-file string Read description from file (use - for stdin) + --claim Atomically claim the issue (sets assignee to you, status to in_progress; idempotent if already claimed by you; issues assigned to a pool alias listed in the claim.pools config are claimable too) + --clear-notes Clear the notes field (--notes "" is refused: an empty value is usually a dead command substitution) + --defer string Defer until date (empty to clear). Issue hidden from bd ready until then, then auto-wakes to open + -d, --description string Issue description + --design string Design notes + --design-file string Read design from file (use - for stdin) + --due string Due date/time (empty to clear). Formats: +6h, +1d, +2w, tomorrow, next monday, 2025-01-15 + --ephemeral Mark issue as ephemeral (wisp) - not exported to JSONL + -e, --estimate int Time estimate in minutes (e.g., 60 for 1 hour) + --external-ref string External reference (e.g., 'gh-9', 'jira-ABC', Linear URL) + --force Override refusals: let -a/--assignee overwrite another actor's live in_progress claim (use only for abandoned claims — crashed agent, expired lease; prefer bd reclaim), let -s/--status move the issue into closed (or a configured done status) despite open children or a live blocker (same as bd close --force), and let --notes overwrite existing non-empty notes (use --append-notes to preserve history instead) + -h, --help help for update + --history Clear no-history flag (re-enable Dolt commit history) + --if-assignee string Apply the update only if the current assignee equals this value (--if-assignee '' requires unassigned); a mismatch writes nothing and exits 13 (vs 1 for other failures). Requires a field update; cannot combine with --claim + --if-revision bd show --json Apply the write only if the issue's current revision equals this value (the decimal int64 from bd show --json's "revision" field). One id only: refused before any write when combined with more than one issue id. A mismatch writes nothing and exits 13 (vs 1 for other failures). Composes with --if-assignee/--if-status: all guards present must hold. + --if-status string Apply the update only if the current status equals this value; a mismatch writes nothing and exits 13 (vs 1 for other failures). Requires a field update; cannot combine with --claim + --metadata string Set custom metadata (JSON object, or @file.json to read from file) + --no-history Mark issue as no-history (skip Dolt commits, not GC-eligible) + --notes string Replace the notes field (requires --force over existing non-empty notes; --clear-notes clears; --append-notes appends instead) + --parent string New parent issue ID (reparents the issue, use empty string to remove parent) + --persistent Mark issue as persistent (promote wisp to regular issue) + -p, --priority string Priority (0-4 or P0-P4, 0=highest) + --remove-label strings Remove labels (repeatable) + --session string Claude Code session ID for status=closed (or set CLAUDE_SESSION_ID env var) + --set-labels strings Set labels, replacing all existing (repeatable) + --set-metadata stringArray Set metadata key=value (repeatable, e.g., --set-metadata team=platform) + --spec-id string Link to specification document + -s, --status string New status + --stdin Read description from stdin (alias for --body-file -) + --title string New title + -t, --type string New type (bug|feature|task|epic|chore|decision|spike|story|milestone); custom types require types.custom config; aliases: enhancement/feat→feature, dec/adr→decision + --unset-metadata stringArray Remove metadata key (repeatable, e.g., --unset-metadata team) + +Global Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output diff --git a/cmd/bd/testdata/command_tree/help/bd_version.golden b/cmd/bd/testdata/command_tree/help/bd_version.golden new file mode 100644 index 000000000..cb5978ad7 --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd_version.golden @@ -0,0 +1,24 @@ +Print version information + +Usage: + bd version [flags] + +Flags: + -h, --help help for version + +Global Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output diff --git a/cmd/bd/testdata/command_tree/help/bd_versions.golden b/cmd/bd/testdata/command_tree/help/bd_versions.golden new file mode 100644 index 000000000..54d21dac5 --- /dev/null +++ b/cmd/bd/testdata/command_tree/help/bd_versions.golden @@ -0,0 +1,60 @@ +List the versions recorded for a bead by versioned history. + +Recording is off by default. Turn it on with: + + bd config set versioned-history.enabled true + +That writes the setting into the store, so every client of that store agrees +about whether a write is recorded, and it is the setting this command reports. +BD_VERSIONED_HISTORY_ENABLED=1 (or a value in config.yaml) also turns recording +on for the writes a process makes, but it says nothing about the store, and +this command only reads, so it does not consult them. + +Either source turning it on is enough for a writer; neither can switch the +other off. The consequence worth knowing: once the store setting says on, +BD_VERSIONED_HISTORY_ENABLED=0 will not turn recording off for one run. Turn +it off where it was turned on: + + bd config set versioned-history.enabled false + +Only beads created while recording is on, or that enter the issues plane +while it is on (a wisp promoted or made persistent), are versioned. A bead +already on the issues plane when recording was turned on is legacy: editing +it records nothing (the write fence, design §16.2b), and nothing is +backfilled. + +Turning recording off does not hide what was already recorded — this command +still lists it, and says that recording is currently off. + +A caution while the feature is young: keep recording ON for every writer of a +shared store, or leave it off for all of them. A write from a client with +recording off advances nothing, and the next recorded write mints a version +that absorbs that unrecorded change under its own actor — the revisions stay +contiguous and the listing looks gapless when it is not. + +Examples: + bd versions bd-123 + bd versions bd-123 --json + +Usage: + bd versions [flags] + +Flags: + -h, --help help for versions + +Global Flags: + --actor string Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER) + --cpu-profile Generate CPU profile for performance analysis + --database string Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only) + --db string Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database) + -C, --directory string Change to this directory before running the command (like git -C) + --dolt-auto-commit string Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit + --global Use the global shared-server database (beads_global) + --ignore-schema-skew Proceed despite forward schema drift (some queries may fail) + --json Output in JSON format + --mem-profile string Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE) + --no-color Disable color output (also: NO_COLOR=1 or CLICOLOR=0) + -q, --quiet Suppress non-essential output (errors only) + --readonly Read-only mode: block write operations (for worker sandboxes) + --sandbox Sandbox mode: disables Dolt auto-push + -v, --verbose Enable verbose/debug output diff --git a/cmd/bd/testdata/command_tree/tree.golden b/cmd/bd/testdata/command_tree/tree.golden new file mode 100644 index 000000000..79d89ce5e --- /dev/null +++ b/cmd/bd/testdata/command_tree/tree.golden @@ -0,0 +1,7461 @@ +# bd command tree (cgo build), pinned by TestCommandTreeGolden. +# Regenerate after an intentional CLI change: +# BD_UPDATE_GOLDEN=1 CGO_ENABLED=1 go test -tags gms_pure_go ./cmd/bd -run '^TestCommand(Tree|Help)Golden$' + +bd + use: "bd" + short: "bd - Dependency-aware issue tracker" + hidden: false + runnable: true + long: + | Issues chained together like beads. A lightweight issue tracker with first-class dependency support. + persistent-flag --actor type=string default="" usage="Actor name for audit trail (default: $BEADS_ACTOR, git user.name, $USER)" + persistent-flag --cpu-profile type=bool default="false" usage="Generate CPU profile for performance analysis" + persistent-flag --database type=string default="" usage="Run against a different server database for this invocation, without changing the project's configured database (proxied-server mode only)" + persistent-flag --db type=string default="" usage="Database path (default: auto-discover .beads/*.db). In proxied-server mode, a value that isn't an existing path is treated as a database name override (see --database)" + persistent-flag --directory -C type=string default="" usage="Change to this directory before running the command (like git -C)" + persistent-flag --dolt-auto-commit type=string default="" usage="Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit" + persistent-flag --format type=string default="" hidden usage="Output format (json). Alias for --json" + persistent-flag --global type=bool default="false" usage="Use the global shared-server database (beads_global)" + persistent-flag --ignore-schema-skew type=bool default="false" usage="Proceed despite forward schema drift (some queries may fail)" + persistent-flag --json type=bool default="false" usage="Output in JSON format" + persistent-flag --mem-profile type=string default="" usage="Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE)" + persistent-flag --no-color type=bool default="false" usage="Disable color output (also: NO_COLOR=1 or CLICOLOR=0)" + persistent-flag --quiet -q type=bool default="false" usage="Suppress non-essential output (errors only)" + persistent-flag --readonly type=bool default="false" usage="Read-only mode: block write operations (for worker sandboxes)" + persistent-flag --sandbox type=bool default="false" usage="Sandbox mode: disables Dolt auto-push" + persistent-flag --verbose -v type=bool default="false" usage="Enable verbose/debug output" + flag --version -V type=bool default="false" usage="Print version information" + +bd admin + use: "admin" + short: "Administrative commands for database maintenance" + group: "advanced" + hidden: false + runnable: false + long: + | Administrative commands for beads database maintenance. + | + | These commands are for advanced users and should be used carefully: + | cleanup Delete closed issues (issue lifecycle) + | compact Compact old closed issues to save space (storage optimization) + | reset Remove all beads data and configuration (full reset) + | + | For routine maintenance, prefer 'bd doctor --fix' which handles common repairs + | automatically. Use these admin commands for targeted database operations. + +bd admin cleanup + use: "cleanup" + short: "Delete closed issues to reduce database size" + hidden: false + runnable: true + long: + | Delete closed issues to reduce database size. + | + | This command permanently removes closed issues from the database. + | + | NOTE: This command only manages issue lifecycle (closed -> deleted). For general + | health checks and automatic repairs, use 'bd doctor --fix' instead. + | + | By default, deletes ALL closed issues. Use --older-than to only delete + | issues closed before a certain date. + | + | EXAMPLES: + | bd admin cleanup --force # Delete all closed issues + | bd admin cleanup --older-than 30 --force # Only issues closed 30+ days ago + | bd admin cleanup --ephemeral --force # Only closed wisps (transient molecules) + | bd admin cleanup --dry-run # Preview what would be deleted + | + | SAFETY: + | - Requires --force flag to actually delete (unless --dry-run) + | - Supports --cascade to delete dependents + | - Shows preview of what will be deleted + | - Use --json for programmatic output + | + | SEE ALSO: + | bd doctor --fix Automatic health checks and repairs (recommended for routine maintenance) + | bd admin compact Compact old closed issues to save space + flag --cascade type=bool default="false" usage="Recursively delete all dependent issues" + flag --dry-run type=bool default="false" usage="Preview what would be deleted without making changes" + flag --ephemeral type=bool default="false" usage="Only delete closed wisps (transient molecules)" + flag --force -f type=bool default="false" usage="Actually delete (without this flag, shows error)" + flag --older-than type=int default="0" usage="Only delete issues closed more than N days ago (0 = all closed issues)" + +bd admin compact + use: "compact" + short: "Compact old closed issues to save space" + hidden: false + runnable: true + long: + | Compact old closed issues using semantic summarization. + | + | Compaction reduces database size by summarizing closed issues that are no longer + | actively referenced. This is permanent graceful decay - original content is discarded. + | + | Modes: + | - Analyze: Export candidates for agent review (no API key needed) + | - Apply: Accept agent-provided summary (no API key needed) + | - Auto: AI-powered compaction (requires ANTHROPIC_API_KEY, MINIMAX_API_KEY, or ai.api_key) + | - Dolt: Run Dolt garbage collection (for Dolt-backend repositories) + | + | Tiers: + | - Tier 1: Semantic compression (30 days closed, 70% reduction) + | - Tier 2: Ultra compression (90 days closed) - planned, not yet implemented + | + | Dolt Garbage Collection: + | With auto-commit per mutation, Dolt commit history grows over time. Use + | --dolt to run Dolt garbage collection and reclaim disk space. + | + | --dolt: Run Dolt GC on the active database to free disk space. + | This removes unreachable commits and compacts storage. + | + | Examples: + | # Dolt garbage collection + | bd compact --dolt # Run Dolt GC + | bd compact --dolt --dry-run # Preview without running GC + | + | # Agent-driven workflow (recommended) + | bd compact --analyze --json # Get candidates with full content + | bd compact --apply --id bd-42 --summary summary.txt + | bd compact --apply --id bd-42 --summary - < summary.txt + | + | # AI-powered workflow + | bd compact --auto --dry-run # Preview candidates + | bd compact --auto --all # Compact all eligible issues + | bd compact --auto --id bd-42 # Compact specific issue + | MINIMAX_API_KEY=... bd compact --auto --all + | + | # Statistics + | bd compact --stats # Show statistics + | + flag --actor type=string default="agent" usage="Actor name for audit trail" + flag --all type=bool default="false" usage="Process all candidates" + flag --analyze type=bool default="false" usage="Analyze mode: export candidates for agent review" + flag --apply type=bool default="false" usage="Apply mode: accept agent-provided summary" + flag --auto type=bool default="false" usage="Auto mode: AI-powered compaction" + flag --batch-size type=int default="10" usage="Issues per batch" + flag --dolt type=bool default="false" usage="Dolt mode: run Dolt garbage collection on the active database" + flag --dry-run type=bool default="false" usage="Preview without compacting" + flag --force type=bool default="false" usage="Force compact (bypass checks, requires --id)" + flag --id type=string default="" usage="Compact specific issue" + flag --limit type=int default="0" usage="Limit number of candidates (0 = no limit)" + flag --stats type=bool default="false" usage="Show compaction statistics" + flag --summary type=string default="" usage="Path to summary file (use '-' for stdin)" + flag --tier type=int default="1" usage="Compaction tier (only tier 1 is implemented)" + flag --workers type=int default="5" usage="Parallel workers" + +bd admin reset + use: "reset" + short: "Remove all beads data and configuration" + hidden: false + runnable: true + long: + | Reset beads to an uninitialized state, removing all local data. + | + | This command removes: + | - The .beads directory (database, JSONL, config) + | - Git hooks installed by bd + | - Sync branch worktrees + | + | By default, shows what would be deleted (dry-run mode). + | Use --force to actually perform the reset. + | + | Examples: + | bd reset # Show what would be deleted + | bd reset --force # Actually delete everything + flag --force type=bool default="false" usage="Actually perform the reset (required)" + +bd ado + use: "ado" + short: "Azure DevOps integration commands" + hidden: false + runnable: false + long: + | Commands for syncing issues between beads and Azure DevOps. + | + | Configuration can be set via 'bd config' or environment variables: + | ado.org / AZURE_DEVOPS_ORG - Organization name + | ado.project / AZURE_DEVOPS_PROJECT - Project name (single) + | ado.projects / AZURE_DEVOPS_PROJECTS - Project names (comma-separated) + | ado.pat / AZURE_DEVOPS_PAT - Personal access token + | ado.url / AZURE_DEVOPS_URL - Custom base URL (on-prem) + +bd ado projects + use: "projects" + short: "List accessible Azure DevOps projects" + hidden: false + runnable: true + long: + | List Azure DevOps projects that the configured token has access to. + +bd ado pull + use: "pull [refs...]" + short: "Pull specific items from Azure DevOps" + hidden: false + runnable: true + long: + | Pull one or more items from Azure DevOps. + | + | Accepts bead IDs or external references as positional arguments. + | Equivalent to: bd ado sync --pull-only --issues + flag --dry-run type=bool default="false" usage="Preview pull without making changes" + +bd ado push + use: "push [bead-ids...]" + short: "Push specific beads to Azure DevOps" + hidden: false + runnable: true + long: + | Push one or more beads issues to Azure DevOps. + | + | Accepts bead IDs as positional arguments. + | Equivalent to: bd ado sync --push-only --issues + flag --dry-run type=bool default="false" usage="Preview push without making changes" + +bd ado status + use: "status" + short: "Show Azure DevOps sync status" + hidden: false + runnable: true + long: + | Display current Azure DevOps configuration and sync status. + +bd ado sync + use: "sync" + short: "Sync issues with Azure DevOps" + hidden: false + runnable: true + long: + | Synchronize issues between beads and Azure DevOps. + | + | By default, performs bidirectional sync: + | - Pulls new/updated work items from Azure DevOps to beads + | - Pushes local beads issues to Azure DevOps + | + | Use --pull-only or --push-only to limit direction. + | + | Filters (--area-path, --iteration-path, --types, --states) restrict + | which work items are synced. On pull, they limit the WIQL query. On push, + | --types and --states filter local beads before pushing to ADO. Use + | --no-create with push to skip creating new ADO work items (only update + | existing linked items). Filters can also be persisted via config: + | ado.filter.area_path, ado.filter.iteration_path, + | ado.filter.types, ado.filter.states + | CLI flags override config values when both are set. + flag --area-path type=string default="" usage="Filter to ADO area path (e.g., \"Project\\Team\")" + flag --bootstrap-match type=bool default="false" usage="Enable heuristic matching for first sync" + flag --dry-run type=bool default="false" usage="Show what would be synced without making changes" + flag --issues type=string default="" usage="Comma-separated bead IDs to sync selectively (e.g., bd-abc,bd-def). Mutually exclusive with --parent." + flag --iteration-path type=string default="" usage="Filter to ADO iteration path (e.g., \"Project\\Sprint 1\")" + flag --no-create type=bool default="false" usage="Never create new items in either direction (pull or push)" + flag --parent type=string default="" usage="Limit push to this bead and its descendants (push only). Mutually exclusive with --issues." + flag --prefer-ado type=bool default="false" usage="On conflict, use Azure DevOps version" + flag --prefer-local type=bool default="false" usage="On conflict, keep local beads version" + flag --prefer-newer type=bool default="false" usage="On conflict, use most recent version (default)" + flag --project type=stringSlice default="[]" usage="Project name(s) to sync (overrides configured project/projects)" + flag --pull-only type=bool default="false" usage="Only pull issues from Azure DevOps" + flag --push-only type=bool default="false" usage="Only push issues to Azure DevOps" + flag --reconcile type=bool default="false" usage="Force reconciliation scan for deleted items" + flag --states type=string default="" usage="Filter to ADO states, comma-separated (e.g., \"New,Active,Resolved\")" + flag --types type=string default="" usage="Filter to work item types, comma-separated (e.g., \"Bug,Task,User Story\")" + +bd assign + use: "assign " + short: "Assign an issue to someone" + group: "issues" + hidden: false + runnable: true + long: + | Assign an issue to someone. + | + | Shorthand for 'bd update --assignee '. + | + | Refuses to overwrite another actor's live in_progress claim without --force + | (bd-98s5c); issues assigned to a claim.pools alias are exempt, matching + | --claim. For a holder-aware transfer prefer + | 'bd update --if-assignee -a '. + | + | Examples: + | bd assign bd-123 alice + | bd assign bd-123 "" # unassign + flag --force type=bool default="false" usage="Allow overwriting another actor's live in_progress claim (use only for abandoned claims — crashed agent, expired lease; prefer bd reclaim)" + flag --if-revision type=string default="" usage="Apply the write only if the issue's current revision equals this value (the decimal int64 from `bd show --json`'s \"revision\" field). One id only: refused before any write when combined with more than one issue id. A mismatch writes nothing and exits 13 (vs 1 for other failures)." + +bd audit + use: "audit" + short: "Record and label agent interactions (append-only JSONL)" + hidden: false + runnable: false + long: + | Record explicit agent/tool interaction audit entries in .beads/interactions.jsonl. + | + | This optional JSONL sidecar is disabled by default. Enable it with: + | + | bd config set audit.enabled true + | + | Issue history is always recorded in the database and is visible with + | bd history --events. The JSONL sidecar is for explicit interaction capture: + | - auditing ("why did the agent do that?") + | - dataset generation (SFT/RL fine-tuning) + | + | Entries are append-only. Labeling creates a new "label" entry that references a parent entry. + +bd audit label + use: "label " + short: "Append a label entry referencing an existing interaction" + hidden: false + runnable: true + flag --label type=string default="" usage="Label value (e.g. \"good\" or \"bad\")" + flag --reason type=string default="" usage="Reason for label" + +bd audit record + use: "record" + short: "Append an audit interaction entry" + hidden: false + runnable: true + flag --error type=string default="" usage="Error string (llm_call/tool_call)" + flag --exit-code type=int default="-1" usage="Exit code (tool_call)" + flag --issue-id type=string default="" usage="Related issue id (bd-...)" + flag --kind type=string default="" usage="Entry kind (e.g. llm_call, tool_call, label)" + flag --model type=string default="" usage="Model name (llm_call)" + flag --prompt type=string default="" usage="Prompt text (llm_call)" + flag --response type=string default="" usage="Response text (llm_call)" + flag --stdin type=bool default="false" usage="Read a JSON object from stdin (must match audit.Entry schema)" + flag --tool-name type=string default="" usage="Tool name (tool_call)" + +bd backup + use: "backup" + short: "Back up your beads database" + group: "sync" + hidden: false + runnable: false + long: + | Back up your beads database for off-machine recovery. + | + | This is a Dolt-native database backup. It preserves the database state, + | including tables, branches, commit history, and working-set data. This is + | different from 'bd export', which writes issue records to JSONL for migration + | and interoperability. + | + | Commands: + | bd backup init Set up a backup destination (filesystem or DoltHub) + | bd backup sync Push to configured backup destination + | bd backup restore [path] Restore from a backup directory + | bd backup remove Remove backup destination + | bd backup status Show backup status + | + | DoltHub is recommended for cloud backup: + | bd backup init https://doltremoteapi.dolthub.com// + | Set DOLT_REMOTE_USER and DOLT_REMOTE_PASSWORD for authentication. + | + | Auto-backup default: + | When backup.enabled is unset, auto-backup turns ON in embedded mode if a + | git remote exists, and stays OFF in sql-server / shared-server mode. In + | server mode many bd clients share one Dolt server, and each would register + | a server-side backup remote under the same name pointing at its own local + | dir and full-sync the whole database — a self-amplifying storm. To back up + | a shared server, run 'bd backup' explicitly (or set backup.enabled=true and + | coordinate destinations). 'bd config get backup.enabled' shows the effective + | value and its source. + | + | A proxied-server workspace is server mode too: auto-backup is OFF by + | default and backup.enabled=true opts in. Both the opt-in and the five + | backup verbs above are honored only on a managed-local proxied server — + | one bd started itself; on a Dolt server bd does not own they are refused. + +bd backup init + use: "init " + aliases: ["add"] + short: "Set up a Dolt backup destination" + hidden: false + runnable: true + long: + | Configure a filesystem path or URL as a backup destination. + | + | The path can be a local directory (external drive, NAS, Dropbox folder) or a + | DoltHub remote URL. If the destination was previously configured, it is + | updated to the new path. + | + | Filesystem examples: + | bd backup add /mnt/usb/beads-backup + | bd backup add ~/Dropbox/beads-backup + | + | DoltHub (recommended for cloud backup): + | bd backup add https://doltremoteapi.dolthub.com/myuser/beads-backup + | + | After adding, run 'bd backup sync' to push your data. + +bd backup remove + use: "remove" + aliases: ["rm"] + short: "Remove the configured backup destination" + hidden: false + runnable: true + long: + | Remove the configured backup destination. + | + | This unregisters the backup remote from Dolt and removes the local + | backup configuration. The backup data at the destination is not deleted. + +bd backup restore + use: "restore [path]" + short: "Restore database from a Dolt backup" + hidden: false + runnable: true + long: + | Restore the beads database from a Dolt-native backup. + | + | By default, reads from .beads/backup/ (or the configured backup directory). + | Optionally specify a path to a directory containing a Dolt backup. + | + | This restores a full database backup created by 'bd backup sync' or an + | equivalent Dolt backup. JSONL files produced by 'bd export' are issue exports, + | not restore targets for this command. + | + | Use --force to overwrite an existing database with the backup contents. + | + | The database must already be initialized (run 'bd init' first if needed). + | To initialize and restore in one step, use: bd init && bd backup restore + flag --force type=bool default="false" usage="Overwrite existing database with backup contents" + +bd backup status + use: "status" + short: "Show last backup status" + hidden: false + runnable: true + +bd backup sync + use: "sync" + short: "Push database to configured Dolt backup" + hidden: false + runnable: true + long: + | Sync the current beads database to the configured Dolt backup destination. + | + | This pushes the entire database state (all branches, full history) to the + | backup location configured with 'bd backup init'. + | + | The backup is atomic — if the sync fails, the previous backup state is preserved. + | + | Run 'bd backup init ' first to configure a destination. + +bd batch + use: "batch" + short: "Run multiple write operations in a single database transaction" + group: "maint" + hidden: false + runnable: true + long: + | Run multiple write operations in a single database transaction. + | + | Commands are read from stdin (one per line) or from a file via -f/--file. + | All operations execute inside a single dolt transaction: on any error the + | whole batch is rolled back, otherwise it is committed with one DOLT_COMMIT. + | + | This is intended for shell scripts that currently invoke 'bd' many times in + | a loop, which causes severe write amplification on a dolt sql-server backed + | by btrfs+compression. Batching collapses N invocations into one transaction + | and one dolt commit. + | + | Grammar (one command per line): + | close [reason...] + | update = [= ...] + | create + | dep add [type] + | dep remove + | #comment (blank lines and '# ...' comments are ignored) + | + | Supported 'update' keys: status, priority, title, assignee, force + | Supported dependency types: see 'bd dep add --help' (default: blocks) + | + | 'force' is not a field. An update whose status moves the issue into closed + | (or a configured done status) is refused when it still has open children or + | a live blocker, the same as 'bd close'; 'force=true' overrides that refusal. + | Because the batch is one transaction, an unforced refusal rolls back EVERY + | operation in the batch, not just the offending line. Note the asymmetry with + | 'close ', which does not apply that policy at all. + | + | Tokens are whitespace-separated. Double-quoted strings ("like this") may + | contain spaces; use \" to embed a quote and \\ for a backslash. + | + | Examples: + | # From a pipe + | bd list --status stale -q | awk '{print "close",$1," stale"}' | bd batch + | + | # From a file + | bd batch -f operations.txt + | + | # Inline + | printf 'close bd-1 done\nupdate bd-2 status=in_progress\n' | bd batch + | + | On success, exits 0 and prints a summary (or JSON with --json). On any error, + | rolls back the entire transaction and exits non-zero with the failing line. + | + | NOTE: This is a narrow subset. Commands like 'show', 'list', 'ready', 'sync', + | complex create flows, or any flag not listed above are NOT accepted. Use + | normal 'bd' subcommands for interactive/read operations. + flag --dry-run type=bool default="false" usage="Parse input and echo commands without executing" + flag --file -f type=string default="" usage="Read commands from file instead of stdin" + flag --message -m type=string default="" usage="DOLT_COMMIT message (default: 'bd: batch N ops by ')" + +bd blocked + use: "blocked" + short: "Show blocked issues" + hidden: false + runnable: true + flag --exclude-label type=stringSlice default="[]" usage="Exclude issues that have ANY of these labels" + flag --flags type=bool default="false" usage="Print the blocked flag for each id read from stdin (one per line) instead of the listing; with --json, [{\"id\",\"is_blocked\"}]" + flag --label -l type=stringSlice default="[]" usage="Filter by labels (AND: must have ALL). Can combine with --label-any" + flag --label-any type=stringSlice default="[]" usage="Filter by labels (OR: must have AT LEAST ONE). Can combine with --label" + flag --parent type=string default="" usage="Filter to descendants of this bead/epic" + flag --raw type=bool default="false" usage="With --flags: the stored blocked column itself, without the external-dependency policy (an unsatisfied external: blocker does not set it)" + +bd bootstrap + use: "bootstrap" + short: "Non-destructive database setup for fresh clones and recovery" + group: "setup" + hidden: false + runnable: true + long: + | Bootstrap sets up the beads database without destroying existing data. + | Unlike 'bd init --force', bootstrap will never delete existing issues. + | + | Bootstrap auto-detects the right action: + | • If sync.remote is configured: clones from the remote + | • If git origin has Dolt data (refs/dolt/data): clones from git and wires origin for future push/pull + | • If .beads/backup/*.jsonl exists: restores from backup + | • If .beads/issues.jsonl exists: imports from git-tracked JSONL + | • If no database exists: creates a fresh one — unless the workspace was already + | initialized (metadata.json has a project_id) and its server-mode database is + | missing, in which case bootstrap refuses rather than strand the existing data + | • If database already exists: validates and reports status + | + | If sync.remote points at a git repository, bootstrap verifies refs/dolt/data + | before cloning. Bootstrap exits non-zero when it cannot set up a database. + | + | This is the recommended command for: + | • Setting up beads on a fresh clone + | • Recovering after moving to a new machine + | • Repairing a broken database configuration + | + | Non-interactive mode (--non-interactive, --yes/-y, or BD_NON_INTERACTIVE=1): + | Skips the confirmation prompt before executing the bootstrap plan. + | Also auto-detected when stdin is not a terminal or CI=true is set. + | + | Examples: + | bd bootstrap # Auto-detect and set up + | bd bootstrap --dry-run # Show what would be done + | bd bootstrap --json # Output plan as JSON + | bd bootstrap --yes # Skip confirmation prompt + | + flag --dry-run type=bool default="false" usage="Show what would be done without doing it" + flag --non-interactive type=bool default="false" usage="Alias for --yes" + flag --yes -y type=bool default="false" usage="Skip confirmation prompts (for CI/automation)" + +bd branch + use: "branch [name]" + short: "List or create branches" + group: "sync" + hidden: false + runnable: true + long: + | List all branches or create a new branch. + | + | This command requires the Dolt storage backend. Without arguments, + | it lists all branches. With an argument, it creates a new branch. + | + | Examples: + | bd branch # List all branches + | bd branch feature-xyz # Create a new branch named feature-xyz + +bd children + use: "children " + short: "List child beads of a parent" + group: "issues" + hidden: false + runnable: true + long: + | List all beads that are children of the specified parent bead. + | + | This is a convenience alias for 'bd list --parent --status all'. + | Unlike plain 'bd list', children includes closed issues by default, + | since the primary use case is inspecting all work under a parent. + | + | Examples: + | bd children hq-abc123 # List all children of hq-abc123 + | bd children hq-abc123 --json # List children in JSON format + | bd children hq-abc123 --pretty # Show children in tree format + flag --pretty type=bool default="false" usage="Show children in tree format" + +bd close + use: "close [id...]" + aliases: ["done"] + short: "Close one or more issues" + group: "issues" + hidden: false + runnable: true + long: + | Close one or more issues. + | + | If no issue ID is provided, closes the last touched issue (from most recent + | create, update, show, or close operation). This fallback only applies in + | interactive sessions (stdin is a terminal); in scripts and agent sessions a + | missing ID is an error, so a command built from an empty variable cannot + | silently close an unrelated issue. Set BD_LAST_TOUCHED_FALLBACK=1 to allow + | the fallback anywhere, or =0 to disable it entirely. + | + | When closing multiple issues, provide one --reason for all IDs or repeat + | --reason once per ID. Reasons map positionally: the first --reason applies + | to the first ID, the second --reason to the second ID, regardless of where + | the flags appear in the command line. + | + | Closing a molecule's last open step also closes the molecule's root, with + | the reason "all steps complete", when the root is a poured molecule, a wisp, + | or an epic labeled "template"; any other epic stays open as close-eligible + | work. A root that would refuse its own close (an open blocker, a pin, another + | assignee) stays open with a warning, and the step's close stands. Under + | --json, the step's entry names the closed root (auto_closed_molecule) or the + | refusal (molecule_auto_close_refusal). --continue claims the next ready step + | in the molecule for you; add --no-auto to only show it. + flag --claim-next type=bool default="false" usage="Automatically claim the next highest priority available issue" + flag --comment type=string default="" hidden usage="Alias for --reason" + flag --continue type=bool default="false" usage="Auto-advance to next step in molecule" + flag --force -f type=bool default="false" usage="Force close pinned issues or unsatisfied gates" + flag --if-revision type=string default="" usage="Apply the write only if the issue's current revision equals this value (the decimal int64 from `bd show --json`'s \"revision\" field). One id only: refused before any write when combined with more than one issue id. A mismatch writes nothing and exits 13 (vs 1 for other failures)." + flag --message -m type=string default="" hidden usage="Alias for --reason (git commit convention)" + flag --no-auto type=bool default="false" usage="With --continue, show next step but don't claim it" + flag --reason -r type=string default="" usage="Reason for closing" + flag --reason-file type=string default="" usage="Read close reason from file (use - for stdin)" + flag --resolution type=string default="" hidden usage="Alias for --reason (Jira CLI convention)" + flag --session type=string default="" usage="Claude Code session ID (or set CLAUDE_SESSION_ID env var)" + flag --suggest-next type=bool default="false" usage="Show newly unblocked issues after closing" + +bd codex-hook + use: "codex-hook " + short: "Run an internal Codex lifecycle hook" + hidden: true + runnable: true + +bd comment + use: "comment [text...]" + short: "Add a comment to an issue" + group: "issues" + hidden: false + runnable: true + long: + | Add a comment to an issue. + | + | Shorthand for 'bd comments add "text"'. + | + | Examples: + | bd comment bd-123 "Working on this now" + | bd comment bd-123 Working on this now + | echo "comment from pipe" | bd comment bd-123 --stdin + | bd comment bd-123 --file notes.txt + | + | Note: "comment" (singular) only adds a comment — it has no "list" subcommand. + | To list comments on an issue, use the plural form: bd comments + flag --file type=string default="" annotations={cobra_annotation_mutually_exclusive=["stdin file"]} usage="Read comment text from file" + flag --stdin type=bool default="false" annotations={cobra_annotation_mutually_exclusive=["stdin file"]} usage="Read comment text from stdin" + +bd comments + use: "comments [issue-id]" + short: "View or manage comments on an issue" + group: "issues" + hidden: false + runnable: true + long: + | View or manage comments on an issue. + | + | Examples: + | # List all comments on an issue (issue id is required — there is no "comments list") + | bd comments bd-123 + | + | # List comments in JSON format + | bd comments bd-123 --json + | + | # Add a comment + | bd comments add bd-123 "This is a comment" + | + | # Add a comment from a file + | bd comments add bd-123 -f notes.txt + flag --local-time type=bool default="false" usage="Show timestamps in local time instead of UTC" + +bd comments add + use: "add [issue-id] [text...]" + short: "Add a comment to an issue" + hidden: false + runnable: true + long: + | Add a comment to an issue. + | + | Examples: + | # Add a comment + | bd comments add bd-123 "Working on this now" + | + | # Add a comment from a file + | bd comments add bd-123 -f notes.txt + flag --author -a type=string default="" usage="Add author to comment" + flag --file -f type=string default="" usage="Read comment text from file" + +bd comments list + use: "list" + short: "Invalid — use bd comments to list comments" + hidden: false + runnable: true + +bd compact + use: "compact" + short: "Squash old Dolt commits to reduce history size" + group: "maint" + hidden: false + runnable: true + long: + | Squash Dolt commits older than N days into a single commit. + | + | Recent commits (within the retention window) are preserved via cherry-pick. + | This reduces Dolt storage overhead from auto-commit history while keeping + | recent change tracking intact. + | + | For semantic issue compaction (summarizing closed issues), use 'bd admin compact'. + | For full history squash, use 'bd flatten'. + | + | How it works: + | 1. Identifies commits older than --days threshold + | 2. Creates a squashed base commit from all old history + | 3. Cherry-picks recent commits on top + | 4. Swaps main branch to the compacted version + | 5. Prunes remote-tracking refs (they would keep the old history alive; + | the next push or fetch re-creates them at the new tip) + | 6. Runs a full Dolt GC (all storage generations) to reclaim the old history + | + | The GC pass is a full collection: Dolt storage is generational, and a default + | GC never revisits data an earlier GC moved to the old generation. On any store + | that has been GC'd before (bd gc, or a previous flatten or compact), only a + | full collection reclaims the squashed history. A full GC can take minutes on + | multi-gigabyte stores. + | + | Examples: + | bd compact --dry-run # Preview: show commit breakdown + | bd compact --force # Squash commits older than 30 days + | bd compact --days 7 --force # Keep only last 7 days of history + | bd compact --days 90 --force # Conservative: squash 90+ day old commits + flag --days type=int default="30" usage="Keep commits newer than N days" + flag --dry-run type=bool default="false" usage="Preview without making changes" + flag --force -f type=bool default="false" usage="Confirm commit squash" + +bd completion + use: "completion" + short: "Generate the autocompletion script for the specified shell" + hidden: false + runnable: false + long: + | Generate the autocompletion script for bd for the specified shell. + | See each sub-command's help for details on how to use the generated script. + | + +bd completion bash + use: "bash" + short: "Generate the autocompletion script for bash" + hidden: false + runnable: true + long: + | Generate the autocompletion script for the bash shell. + | + | This script depends on the 'bash-completion' package. + | If it is not installed already, you can install it via your OS's package manager. + | + | To load completions in your current shell session: + | + | source <(bd completion bash) + | + | To load completions for every new session, execute once: + | + | #### Linux: + | + | bd completion bash > /etc/bash_completion.d/bd + | + | #### macOS: + | + | bd completion bash > $(brew --prefix)/etc/bash_completion.d/bd + | + | You will need to start a new shell for this setup to take effect. + | + flag --no-descriptions type=bool default="false" usage="disable completion descriptions" + +bd completion fish + use: "fish" + short: "Generate the autocompletion script for fish" + hidden: false + runnable: true + long: + | Generate the autocompletion script for the fish shell. + | + | To load completions in your current shell session: + | + | bd completion fish | source + | + | To load completions for every new session, execute once: + | + | bd completion fish > ~/.config/fish/completions/bd.fish + | + | You will need to start a new shell for this setup to take effect. + | + flag --no-descriptions type=bool default="false" usage="disable completion descriptions" + +bd completion powershell + use: "powershell" + short: "Generate the autocompletion script for powershell" + hidden: false + runnable: true + long: + | Generate the autocompletion script for powershell. + | + | To load completions in your current shell session: + | + | bd completion powershell | Out-String | Invoke-Expression + | + | To load completions for every new session, add the output of the above command + | to your powershell profile. + | + flag --no-descriptions type=bool default="false" usage="disable completion descriptions" + +bd completion zsh + use: "zsh" + short: "Generate the autocompletion script for zsh" + hidden: false + runnable: true + long: + | Generate the autocompletion script for the zsh shell. + | + | If shell completion is not already enabled in your environment you will need + | to enable it. You can execute the following once: + | + | echo "autoload -U compinit; compinit" >> ~/.zshrc + | + | To load completions in your current shell session: + | + | source <(bd completion zsh) + | + | To load completions for every new session, execute once: + | + | #### Linux: + | + | bd completion zsh > "${fpath[1]}/_bd" + | + | #### macOS: + | + | bd completion zsh > $(brew --prefix)/share/zsh/site-functions/_bd + | + | You will need to start a new shell for this setup to take effect. + | + flag --no-descriptions type=bool default="false" usage="disable completion descriptions" + +bd config + use: "config" + short: "Manage configuration settings" + group: "setup" + hidden: false + runnable: false + long: + | Manage configuration settings for external integrations and preferences. + | + | Configuration is stored per-project in the beads database and is version-control-friendly. + | + | Common namespaces: + | - export.* Auto-export settings (stored in config.yaml) + | - import.* JSONL import settings (stored in config.yaml) + | - jira.* Jira integration settings + | - linear.* Linear integration settings + | - github.* GitHub integration settings + | - gitlab.* GitLab integration settings + | - ado.* Azure DevOps integration settings + | - notion.* Notion integration settings + | - custom.* Custom integration settings + | - status.* Issue status configuration + | - claim.* Claim arbitration settings (pool-aware claiming) + | - doctor.suppress.* Suppress specific bd doctor warnings (GH#1095) + | - lint.* Additional per-type lint sections (stored in config.yaml) + | + | Auto-Export (config.yaml): + | Optional JSONL export to .beads/issues.jsonl after write commands (throttled). + | Useful for viewers (bv), interchange, and issue-level migration; not a backup. + | It is not cross-machine sync; use bd dolt push/pull with a Dolt remote. + | Disabled by default. Enable only for integrations that need fresh JSONL. + | Auto-staging is separate and disabled by default. + | + | Keys: + | export.auto Enable/disable auto-export (default: false) + | export.path Output filename relative to .beads/ (default: issues.jsonl) + | export.interval Minimum time between exports (default: 60s) + | export.git-add Auto-stage the export file (default: false) + | + | Auto-Import (config.yaml): + | Reads .beads/issues.jsonl by default when a JSONL import path is implied. + | Use a relative filename/path so the import stays within the project .beads/ + | directory and remains portable across machines. + | + | Keys: + | import.path Input filename relative to .beads/ (default: issues.jsonl) + | + | Custom Status States: + | You can define custom status states for multi-step pipelines using the + | status.custom config key. Statuses should be comma-separated. + | + | Example: + | bd config set status.custom "awaiting_review,awaiting_testing,awaiting_docs" + | + | This enables issues to use statuses like 'awaiting_review' in addition to + | the built-in statuses (open, in_progress, blocked, deferred, closed). + | + | Claim Pools: + | A dispatcher can pre-assign issues to a pool pseudo-assignee (e.g. + | "fable-crew") and let any actor take them with --claim. List the pool + | aliases in the claim.pools config key, comma-separated: + | + | bd config set claim.pools "fable-crew,night-crew" + | + | Issues assigned to a real actor (or to an alias not in the list) keep + | their anti-steal protection. Pool takes carry the normal lease; note + | that if a taker's lease expires, bd reclaim returns the issue to the + | unassigned pool, not to the pool alias it was dispatched to. + | + | Suppressing Doctor Warnings: + | Suppress specific bd doctor warnings by check name slug: + | bd config set doctor.suppress.pending-migrations true + | bd config set doctor.suppress.git-hooks true + | Check names are converted to slugs: "Git Hooks" → "git-hooks". + | Only warnings are suppressed (errors and passing checks always show). + | To unsuppress: bd config unset doctor.suppress. + | + | Examples: + | bd config set export.auto true # Enable auto-export for viewer integrations + | bd config set export.path "beads.jsonl" # Custom export filename + | bd config set import.path "beads.jsonl" # Custom import filename + | bd config set export.git-add true # Also stage the export file + | bd config set jira.url "https://company.atlassian.net" + | bd config set jira.project "PROJ" + | bd config set status.custom "awaiting_review,awaiting_testing" + | bd config set claim.pools "fable-crew,night-crew" # Pool aliases claimable by any actor + | bd config set doctor.suppress.pending-migrations true + | bd config set dolt.debug true # Enable Dolt sql-server debug mode (loglevel=debug, --prof cpu) + | bd config set dolt.local-only true # Skip wiring a Dolt sync remote during bd init + | bd config set lint.sections.epic "Standards scorecard, Cost" # Extra sections bd lint requires for epics + | bd config get export.auto + | bd config list + | bd config unset jira.url + +bd config apply + use: "apply" + short: "Reconcile system state to match configuration" + hidden: false + runnable: true + long: + | Reconcile actual system state to match declared configuration. + | + | Runs drift detection and then fixes any mismatches it finds: + | + | - hooks Reinstall git hooks if missing or outdated + | - remote Add/update Dolt origin remote to match federation.remote + | - server Start Dolt server if dolt.shared-server is enabled + | + | This command is idempotent — safe to run multiple times. Use --dry-run + | to preview what would change without making modifications. + | + | Examples: + | bd config apply + | bd config apply --dry-run + | bd config apply --json + flag --dry-run type=bool default="false" usage="Show what would change without making modifications" + +bd config drift + use: "drift" + short: "Detect config-vs-reality inconsistencies" + hidden: false + runnable: true + long: + | Detect drift between declared configuration and actual system state. + | + | This is a read-only diagnostic that answers "is my environment consistent + | with my config?" — no mutations are performed. + | + | Checks: + | - hooks Git hooks installed and up-to-date + | - remote Dolt remote matches federation.remote config + | - server Server state matches dolt.shared-server config + | + | Exit codes: + | 0 No drift detected (all checks ok/info/skipped) + | 1 Drift detected (at least one check has status "drift") + | + | Examples: + | bd config drift + | bd config drift --json + +bd config get + use: "get " + short: "Get a configuration value" + hidden: false + runnable: true + +bd config list + use: "list" + short: "List all configuration" + hidden: false + runnable: true + +bd config set + use: "set " + short: "Set a configuration value" + hidden: false + runnable: true + flag --force-git-tracked type=bool default="false" usage="Allow writing secret keys to git-tracked config files (use with caution)" + +bd config set-many + use: "set-many ..." + short: "Set multiple configuration values in one operation" + hidden: false + runnable: true + long: + | Set multiple configuration values at once with a single auto-commit and auto-push. + | + | Each argument must be in key=value format. All values are validated before + | any writes occur. This is faster and less noisy than separate 'bd config set' + | calls, especially in CI. + | + | Examples: + | bd config set-many ado.state_map.open=New ado.state_map.closed=Closed + | bd config set-many jira.url=https://example.atlassian.net jira.project=PROJ + flag --force-git-tracked type=bool default="false" usage="Allow writing secret keys to git-tracked config files (use with caution)" + +bd config show + use: "show" + short: "Show all effective configuration with provenance" + hidden: false + runnable: true + long: + | Display a unified view of all effective configuration across all sources + | with annotations showing where each value comes from. + | + | Sources (by precedence for Viper-managed keys): + | - env Environment variable (BD_* or BEADS_*) + | - config.yaml Project config file (.beads/config.yaml) + | - default Built-in default value + | + | Additional sources: + | - metadata Connection settings from .beads/metadata.json + | - database Integration config stored in the Dolt database + | - git Git config (e.g., beads.role) + | + | Examples: + | bd config show + | bd config show --json + | bd config show --source config.yaml + flag --source type=string default="" usage="Filter by source (e.g., config.yaml, env, default, metadata, database, git)" + +bd config unset + use: "unset " + short: "Delete a configuration value" + hidden: false + runnable: true + +bd config validate + use: "validate" + short: "Validate sync-related configuration" + hidden: false + runnable: true + long: + | Validate sync-related configuration settings. + | + | Checks: + | - federation.sovereignty is valid (T1, T2, T3, T4, or empty) + | - federation.remote is set for Dolt sync + | - Remote URL format is valid (dolthub://, gs://, s3://, az://, file://) + | - routing.mode is valid (auto, maintainer, contributor, explicit) + | + | Examples: + | bd config validate + | bd config validate --json + +bd conflicts + use: "conflicts" + short: "Inspect and resolve live merge conflicts" + group: "sync" + hidden: false + runnable: false + long: + | Inspect and resolve the merge conflicts sitting in the working set. + | + | Conflicts appear when a pull or merge brought in changes that collide with + | local ones and could not be settled automatically. These commands present them + | per issue and per field, and resolve them without the raw dolt CLI. + | + | Examples: + | bd conflicts list # which tables and issues are conflicted + | bd conflicts show # every conflicted row, field by field + | bd conflicts show bd-1234 # one issue + | bd conflicts resolve bd-1234 --ours # keep our side of one issue + | bd conflicts resolve --all --theirs # take their side of everything + +bd conflicts list + use: "list" + short: "List tables and issues with live merge conflicts" + hidden: false + runnable: true + +bd conflicts resolve + use: "resolve [...]" + short: "Resolve merge conflicts with --ours or --theirs" + hidden: false + runnable: true + long: + | Resolve live merge conflicts, then conclude the merge with a commit. + | + | Named issue IDs are resolved row by row, leaving every other conflicted row + | alone. --all resolves whole tables at once (dolt's own table-level + | resolution). The merge is committed only once NO conflicts remain, so a + | partial resolution leaves the merge open for the next pass. + | + | Row-by-row resolution requires the row to exist on both sides: when one side + | deleted it, resolve that table wholesale or edit the row directly. + | + | Examples: + | bd conflicts resolve bd-1234 --ours # keep our side of one issue + | bd conflicts resolve bd-1234 bd-5678 --theirs # take their side of two + | bd conflicts resolve --all --ours # every conflicted table + | bd conflicts resolve --all --table config --theirs + | bd conflicts resolve --conclude # commit an already-resolved merge + flag --all type=bool default="false" usage="Resolve whole tables instead of named issues" + flag --conclude type=bool default="false" usage="Commit a merge whose conflicts are already resolved" + flag --no-commit type=bool default="false" usage="Resolve without committing the merge" + flag --ours type=bool default="false" usage="Keep our side" + flag --strategy type=string default="" usage="Resolution strategy: ours|theirs" + flag --table type=string default="" usage="Table to resolve (default: issues)" + flag --theirs type=bool default="false" usage="Take their side" + +bd conflicts show + use: "show []" + short: "Show conflicted rows field by field (base/ours/theirs)" + hidden: false + runnable: true + long: + | Show each conflicted row with its fields side by side. + | + | Only fields where our side and their side disagree are shown; --all-fields + | shows every column. Without an issue ID, every conflicted row of every + | conflicted table is shown. + flag --all-fields type=bool default="false" usage="Show every column, not just the fields that diverged" + flag --table type=string default="" usage="Restrict to one conflicted table (default: all)" + +bd connect + use: "connect [url]" + short: "Attach this workspace to a remote bd serve over HTTP" + group: "setup" + hidden: false + runnable: true + annotations: {bd:skip_store="1"} + long: + | Attach this workspace to a remote bd serve over HTTP. + | + | bd connect verifies the server first (a handshake: api_version, wire + | revision, and — when --expect-project-id is given — workspace identity) and + | writes nothing until that succeeds. On success it writes the per-user + | activation sidecar (.beads/http_target.json, never git-tracked) and sets this + | workspace's metadata.json to backend "http". + | + | A credential, if the server requires one, is never read from a flag or + | recorded on disk by this command. It comes from the same ladder every http + | request does: the BEADS_HTTP_TOKEN environment variable, then + | BEADS_HTTP_TOKEN_COMMAND (a helper that prints a token), then the credentials + | file's [host:port] section, then no credential at all — which the tip OSS + | server's loopback-trust posture answers legitimately, not as a failure. Both + | variables name the server they are for, host[:port]=value (for example + | BEADS_HTTP_TOKEN=bd.example.com=); a bare value is refused, since it + | would be sent to whatever server the workspace's http_target.json names. + | + | --allow-plaintext IS recorded on disk, in the sidecar alongside the server + | url: once granted here, a credential may cross this one server in the clear + | on every later command too, not only this one, without re-passing the flag + | or setting BEADS_HTTP_ALLOW_INSECURE=1 yourself. A later connect to a + | DIFFERENT url starts that grant over; it is never carried forward to a + | server it was not given for. + | + | Examples: + | bd connect http://127.0.0.1:8080 # loopback, no TLS needed + | bd connect https://bd.example.com --expect-project-id p1 # pin workspace identity + | bd connect https://bd.example.com --ca-file ./ca.pem # trust only this CA + | bd connect --clear # forget the server, restoring the previous backend + | + flag --allow-plaintext type=bool default="false" usage="allow a plain http connection to a non-loopback host; remembered in the sidecar for every later command against this server" + flag --ca-file type=string default="" usage="trust ONLY this PEM file for this server (resolved and recorded as an absolute path)" + flag --clear type=bool default="false" usage="remove this workspace's http activation sidecar and restore its previous backend selection in metadata.json" + flag --expect-project-id type=string default="" usage="pin the workspace identity the server must present at every handshake" + flag --force type=bool default="false" usage="allow switching a workspace that already selects a different backend" + +bd context + use: "context" + short: "Show effective backend identity and repository context" + group: "setup" + hidden: false + runnable: true + long: + | Show the effective backend identity information including repository paths, + | backend configuration, and sync settings. + | + | This command reads directly from config files and does not require the + | database to be open, making it useful for diagnostics in degraded states. + | + | Examples: + | bd context # Show context information + | bd context --json # Output in JSON format + | + +bd cook + use: "cook " + short: "Compile a formula into a proto (ephemeral by default)" + hidden: false + runnable: true + long: + | Cook transforms a .formula.json file into a proto. + | + | By default, cook outputs the resolved formula as JSON to stdout for + | ephemeral use. The output can be inspected, piped, or saved to a file. + | + | Two cooking modes are available: + | + | COMPILE-TIME (default, --mode=compile): + | Produces a proto with {{variable}} placeholders intact. + | Use for: modeling, estimation, contractor handoff, planning. + | Variables are NOT substituted - the output shows the template structure. + | + | RUNTIME (--mode=runtime or when --var flags provided): + | Produces a fully-resolved proto with variables substituted. + | Use for: final validation before pour, seeing exact output. + | Requires all variables to have values (via --var or defaults). + | + | Formulas are high-level workflow templates that support: + | - Variable definitions with defaults and validation + | - Step definitions that become issue hierarchies + | - Composition rules for bonding formulas together + | - Inheritance via extends + | + | The --persist flag enables the legacy behavior of writing the proto + | to the database. This is useful when you want to reuse the same + | proto multiple times without re-cooking. + | + | For most workflows, prefer ephemeral protos: pour and wisp commands + | accept formula names directly and cook inline. + | + | Examples: + | bd cook mol-feature.formula.json # Compile-time: keep {{vars}} + | bd cook mol-feature --var name=auth # Runtime: substitute vars + | bd cook mol-feature --mode=runtime --var name=auth # Explicit runtime mode + | bd cook mol-feature --dry-run # Preview steps + | bd cook mol-release.formula.json --persist # Write to database + | bd cook mol-release.formula.json --persist --force # Replace existing + | + | Output (default): + | JSON representation of the resolved formula with all steps. + | + | Output (--persist): + | Creates a proto bead in the database with: + | - ID matching the formula name (e.g., mol-feature) + | - The "template" label for proto identification + | - Child issues for each step + | - Dependencies matching depends_on relationships + flag --dry-run type=bool default="false" usage="Preview what would be created" + flag --force type=bool default="false" usage="Replace existing proto if it exists (requires --persist)" + flag --mode type=string default="" usage="Cooking mode: compile (keep placeholders) or runtime (substitute vars)" + flag --persist type=bool default="false" usage="Persist proto to database (legacy behavior)" + flag --prefix type=string default="" usage="Prefix to prepend to proto ID (e.g., 'gt-' creates 'gt-mol-feature')" + flag --search-path type=stringSlice default="[]" usage="Additional paths to search for formula inheritance" + flag --var type=stringArray default="[]" usage="Variable substitution (key=value), enables runtime mode" + +bd count + use: "count" + short: "Count issues matching filters" + group: "views" + hidden: false + runnable: true + long: + | Count issues matching the specified filters. + | + | By default, returns the total count of issues matching the filters. + | Use --by-* flags to group counts by different attributes. + | + | Examples: + | bd count # Count all issues + | bd count --status open # Count open issues + | bd count --by-status # Group count by status + | bd count --by-priority # Group count by priority + | bd count --by-type # Group count by issue type + | bd count --by-assignee # Group count by assignee + | bd count --by-label # Group count by label + | bd count --assignee alice --by-status # Count alice's issues by status + | bd count --include-infra # Count issues + wisps tier (matches 'bd list --include-infra --all' cardinality) + | bd count --type task --include-ephemeral # Count tasks in the durable AND wisps tiers + | + flag --assignee -a type=string default="" usage="Filter by assignee" + flag --by-assignee type=bool default="false" usage="Group count by assignee" + flag --by-label type=bool default="false" usage="Group count by label" + flag --by-priority type=bool default="false" usage="Group count by priority" + flag --by-status type=bool default="false" usage="Group count by status" + flag --by-type type=bool default="false" usage="Group count by issue type" + flag --closed-after type=string default="" usage="Filter issues closed after date (YYYY-MM-DD or RFC3339)" + flag --closed-before type=string default="" usage="Filter issues closed before date (YYYY-MM-DD or RFC3339)" + flag --created-after type=string default="" usage="Filter issues created after date (YYYY-MM-DD or RFC3339)" + flag --created-before type=string default="" usage="Filter issues created before date (YYYY-MM-DD or RFC3339)" + flag --desc-contains type=string default="" usage="Filter by description substring" + flag --empty-description type=bool default="false" usage="Filter issues with empty description" + flag --exclude-status type=stringSlice default="[]" usage="Exclude statuses from the count (comma-separated or repeatable)" + flag --exclude-type type=stringSlice default="[]" usage="Exclude issue types from the count (comma-separated or repeatable)" + flag --has-metadata-key type=string default="" usage="Filter issues that have this metadata key set" + flag --id type=string default="" usage="Filter by specific issue IDs (comma-separated)" + flag --include-ephemeral type=bool default="false" usage="Include ephemeral wisp-plane rows in the count (normally hidden), without lifting type exclusions" + flag --include-infra type=bool default="false" usage="Include infrastructure beads and the wisps tier (matches 'bd list --include-infra --all' cardinality)" + flag --label -l type=stringSlice default="[]" usage="Filter by labels (AND: must have ALL)" + flag --label-any type=stringSlice default="[]" usage="Filter by labels (OR: must have AT LEAST ONE)" + flag --metadata-field type=stringArray default="[]" usage="Filter by metadata field (key=value, repeatable)" + flag --no-assignee type=bool default="false" usage="Filter issues with no assignee" + flag --no-labels type=bool default="false" usage="Filter issues with no labels" + flag --no-parent type=bool default="false" usage="Count only top-level issues (exclude child issues)" + flag --notes-contains type=string default="" usage="Filter by notes substring" + flag --parent type=string default="" usage="Filter by parent issue ID (count children of specified issue)" + flag --priority -p type=int default="0" usage="Filter by priority (0-4: 0=critical, 1=high, 2=medium, 3=low, 4=backlog)" + flag --priority-max type=int default="0" usage="Filter by maximum priority (inclusive)" + flag --priority-min type=int default="0" usage="Filter by minimum priority (inclusive)" + flag --status -s type=string default="" usage="Filter by stored status (open, in_progress, blocked, deferred, closed). Note: dependency-blocked issues use 'bd blocked'" + flag --title type=string default="" usage="Filter by title text (case-insensitive substring match)" + flag --title-contains type=string default="" usage="Filter by title substring" + flag --type -t type=string default="" usage="Filter by type (bug, feature, task, epic, chore, decision, merge-request, molecule, gate)" + flag --updated-after type=string default="" usage="Filter issues updated after date (YYYY-MM-DD or RFC3339)" + flag --updated-before type=string default="" usage="Filter issues updated before date (YYYY-MM-DD or RFC3339)" + +bd create + use: "create [title]" + aliases: ["new"] + short: "Create a new issue (or batch from markdown/graph JSON)" + group: "issues" + hidden: false + runnable: true + flag --acceptance type=string default="" usage="Acceptance criteria" + flag --allow-empty-description type=bool default="false" usage="Allow empty description input from stdin or file" + flag --append-notes type=string default="" usage="Append to existing notes (with newline separator)" + flag --assignee -a type=string default="" usage="Assignee" + flag --body type=string default="" hidden annotations={cobra_annotation_mutually_exclusive=["stdin body"]} usage="Alias for --description (GitHub CLI convention)" + flag --body-file type=string default="" annotations={cobra_annotation_mutually_exclusive=["stdin body-file"]} usage="Read description from file (use - for stdin)" + flag --context type=string default="" usage="Additional context for the issue" + flag --defer type=string default="" usage="Defer until date (issue hidden from bd ready until then). Same formats as --due" + flag --deps type=stringSlice default="[]" usage="Dependencies as 'type:id' or bare 'id'. Bare 'id', 'depends-on:id', and 'blocked-by:id' all make THIS issue depend on id; 'blocks:id' reverses direction (id depends on this issue). E.g. 'blocked-by:bd-20,discovered-from:bd-15'" + flag --description -d type=string default="" annotations={cobra_annotation_mutually_exclusive=["stdin description"]} usage="Issue description" + flag --description-file type=string default="" hidden annotations={cobra_annotation_mutually_exclusive=["stdin description-file"]} usage="Alias for --body-file" + flag --design type=string default="" annotations={cobra_annotation_mutually_exclusive=["design design-file"]} usage="Design notes" + flag --design-file type=string default="" annotations={cobra_annotation_mutually_exclusive=["design design-file"]} usage="Read design from file (use - for stdin)" + flag --dry-run type=bool default="false" usage="Preview what would be created without actually creating" + flag --due type=string default="" usage="Due date/time. Formats: +6h, +1d, +2w, tomorrow, next monday, 2025-01-15" + flag --ephemeral type=bool default="false" usage="Create as ephemeral (short-lived, subject to TTL compaction)" + flag --estimate -e type=int default="0" usage="Time estimate in minutes (e.g., 60 for 1 hour)" + flag --event-actor type=string default="" usage="Entity URI who caused this event (requires --type=event)" + flag --event-category type=string default="" usage="Event category (e.g., patrol.muted, agent.started) (requires --type=event)" + flag --event-payload type=string default="" usage="Event-specific JSON data (requires --type=event)" + flag --event-target type=string default="" usage="Entity URI or bead ID affected (requires --type=event)" + flag --external-ref type=string default="" usage="External reference (e.g., 'gh-9', 'jira-ABC', Linear URL)" + flag --file -f type=string default="" usage="Create one issue per ## heading (### fields attach to that issue); for title plus description-from-file, use --body-file" + flag --force type=bool default="false" usage="Force creation even if prefix doesn't match database prefix" + flag --graph type=string default="" usage="Create a graph of issues with dependencies from JSON plan file" + flag --id type=string default="" usage="Explicit issue ID (e.g., 'bd-42' for partitioning)" + flag --label type=stringSlice default="[]" hidden usage="Alias for --labels" + flag --labels -l type=stringSlice default="[]" usage="Labels (comma-separated)" + flag --message -m type=string default="" hidden annotations={cobra_annotation_mutually_exclusive=["stdin message"]} usage="Alias for --description (git commit convention)" + flag --metadata type=string default="" usage="Set custom metadata (JSON object, or @file.json to read from file)" + flag --mol-type type=string default="" usage="Molecule type: swarm (multi-agent), patrol (recurring ops), work (default)" + flag --no-history type=bool default="false" usage="Skip Dolt commit history without making GC-eligible (for permanent agent beads)" + flag --no-inherit-labels type=bool default="false" usage="Don't inherit labels from parent issue" + flag --notes type=string default="" usage="Notes" + flag --parent type=string default="" usage="Parent issue ID for hierarchical child (e.g., 'bd-a3f8e9')" + flag --priority -p type=string default="2" usage="Priority (0-4 or P0-P4, 0=highest)" + flag --repo type=string default="" usage="Target repository for issue (overrides auto-routing)" + flag --silent type=bool default="false" usage="Output only the issue ID (for scripting)" + flag --skills type=string default="" usage="Required skills for this issue" + flag --spec-id type=string default="" usage="Link to specification document" + flag --status -s type=string default="" usage="Initial status" + flag --stdin type=bool default="false" annotations={cobra_annotation_mutually_exclusive=["stdin body-file" "stdin description-file" "stdin description" "stdin body" "stdin message"]} usage="Read description from stdin (alias for --body-file -)" + flag --storage-class type=string default="" usage="Storage class: versioned, unversioned, or ephemeral (default: storage-class. config, else versioned)" + flag --title type=string default="" usage="Issue title (alternative to positional argument)" + flag --type -t type=string default="task" usage="Issue type (bug|feature|task|epic|chore|decision|spike|story|milestone); custom types require types.custom config; aliases: enhancement/feat→feature, dec/adr→decision" + flag --validate type=bool default="false" usage="Validate description contains required sections for issue type" + flag --waits-for type=string default="" usage="Spawner issue ID to wait for (creates waits-for dependency for fanout gate)" + flag --waits-for-gate type=string default="all-children" usage="Gate type: all-children (wait for all) or any-children (wait for first)" + flag --wisp-type type=string default="" usage="Wisp type for TTL-based compaction: heartbeat, ping, patrol, gc_report, recovery, error, escalation" + +bd create-form + use: "create-form" + short: "Create a new issue using an interactive form" + group: "issues" + hidden: false + runnable: true + long: + | Create a new issue using an interactive terminal form. + | + | This command provides a user-friendly form interface for creating issues, + | with fields for title, description, type, priority, labels, and more. + | + | Use --parent to create a sub-issue under an existing parent issue. + | The child will get an auto-generated hierarchical ID (e.g., parent-id.1). + | + | The form uses keyboard navigation: + | - Tab/Shift+Tab: Move between fields + | - Enter: Submit the form (on the last field or submit button) + | - Ctrl+C: Cancel and exit + | - Arrow keys: Navigate within select fields + flag --parent type=string default="" usage="Parent issue ID for creating a hierarchical child (e.g., 'bd-a3f8e9')" + +bd cursor-hook + use: "cursor-hook " + short: "Run an internal Cursor agent lifecycle hook" + hidden: true + runnable: true + +bd db-proxy-child + use: "db-proxy-child" + short: "Internal: run as the database proxy child process" + hidden: true + runnable: true + long: + | db-proxy-child runs the long-lived per-rootDir TCP proxy that fronts a + | DatabaseServer. It is spawned by the parent bd process via fork+exec and is + | not intended to be invoked directly by users. + flag --backend type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="backend kind: external | local-server | local-shared-server" + flag --config type=string default="" usage="path to backend server config (e.g. dolt sql-server YAML)" + flag --database type=string default="" usage="database to select when running shutdown maintenance (local-server backend)" + flag --dolt-bin type=string default="" usage="path to the dolt executable" + flag --external-host type=string default="" usage="external backend: hostname or IP of the dolt sql-server" + flag --external-keep-alive type=duration default="0s" usage="external backend: TCP keepalive period (default 30s)" + flag --external-port type=int default="0" usage="external backend: TCP port of the dolt sql-server" + flag --external-socket-path type=string default="" usage="external backend: absolute path to a unix domain socket (overrides host/port)" + flag --idle-timeout type=duration default="0s" usage="idle timeout before shutdown (0 or negative = never shut down)" + flag --logpath type=string default="" usage="path the backend server should write its stdout/stderr to" + flag --port type=int default="0" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="port to listen on" + flag --root type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="root directory holding proxy.lock, proxy.pid, proxy.log" + flag --stop-epoch type=string default="" usage="stop epoch the parent observed at spawn; the proxy aborts before publishing if it advances" + +bd defer + use: "defer [id...]" + short: "Defer one or more issues for later" + hidden: false + runnable: true + long: + | Defer issues to put them on ice for later. + | + | Deferred issues are deliberately set aside - not blocked by anything specific, + | just postponed for future consideration. Unlike blocked issues, there's no + | dependency keeping them from being worked. Unlike closed issues, they will + | be revisited. + | + | Deferred issues don't show in 'bd ready' but remain visible in 'bd list'. + | + | A defer WITH a date is a snooze: once --until passes, the next ready-front + | read returns the issue to open automatically (same shape as 'bd undefer'). + | A defer WITHOUT a date is the indefinite icebox: it stays deferred until + | someone runs 'bd undefer'. + | + | Examples: + | bd defer bd-abc # Icebox indefinitely (until bd undefer) + | bd defer bd-abc --until=tomorrow # Snooze: auto-wakes once the date passes + | bd defer bd-abc --reason="waiting on API access" + | bd defer bd-abc bd-def # Defer multiple issues + flag --reason type=string default="" usage="Record why this issue is being deferred (appended to notes)" + flag --until type=string default="" usage="Defer until specific time (e.g., +1h, tomorrow, next monday)" + +bd delete + use: "delete [issue-id...]" + short: "Delete one or more issues and clean up references" + group: "issues" + hidden: false + runnable: true + long: + | Delete one or more issues and clean up all references to them. + | This command will: + | 1. Remove all dependency links (any type, both directions) involving the issues + | 2. Update text references to "[deleted:ID]" in directly connected issues + | 3. Permanently delete the issues from the database + | + | This is a destructive operation that cannot be undone. Use with caution. + | + | BATCH DELETION: + | Delete multiple issues at once: + | bd delete bd-1 bd-2 bd-3 --force + | + | Delete from file (one ID per line): + | bd delete --from-file deletions.txt --force + | + | Preview before deleting: + | bd delete --from-file deletions.txt --dry-run + | + | DEPENDENCY HANDLING (the same on a local database and against a team server): + | Default: Fails if any issue has dependents not in deletion set + | bd delete bd-1 bd-2 + | + | Cascade: Recursively delete all dependents + | bd delete bd-1 --cascade --force + | + | Force: Delete and orphan dependents + | bd delete bd-1 --force + flag --cascade type=bool default="false" usage="Recursively delete all dependent issues" + flag --dry-run type=bool default="false" usage="Preview what would be deleted without making changes" + flag --force -f type=bool default="false" usage="Actually delete (without this flag, shows preview)" + flag --from-file type=string default="" usage="Read issue IDs from file (one per line)" + flag --if-revision type=string default="" usage="Apply the write only if the issue's current revision equals this value (the decimal int64 from `bd show --json`'s \"revision\" field). One id only: refused before any write when combined with more than one issue id. A mismatch writes nothing and exits 13 (vs 1 for other failures)." + +bd dep + use: "dep [issue-id]" + short: "Manage dependencies" + group: "deps" + hidden: false + runnable: true + long: + | Manage dependencies between issues. + | + | When called with an issue ID and --blocks flag, creates a blocking dependency: + | bd dep --blocks + | + | This is equivalent to: + | bd dep add + | + | Examples: + | bd dep bd-xyz --blocks bd-abc # bd-xyz blocks bd-abc + | bd dep add bd-abc bd-xyz # Same as above (bd-abc depends on bd-xyz) + flag --blocks -b type=string default="" usage="Issue ID that this issue blocks (shorthand for: bd dep add )" + flag --no-cycle-check type=bool default="false" usage="Skip the post-add cycle warning (the per-edge cycle check still runs)" + +bd dep add + use: "add [issue-id] [depends-on-id]" + short: "Add a dependency" + hidden: false + runnable: true + long: + | Add a dependency between two issues. + | + | The depends-on-id can be provided as: + | - A positional argument: bd dep add issue-123 issue-456 + | - A flag: bd dep add issue-123 --blocked-by issue-456 + | - A flag: bd dep add issue-123 --depends-on issue-456 + | + | The --blocked-by and --depends-on flags are aliases and both mean "issue-123 + | depends on (is blocked by) the specified issue." + | + | The depends-on-id can be: + | - A local issue ID (e.g., bd-xyz) + | - An external reference: external:: + | + | For bulk wiring, pass newline-delimited JSON with --file. Each line must be an + | object with "from" and "to" fields, and may include "type". The aliases + | "issue_id" and "depends_on_id" are also accepted. Use --file - to read stdin. + | + | External references are stored as-is and resolved at query time using + | the external_projects config. They block the issue until the capability + | is "shipped" in the target project. + | + | With no -t/--type the edge is created as type=blocks, which excludes the + | dependent from bd ready. When stderr is an interactive terminal, an advisory + | note says so once per command; it is silent for scripted and agent callers + | (non-TTY stderr) and can be turned off with --quiet or BD_NO_DEP_TYPE_WARNING=1. + | + | Examples: + | bd dep add bd-42 bd-41 # Positional args + | bd dep add bd-42 --blocked-by bd-41 # Flag syntax (same effect) + | bd dep add bd-42 --depends-on bd-41 # Alias (same effect) + | bd dep add gt-xyz external:beads:mol-run-assignee # Cross-project dependency + | bd dep add bd-42 bd-41 --no-cycle-check # Skip the post-add cycle warning + | bd dep add --file deps.jsonl # Bulk JSONL: {"from":"bd-42","to":"bd-41"} + flag --blocked-by type=string default="" usage="Issue ID that blocks the first issue (alternative to positional arg)" + flag --depends-on type=string default="" usage="Issue ID that the first issue depends on (alias for --blocked-by)" + flag --file type=string default="" usage="Read dependency edges from JSONL file, or '-' for stdin" + flag --no-cycle-check type=bool default="false" usage="On bulk --file adds, skip per-edge cycle checks and the post-add cycle warning (one final whole-graph check still runs before commit); on a single-edge add, skip only the post-add cycle warning" + flag --type -t type=string default="blocks" usage="Dependency type (blocks|tracks|related|parent-child|discovered-from|until|caused-by|validates|relates-to|supersedes); 'blocked-by' and 'depends-on' are accepted as aliases for 'blocks'" + +bd dep cycles + use: "cycles" + short: "Detect dependency cycles" + hidden: false + runnable: true + flag --include-tracks type=bool default="false" usage="Also walk 'tracks' edges: reports every cycle the default walk finds plus cycles that close through a tracks edge, never a loop made only of tracks edges (diagnostic for molecule-root deadlocks hidden by tracks-only propagation)" + flag --limit type=int default="50" usage="Maximum cycles to print (0 = no limit); the reported total is always the whole count, and --json is never truncated" + +bd dep list + use: "list [issue-id...]" + short: "List dependencies or dependents of one or more issues" + hidden: false + runnable: true + long: + | List dependencies or dependents of one or more issues with optional type filtering. + | + | By default shows dependencies (what issues depend on). Use --direction to control: + | - down: Show dependencies (what this issue depends on) - default + | - up: Show dependents (what depends on this issue) + | + | Multiple IDs can be provided for batch dep listing. With --json, the output + | is a flat array of dependency records across all requested issues. + | + | Use --type to filter by dependency type (e.g., tracks, blocks, parent-child). + | + | Examples: + | bd dep list gt-abc # Show what gt-abc depends on + | bd dep list gt-abc gt-def # Batch: deps for both issues + | bd dep list gt-abc --direction=up # Show what depends on gt-abc + | bd dep list gt-abc --direction=up -t tracks # Show what tracks gt-abc (convoy tracking) + flag --direction type=string default="down" usage="Direction: 'down' (dependencies), 'up' (dependents)" + flag --type -t type=string default="" usage="Filter by dependency type (e.g., tracks, blocks, parent-child)" + +bd dep relate + use: "relate " + short: "Create a bidirectional relates_to link between issues" + hidden: false + runnable: true + long: + | Create a loose 'see also' relationship between two issues. + | + | The relates_to link is bidirectional - both issues will reference each other. + | This enables knowledge graph connections without blocking or hierarchy. + | + | Examples: + | bd relate bd-abc bd-xyz # Link two related issues + | bd relate bd-123 bd-456 # Create see-also connection + +bd dep remove + use: "remove [issue-id] [depends-on-id]" + aliases: ["rm"] + short: "Remove a dependency" + hidden: false + runnable: true + +bd dep tree + use: "tree [issue-id]" + short: "Show dependency tree" + hidden: false + runnable: true + long: + | Show dependency tree rooted at the given issue. + | + | By default, shows dependencies (what blocks this issue). Use --direction to control: + | - down: Show dependencies (what blocks this issue) - default + | - up: Show dependents (what this issue blocks) + | - both: Show full graph in both directions + | + | Examples: + | bd dep tree gt-0iqq # Show what blocks gt-0iqq + | bd dep tree gt-0iqq --direction=up # Show what gt-0iqq blocks + | bd dep tree gt-0iqq --status=open # Only show open issues + | bd dep tree gt-0iqq --depth=3 # Limit to 3 levels deep + | + | A node reached by two paths is shown ONCE, under the first path that got + | there, and a cycle simply ends the descent. --show-all-paths is a deprecated + | no-op; use 'bd dep cycles' to find circular dependencies. + | + | --max-rows / BEADS_MAX_ROWS caveat: the tree walk has no query filter to + | thread the cap through, so the full tree is always built first and the + | node count is checked afterward (post-hoc), not during the walk. The cap is + | honored on the --proxied-server route too, which it was not before. + flag --direction type=string default="" usage="Tree direction: 'down' (dependencies), 'up' (dependents), or 'both'" + flag --format type=string default="" usage="Output format: 'mermaid' for Mermaid.js flowchart" + flag --max-depth -d type=int default="50" usage="Maximum tree depth to display (safety limit)" + flag --max-rows type=int default="0" usage="Hard upper bound on rows returned. Returns a non-zero exit (code 2) and an error to stderr if exceeded. 0 disables (the default). Overrides BEADS_MAX_ROWS for this invocation. Useful in CI/agent rigs that want a circuit breaker against pathological queries. Honored on both the direct and the --proxied-server route." + flag --reverse type=bool default="false" usage="Show dependent tree (deprecated: use --direction=up)" + flag --show-all-paths type=bool default="false" usage="Deprecated no-op: accepted and ignored. A node reached by two paths is shown once, under the first." + flag --status type=string default="" usage="Filter to only show issues with this status (open, in_progress, blocked, deferred, closed)" + +bd dep unrelate + use: "unrelate " + short: "Remove a relates_to link between issues" + hidden: false + runnable: true + long: + | Remove a relates_to relationship between two issues. + | + | Removes the link in both directions. + | + | Example: + | bd unrelate bd-abc bd-xyz + +bd diff + use: "diff " + short: "Show changes between two commits or branches" + group: "views" + hidden: false + runnable: true + long: + | Show the differences in issues between two commits or branches. + | + | The refs can be: + | - Commit hashes (e.g., abc123def) + | - Branch names (e.g., main, feature-branch) + | - Special refs like HEAD, HEAD~1 + | + | Examples: + | bd diff main feature-branch # Compare main to feature branch + | bd diff HEAD~5 HEAD # Show changes in last 5 commits + | bd diff abc123 def456 # Compare two specific commits + +bd doctor + use: "doctor [path]" + short: "Check and fix beads installation health (start here)" + group: "maint" + hidden: false + runnable: true + annotations: {bd:skip_store="1"} + long: + | Sanity check the beads installation for the current directory or specified path. + | + | This command checks: + | - If .beads/ directory exists + | - Database version and migration status + | - Schema compatibility (all required tables and columns present) + | - Whether using hash-based vs sequential IDs + | - If CLI version is current (checks GitHub releases) + | - If Claude plugin is current (when running in Claude Code) + | - File permissions + | - Circular dependencies + | - Git hooks (pre-commit, post-merge, pre-push) + | - .beads/.gitignore up to date + | - Metadata.json version tracking (LastBdVersion field) + | + | Storage Availability: + | Full diagnostics, --perf, --deep, --server, --migration, and + | --check=validate currently require Dolt server mode. Embedded Dolt + | supports --check=artifacts, --check=conventions, and + | --check=pollution. --check-health has a limited hook-health fallback. + | Unsupported combinations return a notice without changing storage. + | + | Performance Mode (--perf): + | Run performance diagnostics on your database: + | - Times key operations (bd ready, bd list, bd show, etc.) + | - Collects system info (OS, arch, database stats) + | - Generates CPU profile for analysis + | - Outputs shareable report for bug reports + | + | Export Mode (--output): + | Save diagnostics to a JSON file for historical analysis and bug reporting. + | Includes timestamp and platform info for tracking intermittent issues. + | + | Specific Check Mode (--check): + | Run a specific check in detail. Available checks: + | - artifacts: Detect and optionally clean beads classic artifacts + | (stale JSONL, SQLite files, cruft .beads dirs). Use with --clean. + | - conventions: Check for convention drift (lint warnings, stale + | issues, orphaned issues). Advisory only - warns, never blocks. + | - pollution: Detect and optionally clean test issues from database + | - validate: Run focused data-integrity checks (duplicates, orphaned + | deps, test pollution, git conflicts). Use with --fix to auto-repair. + | + | Deep Validation Mode (--deep): + | Validate full graph integrity. May be slow on large databases. + | Additional checks: + | - Parent consistency: All parent-child deps point to existing issues + | - Dependency integrity: All deps reference valid issues + | - Epic completeness: Find epics ready to close (all children closed as + | completed work; duplicate/wontfix/superseded closes do not count) + | - Agent bead integrity: Agent beads have valid state values + | - Mail thread integrity: Thread IDs reference existing issues + | - Molecule integrity: Molecules have valid parent-child structures + | + | Server Mode (--server): + | Run health checks for Dolt server mode connections (bd-dolt.2.3): + | - Server reachable: Can connect to configured host:port? + | - Dolt version: Is it a Dolt server (not vanilla MySQL)? + | - Database exists: Does the 'beads' database exist? + | - Schema compatible: Can query beads tables? + | - Connection pool: Pool health metrics + | + | Legacy Dolt Migration Validation Mode (--migration): + | Retained for older SQLite-to-Dolt migration workflows and available only in + | Dolt server mode. It is not the migration path for removed backends + | (PostgreSQL, MySQL, SQLite); those fail closed with export/import guidance. + | Combine + | with --json for machine-parseable diagnostic output. + | + | Agent Mode (--agent): + | Output diagnostics designed for AI agent consumption. Instead of terse + | pass/fail messages, each issue includes: + | - Observed state: what the system actually looks like + | - Expected state: what it should look like + | - Explanation: full prose context about the issue and why it matters + | - Commands: exact remediation commands to run + | - Source files: where in the codebase to investigate further + | - Severity: blocking (prevents operation), degraded (partial function), + | or advisory (informational only) + | ZFC-compliant: Go observes and reports, the agent decides and acts. + | Combine with --json for structured agent-facing output. + | + | Suppressing Warnings: + | Suppress specific warnings by setting doctor.suppress. config: + | bd config set doctor.suppress.pending-migrations true + | bd config set doctor.suppress.git-hooks true + | Check names are converted to slugs: "Git Hooks" → "git-hooks". + | Only warnings are suppressed; errors and passing checks always show. + | To unsuppress: bd config unset doctor.suppress. + | + | Examples: + | bd doctor # Check current directory + | bd doctor /path/to/repo # Check specific repository + | bd doctor --json # Machine-readable output + | bd doctor --agent # Agent-facing diagnostic output + | bd doctor --agent --json # Structured agent diagnostics (JSON) + | bd doctor --fix # Automatically fix issues (with confirmation) + | bd doctor --fix --yes # Automatically fix issues (no confirmation) + | bd doctor --fix -i # Confirm each fix individually + | bd doctor --fix --fix-child-parent # Also fix child→parent deps (opt-in) + | bd doctor --fix --force # Force repair even when database can't be opened + | bd doctor --fix --source=jsonl # Rebuild database from a JSONL export + | bd doctor --dry-run # Preview what --fix would do without making changes + | bd doctor --perf # Performance diagnostics + | bd doctor --output diagnostics.json # Export diagnostics to file + | bd doctor --check=artifacts # Show classic artifacts (JSONL, SQLite, cruft dirs) + | bd doctor --check=artifacts --clean # Delete safe-to-delete artifacts (with confirmation) + | bd doctor --check=conventions # Convention drift check (lint, stale, orphans) + | bd doctor --check=pollution # Show potential test issues + | bd doctor --check=pollution --clean # Delete test issues (with confirmation) + | bd doctor --check=validate # Data-integrity checks only + | bd doctor --check=validate --fix # Auto-fix data-integrity issues + | bd doctor --deep # Full graph integrity validation + | bd doctor --server # Dolt server mode health checks + | bd doctor --migration=pre # Legacy Dolt-server migration diagnostic + | bd doctor --migration=post # Legacy Dolt-server completion diagnostic + | bd doctor --migration=pre --json # Machine-parseable legacy diagnostic + flag --agent type=bool default="false" usage="Agent-facing diagnostic mode: rich context for AI agents (ZFC-compliant)" + flag --check type=string default="" usage="Run specific check in detail (e.g., 'pollution')" + flag --check-health type=bool default="false" usage="Quick health check for git hooks (silent on success)" + flag --clean type=bool default="false" usage="For pollution check: delete detected test issues" + flag --deep type=bool default="false" usage="Validate full graph integrity" + flag --dry-run type=bool default="false" usage="Preview fixes without making changes" + flag --fix type=bool default="false" usage="Automatically fix issues where possible" + flag --fix-child-parent type=bool default="false" usage="Remove child→parent dependencies (opt-in)" + flag --interactive -i type=bool default="false" usage="Confirm each fix individually" + flag --migration type=string default="" usage="Run legacy Dolt-server migration diagnostics: 'pre' or 'post'" + flag --orchestrator type=bool default="false" usage="Running in orchestrator multi-workspace mode (routes.jsonl is expected, higher duplicate tolerance)" + flag --orchestrator-duplicates-threshold type=int default="1000" usage="Duplicate tolerance threshold for orchestrator mode (wisps are ephemeral)" + flag --output -o type=string default="" usage="Export diagnostics to JSON file" + flag --perf type=bool default="false" usage="Run performance diagnostics and generate CPU profile" + flag --server type=bool default="false" usage="Run Dolt server mode health checks (connectivity, version, schema)" + flag --verbose -v type=bool default="false" usage="Show all checks (default shows only warnings/errors)" + flag --yes -y type=bool default="false" usage="Skip confirmation prompt (for non-interactive use)" + +bd dolt + use: "dolt" + short: "Configure Dolt database settings" + group: "setup" + hidden: false + runnable: false + long: + | Configure and manage Dolt database settings and server lifecycle. + | + | Beads runs Dolt embedded (in-process) by default: there is no sql-server and + | nothing is auto-started. A database only uses a dolt sql-server when it is + | configured for one: shared-server mode, an explicit server mode, or a + | non-localhost dolt_server_host. The server-only commands below fail with + | "not supported in embedded mode (no Dolt server)" on an embedded database. + | + | Server lifecycle (server mode only): + | bd dolt start Start the Dolt server for this project + | bd dolt stop Stop the Dolt server for this project + | + | Diagnostics (both modes): + | bd dolt status Show Dolt engine status (embedded: in-process, data dir) + | bd dolt show Show current Dolt configuration with connection test + | + | Configuration (server mode only): + | bd dolt set Set a configuration value + | bd dolt test Test server connection + | + | Version control: + | bd dolt commit Commit pending changes + | bd dolt push Push commits to Dolt remote + | bd dolt pull Pull commits from Dolt remote + | + | Remote management: + | bd dolt remote add Add a Dolt remote + | bd dolt remote list List configured remotes + | bd dolt remote remove Remove a Dolt remote + | + | Configuration keys for 'bd dolt set': + | database Database name (default: issue prefix or "beads") + | host Server host (default: 127.0.0.1) + | port Server port (auto-detected; override with bd dolt set port ) + | user MySQL user (default: root) + | data-dir Custom dolt data directory (absolute path; default: .beads/dolt) + | + | Remote server authentication (password + TLS) is NOT stored via 'bd dolt set' + | (keeps secrets out of metadata.json). Configure them with: + | + | BEADS_DOLT_PASSWORD Server password (highest priority) + | BEADS_DOLT_SERVER_TLS Enable TLS (set to "1" or "true") + | BEADS_DOLT_SERVER_USER MySQL user override (else use 'bd dolt set user') + | BEADS_CREDENTIALS_FILE Optional path to credentials file + | + | Default credentials file: ~/.config/beads/credentials (Linux/macOS) + | %APPDATA%\beads\credentials (Windows) + | Format (INI, section = host:port of the resolved connection): + | [127.0.0.1:3307] + | password = secret + | + | Password resolution: BEADS_DOLT_PASSWORD → credentials [host:port] → empty. + | Full reference: docs/architecture/dolt.md (Environment Variables / Credentials). + | + | Flags for 'bd dolt set': + | --update-config Also write to config.yaml for team-wide defaults + | + | Examples: + | bd dolt set database myproject + | bd dolt set host 192.168.1.100 --update-config + | bd dolt set data-dir /home/user/.beads-dolt/myproject + | export BEADS_DOLT_PASSWORD=... BEADS_DOLT_SERVER_TLS=1 + | bd dolt test + +bd dolt clean-databases + use: "clean-databases" + short: "Drop stale test databases from the Dolt server" + hidden: false + runnable: true + long: + | Identify and drop leftover test and agent databases that accumulate + | on the shared Dolt server from interrupted test runs and terminated agents. + | + | Stale database prefixes: testdb_*, beads_test*, beads_pt*, beads_vr*, doctest_*, doctortest_*, benchdb_* + | + | These waste server memory and can degrade performance under concurrent load. + | Use --dry-run to see what would be dropped without actually dropping. + | + | DROP DATABASE only marks a database as dropped; Dolt keeps its directory + | under .dolt_dropped_databases/ so it can be restored with + | CALL DOLT_UNDROP(name) until an explicit purge — disk is not reclaimed + | until then. Pass --purge-dropped to run CALL DOLT_PURGE_DROPPED_DATABASES() + | after cleanup. + | + | --purge-dropped is SERVER-GLOBAL and IRREVERSIBLE. Dolt has no way to scope + | the purge to only the databases this run dropped: it permanently deletes + | every dropped-but-not-yet-purged database on the server, including ones + | dropped by something else entirely (e.g. an operator's accidental + | DROP DATABASE on an unrelated database that was still recoverable via + | DOLT_UNDROP). It also purges pre-existing residue from earlier + | clean-databases runs even if this run finds no stale databases to drop. + | Only pass it when nothing else on the server may be relying on DOLT_UNDROP + | recovery. + flag --dry-run type=bool default="false" usage="Show what would be dropped without dropping" + flag --purge-dropped type=bool default="false" usage="After dropping, also run CALL DOLT_PURGE_DROPPED_DATABASES() — server-global and irreversible, see --help" + +bd dolt commit + use: "commit" + short: "Create a Dolt commit from pending changes" + hidden: false + runnable: true + long: + | Create a Dolt commit from any uncommitted changes in the working set. + | + | This is the primary commit point for batch mode. When auto-commit is set to + | "batch", changes accumulate in the working set across multiple bd commands and + | are committed together here with a descriptive summary message. + | + | Also useful before push operations that require a clean working set, or when + | auto-commit was off or changes were made externally. + | + | For more options (--stdin, custom messages), see: bd vc commit + flag --message -m type=string default="" usage="Commit message (default: auto-generated)" + +bd dolt killall + use: "killall" + short: "Kill all orphan Dolt server processes" + hidden: false + runnable: true + long: + | Find and kill orphan dolt sql-server processes not tracked by the + | canonical PID file for the current repo's Dolt data directory. + | + | Under an orchestrator, the canonical server lives at $GT_ROOT/.beads/. Any other + | dolt sql-server processes using that shared data directory are considered + | orphans and will be killed. + | + | In standalone mode, only dolt sql-server processes using the current + | project's Dolt data directory are eligible for cleanup. Other projects' + | servers are preserved. + +bd dolt pull + use: "pull" + short: "Pull commits from Dolt remote" + hidden: false + runnable: true + long: + | Pull commits from the configured Dolt remote into the local database. + | + | Requires a Dolt remote to be configured in the database directory. + | For Hosted Dolt, set DOLT_REMOTE_USER and DOLT_REMOTE_PASSWORD environment + | variables for authentication. + | + | Use --remote to pull from a specific named remote instead of the default. + | The remote must already exist (see 'bd dolt remote add'). + | + | Use --strategy ours|theirs to resolve conflicts the auto-resolver declines + | (e.g. both sides edited the same issue since the last sync) instead of + | aborting the pull for manual resolution. Embedded storage only (#4992); on + | server-mode/sql-server storage use 'bd conflicts resolve' after a pull that + | reports conflicts. + flag --remote type=string default="" usage="Pull from a specific named remote instead of the default" + flag --strategy type=string default="" usage="Conflict resolution strategy for conflicts the auto-resolver declines: 'ours' or 'theirs' (embedded storage only, #4992)" + +bd dolt push + use: "push" + short: "Push commits to Dolt remote" + hidden: false + runnable: true + long: + | Push local Dolt commits to the configured remote. + | + | Requires a Dolt remote to be configured in the database directory. With no + | remote configured, bd can adopt one derived from git origin — only with + | consent: interactively, or via --yes; --no-adopt or BD_NO_REMOTE_ADOPT=1 + | disables adoption entirely. + | For Hosted Dolt, set DOLT_REMOTE_USER and DOLT_REMOTE_PASSWORD environment + | variables for authentication. + | + | Use --force to overwrite remote changes (e.g., when the remote has + | uncommitted changes in its working set). + | + | Use --remote to push to a specific named remote instead of the default. + | The remote must already exist (see 'bd dolt remote add'). + flag --force type=bool default="false" usage="Force push (overwrite remote changes)" + flag --no-adopt type=bool default="false" usage="Never derive a Dolt remote from git origin (also BD_NO_REMOTE_ADOPT=1)" + flag --remote type=string default="" usage="Push to a specific named remote instead of the default" + flag --yes -y type=bool default="false" usage="Consent to adopting a Dolt remote derived from git origin when none is configured" + +bd dolt remote + use: "remote" + short: "Manage Dolt remotes" + hidden: false + runnable: false + long: + | Manage Dolt remotes for push/pull replication. + | + | Subcommands: + | add Add a new remote + | list List all configured remotes + | remove Remove a remote + | reset-data Replace a remote's data plane after a history squash + +bd dolt remote add + use: "add " + short: "Add a Dolt remote" + hidden: false + runnable: true + flag --allow-git-origin type=bool default="false" usage="Allow adding a Dolt remote whose URL matches the git origin (proceed with a warning instead of aborting)" + +bd dolt remote list + use: "list" + short: "List configured Dolt remotes" + hidden: false + runnable: true + +bd dolt remote remove + use: "remove " + short: "Remove a Dolt remote" + hidden: false + runnable: true + +bd dolt remote reset-data + use: "reset-data " + short: "Replace a remote's data plane in place after a history squash" + hidden: false + runnable: true + long: + | Replace a Dolt remote's stored data with a fresh copy of local HEAD. + | + | After a history squash (see the History Bloat recovery runbook), a plain + | 'bd dolt push --force' re-points the remote's refs but deletes nothing: + | Dolt remotes accumulate chunks monotonically, so the remote keeps the full + | pre-squash store. This command rebuilds the remote's data plane so it holds + | only live chunks: + | + | - Git-backed remotes (issue data riding a git remote under refs/dolt/data): + | deletes the Dolt data refs on the git remote, then force-pushes to + | rebuild a fresh store. Code branches are untouched. + | - Native file remotes (file:// paths): clears the store directory, then + | force-pushes to rebuild it. + | - Cloud/hosted remotes (aws://, gs://, dolthub://, ...): bd cannot clear + | the stored data safely — replace the remote with a fresh URL or prefix: + | bd dolt remote remove + | bd dolt remote add + | bd dolt push --force + | + | This rewrites the remote's data plane. Every other clone must re-clone from + | the reset remote (that is already true after the squash itself). Refuses to + | run with uncommitted working-set changes: the rebuilt remote holds exactly + | HEAD, and anything uncommitted would not be part of it. + | + | Examples: + | bd dolt remote reset-data origin # prompts for confirmation + | bd dolt remote reset-data origin --yes # no prompt (scripts, agents) + flag --yes -y type=bool default="false" usage="Skip the confirmation prompt (required in non-interactive use)" + +bd dolt set + use: "set " + short: "Set a Dolt configuration value" + hidden: false + runnable: true + long: + | Set a Dolt configuration value in metadata.json. + | + | Keys: + | database Database name (default: issue prefix or "beads") + | host Server host (default: 127.0.0.1) + | port Server port (auto-detected; override with bd dolt set port ) + | user MySQL user (default: root) + | data-dir Custom dolt data directory (absolute path; default: .beads/dolt) + | + | There is no 'password' or 'tls' key here on purpose — secrets and TLS must + | not land in metadata.json. Use environment variables or the credentials file: + | + | BEADS_DOLT_PASSWORD Server password (highest priority) + | BEADS_DOLT_SERVER_TLS Enable TLS ("1" or "true") + | BEADS_CREDENTIALS_FILE Optional override path for credentials + | + | Default credentials file: ~/.config/beads/credentials + | Format: + | [host:port] + | password = secret + | + | See: bd dolt --help and docs/architecture/dolt.md + | + | Use --update-config to also write to config.yaml for team-wide defaults. + | + | Examples: + | bd dolt set database myproject + | bd dolt set host 192.168.1.100 + | bd dolt set port 3307 --update-config + | bd dolt set data-dir /home/user/.beads-dolt/myproject + | export BEADS_DOLT_PASSWORD=... BEADS_DOLT_SERVER_TLS=1 + flag --update-config type=bool default="false" usage="Also write to config.yaml for team-wide defaults" + +bd dolt show + use: "show" + short: "Show current Dolt configuration with connection status" + hidden: false + runnable: true + +bd dolt start + use: "start" + short: "Start the Dolt SQL server for this project" + hidden: false + runnable: true + long: + | Start a dolt sql-server for the current beads project. + | + | The server runs in the background on a per-project port derived from the + | project path. PID and logs are stored in .beads/. + | + | The server auto-starts transparently when needed, so manual start is rarely + | required. Use this command for explicit control or diagnostics. + | + | Not available in proxied-server mode: there the proxy owns its dolt backend's + | lifecycle and starts it on demand, so a server started here would be a second, + | unsupervised one over the same data directory. Use 'bd dolt status' to see what + | is running and 'bd dolt stop' to shut it down. + +bd dolt status + use: "status" + short: "Show Dolt engine status" + hidden: false + runnable: true + long: + | Show the status of the Dolt engine for the current project. + | + | In embedded mode, reports that the Dolt engine runs in-process and shows + | the on-disk data directory. For beads-managed (local) servers, displays + | PID, port, and data directory from the local PID file. For externally- + | managed servers — a shared server (dolt.shared-server: true), a remote + | dolt_server_host, or a local server managed outside bd (dolt.auto-start: + | false, e.g. an orchestrator-shared sql-server) — pings the configured + | endpoint via SQL and reports reachability, server version, and database. + | + | In proxied-server mode, reports the proxy (the endpoint every bd command + | connects through) and the dolt backend behind it separately, read from the + | proxy's own process records. Reporting starts nothing: a quiesced workspace + | is shown as not running, and the next bd command launches it. + +bd dolt stop + use: "stop" + short: "Stop the Dolt SQL server for this project" + hidden: false + runnable: true + long: + | Stop the dolt sql-server managed by beads for the current project. + | + | This sends a graceful shutdown signal. The server will restart automatically + | on the next bd command unless auto-start is disabled. + | + | For a managed proxied server, --force can recover unverifiable or legacy + | process records (both the proxy and its backend) only after each live process + | executable is matched to bd or dolt and its command line ties it to this + | workspace. In that recovery path, force still refuses to signal a process + | whose executable identity cannot be matched to bd or dolt, or whose workspace + | scope cannot be established. + flag --force type=bool default="false" usage="Force stop (proxied recovery still requires a bd/dolt executable match)" + +bd dolt test + use: "test" + short: "Test connection to Dolt server" + hidden: false + runnable: true + long: + | Test the connection to the configured Dolt server. + | + | This verifies that: + | 1. The server is reachable at the configured host:port + | 2. The connection can be established + | + | Use this before switching to server mode to ensure the server is running. + +bd duplicate + use: "duplicate --of " + short: "Mark an issue as a duplicate of another" + group: "deps" + hidden: false + runnable: true + long: + | Mark an issue as a duplicate of a canonical issue. + | + | The duplicate issue is automatically closed with a reference to the canonical. + | This is essential for large issue databases with many similar reports. + | + | Examples: + | bd duplicate bd-abc --of bd-xyz # Mark bd-abc as duplicate of bd-xyz + flag --of type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="Canonical issue ID (required)" + +bd duplicates + use: "duplicates" + short: "Find and optionally merge duplicate issues" + group: "deps" + hidden: false + runnable: true + long: + | Find issues with identical content (title, description, design, acceptance criteria). + | Groups issues by content hash and reports duplicates with suggested merge targets. + | The merge target is chosen by: + | 1. Reference count (most referenced issue wins) + | 2. Lexicographically smallest ID if reference counts are equal + | Only groups issues with matching status (open with open, closed with closed). + | Example: + | bd duplicates # Show all duplicate groups + | bd duplicates --auto-merge # Automatically merge all duplicates + | bd duplicates --dry-run # Show what would be merged + flag --auto-merge type=bool default="false" usage="Automatically merge all duplicates" + flag --dry-run type=bool default="false" usage="Show what would be merged without making changes" + +bd edit + use: "edit [id]" + short: "Edit an issue field in $EDITOR" + group: "issues" + hidden: false + runnable: true + long: + | Edit an issue field using your configured $EDITOR. + | + | By default, edits the description. Use flags to edit other fields. + | + | Examples: + | bd edit bd-42 # Edit description + | bd edit bd-42 --title # Edit title + | bd edit bd-42 --design # Edit design notes + | bd edit bd-42 --notes # Edit notes + | bd edit bd-42 --acceptance # Edit acceptance criteria + flag --acceptance type=bool default="false" usage="Edit the acceptance criteria" + flag --description type=bool default="false" usage="Edit the description (default)" + flag --design type=bool default="false" usage="Edit the design notes" + flag --notes type=bool default="false" usage="Edit the notes" + flag --title type=bool default="false" usage="Edit the title" + +bd epic + use: "epic" + short: "Epic management commands" + group: "deps" + hidden: false + runnable: false + +bd epic close-eligible + use: "close-eligible" + short: "Close epics where all children are complete" + hidden: false + runnable: true + flag --dry-run type=bool default="false" usage="Preview what would be closed without making changes" + flag --reason -r type=string default="All children completed" usage="Close reason applied to every epic closed" + +bd epic status + use: "status" + short: "Show epic completion status" + hidden: false + runnable: true + flag --eligible-only type=bool default="false" usage="Show only epics eligible for closure" + +bd events + use: "events" + short: "Read and manage the durable events journal" + group: "maint" + hidden: false + runnable: false + long: + | Read and manage the durable events journal (bd_events_journal). + | + | The journal records every committed issue mutation as an ordered, replayable + | row. Enable it with 'bd config set events-journal true' (or + | BD_EVENTS_JOURNAL=1). Records are emitted only while it is enabled. + | + | Retention is automatic: an enabled journal is bounded to the retention floors + | (events-journal-retain-days / -rows, 7 days / 100k rows by default) without + | anyone running a command. Disable both floors for an unbounded ledger, or + | events-journal-auto-prune for manual control; 'bd events prune' remains for an + | earlier, on-demand cut below the floors. + | + | Coverage and scope: + | - Every mutation through bd's normal write paths (create, update, close, + | reopen, delete, claim, dependency add/remove, label add/remove, comment) is + | journaled in the same transaction as the change. Raw DML run through + | 'bd sql' bypasses those paths and is NOT journaled — a known non-coverage. + | - The journal is per-branch working-set state (dolt_ignored): it records the + | mutations committed on the writer's active branch. Rows arrive by direct + | write, not by merge, so a consumer must read the journal on the same branch + | the writer commits to; a branch checkout or merge does not carry journal + | rows across branches. + | - For the same reason the journal is per REPLICA. 'bd dolt pull' and the + | changes a merge settles into this clone are not journaled: those rows + | arrived as data, not as local mutations, and nothing on this clone wrote + | them through the mutation seam. A consumer that mirrors a synced workspace + | must re-baseline (a fresh export or a full re-read) after a sync, because + | the journal describes only what THIS clone mutated. + | Each replica also has its OWN seq space, counted from its own first + | mutation. A checkpoint taken against one replica is meaningless against + | another — the same seq names a different record, and a seq above the other + | replica's head reads as "caught up" and stalls forever. Track a checkpoint + | per replica, and re-baseline rather than carry one across. + | - A few writes that happen while a store is being OPENED are unjournaled by + | design: schema migrations and the version reconciliation that runs before + | the workspace's configuration has been applied to the store. They touch + | schema and clone-local metadata, never a bead, so a replaying consumer has + | nothing to apply them to. + | - Dependency records are not symmetric, in two ways. + | Count: a dep_add is emitted for every accepted add, INCLUDING an idempotent + | same-type re-add that only refreshes the edge's metadata. The audit 'events' + | table deduplicates that case and writes nothing; the journal does not. Treat + | dep_add as an upsert of the edge, not as proof the edge is new. A dep_remove + | naming an edge that is already gone emits nothing at all. + | Payload: dep.metadata differs in provenance between the two ops. On dep_add + | it is the value being written, as the caller supplied it; on dep_remove it is + | the raw stored column read back just before the delete. The two can differ + | byte for byte while meaning the same thing, so compare parsed values. + | + | Structural dependency edits — the ones bd wires up itself rather than a 'bd dep' + | verb — write no audit event but DO journal, by design: a replaying consumer + | needs the edge either way. + +bd events export + use: "export" + short: "Print the entire journal from the beginning (JSON lines)" + hidden: false + runnable: true + long: + | Print every events journal record from seq 1, in order, as JSON lines. + | + | Equivalent to 'bd events tail --since 0'. Like tail, it FAILS rather than + | present a pruned journal's surviving suffix as a complete history. + flag --limit type=int default="0" usage="maximum number of records to return (0 = no limit)" + +bd events prune + use: "prune" + short: "Delete journal records below a sequence number (retention)" + hidden: false + runnable: true + long: + | Delete events journal records with seq less than --before. + | + | Retention is already enforced automatically: after a mutating command commits, + | and on a timer in 'bd serve', bd deletes everything the floors below do not + | protect. This command is for an EARLIER, on-demand cut BELOW the floors — after + | a consumer has durably processed a span you do not want to wait out. It cannot + | cut deeper than the floors: shrinking the retained window itself means lowering + | them. The journal is clone-local operational state, so pruning never affects + | issue data. + | + | Two retention floors compose onto --before and can only reduce what a prune + | removes. They bound the automatic prune and this one identically: + | events-journal-retain-days keep every row younger than N days (default 7) + | events-journal-retain-rows always keep the newest N rows (default 100000) + | + | Set BOTH floors to 0 for an unbounded ledger: automatic pruning then does + | nothing, and this command becomes the only thing that deletes a record. To keep + | the floors but own deletion yourself, set events-journal-auto-prune false. + | + | Note the floors are time-based and count-based — they are NOT a consumer + | watermark. They protect only the recent window; a consumer that has fallen + | further behind than both floors allow will be pruned past and lose records. + | Consumers are responsible for tracking their own watermark (the highest seq they + | have durably processed) and for sizing the floors to the longest outage they + | intend to survive. Pruned history cannot be recovered from the workspace — the + | journal is the only local copy. Pruning frees rows, not disk: pair it with + | 'dolt gc' to reclaim the space, since the table is working-set (dolt_ignored) + | state that ordinary Dolt commits never garbage-collect. + flag --before type=int64 default="0" usage="delete records with seq less than this value" + +bd events tail + use: "tail" + short: "Print journal records after a sequence number (JSON lines)" + hidden: false + runnable: true + long: + | Print events journal records with seq greater than --since, in order. + | + | Each line is a JSON record: + | {"seq":N,"ts":"...","op":"create|update|close|delete|dep_add|dep_remove|comment", + | "issue_id":"...","actor":"...","issue":{...|null},"dep":{"kind":..,"target":..,"metadata":..},"comment":{...}} + | + | Record contract (stable for external consumers): + | seq int64 counter-assigned inside the mutation's transaction; gapless, + | strictly increasing in commit order, never reused or reset + | ts string UTC insert time, stamped inside the committing transaction + | op string one of the seven ops above + | issue_id string the mutated issue's id + | actor string the acting identity that performed the mutation, as resolved + | for the audit-events table (on a comment row: the comment's + | author). A delete — and the dep_remove rows a cascading + | delete produces — carries the identity that REQUESTED it. + | Empty (omitted) only when the path genuinely has no actor: + | derived maintenance, system cleanup with no request behind + | it, and rows older than the column. Never user attribution + | when empty. + | issue object full issue state AFTER the mutation; null on delete + | dep object {"kind","target","metadata"} for dep_add / dep_remove; omitted otherwise + | comment object {"id","author","text","created_at","source"} for comment; omitted otherwise + | + | Poll with the highest seq seen to consume new mutations incrementally, or pass + | --follow to keep printing new records as they are committed (Ctrl-C to stop). + | + | Retention boundary: if --since is below the oldest retained record — the prefix + | you asked for was pruned — the read FAILS instead of silently skipping ahead or + | returning an empty success. With --json the failure carries + | {"code":"events_journal_truncated","since":N,"floor":F,"head":H}: floor is the + | oldest seq still retained, head the highest ever assigned. Resume from floor-1 + | to continue with a known gap, or rebuild from a full export. + flag --follow type=bool default="false" usage="keep printing new records as they are committed (Ctrl-C to stop)" + flag --limit type=int default="0" usage="maximum number of records to return (0 = no limit)" + flag --since type=int64 default="0" usage="return records with seq greater than this value" + +bd export + use: "export" + short: "Export issues to JSONL format" + group: "sync" + hidden: false + runnable: true + long: + | Export all issues to JSONL (newline-delimited JSON) format. + | + | Each line is a complete JSON object representing one issue, including its + | labels, dependencies, and comments. + | + | This command is for issue export, migration, and interoperability. It exports + | records from the issues table; it is not a full database backup and does not + | capture Dolt branches, commit history, working-set state, or non-issue tables. + | For supported full backup/restore flows, use 'bd backup init', 'bd backup sync', + | and 'bd backup restore'. + | + | By default, exports only regular issues (excluding infrastructure beads + | like agents, roles, and messages). Use --all to include everything. + | + | Memories (from 'bd remember') are excluded by default because they may + | contain sensitive agent context. Use --include-memories or --all to + | include them. + | + | EXAMPLES: + | bd export # Export issues to stdout + | bd export -o issues.jsonl # Export issues to file + | bd export --include-memories # Export issues + memories + | bd export --all -o full.jsonl # Include infra + templates + gates + memories + | bd export --scrub -o clean.jsonl # Exclude test/pollution records + flag --all type=bool default="false" usage="Include all records (infra, templates, gates, memories)" + flag --exclude-owner type=stringArray default="[]" usage="Exclude issues created by this identity (repeatable; also reads export.exclude_owners config)" + flag --include-infra type=bool default="false" usage="Include infrastructure beads (agents, roles, messages)" + flag --include-memories type=bool default="false" usage="Include persistent memories (from 'bd remember') in the export" + flag --no-memories type=bool default="false" hidden usage="Exclude persistent memories (deprecated: now the default)" + flag --output -o type=string default="" usage="Output file path (default: stdout)" + flag --scrub type=bool default="false" usage="Exclude test/pollution records" + flag --verbose type=bool default="false" usage="Print filtered issue count when owners are excluded" + +bd federation + use: "federation" + short: "Manage peer-to-peer federation with other workspaces" + group: "sync" + hidden: false + runnable: false + long: + | Manage peer-to-peer federation between Dolt-backed beads databases. + | + | Federation enables synchronized issue tracking across multiple workspaces, + | each maintaining their own Dolt database while sharing updates via remotes. + | + | Requires the Dolt storage backend. + +bd federation add-peer + use: "add-peer " + short: "Add a federation peer with optional SQL credentials" + hidden: false + runnable: true + long: + | Add a new federation peer remote with optional SQL user authentication. + | + | The URL can be: + | - dolthub://org/repo DoltHub hosted repository + | - host:port/database Direct dolt sql-server connection + | - file:///path/to/repo Local file path (for testing) + | + | Credentials are encrypted and stored locally. They are used automatically + | when syncing with the peer. If --user is provided without --password, + | you will be prompted for the password interactively. + | + | Examples: + | bd federation add-peer town-beta dolthub://acme/town-beta-beads + | bd federation add-peer town-gamma 192.168.1.100:3306/beads --user sync-bot + | bd federation add-peer partner https://partner.example.com/beads --user admin --password secret + flag --password -p type=string default="" usage="SQL password (prompted if --user set without --password)" + flag --sovereignty type=string default="" usage="Sovereignty tier (T1, T2, T3, T4)" + flag --user -u type=string default="" usage="SQL username for authentication" + +bd federation list-peers + use: "list-peers" + short: "List configured federation peers" + hidden: false + runnable: true + +bd federation remove-peer + use: "remove-peer " + short: "Remove a federation peer" + hidden: false + runnable: true + +bd federation status + use: "status [--peer name]" + short: "Show federation sync status" + hidden: false + runnable: true + long: + | Show synchronization status with peer towns. + | + | Displays: + | - Configured peers and their URLs + | - Commits ahead/behind each peer + | - Whether there are unresolved conflicts + | + | Examples: + | bd federation status # Status for all peers + | bd federation status --peer town-beta # Status for specific peer + flag --peer type=string default="" usage="Specific peer to check" + +bd federation sync + use: "sync [--peer name]" + short: "Synchronize with a peer town" + hidden: false + runnable: true + long: + | Pull from and push to peer towns. + | + | Without --peer, syncs with all configured peers. + | With --peer, syncs only with the specified peer. + | + | Handles merge conflicts using the configured strategy: + | --strategy ours Keep local changes on conflict + | --strategy theirs Accept remote changes on conflict + | + | If no strategy is specified and conflicts occur, the sync will pause + | and report which tables have conflicts for manual resolution. + | + | Examples: + | bd federation sync # Sync with all peers + | bd federation sync --peer town-beta # Sync with specific peer + | bd federation sync --strategy theirs # Auto-resolve using remote values + flag --peer type=string default="" usage="Specific peer to sync with" + flag --strategy type=string default="" usage="Conflict resolution strategy (ours|theirs)" + +bd find-duplicates + use: "find-duplicates" + aliases: ["find-dups"] + short: "Find semantically similar issues using text analysis or AI" + group: "views" + hidden: false + runnable: true + long: + | Find issues that are semantically similar but not exact duplicates. + | + | Unlike 'bd duplicates' which finds exact content matches, find-duplicates + | uses text similarity or AI to find issues that discuss the same topic + | with different wording. + | + | Approaches: + | mechanical Token-based text similarity (default, no API key needed) + | ai LLM-based semantic comparison (requires ANTHROPIC_API_KEY, MINIMAX_API_KEY, or ai.api_key) + | + | The mechanical approach tokenizes titles and descriptions, then computes + | Jaccard similarity between all issue pairs. It's fast and free but may + | miss semantically similar issues with very different wording. + | + | The AI approach sends candidate pairs to an Anthropic-compatible model for semantic comparison. + | It first uses mechanical pre-filtering to reduce the number of API calls, + | then asks the LLM to judge whether the remaining pairs are true duplicates. + | + | Examples: + | bd find-duplicates # Mechanical similarity (default) + | bd find-duplicates --threshold 0.4 # Lower threshold = more results + | bd find-duplicates --method ai # Use AI for semantic comparison + | bd find-duplicates --status open # Only check open issues + | bd find-duplicates --limit 20 # Show top 20 pairs + | bd find-duplicates --json # JSON output + flag --limit -n type=int default="50" usage="Maximum number of pairs to show" + flag --max-rows type=int default="0" usage="Hard upper bound on rows fetched from storage. Returns a non-zero exit (code 2) and an error to stderr if exceeded. 0 disables (the default). Overrides BEADS_MAX_ROWS for this invocation. Useful in CI/agent rigs that want a circuit breaker against pathological queries. Not supported under --proxied-server: an explicit --max-rows or BEADS_MAX_ROWS cap errors out rather than silently going unenforced." + flag --method type=string default="mechanical" usage="Detection method: mechanical, ai" + flag --model type=string default="" usage="AI model to use (only with --method ai; default from config ai.model)" + flag --status -s type=string default="" usage="Filter by status (default: non-closed)" + flag --threshold type=float64 default="0.5" usage="Similarity threshold (0.0-1.0, lower = more results)" + +bd flatten + use: "flatten" + short: "Squash all Dolt history into a single commit" + group: "maint" + hidden: false + runnable: true + long: + | Nuclear option: squash ALL Dolt commit history into a single commit. + | + | This uses the Tim Sehn recipe: + | 1. Create a new branch from the current state + | 2. Soft-reset to the initial commit (preserving all data) + | 3. Commit everything as a single snapshot + | 4. Swap main branch to the new flattened branch + | 5. Prune remote-tracking refs (they would keep the old history alive; + | the next push or fetch re-creates them at the new tip) + | 6. Run a full Dolt GC (all storage generations) to reclaim the old history + | + | The GC pass is a full collection: Dolt storage is generational, and a default + | GC never revisits data an earlier GC moved to the old generation. On any store + | that has been GC'd before (bd gc, or a previous flatten or compact), only a + | full collection reclaims the squashed history. A full GC can take minutes on + | multi-gigabyte stores. + | + | This is irreversible — all commit history is lost. The resulting database + | has exactly one commit containing all current data. + | + | Use this when: + | - Your active database has grown very large + | - You don't need commit-level history (time travel) + | - You want to start fresh with minimal storage + | + | Examples: + | bd flatten --dry-run # Preview: show commit count and disk usage + | bd flatten --force # Actually squash all history + | bd flatten --force --json # JSON output + flag --dry-run type=bool default="false" usage="Preview without making changes" + flag --force -f type=bool default="false" usage="Confirm irreversible history squash" + +bd forget + use: "forget " + short: "Remove a persistent memory" + group: "setup" + hidden: false + runnable: true + long: + | Remove a memory by its key. + | + | Use 'bd memories' to see available keys. + | + | Examples: + | bd forget dolt-phantoms + | bd forget auth-jwt + +bd formula + use: "formula" + short: "Manage workflow formulas" + hidden: false + runnable: false + long: + | Manage workflow formulas - the source layer for molecule templates. + | + | Formulas are TOML/JSON files that define workflows with composition rules. + | Define formulas, cook them into protos, then pour or wisp them into work. + | + | Search paths (in order): + | 1. /formulas/ (active project) + | 2. /.beads/formulas/ (repo-local formulas) + | 3. ~/.beads/formulas/ (user) + | 4. $GT_ROOT/.beads/formulas/ (shared workspace root, if GT_ROOT set) + | + | Commands: + | list List available formulas from all search paths + | show Show formula details, steps, and composition rules + | schema Show the formula schema index (alias: primitives) + | + | Discovering primitives: + | bd formula schema # list every declared formula struct + | bd formula schema loop # show LoopSpec fields, types, and tags + | bd formula primitives gate # alias; same handler as 'schema' + | examples/formulas/primitives/ # curated, smoke-tested wired fixtures + | docs/workflows/formulas.md # narrative reference + +bd formula convert + use: "convert [--all]" + short: "Convert formula from JSON to TOML" + hidden: false + runnable: true + long: + | Convert formula files from JSON to TOML format. + | + | TOML format provides better ergonomics: + | - Multi-line strings without \n escaping + | - Human-readable diffs + | - Comments allowed + | + | The convert command reads a .formula.json file and outputs .formula.toml. + | The original JSON file is preserved (use --delete to remove it). + | + | Examples: + | bd formula convert shiny # Convert shiny.formula.json to .toml + | bd formula convert ./my.formula.json # Convert specific file + | bd formula convert --all # Convert all JSON formulas + | bd formula convert shiny --delete # Convert and remove JSON file + | bd formula convert shiny --stdout # Print TOML to stdout + flag --all type=bool default="false" usage="Convert all JSON formulas" + flag --delete type=bool default="false" usage="Delete JSON file after conversion" + flag --stdout type=bool default="false" usage="Print TOML to stdout instead of file" + +bd formula list + use: "list" + short: "List available formulas" + hidden: false + runnable: true + long: + | List all formulas from search paths. + | + | Search paths (in order of priority): + | 1. /formulas/ (active project - highest priority) + | 2. /.beads/formulas/ (repo-local formulas) + | 3. ~/.beads/formulas/ (user) + | 4. $GT_ROOT/.beads/formulas/ (shared workspace root, if GT_ROOT set) + | + | Formulas in earlier paths shadow those with the same name in later paths. + | + | To list the declared formula schema structs an agent can write inside a .formula.toml, + | use 'bd formula schema' (alias: 'bd formula primitives'). + | + | Examples: + | bd formula list + | bd formula list --json + | bd formula list --type workflow + | bd formula list --type convoy + flag --type type=string default="" usage="Filter by type (workflow, expansion, aspect, convoy)" + +bd formula schema + use: "schema [primitive]" + aliases: ["primitives"] + short: "Show the formula schema index (every exported struct in types.go)" + hidden: false + runnable: true + long: + | Show the formula schema index: every exported struct declared + | in a .formula.toml/.formula.json, with field names, types, and tags. + | + | The index is generated from internal/formula/types.go via go:generate; the + | struct definitions are the source of truth, so this list cannot drift. It is + | structural reference, not proof that every declared runtime behavior is wired. + | + | Examples: + | bd formula schema # list every declared schema struct + | bd formula schema loop # show LoopSpec fields + | bd formula primitives gate # alias; shows Gate fields + | bd formula schema --json # machine-readable index + | + | Curated smoke-tested fixtures for wired primitives live in + | examples/formulas/primitives/ (with a smoke harness that proves they work). + +bd formula show + use: "show " + short: "Show formula details" + hidden: false + runnable: true + long: + | Show detailed information about a formula. + | + | Displays: + | - Formula metadata (name, type, description) + | - Variables with defaults and constraints + | - Steps with dependencies + | - Composition rules (extends, aspects, expansions) + | - Bond points for external composition + | + | To inspect the structure of an individual primitive (e.g. LoopSpec, Gate) + | rather than a user-authored formula, use 'bd formula schema '. + | + | Examples: + | bd formula show shiny + | bd formula show rule-of-five + | bd formula show security-audit --json + +bd gate + use: "gate" + short: "Manage async coordination gates" + group: "issues" + hidden: false + runnable: false + long: + | Gates are async wait conditions that block workflow steps. + | + | Gates are created automatically when a formula step has a gate field. + | They must be closed (manually or via watchers) for the blocked step to proceed. + | + | Gate types: + | human - Requires manual bd close (Phase 1) + | timer - Expires after timeout (Phase 2) + | gh:run - Waits for GitHub workflow (Phase 3) + | gh:pr - Waits for PR merge (Phase 3) + | bead - Waits for another bead to close (Phase 4) + | + | For bead gates, await_id may be a bead ID in this rig's database (e.g., + | "bd-abc123") or the historical cross-rig form :. Cross-rig + | targets resolve through the bead ID's prefix route in routes.jsonl. + | + | Examples: + | bd gate list # Show all open gates + | bd gate list --all # Show all gates including closed + | bd gate check # Evaluate all open gates + | bd gate check --type=bead # Evaluate only bead gates + | bd gate resolve # Close a gate manually + +bd gate add-waiter + use: "add-waiter " + short: "Add a waiter to a gate" + hidden: false + runnable: true + long: + | Register an agent as a waiter on a gate bead. + | + | When the gate closes, the waiter will receive a wake notification via 'bd gate wake'. + | The waiter is typically the worker's address (e.g., "my-project/workers/agent-1"). + | + | This is used by 'bd done --phase-complete' to register for gate wake notifications. + +bd gate check + use: "check" + short: "Evaluate gates and close resolved ones" + hidden: false + runnable: true + long: + | Evaluate gate conditions and automatically close resolved gates. + | + | By default, checks all open gates. Use --type to filter by gate type. + | + | Gate types: + | gh - Check all GitHub gates (gh:run and gh:pr) + | gh:run - Check GitHub Actions workflow runs + | gh:pr - Check pull request merge status + | timer - Check timer gates (auto-expire based on timeout) + | bead - Check bead gates + | all - Check all gate types + | + | GitHub gates use the 'gh' CLI to query status: + | - gh:run checks 'gh run view --json status,conclusion' + | - gh:pr checks 'gh pr view --json state,title' + | + | A gate is resolved when: + | - gh:run: status=completed AND conclusion=success + | - gh:pr: state=MERGED + | - timer: current time > created_at + timeout + | - bead: target bead status=closed, or a bead an earlier check saw in + | this rig no longer exists + | + | A gate is escalated when: + | - gh:run: status=completed AND conclusion in (failure, canceled) + | - gh:pr: state=CLOSED + | + | Examples: + | bd gate check # Check all gates + | bd gate check --type=gh # Check only GitHub gates + | bd gate check --type=gh:run # Check only workflow run gates + | bd gate check --type=timer # Check only timer gates + | bd gate check --type=bead # Check only bead gates + | bd gate check --dry-run # Show what would happen without changes + | bd gate check --escalate # Escalate expired/failed gates + flag --dry-run type=bool default="false" usage="Show what would happen without making changes" + flag --escalate -e type=bool default="false" usage="Escalate failed/expired gates" + flag --limit -l type=int default="100" usage="Limit results (default 100)" + flag --type -t type=string default="" usage="Gate type to check (gh, gh:run, gh:pr, timer, bead, all)" + +bd gate create + use: "create" + short: "Create a gate that blocks an issue" + hidden: false + runnable: true + long: + | Create an ad-hoc gate issue that blocks another issue until resolved. + | + | The blocked issue will not appear in 'bd ready' until the gate is resolved + | via 'bd gate resolve'. + | + | Gate types: + | human - Requires manual 'bd gate resolve' (default) + | timer - Auto-resolves after --timeout duration + | gh:run - Waits for GitHub Actions workflow + | gh:pr - Waits for PR merge + | + | gh:run and gh:pr gates are checked in the current Git repository unless + | --repo names another, or the blocked issue carries a metadata.repo value. + | + | Examples: + | bd gate create --blocks bd-abc + | bd gate create --type=human --blocks bd-abc --reason="Need design review" + | bd gate create --type=timer --blocks bd-abc --timeout=2h + | bd gate create --type=gh:pr --blocks bd-abc --await-id=42 + | bd gate create --type=gh:pr --blocks bd-abc --await-id=42 --repo=owner/other-repo + | bd gate create --blocks bd-abc --title="Gate: awaiting owner sign-off" + flag --await-id type=string default="" usage="Condition identifier (run ID, PR number, etc.)" + flag --blocks type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="Issue ID to block (required)" + flag --reason -r type=string default="" usage="Reason for the gate" + flag --repo type=string default="" usage="GitHub repository the gh:run/gh:pr condition is checked in (OWNER/REPO or HOST/OWNER/REPO); default: the blocked issue's metadata.repo, else the current repository" + flag --timeout type=string default="" usage="Timeout duration (e.g., 2h, 30m)" + flag --title type=string default="" usage="Custom gate title (default: \"Gate: \")" + flag --type -t type=string default="human" usage="Gate type (human, timer, gh:run, gh:pr)" + +bd gate discover + use: "discover" + short: "Discover await_id for gh:run gates" + hidden: false + runnable: true + long: + | Discovers GitHub workflow run IDs for gates awaiting CI/CD completion. + | + | This command finds open gates with await_type="gh:run" that don't have an await_id, + | queries recent GitHub workflow runs, and matches them using heuristics: + | - Branch name matching + | - Commit SHA matching + | - Time proximity (runs within 5 minutes of gate creation) + | + | Once matched, the gate's await_id is updated with the GitHub run ID, enabling + | subsequent polling to check the run's status. + | + | A gate whose metadata.repo targets another repository is only matched + | against runs queried from that repository, never against the current + | repository's runs of a same-named workflow. + | + | Examples: + | bd gate discover # Auto-discover run IDs for all matching gates + | bd gate discover --dry-run # Preview what would be matched (no updates) + | bd gate discover --branch main --limit 10 # Only match runs on 'main' branch + flag --branch -b type=string default="" usage="Filter runs by branch (default: current branch)" + flag --dry-run -n type=bool default="false" usage="Preview mode: show matches without updating" + flag --limit -l type=int default="10" usage="Max runs to query from GitHub" + flag --max-age -a type=duration default="30m0s" usage="Max age for gate/run matching" + +bd gate list + use: "list [issue-id]" + short: "List gate issues" + hidden: false + runnable: true + long: + | List gate issues. + | + | With no argument, lists all gate issues in the current beads database. + | With an [issue-id] argument, lists ONLY the gates that block that issue + | (its own dependency gates) — not every gate in the database. + | + | By default, shows only open gates. Use --all to include closed gates. + flag --all -a type=bool default="false" usage="Show all gates including closed" + flag --limit -n type=int default="50" usage="Limit results (default 50)" + +bd gate resolve + use: "resolve " + short: "Manually resolve (close) a gate" + hidden: false + runnable: true + long: + | Close a gate issue to unblock the step waiting on it. + | + | This is equivalent to 'bd close ' but with a more explicit name. + | Use --reason to provide context for why the gate was resolved. + flag --reason -r type=string default="" usage="Reason for resolving the gate" + +bd gate show + use: "show " + short: "Show a gate issue" + hidden: false + runnable: true + long: + | Display details of a gate issue including its waiters. + | + | This is similar to 'bd show' but validates that the issue is a gate. + +bd gc + use: "gc" + short: "Garbage collect: decay old issues, compact Dolt commits, run Dolt GC" + group: "maint" + hidden: false + runnable: true + long: + | Full lifecycle garbage collection for standalone Beads databases. + | + | Runs three phases in sequence: + | 1. DECAY — Delete closed issues older than N days (default 90) + | 2. COMPACT — Squash old Dolt commits into fewer commits (bd compact) + | 3. GC — Run Dolt garbage collection to reclaim disk space + | + | Each phase can be skipped individually. Use --dry-run to preview all phases + | without making changes. + | + | Phase 3 runs Dolt's default, generational GC: it only examines data written + | since the last GC. Data that survived an earlier GC lives in the old generation + | and is never revisited, so on a long-lived store the space freed by decay may + | not be reclaimed. Use --full to collect all generations (slower on large + | stores). + | + | Examples: + | bd gc # Full GC with defaults (90 day decay) + | bd gc --dry-run # Preview what would happen + | bd gc --older-than 30 # Decay issues closed 30+ days ago + | bd gc --skip-decay # Skip issue deletion, just compact+GC + | bd gc --skip-dolt # Skip Dolt GC, just decay+compact + | bd gc --full # Collect all storage generations + | bd gc --force # Skip confirmation prompt + flag --dry-run type=bool default="false" usage="Preview without making changes" + flag --force -f type=bool default="false" usage="Skip confirmation prompts" + flag --full type=bool default="false" usage="Run a full Dolt GC (all generations; slower, reclaims space default passes cannot)" + flag --older-than type=int default="90" usage="Delete closed issues older than N days" + flag --skip-decay type=bool default="false" usage="Skip issue deletion phase" + flag --skip-dolt type=bool default="false" usage="Skip Dolt garbage collection phase" + +bd github + use: "github" + short: "GitHub integration commands" + hidden: false + runnable: false + long: + | Commands for syncing issues between beads and GitHub. + | + | Configuration can be set via 'bd config' or environment variables: + | github.token / GITHUB_TOKEN - Personal access token + | github.owner / GITHUB_OWNER - Repository owner + | github.repo / GITHUB_REPO - Repository name + | github.repository / GITHUB_REPOSITORY - Combined "owner/repo" format + | github.url / GITHUB_API_URL - Custom API URL (GitHub Enterprise) + +bd github pull + use: "pull [refs...]" + short: "Pull specific items from GitHub" + hidden: false + runnable: true + long: + | Pull one or more items from GitHub. + | + | Accepts bead IDs or external references as positional arguments. + | Equivalent to: bd github sync --pull-only --issues + flag --dry-run type=bool default="false" usage="Preview pull without making changes" + +bd github push + use: "push [bead-ids...]" + short: "Push specific beads to GitHub" + hidden: false + runnable: true + long: + | Push one or more beads issues to GitHub. + | + | Accepts bead IDs as positional arguments. + | Equivalent to: bd github sync --push-only --issues + flag --dry-run type=bool default="false" usage="Preview push without making changes" + +bd github repos + use: "repos" + short: "List accessible GitHub repositories" + hidden: false + runnable: true + long: + | List GitHub repositories that the configured token has access to. + +bd github status + use: "status" + short: "Show GitHub sync status" + hidden: false + runnable: true + long: + | Display current GitHub configuration and sync status. + +bd github sync + use: "sync" + short: "Sync issues with GitHub" + hidden: false + runnable: true + long: + | Synchronize issues between beads and GitHub. + | + | By default, performs bidirectional sync: + | - Pulls new/updated issues from GitHub to beads + | - Pushes local beads issues to GitHub + | + | Use --pull-only or --push-only to limit direction. + flag --dry-run type=bool default="false" usage="Show what would be synced without making changes" + flag --issues type=string default="" usage="Comma-separated bead IDs to sync selectively (e.g., bd-abc,bd-def). Mutually exclusive with --parent." + flag --parent type=string default="" usage="Limit push to this bead and its descendants (push only). Mutually exclusive with --issues." + flag --prefer-github type=bool default="false" usage="On conflict, use GitHub version" + flag --prefer-local type=bool default="false" usage="On conflict, keep local beads version" + flag --prefer-newer type=bool default="false" usage="On conflict, use most recent version (default)" + flag --pull-only type=bool default="false" usage="Only pull issues from GitHub" + flag --push-only type=bool default="false" usage="Only push issues to GitHub" + +bd gitlab + use: "gitlab" + short: "GitLab integration commands" + hidden: false + runnable: false + long: + | Commands for syncing issues between beads and GitLab. + | + | Configuration can be set via 'bd config' or environment variables: + | gitlab.url / GITLAB_URL - GitLab instance URL + | gitlab.token / GITLAB_TOKEN - Personal access token + | gitlab.project_id / GITLAB_PROJECT_ID - Project ID or path + | gitlab.group_id / GITLAB_GROUP_ID - Group ID for group-level sync + | gitlab.default_project_id / GITLAB_DEFAULT_PROJECT_ID - Project for creating issues in group mode + +bd gitlab projects + use: "projects" + short: "List accessible GitLab projects" + hidden: false + runnable: true + long: + | List GitLab projects that the configured token has access to. + +bd gitlab pull + use: "pull [refs...]" + short: "Pull specific items from GitLab" + hidden: false + runnable: true + long: + | Pull one or more items from GitLab. + | + | Accepts bead IDs or external references as positional arguments. + | Equivalent to: bd gitlab sync --pull-only --issues + flag --dry-run type=bool default="false" usage="Preview pull without making changes" + +bd gitlab push + use: "push [bead-ids...]" + short: "Push specific beads to GitLab" + hidden: false + runnable: true + long: + | Push one or more beads issues to GitLab. + | + | Accepts bead IDs as positional arguments. + | Equivalent to: bd gitlab sync --push-only --issues + flag --dry-run type=bool default="false" usage="Preview push without making changes" + +bd gitlab status + use: "status" + short: "Show GitLab sync status" + hidden: false + runnable: true + long: + | Display current GitLab configuration and sync status. + +bd gitlab sync + use: "sync" + short: "Sync issues with GitLab" + hidden: false + runnable: true + long: + | Synchronize issues between beads and GitLab. + | + | By default, performs bidirectional sync: + | - Pulls new/updated issues from GitLab to beads + | - Pushes local beads issues to GitLab + | + | Use --pull-only or --push-only to limit direction. + flag --assignee type=string default="" usage="Filter by assignee username" + flag --dry-run type=bool default="false" usage="Show what would be synced without making changes" + flag --exclude-type type=string default="" usage="Exclude these issue types from sync (comma-separated)" + flag --issues type=string default="" usage="Comma-separated bead IDs to sync selectively (e.g., bd-abc,bd-def). Mutually exclusive with --parent." + flag --label type=string default="" usage="Filter by labels (comma-separated, AND logic)" + flag --milestone type=string default="" usage="Filter by milestone title" + flag --no-ephemeral type=bool default="true" usage="Exclude ephemeral/wisp issues from push (default: true)" + flag --parent type=string default="" usage="Limit push to this bead and its descendants (push only). Mutually exclusive with --issues." + flag --prefer-gitlab type=bool default="false" usage="On conflict, use GitLab version" + flag --prefer-local type=bool default="false" usage="On conflict, keep local beads version" + flag --prefer-newer type=bool default="false" usage="On conflict, use most recent version (default)" + flag --project type=string default="" usage="Filter to issues from this project ID (group mode)" + flag --pull-only type=bool default="false" usage="Only pull issues from GitLab" + flag --push-only type=bool default="false" usage="Only push issues to GitLab" + flag --type type=string default="" usage="Only sync these issue types (comma-separated, e.g. 'epic,feature,task')" + +bd graph + use: "graph [issue-id]" + short: "Display issue dependency graph" + group: "deps" + hidden: false + runnable: true + long: + | Display a visualization of an issue's dependency graph. + | + | For epics, shows all children and their dependencies. + | For regular issues, shows the issue and its direct dependencies. + | + | With --all, shows all open issues grouped by connected component. + | With --open, filters to only open/actionable issues (compact layer format). + | + | Display formats: + | (default) DAG with columns and box-drawing edges (terminal-native) + | --box ASCII boxes showing layers, more detailed + | --compact Tree format, one line per issue, more scannable + | --dot Graphviz DOT format (pipe to dot -Tsvg > graph.svg) + | --html Self-contained interactive HTML with D3.js visualization + | --open Open issues only, compact layers (LLM-friendly) + | + | The graph shows execution order: + | - Layer 0 / leftmost = no dependencies (can start immediately) + | - Higher layers depend on lower layers + | - Nodes in the same layer can run in parallel + | + | Status icons: ○ open ◐ in_progress ● blocked ✓ closed ❄ deferred + | + | Examples: + | bd graph issue-id # Terminal DAG visualization (default) + | bd graph --box issue-id # ASCII boxes with layer grouping + | bd graph --dot issue-id | dot -Tsvg > graph.svg # SVG via Graphviz + | bd graph --dot issue-id | dot -Tpng > graph.png # PNG via Graphviz + | bd graph --html issue-id > graph.html # Interactive browser view + | bd graph --all --html > all.html # All issues, interactive + | bd graph --open issue-id # Open issues only, layered by blocking order + | bd graph --all --open # All open issues, compact layers + | + | --max-rows / BEADS_MAX_ROWS caveat: the cap is checked differently per mode. + | Single-issue graphs (no --all) check the connected-component node count + | after the BFS traversal completes — the whole subgraph is always walked + | first, then rejected if it's over cap. --all checks each status + | (open/in_progress/blocked) independently, so up to 3x the cap can be loaded + | in total before any individual status trips it. + flag --all type=bool default="false" usage="Show graph for all open issues" + flag --box type=bool default="false" usage="ASCII boxes showing layers" + flag --compact type=bool default="false" usage="Tree format, one line per issue, more scannable" + flag --dot type=bool default="false" usage="Output Graphviz DOT format (pipe to: dot -Tsvg > graph.svg)" + flag --html type=bool default="false" usage="Output self-contained interactive HTML (redirect to file)" + flag --max-rows type=int default="0" usage="Hard upper bound on rows fetched from storage. Returns a non-zero exit (code 2) and an error to stderr if exceeded. 0 disables (the default). Overrides BEADS_MAX_ROWS for this invocation. Useful in CI/agent rigs that want a circuit breaker against pathological queries. Not supported under --proxied-server: an explicit --max-rows or BEADS_MAX_ROWS cap errors out rather than silently going unenforced." + flag --open type=bool default="false" usage="Show only open issues (filters out closed/deferred), forces compact layer format" + +bd graph check + use: "check" + short: "Check dependency graph integrity" + hidden: false + runnable: true + long: + | Check the dependency graph for cycles, orphans, and other integrity issues. + | + | Returns exit code 0 if the graph is clean, 1 if issues are found. + +bd heartbeat + use: "heartbeat " + aliases: ["hb"] + short: "Refresh the lease on an issue you hold in_progress" + group: "issues" + hidden: false + runnable: true + long: + | Refresh the lease on an issue you currently hold in_progress. + | + | A claim carries a lease that expires after a TTL. A worker keeps its claim alive + | by heartbeating faster than the TTL; once it stops (because it died), the lease + | goes stale and 'bd reclaim' reverts the issue to ready so another worker can pick + | it up. Heartbeat pushes lease_expires_at forward and stamps heartbeat_at = now. + | + | Only the current owner may heartbeat. If the lease has already been reclaimed or + | the issue closed, heartbeat fails so the worker learns to stop. + | + | Leases live in an ephemeral, node-local table: a heartbeat that refreshes your + | lease writes no Dolt commit and no history, so any cadence comfortably below + | the TTL is fine. Only a re-arm (the first heartbeat of a claim that has no + | lease, such as one set by a plain update) is recorded as a change to the issue + | and committed, once. Leases are only enforceable on the node that granted them; + | cross-machine claim visibility rides the issue's status and assignee, which do + | commit. + | + | Examples: + | bd heartbeat bd-123 + | bd hb bd-123 + +bd help + use: "help [command]" + short: "Help about any command" + hidden: false + runnable: true + long: + | Help provides help for any command in the application. + | Simply type bd help [path to command] for full details. + flag --all type=bool default="false" usage="Show help for all commands in a single document" + flag --doc type=string default="" usage="Generate markdown docs for a single command" + flag --docs-root type=string default="" usage="Generate repository CLI docs under this root" + flag --list type=bool default="false" usage="List all available commands" + +bd history + use: "history " + short: "Show version history for an issue" + group: "views" + hidden: false + runnable: true + long: + | Show the complete version history of an issue, including all commits + | where the issue was modified. + | + | Examples: + | bd history bd-123 # Show all history for issue bd-123 + | bd history bd-123 --limit 5 # Show last 5 changes + | bd history bd-123 --events # Show database audit events + flag --events type=bool default="false" usage="Show database audit events instead of commit snapshots" + flag --limit type=int default="0" usage="Limit number of history entries (0 = all)" + +bd hooks + use: "hooks" + short: "Manage git hooks for beads integration" + group: "setup" + hidden: false + runnable: false + long: + | Install, uninstall, or list git hooks for beads integration. + | + | The hooks provide: + | - pre-commit: Run chained hooks before commit + | - post-merge: Run chained hooks after pull/merge + | - pre-push: Run chained hooks before push + | - post-checkout: Run chained hooks after branch checkout + | - prepare-commit-msg: Add agent identity trailers for forensics + +bd hooks install + use: "install" + short: "Install bd git hooks" + hidden: false + runnable: true + long: + | Install git hooks for beads integration. + | + | By default, hooks are installed to .git/hooks/ in the current repository. + | Use --beads to install to .beads/hooks/ (recommended for Dolt backend). + | Use --shared to install to a versioned directory (.beads-hooks/) that can be + | committed to git and shared with team members. + | + | Hooks use section markers to coexist with existing hooks — any user content + | outside the markers is preserved across installs and upgrades. + | + | Installed hooks: + | - pre-commit: Run chained hooks before commit + | - post-merge: Run chained hooks after pull/merge + | - pre-push: Run chained hooks before push + | - post-checkout: Run chained hooks after branch checkout + | - prepare-commit-msg: Add agent identity trailers (for orchestrator agents) + flag --beads type=bool default="false" usage="Install hooks to .beads/hooks/ (recommended for Dolt backend)" + flag --chain type=bool default="false" usage="No-op, kept for compatibility (existing hook content always runs alongside the bd section)" + flag --force type=bool default="false" usage="No-op, kept for compatibility (section markers always preserve non-bd content)" + flag --shared type=bool default="false" usage="Install hooks to .beads-hooks/ (versioned) instead of .git/hooks/" + +bd hooks list + use: "list" + short: "List installed git hooks status" + hidden: false + runnable: true + long: + | Show the status of bd git hooks (installed, outdated, missing). + +bd hooks run + use: "run [args...]" + short: "Execute a git hook (called by thin shims)" + hidden: false + runnable: true + long: + | Execute the logic for a git hook. This command is typically called by + | thin shim scripts installed in .git/hooks/. + | + | Supported hooks: + | - pre-commit: Run chained hooks before commit + | - post-merge: Run chained hooks after pull/merge + | - pre-push: Run chained hooks before push + | - post-checkout: Run chained hooks after branch checkout + | - prepare-commit-msg: Add agent identity trailers for forensics + | + | The thin shim keeps delegated hook logic in sync with the installed bd + | version. Upgrading bd updates that delegated behavior. To adopt changes to the + | shim's generated shell policy, refresh it with 'bd hooks install'. + +bd hooks uninstall + use: "uninstall" + short: "Uninstall bd git hooks" + hidden: false + runnable: true + long: + | Remove bd git hooks from .git/hooks/ directory. + +bd human + use: "human" + short: "Show essential commands for human users" + group: "setup" + hidden: false + runnable: true + long: + | Display a focused help menu showing only the most common commands. + | + | bd has 70+ commands - many for AI agents, integrations, and advanced workflows. + | This command shows the ~15 essential commands that human users need most often. + | + | For the full command list, run: bd --help + | + | SUBCOMMANDS: + | human list List human-needed beads (issues with 'human' label; hides closed by default) + | human respond Respond to a human-needed bead (adds comment and closes) + | human dismiss Dismiss a human-needed bead permanently + | human stats Show summary statistics for human-needed beads + +bd human dismiss + use: "dismiss [reason...]" + short: "Dismiss a human-needed bead" + hidden: false + runnable: true + long: + | Dismiss a human-needed bead permanently without responding. + | + | The issue is closed with a "Dismissed" reason and optional note. + | The reason can be given as positional arguments or --reason. + | + | Examples: + | bd human dismiss bd-123 + | bd human dismiss bd-123 "No longer applicable" + | bd human dismiss bd-123 --reason "No longer applicable" + flag --reason type=string default="" usage="Reason for dismissal (optional)" + +bd human list + use: "list" + short: "List human-needed beads" + hidden: false + runnable: true + long: + | List issues labeled with 'human' tag. + | + | These are issues that require human intervention or input. Every + | human-labeled bead shows regardless of type (including gates and wisps). + | By default closed, pinned, and other done/frozen beads are hidden; use + | --status to select specific statuses, or --status=all to include every + | status. + | + | Examples: + | bd human list + | bd human list --status=closed + | bd human list --status=all + | bd human list --json + flag --status -s type=string default="" usage="Filter by status (open, closed, etc.; comma-separated for multiple, 'all' for every status)" + +bd human respond + use: "respond [response...]" + short: "Respond to a human-needed bead" + hidden: false + runnable: true + long: + | Respond to a human-needed bead by adding a comment and closing it. + | + | The response is added as a comment and the issue is closed with reason "Responded". + | The response text can be given as positional arguments, --response, --file, or --stdin. + | + | Examples: + | bd human respond bd-123 "Use OAuth2 for authentication" + | bd human respond bd-123 -r "Approved, proceed with implementation" + | bd human respond bd-123 --file response.md + | echo "Approved" | bd human respond bd-123 --stdin + flag --file type=string default="" annotations={cobra_annotation_mutually_exclusive=["stdin file response"]} usage="Read response text from file" + flag --response -r type=string default="" annotations={cobra_annotation_mutually_exclusive=["stdin file response"]} usage="Response text" + flag --stdin type=bool default="false" annotations={cobra_annotation_mutually_exclusive=["stdin file response"]} usage="Read response text from stdin" + +bd human stats + use: "stats" + short: "Show summary statistics for human-needed beads" + hidden: false + runnable: true + long: + | Display summary statistics for human-needed beads. + | + | Shows counts for total, pending (open), responded (closed without dismiss), + | and dismissed beads. + | + | Example: + | bd human stats + +bd import + use: "import [file|-]" + short: "Import issues from a JSONL file or stdin into the database" + group: "sync" + hidden: false + runnable: true + long: + | Import issues from a JSONL file (newline-delimited JSON) into the database. + | + | If no file is specified, imports from the configured import.path under .beads/ + | (default: issues.jsonl). Use "-" to read from stdin; redirecting stdin without + | "-" or a file argument is an error, so a typo'd 'bd import < file' cannot + | silently import the default file instead. This is the incremental counterpart to + | 'bd export': new issues are created and existing issues are updated (upsert + | semantics). + | + | Memory records (lines with "_type":"memory") are automatically detected and + | imported as persistent memories (equivalent to 'bd remember'). This makes + | 'bd export | bd import' a full round-trip for both issues and memories. + | + | Each JSONL line should map to an issue. The importer accepts every field + | 'bd export' emits — see 'bd export' output for the canonical schema. Only + | "title" is required; everything else is optional. + | + | Common fields: + | title Required. Short summary. + | description Long-form body. + | design, notes, Additional content sections. + | acceptance_criteria + | issue_type bug | feature | task | epic | chore | ... + | priority 0-4 (0 = critical). 0 is preserved (no omitempty). + | status open | in_progress | blocked | closed | ... + | (rows with status "tombstone" are skipped) + | assignee, owner, Ownership metadata. + | created_by + | labels Array of strings. + | dependencies Array of {issue_id, depends_on_id, type, ...}. + | comments Array of comment objects. + | external_ref, Cross-system identifiers (e.g. "gh-9"). + | source_system + | due_at, defer_until RFC3339 timestamps for scheduling. + | metadata Arbitrary JSON object preserved verbatim. + | + | Timestamps (created_at, updated_at, started_at, closed_at) are preserved + | when present in the JSONL and otherwise filled in by the importer. The + | legacy "wisp" boolean is accepted as an alias for "ephemeral". + | + | By default a row only rewrites an existing local issue when its + | updated_at is strictly newer. Older rows are skipped (reported as + | stale_skipped_ids) and rows with the same updated_at keep every local + | column — updated_at has second granularity, so a timestamp tie can be + | two distinct same-second updates, and the local row wins the tie + | (reported as tie_kept_local_ids; the row's labels/comments/dependencies + | still merge). The guard is also enforced inside the upsert itself, so a + | local update that lands while the import is running is preserved rather + | than overwritten. Existing issues that the import did rewrite are listed + | with a field-level summary (updated_issues), so local state changed by + | an import is visible. To deliberately restore an older snapshot, pass + | --allow-stale, which imports every row even when it overwrites newer + | local state. + | + | The staleness guard is import's ONLY overwrite guard: the per-field + | fences that bd update enforces (notes overwrite, live-claim reassign, + | close policy) do not apply to imported rows. Import replaces rows + | wholesale by design — a restore must not fail field-by-field — so + | --allow-stale is the single deliberate-overwrite opt-in, and + | updated_issues is where to look for what an import rewrote. + | + | Large imports are written in bounded transactions (a few hundred issues + | each, with a short pause between commits) with progress on stderr, so + | concurrent bd commands keep working while the import runs instead of + | stalling on one batch-wide write lock. Rows land in dependency order + | with their blocking edges in the same transaction, so a half-finished + | import never shows a blocked issue as ready. If an import fails partway, + | the already-committed chunks are durable and the command exits nonzero; + | re-running the same import is safe and converges (rows upsert, + | labels/comments/dependencies deduplicate). + | + | EXAMPLES: + | bd import # Import from configured import.path + | bd import backup.jsonl # Import from a specific file + | bd import -i backup.jsonl # Legacy alias for a specific file + | bd import - # Read JSONL from stdin + | cat issues.jsonl | bd import - # Pipe JSONL from another tool + | bd import --dry-run # Show what would be imported + | bd import --dedup # Skip issues with duplicate titles + | bd import --allow-stale old.jsonl # Restore an older snapshot (overwrites newer local rows) + | bd import --json # Structured output with created and skipped IDs + flag --allow-stale type=bool default="false" usage="Import rows even when older than the local issue (required to restore an older snapshot)" + flag --dedup type=bool default="false" usage="Skip lines whose title matches an existing open issue" + flag --dry-run type=bool default="false" usage="Show what would be imported without importing" + flag --input -i type=string default="" usage="Read JSONL from a specific file" + +bd info + use: "info" + short: "Show database information" + group: "setup" + hidden: false + runnable: true + long: + | Display information about the current database. + | + | This command helps debug issues where bd is using an unexpected database. It shows: + | - The absolute path to the database file + | - Database statistics (issue count) + | - Schema information (with --schema flag) + | - What's new in recent versions (with --whats-new flag) + | + | Examples: + | bd info + | bd info --json + | bd info --schema --json + | bd info --whats-new + | bd info --whats-new --json + | bd info --thanks + flag --schema type=bool default="false" usage="Include schema information in output" + flag --thanks type=bool default="false" usage="Show thank you page for contributors" + flag --whats-new type=bool default="false" usage="Show agent-relevant changes from recent versions" + +bd init + use: "init" + short: "Initialize bd in the current directory" + group: "setup" + hidden: false + runnable: true + long: + | Initialize bd in the current directory by creating a .beads/ directory + | and its storage (a Dolt database by default). Optionally specify a custom issue prefix. + | + | Dolt is the default and only supported storage backend, with full version + | control (history, branching, sync). + | + | Use --database to specify an existing server database name, overriding the + | default prefix-based naming. This is useful when an external tool (e.g. an orchestrator) + | has already created the database. + | + | With --stealth: configures per-repository git settings for invisible beads usage: + | • .git/info/exclude to prevent beads files from being committed + | Perfect for personal use without affecting repo collaborators. + | To set up a specific AI tool, run: bd setup --stealth + | + | By default, beads uses an embedded Dolt engine (no external server needed). + | Pass --server to use an external dolt sql-server instead. In server mode, + | set connection details with --server-host, --server-port, and --server-user. + | Password should be set via BEADS_DOLT_PASSWORD environment variable. + | + | Auto-export is optional. When enabled, bd exports issues to + | .beads/issues.jsonl after write commands (throttled to once per 60s). This is + | for viewers (bv), interchange, and issue-level migration; not backup. + | Cross-machine sync and backups use Dolt remotes/backups, not JSONL import/export. + | To enable: bd config set export.auto true + | + | Non-interactive mode (--non-interactive or BD_NON_INTERACTIVE=1): + | Skips all interactive prompts, using sensible defaults: + | • Role defaults to "maintainer" (override with --role) + | • Fork exclude auto-configured when fork detected + | • Auto-export left at default (disabled) + | • --contributor and --team flags are rejected (wizards require interaction) + | Also auto-detected when stdin is not a terminal or CI=true is set. + flag --agents-file type=string default="" usage="Custom filename for agent instructions (default: AGENTS.md)" + flag --agents-profile type=string default="" usage="AGENTS.md profile: 'minimal' (default, pointer to bd prime) or 'full' (complete command reference)" + flag --agents-template type=string default="" usage="Path to custom AGENTS.md template (overrides embedded default)" + flag --backend type=string default="" usage="Storage backend: dolt (default). Removed backends (postgres, mysql, sqlite) print migration guidance." + flag --contributor type=bool default="false" usage="Run OSS contributor setup wizard" + flag --debug type=bool default="false" usage="Run the managed Dolt sql-server with --loglevel=debug and CPU profiling (--prof cpu). Persisted to config.yaml as dolt.debug. No effect on externally-managed servers." + flag --destroy-token type=string default="" usage="Explicit confirmation token for destructive re-init in non-interactive mode (format: 'DESTROY-')" + flag --discard-remote type=bool default="false" usage="Authorize discarding the configured remote's Dolt history when re-initializing. Requires --destroy-token in non-interactive mode; see 'bd help init-safety'." + flag --external type=bool default="false" usage="Server is externally managed (skip server startup); use with --shared-server or --server" + flag --force type=bool default="false" usage="Deprecated alias for --reinit-local. Bypasses only the LOCAL data-safety guard; does NOT authorize remote divergence or recreating a missing server-mode database (see 'bd help init-safety')." + flag --from-jsonl type=bool default="false" usage="Import issues from configured import.path; refuses remote history unless --discard-remote authorizes replacement" + flag --init-if-missing type=bool default="false" usage="If the workspace is already initialized, skip init and exit 0 instead of failing (idempotent init for scaffolds)" + flag --mysql-database type=string default="" hidden usage="Legacy MySQL database (removed backend compatibility only)" + flag --mysql-url type=string default="" hidden usage="Legacy MySQL connection URL (removed backend compatibility only)" + flag --non-interactive type=bool default="false" usage="Skip all interactive prompts (auto-detected in CI or non-TTY environments)" + flag --pg-schema type=string default="" hidden usage="Legacy PostgreSQL schema (removed backend compatibility only)" + flag --pg-url type=string default="" hidden usage="Legacy PostgreSQL connection URL (removed backend compatibility only)" + flag --prefix -p type=string default="" usage="Issue prefix (default: current directory name)" + flag --proxied-server type=bool default="false" usage="[EXPERIMENTAL] Use a per-workspace proxied dolt sql-server (proxy + child dolt) rooted at .beads/dolt" + flag --proxied-server-config-path type=string default="" usage="[EXPERIMENTAL] Absolute path to an existing dolt sql-server YAML config (proxied-server mode only). When set, bd uses this file instead of auto-generating one. Relative paths are rejected. Managed mode requires listener.host to be a numeric loopback IP (hostnames including localhost, non-loopback addresses, listener.socket, remotesapi, and cluster config are rejected); the same policy applies to BEADS_PROXIED_SERVER_CONFIG." + flag --proxied-server-external-host type=string default="" usage="[EXPERIMENTAL] Hostname or IP of an externally-managed dolt sql-server the proxy should front (proxied-server mode only). Mutually exclusive with --proxied-server-external-socket-path." + flag --proxied-server-external-keep-alive type=duration default="0s" usage="[EXPERIMENTAL] TCP keepalive period for the proxy→external connection. Zero uses the package default (30s)." + flag --proxied-server-external-port type=int default="0" usage="[EXPERIMENTAL] TCP port of the externally-managed dolt sql-server (proxied-server mode only). Required when --proxied-server-external-host is set." + flag --proxied-server-external-socket-path type=string default="" usage="[EXPERIMENTAL] Absolute unix socket path of the externally-managed dolt sql-server (proxied-server mode only). Mutually exclusive with --proxied-server-external-host. Relative paths are rejected." + flag --proxied-server-external-tls type=bool default="false" usage="[EXPERIMENTAL] Require TLS when connecting to the externally-managed dolt sql-server (proxied-server mode only)." + flag --proxied-server-external-tls-ca-cert-path type=string default="" usage="[EXPERIMENTAL] Absolute path to a CA certificate (PEM) used to verify the externally-managed dolt sql-server. Empty uses the system trust store. Relative paths are rejected." + flag --proxied-server-external-tls-cert-path type=string default="" usage="[EXPERIMENTAL] Absolute path to a client TLS certificate (for mTLS to the externally-managed dolt sql-server). Must be paired with --proxied-server-external-tls-key-path. Relative paths are rejected." + flag --proxied-server-external-tls-key-path type=string default="" usage="[EXPERIMENTAL] Absolute path to the client TLS private key (for mTLS to the externally-managed dolt sql-server). Must be paired with --proxied-server-external-tls-cert-path. Relative paths are rejected." + flag --proxied-server-external-tls-server-name type=string default="" usage="[EXPERIMENTAL] Server name to verify in the external dolt sql-server's TLS certificate. Defaults to the external host. Required with a unix socket unless --proxied-server-external-tls-skip-verify is set." + flag --proxied-server-external-tls-skip-verify type=bool default="false" usage="[EXPERIMENTAL] Skip TLS certificate verification for the external dolt sql-server. Insecure; testing only." + flag --proxied-server-external-user type=string default="" usage="[EXPERIMENTAL] MySQL user for the externally-managed dolt sql-server (proxied-server mode only). Defaults to \"root\" when empty. Password is read at runtime from $BEADS_PROXIED_SERVER_EXTERNAL_PASSWORD and is never persisted to disk." + flag --proxied-server-idle-timeout type=duration default="0s" usage="[EXPERIMENTAL] Idle duration after which the proxy shuts down its loopback listener and backend (proxied-server mode only). Omit for the built-in default (30s); 0 keeps the proxy and backend alive indefinitely; a positive value sets the window." + flag --proxied-server-log-path type=string default="" usage="[EXPERIMENTAL] Absolute path to the proxied dolt sql-server log file (proxied-server mode only). Default: /dolt/server.log. Relative paths are rejected." + flag --proxied-server-port type=int default="0" usage="[EXPERIMENTAL] Fixed TCP port for the proxy's loopback listener (proxied-server mode only). Default 0 = an OS-assigned free port. Startup fails if the port is already in use." + flag --proxied-server-root-path type=string default="" usage="[EXPERIMENTAL] Absolute directory holding the proxied dolt sql-server's lockfiles, pidfiles, and child .dolt repository (proxied-server mode only). Default: /dolt. May not exist yet — bd will create it. Relative paths are rejected." + flag --quiet -q type=bool default="false" usage="Suppress output (quiet mode)" + flag --recreate-missing type=bool default="false" usage="Explicitly authorize creating a fresh, empty database when this project's configured server-mode database is missing or unreachable. Opt-in per invocation only; never implied by --force, config, or env (see 'bd help init-safety')." + flag --reinit-local type=bool default="false" usage="Re-initialize local .beads/ over existing local data. Does NOT authorize remote divergence (see --discard-remote) or recreating a missing server-mode database (see --recreate-missing)." + flag --remote type=string default="" usage="Dolt remote URL to clone from and persist as sync.remote" + flag --role type=string default="" usage="Set beads role without prompting: \"maintainer\" or \"contributor\"" + flag --server type=bool default="false" usage="Use external dolt sql-server instead of embedded engine" + flag --server-host type=string default="" usage="Dolt server host (default: 127.0.0.1)" + flag --server-port type=int default="0" usage="Dolt server port (default: 3307)" + flag --server-socket type=string default="" usage="Unix domain socket path (overrides host/port; pass '' to ignore an ambient BEADS_DOLT_SERVER_SOCKET and use TCP)" + flag --server-tls type=bool default="false" usage="Require TLS for the init-time Dolt server connection (overrides BEADS_DOLT_SERVER_TLS for this run; not persisted - set the env var or credentials file for later commands)" + flag --server-user type=string default="" usage="Dolt server MySQL user (default: root)" + flag --setup-exclude type=bool default="false" usage="Configure .git/info/exclude to keep beads files local (for forks)" + flag --shared-server type=bool default="false" usage="Enable shared Dolt server mode (all projects share one server at ~/.beads/shared-server/)" + flag --skip-agents type=bool default="false" usage="Skip AGENTS.md and Claude/Codex/Cursor setup generation" + flag --skip-hooks type=bool default="false" usage="Skip git hooks installation" + flag --sqlite-path type=string default="" hidden usage="Legacy SQLite database file (removed backend compatibility only)" + flag --stealth type=bool default="false" usage="Enable stealth mode: keep beads files out of git via .git/info/exclude (sets no-git-ops)" + flag --team type=bool default="false" usage="Run team workflow setup wizard" + flag --team-server type=bool default="false" usage="[EXPERIMENTAL] The shared database's schema is managed by beads-team-server (bts): bd never creates the database or runs schema migrations, only verifies the schema version (proxied-server mode only). Not related to --team." + +bd init-safety + use: "init-safety" + short: "Explain bd init flag semantics and the destroy-token format" + hidden: false + runnable: true + long: + | bd init flag safety contract. + | + | Every bd init invocation resolves project_id from exactly one explicitly + | named source (local reinit, remote adoption, or a fresh mint). When the + | source is ambiguous, bd init refuses. + | + | FLAG SURFACE + | + | bd init Mint a new identity. Bootstraps from + | origin if it has refs/dolt/data. + | + | bd init --reinit-local Re-initialize local .beads/ over existing + | local data. Does NOT authorize discarding + | remote history. If origin has Dolt data + | this will refuse — pair with + | --discard-remote to override. Nor does it + | authorize recreating a missing server-mode + | database; that takes --recreate-missing. + | + | bd init --reinit-local \ Discard the remote's Dolt history and + | --discard-remote replace it with the local reinit. First + | bd dolt push after this will be a + | history-replacing force-push. + | + | bd init --force Deprecated alias for --reinit-local, with + | the same limits. Kept working for ≥2 + | releases. + | + | bd init --recreate-missing Authorize creating a fresh, empty database + | when this project's configured server-mode + | database is missing or unreachable. Opt-in + | per invocation only: never implied by + | --force/--reinit-local, config, or env. + | Without it, bd init REFUSES rather than + | silently recreating a lost database as + | empty (be-5up5). First rule out a stopped + | server or a wrong port or data dir, and + | restore if you can; the playbook is + | init-missing-server-db in + | docs/recovery/init-safety.md. Only when + | nothing is recoverable: + | + | bd init --recreate-missing --prefix + | + | bd init --from-jsonl Import from configured import.path. If + | origin has Dolt data, this refuses unless + | --discard-remote authorizes replacing that + | remote history. + | + | ADOPTING A REMOTE + | + | If you want to use the remote's existing history, use: + | + | bd bootstrap + | + | bd init will automatically suggest this when a remote is detected. + | + | DESTROY-TOKEN (non-interactive only) + | + | When running with no TTY (CI, agents, piped input), a destructive + | re-init requires an explicit --destroy-token value. That covers both + | --discard-remote and --reinit-local over existing issues. The token + | format is: + | + | DESTROY- + | + | For example, if your issue prefix is "bd", the token is "DESTROY-bd": + | + | bd init --reinit-local --discard-remote --destroy-token=DESTROY-bd + | + | In interactive (TTY) mode you confirm via a typed prompt instead. The + | token is not echoed by bd's runtime error messages — this is a + | deliberate guard against pattern-matched one-liners (see + | engdocs/adr/0002-init-safety-invariants.md). + | + | EXIT CODES + | + | 10 refused: remote has Dolt history and you selected local history + | without --discard-remote + | 11 refused: existing local data and you declined the destroy confirm + | (interactive mode only) + | 12 refused: destructive re-init (--discard-remote, or --reinit-local + | over existing issues) without a valid --destroy-token + | (non-interactive mode); also returned when the interactive + | --discard-remote typed-token confirmation is declined + | 15 refused: run directly in your home directory, which is not itself + | a git repository, with no BEADS_DIR named by the caller + | + | RECOVERY + | + | If you hit a refusal, see docs/recovery/init-safety.md for step-by-step recovery + | playbooks for each exit code. + | + | RE-CLONE GOTCHAS + | + | Setting a damaged or superseded database directory aside by hand, or + | relying on a fresh clone right away? Two gotchas from live recovery: + | + | Crash-loop: a set-aside store left INSIDE data_dir makes the sql-server + | treat it as a database and crash-loop with "root hash doesn't exist: + | ". Move it OUTSIDE data_dir instead. + | + | Missing tables: a fresh clone lacks clone-local tables (leases, wisps, + | events, ...) until you run "bd migrate schema" (no --force). You'll see + | "table not found: leases" until then; "Schema already at v" after + | running it is expected, not an error. + | + | See docs/recovery/init-safety.md#re-clone-gotchas for full detail. + | + +bd jira + use: "jira" + short: "Jira integration commands" + group: "advanced" + hidden: false + runnable: false + long: + | Synchronize issues between beads and Jira. + | + | Configuration: + | bd config set jira.url "https://company.atlassian.net" + | bd config set jira.project "PROJ" + | bd config set jira.projects "PROJ1,PROJ2" # Multiple projects + | bd config set jira.api_token "YOUR_TOKEN" + | bd config set jira.username "your_email@company.com" # For Jira Cloud + | bd config set jira.push_prefix "hippo" # Only push hippo-* issues to Jira + | bd config set jira.push_prefix "proj1,proj2" # Multiple prefixes (comma-separated) + | + | Environment variables (alternative to config): + | JIRA_API_TOKEN - Jira API token + | JIRA_USERNAME - Jira username/email + | JIRA_PROJECTS - Comma-separated project keys + | + | Examples: + | bd jira sync --pull # Import issues from Jira + | bd jira sync --push # Export issues to Jira + | bd jira sync # Bidirectional sync (pull then push) + | bd jira sync --dry-run # Preview sync without changes + | bd jira status # Show sync status + +bd jira pull + use: "pull [refs...]" + short: "Pull specific items from Jira" + hidden: false + runnable: true + long: + | Pull one or more items from Jira. + | + | Accepts bead IDs or external references as positional arguments. + | Equivalent to: bd jira sync --pull --issues + flag --dry-run type=bool default="false" usage="Preview pull without making changes" + +bd jira push + use: "push [bead-ids...]" + short: "Push specific beads to Jira" + hidden: false + runnable: true + long: + | Push one or more beads issues to Jira. + | + | Accepts bead IDs as positional arguments. + | Equivalent to: bd jira sync --push --issues + flag --dry-run type=bool default="false" usage="Preview push without making changes" + +bd jira status + use: "status" + short: "Show Jira sync status" + hidden: false + runnable: true + long: + | Show the current Jira sync status, including: + | - Last sync timestamp + | - Configuration status + | - Number of issues with Jira links + | - Issues pending push (no external_ref) + +bd jira sync + use: "sync" + short: "Synchronize issues with Jira" + hidden: false + runnable: true + long: + | Synchronize issues between beads and Jira. + | + | Modes: + | --pull Import issues from Jira into beads + | --push Export issues from beads to Jira + | (no flags) Bidirectional sync: pull then push, with conflict resolution + | + | Conflict Resolution: + | By default, newer timestamp wins. Override with: + | --prefer-local Always prefer local beads version + | --prefer-jira Always prefer Jira version + | + | Examples: + | bd jira sync --pull # Import from Jira + | bd jira sync --push --create-only # Push new issues only + | bd jira sync --dry-run # Preview without changes + | bd jira sync --prefer-local # Bidirectional, local wins + flag --create-only type=bool default="false" usage="Only create new issues, don't update existing" + flag --dry-run type=bool default="false" usage="Preview sync without making changes" + flag --issues type=string default="" usage="Comma-separated bead IDs to sync selectively (e.g., bd-abc,bd-def). Mutually exclusive with --parent." + flag --parent type=string default="" usage="Limit push to this bead and its descendants (push only). Mutually exclusive with --issues." + flag --prefer-jira type=bool default="false" usage="Prefer Jira version on conflicts" + flag --prefer-local type=bool default="false" usage="Prefer local version on conflicts" + flag --project type=stringSlice default="[]" usage="Project key(s) to sync (overrides configured project/projects)" + flag --pull type=bool default="false" usage="Pull issues from Jira" + flag --push type=bool default="false" usage="Push issues to Jira" + flag --state type=string default="all" usage="Issue state to sync: open, closed, all" + +bd kv + use: "kv" + short: "Key-value store commands" + group: "setup" + hidden: false + runnable: false + long: + | Commands for working with the beads key-value store. + | + | The key-value store is useful for storing flags, environment variables, + | or other user-defined data that persists across sessions. + | + | Examples: + | bd kv set mykey myvalue # Set a value + | bd kv get mykey # Get a value + | bd kv clear mykey # Delete a key + | bd kv list # List all key-value pairs + +bd kv clear + use: "clear " + short: "Delete a key-value pair" + hidden: false + runnable: true + long: + | Delete a key from the beads key-value store. + | + | Examples: + | bd kv clear feature_flag + | bd kv clear api_endpoint + +bd kv get + use: "get " + short: "Get a value by key" + hidden: false + runnable: true + long: + | Get a value from the beads key-value store. + | + | Examples: + | bd kv get feature_flag + | bd kv get api_endpoint + +bd kv list + use: "list" + short: "List all key-value pairs" + hidden: false + runnable: true + long: + | List all key-value pairs in the beads key-value store. + | + | With --prefix, list only the keys starting with that prefix. The filter is + | pushed into SQL on stores that support it, so a scoped read of a large kv + | store does not serialize the whole table. + | + | Examples: + | bd kv list + | bd kv list --json + | bd kv list --prefix mail.dog. --json + flag --prefix type=string default="" usage="list only keys starting with this prefix (filtered in SQL where supported)" + +bd kv set + use: "set " + short: "Set a key-value pair" + hidden: false + runnable: true + long: + | Set a key-value pair in the beads key-value store. + | + | This is useful for storing flags, environment variables, or other + | user-defined data that persists across sessions. + | + | Examples: + | bd kv set feature_flag true + | bd kv set api_endpoint https://api.example.com + | bd kv set max_retries 3 + +bd label + use: "label" + short: "Manage issue labels" + group: "issues" + hidden: false + runnable: false + +bd label add + use: "add [issue-id...] [label[,label...]]" + short: "Add one or more labels to one or more issues" + hidden: false + runnable: true + long: + | Add labels to issues. Issue IDs come first; the final argument is the label. Pass multiple labels comma-separated: bd label add bd-123 label1,label2 + +bd label list + use: "list [issue-id]" + short: "List labels for an issue" + hidden: false + runnable: true + +bd label list-all + use: "list-all" + short: "List all unique labels in the database" + hidden: false + runnable: true + +bd label propagate + use: "propagate [parent-id] [label]" + short: "Propagate a label from a parent issue to all its children" + hidden: false + runnable: true + long: + | Push a label from a parent down to all direct children that don't already have it. Useful for applying branch: labels across an epic's subtasks. + +bd label remove + use: "remove [issue-id...] [label[,label...]]" + short: "Remove one or more labels from one or more issues" + hidden: false + runnable: true + long: + | Remove labels from issues. Issue IDs come first; the final argument is the label. Pass multiple labels comma-separated: bd label remove bd-123 label1,label2 + | + | With --prefix, no label argument is needed: every label on the given issue(s) starting with the prefix is removed, e.g. bd label remove bd-123 --prefix pool:refused: + flag --prefix type=string default="" usage="Remove every label matching this prefix instead of an exact label (e.g., 'pool:refused:' removes all pool:refused:* labels). No label positional argument is needed when set. CAUTION: a short prefix removes every match on an issue in a single edit, including labels you may not have meant to touch (e.g. --prefix t removes both 'tier:opus' and 'test-needed') — check `bd label list ` first if unsure. State-dimension labels (e.g. 'launch:', 'scalegate:') should move via `bd set-state` instead, which this flag bypasses without its event-bead audit trail." + +bd label rename + use: "rename " + short: "Rename a label across every issue and wisp that carries it" + hidden: false + runnable: true + long: + | Rename a label everywhere it appears. An issue that already carries the new label keeps it and drops the old one instead of erroring - a merge, reported honestly (the write path measures merged from what the insert actually affected, never from a snapshot check, so its count is correct under concurrent writes). Pass --dry-run to preview the blast radius (a count and the first issues) without writing anything; the preview's merged count is a plain snapshot intersection of two separate reads, so treat it as an estimate, not the number the rename itself will report. + flag --dry-run type=bool default="false" usage="Preview the blast radius without renaming anything (merged count is a snapshot intersection, not authoritative - see --help)" + +bd linear + use: "linear" + short: "Linear integration commands" + group: "advanced" + hidden: false + runnable: false + long: + | Synchronize issues between beads and Linear. + | + | Configuration: + | bd config set linear.api_key "YOUR_API_KEY" + | bd config set linear.team_id "TEAM_ID" + | bd config set linear.team_ids "TEAM_ID1,TEAM_ID2" # Multiple teams (comma-separated) + | bd config set linear.project_id "PROJECT_ID" # Optional: sync only this project + | + | Environment variables (alternative to config): + | LINEAR_API_KEY - Linear API key (for individual developers) + | LINEAR_TEAM_ID - Linear team ID (UUID, singular) + | LINEAR_TEAM_IDS - Linear team IDs (comma-separated UUIDs) + | + | OAuth (for CI workers / automated sync): + | LINEAR_OAUTH_CLIENT_ID - OAuth app client ID + | LINEAR_OAUTH_CLIENT_SECRET - OAuth app client secret + | + | When both OAuth env vars are set, OAuth client_credentials flow is used + | instead of the API key. This allows CI workers to authenticate as an + | application (actor=application) rather than impersonating a user. + | Precedence: OAuth > LINEAR_API_KEY > config file. + | + | Data Mapping (optional, sensible defaults provided): + | Priority mapping (Linear 0-4 to Beads 0-4): + | bd config set linear.priority_map.0 4 # No priority -> Backlog + | bd config set linear.priority_map.1 0 # Urgent -> Critical + | bd config set linear.priority_map.2 1 # High -> High + | bd config set linear.priority_map.3 2 # Medium -> Medium + | bd config set linear.priority_map.4 3 # Low -> Low + | + | State mapping (Linear state type to Beads status): + | bd config set linear.state_map.backlog open + | bd config set linear.state_map.unstarted open + | bd config set linear.state_map.started in_progress + | bd config set linear.state_map.completed closed + | bd config set linear.state_map.canceled closed + | bd config set linear.state_map.my_custom_state in_progress # Custom state names + | + | Label to issue type mapping: + | bd config set linear.label_type_map.bug bug + | bd config set linear.label_type_map.feature feature + | bd config set linear.label_type_map.epic epic + | + | Relation type mapping (Linear relations to Beads dependencies): + | bd config set linear.relation_map.blocks blocks + | bd config set linear.relation_map.blockedBy blocks + | bd config set linear.relation_map.duplicate duplicates + | bd config set linear.relation_map.related related + | + | ID generation (optional, hash IDs to match bd/Jira hash mode): + | bd config set linear.id_mode "hash" # hash (default) + | bd config set linear.hash_length "6" # hash length 3-8 (default: 6) + | + | Examples: + | bd linear sync --pull # Import issues from Linear + | bd linear sync --push # Export issues to Linear + | bd linear sync # Bidirectional sync (pull then push) + | bd linear sync --dry-run # Preview sync without changes + | bd create "Fix login" --external-ref https://linear.app/team/issue/TEAM-123 + | # Link a local issue to an existing Linear issue + | bd linear status # Show sync status + +bd linear pull + use: "pull [refs...]" + short: "Pull specific items from Linear" + hidden: false + runnable: true + long: + | Pull one or more items from Linear. + | + | Accepts bead IDs or external references as positional arguments. + | Equivalent to: bd linear sync --pull --issues + flag --dry-run type=bool default="false" usage="Preview pull without making changes" + flag --relations type=bool default="false" usage="Import Linear relations as bd dependencies when pulling" + +bd linear push + use: "push [bead-ids...]" + short: "Push specific beads to Linear" + hidden: false + runnable: true + long: + | Push one or more beads issues to Linear. + | + | Accepts bead IDs as positional arguments. + | Equivalent to: bd linear sync --push --issues + flag --dry-run type=bool default="false" usage="Preview push without making changes" + +bd linear status + use: "status" + short: "Show Linear sync status" + hidden: false + runnable: true + long: + | Show the current Linear sync status, including: + | - Last sync timestamp + | - Configuration status + | - Number of issues with Linear links + | - Issues pending push (no external_ref) + +bd linear sync + use: "sync" + short: "Synchronize issues with Linear" + hidden: false + runnable: true + long: + | Synchronize issues between beads and Linear. + | + | Modes: + | --pull Import issues from Linear into beads + | --push Export issues from beads to Linear + | --pull-if-stale Pull only if data is stale (skip if fresh) + | (no flags) Bidirectional sync: pull then push, with conflict resolution + | + | Staleness (--pull-if-stale): + | --threshold 20m How old data must be before pulling (default 20m) + | A 5-minute debounce prevents agent loops: if a pull completed within 5 minutes, + | data is always treated as fresh regardless of the threshold. + | + | Team Selection: + | --team ID1,ID2 Override configured team IDs for this sync + | Multiple teams can be configured via linear.team_ids (comma-separated). + | Falls back to linear.team_id for backward compatibility. + | Push requires explicit --team when multiple teams are configured. + | + | Pull Options: + | --milestones Reconstruct Linear project milestones as local epic parents + | + | Type Filtering (--push only): + | --type task,feature Only sync issues of these types + | --exclude-type wisp Exclude issues of these types + | --include-ephemeral Include ephemeral issues (wisps, etc.); default is to exclude + | --parent TICKET Only push this ticket and its descendants + | --relations Import Linear relations as bd dependencies on pull + | + | Persistent push-direction ID filters (workflow artifacts, sandbox beads, etc.): + | bd config set linear.exclude_id_prefix "hw-mol-" + | bd config set linear.exclude_id_patterns "-wisp-,sandbox-,scratch-" + | + | exclude_id_prefix is a single case-sensitive prefix on the bead ID. + | exclude_id_patterns is a comma-separated list of case-sensitive substrings + | (matched anywhere in the ID). Both are combined as a union: a bead + | matching either rule is skipped from push (no create, no update). Beads + | with an existing external_ref that NOW match are silently skipped on + | future syncs; the Linear-side issue persists — archive/delete it manually + | if desired. + | + | Conflict Resolution: + | By default, newer timestamp wins. Override with: + | --prefer-local Always prefer local beads version + | --prefer-linear Always prefer Linear version + | + | Examples: + | bd linear sync --pull # Import from Linear + | bd linear sync --pull-if-stale # Pull only if data is stale + | bd linear sync --pull-if-stale --threshold 5m # Pull if older than 5 minutes + | bd linear sync --pull --relations # Import Linear blocking relations as bd deps + | bd linear sync --push --create-only # Push new issues only + | bd linear sync --push --type=task,feature # Push only tasks and features + | bd linear sync --push --exclude-type=wisp # Push all except wisps + | bd linear sync --push --parent=bd-abc123 # Push one ticket tree + | bd linear sync --dry-run # Preview without changes + | bd linear sync --prefer-local # Bidirectional, local wins + flag --create-only type=bool default="false" usage="Only create new issues, don't update existing" + flag --dry-run type=bool default="false" usage="Preview sync without making changes" + flag --exclude-type type=stringSlice default="[]" usage="Exclude issues of these types (can be repeated)" + flag --include-ephemeral type=bool default="false" usage="Include ephemeral issues (wisps, etc.) when pushing to Linear" + flag --issues type=string default="" usage="Comma-separated bead IDs to sync selectively (e.g., bd-abc,bd-def). Mutually exclusive with --parent." + flag --milestones type=bool default="false" usage="Reconstruct Linear project milestones as local epic parents when pulling" + flag --no-wait type=bool default="false" usage="Fail immediately if another sync is running instead of waiting" + flag --parent type=string default="" usage="Limit push to this beads ticket and its descendants" + flag --prefer-linear type=bool default="false" usage="Prefer Linear version on conflicts" + flag --prefer-local type=bool default="false" usage="Prefer local version on conflicts" + flag --pull type=bool default="false" usage="Pull issues from Linear" + flag --pull-if-stale type=bool default="false" usage="Pull only if Linear data is stale (skip if fresh)" + flag --push type=bool default="false" usage="Push issues to Linear" + flag --relations type=bool default="false" usage="Import Linear relations as bd dependencies when pulling" + flag --state type=string default="all" usage="Issue state to sync: open, closed, all" + flag --team type=stringSlice default="[]" usage="Team ID(s) to sync (overrides configured team_id/team_ids)" + flag --threshold type=duration default="20m0s" usage="Staleness threshold for --pull-if-stale (default 20m)" + flag --type type=stringSlice default="[]" usage="Only sync issues of these types (can be repeated)" + flag --update-refs type=bool default="true" usage="Update external_ref after creating Linear issues" + +bd linear teams + use: "teams" + short: "List available Linear teams" + hidden: false + runnable: true + long: + | List all teams accessible with your Linear API key. + | + | Use this to find the team ID (UUID) needed for configuration. + | + | Example: + | bd linear teams + | bd config set linear.team_id "12345678-1234-1234-1234-123456789abc" + +bd link + use: "link " + short: "Link two issues with a dependency" + group: "issues" + hidden: false + runnable: true + long: + | Link two issues with a dependency. + | + | Shorthand for 'bd dep add '. By default creates a "blocks" + | dependency (id2 blocks id1). Use --type to specify a different relationship. + | + | Examples: + | bd link bd-123 bd-456 # bd-456 blocks bd-123 + | bd link bd-123 bd-456 --type related # bd-123 related to bd-456 + | bd link bd-123 bd-456 --type parent-child + flag --type -t type=string default="blocks" usage="Dependency type (blocks|tracks|related|parent-child|discovered-from)" + +bd lint + use: "lint [issue-id...]" + short: "Check issues for missing template sections" + group: "views" + hidden: false + runnable: true + long: + | Check issues for missing recommended sections based on issue type. + | + | By default, lints all open issues. Specify issue IDs to lint specific issues. + | + | Section requirements by type: + | bug: Steps to Reproduce, Acceptance Criteria + | task: Acceptance Criteria + | feature: Acceptance Criteria + | epic: Success Criteria (or Acceptance Criteria) + | chore: (none) + | + | Additional per-type sections can be required via config; they are ADDITIVE + | to the built-ins above (built-in requirements are never relaxed): + | + | bd config set lint.sections.epic "Standards scorecard, Cost" + | + | Examples: + | bd lint # Lint all open issues + | bd lint bd-abc # Lint specific issue + | bd lint bd-abc bd-def # Lint multiple issues + | bd lint --type bug # Lint only bugs + | bd lint --status all # Lint all issues (including closed) + | + flag --status -s type=string default="" usage="Filter by status (default: open, use 'all' for all)" + flag --type -t type=string default="" usage="Filter by issue type (bug, task, feature, epic, decision, spike, story, chore, milestone)" + +bd list + use: "list" + short: "List issues" + group: "issues" + hidden: false + runnable: true + flag --all type=bool default="false" usage="Show all issues including closed (overrides default filter)" + flag --assignee -a type=string default="" usage="Filter by assignee" + flag --brief type=bool default="false" usage="Omit the free-form text (description, design, acceptance criteria, notes, payload, waiters) from each row. Filters that read those fields, such as --desc-contains, still select on them. An omitted field is indistinguishable from an empty one in --json; fetch a whole issue with bd show." + flag --closed-after type=string default="" usage="Filter issues closed after date (YYYY-MM-DD or RFC3339)" + flag --closed-before type=string default="" usage="Filter issues closed before date (YYYY-MM-DD or RFC3339)" + flag --created-after type=string default="" usage="Filter issues created after date (YYYY-MM-DD or RFC3339)" + flag --created-before type=string default="" usage="Filter issues created before date (YYYY-MM-DD or RFC3339)" + flag --defer-after type=string default="" usage="Filter issues deferred after date (supports relative: +6h, tomorrow)" + flag --defer-before type=string default="" usage="Filter issues deferred before date (supports relative: +6h, tomorrow)" + flag --deferred type=bool default="false" usage="Show only issues with defer_until set" + flag --deps type=string default="" no-opt-default="scheduling" usage="Annotate tree with dependency edges and order siblings by them: 'scheduling' (bare --deps) or 'all'" + flag --desc-contains type=string default="" usage="Filter by description substring (case-insensitive)" + flag --due-after type=string default="" usage="Filter issues due after date (supports relative: +6h, tomorrow)" + flag --due-before type=string default="" usage="Filter issues due before date (supports relative: +6h, tomorrow)" + flag --empty-description type=bool default="false" usage="Filter issues with empty or missing description" + flag --exclude-label type=stringSlice default="[]" usage="Exclude issues that have ANY of these labels" + flag --exclude-type type=stringSlice default="[]" usage="Exclude issue types from results (comma-separated or repeatable, e.g., --exclude-type=convoy,epic)" + flag --external-contains type=string default="" usage="Filter by external ref substring (case-insensitive)" + flag --external-ref type=string default="" usage="Filter by exact external_ref value" + flag --filter-parent type=string default="" hidden usage="Alias for --parent" + flag --flat type=bool default="false" usage="Disable tree format and use legacy flat list output" + flag --format type=string default="" usage="Export the dependency graph of the listed issues: 'digraph' (edge list for golang.org/x/tools/cmd/digraph), 'dot' (Graphviz), or a Go template rendered once per edge over .IssueID, .DependsOnID, .Type, .Issue, .Dependency. 'json' is the exception and does not export the graph: it is equivalent to --json, which also wins when both flags are given. 'digraph' and templates print nothing when no listed issue depends on another; 'dot' still prints the nodes" + flag --has-metadata-key type=string default="" usage="Filter issues that have this metadata key set" + flag --id type=string default="" usage="Filter by specific issue IDs (comma-separated or repeated, e.g., bd-1,bd-5,bd-10)" + flag --include-comments type=bool default="false" usage="Populate each row's comments field with full comment bodies in JSON output (--json only; costs one query per row that has comments). Without it, a row with comments carries comments_omitted=true, so an absent comments field is never mistaken for having none. Text output is unaffected either way." + flag --include-ephemeral type=bool default="false" usage="Include ephemeral wisp-plane rows in output (normally hidden)" + flag --include-gates type=bool default="false" usage="Include gate issues in output (normally hidden)" + flag --include-infra type=bool default="false" usage="Include infrastructure beads (agent/role/message) in output" + flag --include-templates type=bool default="false" usage="Include template molecules in output" + flag --label -l type=stringSlice default="[]" usage="Filter by labels (AND: must have ALL). Can combine with --label-any" + flag --label-any type=stringSlice default="[]" usage="Filter by labels (OR: must have AT LEAST ONE). Can combine with --label" + flag --label-pattern type=string default="" usage="Filter by label glob pattern (e.g., 'tech-*' matches tech-debt, tech-legacy)" + flag --label-regex type=string default="" usage="Filter by label regex pattern (e.g., 'tech-(debt|legacy)')" + flag --limit -n type=int default="50" usage="Limit results (an explicit --limit always wins, 0 meaning unlimited; otherwise --all is unlimited; otherwise a configured list.limit applies; otherwise unlimited when piped, else 20 in agent mode at a terminal, else 50)" + flag --long type=bool default="false" usage="Show detailed multi-line output for each issue" + flag --max-rows type=int default="0" usage="Hard upper bound on rows returned. Returns a non-zero exit (code 2) and an error to stderr if exceeded. 0 disables (the default). Overrides BEADS_MAX_ROWS for this invocation. Useful in CI/agent rigs that want a circuit breaker against pathological queries. Honored on both the direct and the --proxied-server route." + flag --metadata-field type=stringArray default="[]" usage="Filter by metadata field (key=value, repeatable)" + flag --mol-type type=string default="" usage="Filter by molecule type: swarm, patrol, or work" + flag --no-assignee type=bool default="false" usage="Filter issues with no assignee" + flag --no-labels type=bool default="false" usage="Filter issues with no labels" + flag --no-pager type=bool default="false" usage="Disable pager output" + flag --no-parent type=bool default="false" usage="Exclude child issues (show only top-level issues)" + flag --no-pinned type=bool default="false" usage="Exclude pinned issues" + flag --notes-contains type=string default="" usage="Filter by notes substring (case-insensitive)" + flag --offset type=int default="0" usage="Skip the first N matching results (0-based). Only supported under --proxied-server." + flag --overdue type=bool default="false" usage="Show only issues with due_at in the past (not closed)" + flag --parent type=string default="" usage="Filter by parent issue ID (shows children of specified issue)" + flag --pinned type=bool default="false" usage="Show only pinned issues" + flag --pretty type=bool default="false" usage="Display issues in a tree format with status/priority symbols" + flag --priority -p type=string default="" usage="Priority (0-4 or P0-P4, 0=highest)" + flag --priority-max type=string default="" usage="Filter by maximum priority (inclusive, 0-4 or P0-P4)" + flag --priority-min type=string default="" usage="Filter by minimum priority (inclusive, 0-4 or P0-P4)" + flag --ready type=bool default="false" usage="Show only ready issues (no active blockers, same semantics as bd ready)" + flag --reverse -r type=bool default="false" usage="Invert the sort field's default direction (created/updated/closed default to newest-first, so --sort updated --reverse is oldest-first)" + flag --skip-labels type=bool default="false" usage="Skip label hydration. The labels field in output will be empty regardless of actual labels. Use only when the caller does not depend on label data. Cannot combine with --label, --label-any, --label-pattern, --label-regex, --exclude-label, or --no-labels." + flag --sort type=string default="" usage="Sort by field: priority, created, updated, closed, status, id, title, type, assignee" + flag --spec type=string default="" usage="Filter by spec_id prefix" + flag --state type=string default="" hidden usage="Alias for --status" + flag --status -s type=string default="" usage="Filter by stored status (open, in_progress, blocked, deferred, closed). Comma-separated or repeated for multiple: --status open,in_progress" + flag --title type=string default="" usage="Filter by title text (case-insensitive substring match)" + flag --title-contains type=string default="" usage="Filter by title substring (case-insensitive)" + flag --tree type=bool default="true" usage="Hierarchical tree format (default: true; use --flat to disable)" + flag --type -t type=string default="" usage="Filter by type (bug, feature, task, epic, chore, decision, merge-request, molecule, gate, convoy). Aliases: mr→merge-request, feat→feature, mol→molecule, dec/adr→decision" + flag --updated-after type=string default="" usage="Filter issues updated after date (YYYY-MM-DD or RFC3339)" + flag --updated-before type=string default="" usage="Filter issues updated before date (YYYY-MM-DD or RFC3339)" + flag --watch -w type=bool default="false" usage="Watch for changes and auto-update display (implies --pretty)" + flag --wisp-type type=string default="" usage="Filter by wisp type: heartbeat, ping, patrol, gc_report, recovery, error, escalation" + +bd mail + use: "mail [subcommand] [args...]" + short: "Delegate to mail provider (e.g., gt mail)" + hidden: false + runnable: true + disable-flag-parsing: true + long: + | Delegates mail operations to an external mail provider. + | + | Agents often type 'bd mail' when working with beads, but mail functionality + | is typically provided by the orchestrator. This command bridges that gap + | by delegating to the configured mail provider. + | + | Configuration (checked in order): + | 1. BEADS_MAIL_DELEGATE or BD_MAIL_DELEGATE environment variable + | 2. 'mail.delegate' config setting (bd config set mail.delegate "gt mail") + | + | Examples: + | # Configure delegation (one-time setup) + | export BEADS_MAIL_DELEGATE="gt mail" + | # or + | bd config set mail.delegate "gt mail" + | + | # Then use bd mail as if it were gt mail + | bd mail inbox # Lists inbox + | bd mail send mayor/ -s "Hi" # Sends mail + | bd mail read msg-123 # Reads a message + +bd memories + use: "memories [search]" + short: "List or search persistent memories" + group: "setup" + hidden: false + runnable: true + long: + | List all memories, or search by keyword. + | + | Examples: + | bd memories # list all memories + | bd memories dolt # search for memories about dolt + | bd memories "race flag" # search for a phrase + +bd merge-slot + use: "merge-slot" + short: "Manage merge-slot gates for serialized conflict resolution" + group: "issues" + hidden: false + runnable: false + long: + | Merge-slot gates serialize conflict resolution in the merge queue. + | + | A merge slot is an exclusive access primitive: only one agent can hold it at a time. + | This prevents "monkey knife fights" where multiple polecats race to resolve conflicts + | and create cascading conflicts. + | + | Each rig has one merge slot bead: -merge-slot (labeled gt:slot). + | The slot uses: + | - status=open: slot is available + | - status=in_progress: slot is held + | - metadata.holder: who currently holds the slot + | - metadata.waiters: priority-ordered queue of waiters + | + | Examples: + | bd merge-slot create # Create merge slot for current rig + | bd merge-slot check # Check if slot is available + | bd merge-slot acquire # Try to acquire the slot + | bd merge-slot release # Release the slot + +bd merge-slot acquire + use: "acquire" + short: "Acquire the merge slot" + hidden: false + runnable: true + long: + | Attempt to acquire the merge slot for exclusive access. + | + | If the slot is available (status=open), it will be acquired: + | - status set to in_progress + | - holder set to the requester + | + | If the slot is held (status=in_progress), the command fails unless + | --wait is passed, which adds the requester to the waiters queue. + | + | Use --holder to specify who is acquiring (default: BEADS_ACTOR env var). + flag --holder type=string default="" usage="Who is acquiring the slot (default: BEADS_ACTOR)" + flag --wait type=bool default="false" usage="Add to waiters list if slot is held" + +bd merge-slot check + use: "check" + short: "Check merge slot availability" + hidden: false + runnable: true + long: + | Check if the merge slot is available or held. + | + | Returns: + | - available: slot can be acquired + | - held by : slot is currently held + | - not found: no merge slot exists for this rig + +bd merge-slot create + use: "create" + short: "Create a merge slot bead for the current rig" + hidden: false + runnable: true + long: + | Create a merge slot bead for serialized conflict resolution. + | + | The slot ID is automatically generated based on the beads prefix (e.g., gt-merge-slot). + | The slot is created with status=open (available). + +bd merge-slot release + use: "release" + short: "Release the merge slot" + hidden: false + runnable: true + long: + | Release the merge slot after conflict resolution is complete. + | + | Sets status back to open and clears the holder field. + | If there are waiters, the highest-priority waiter should then acquire. + flag --holder type=string default="" usage="Who is releasing the slot (for verification)" + +bd metrics + use: "metrics" + short: "Show or change anonymous usage-metrics settings" + hidden: false + runnable: true + long: + | Show whether anonymous usage metrics are on, see exactly what is sent, and + | turn them on or off. + | + | bd shares anonymous usage metrics to learn how people actually use it — just + | which commands get run, plus the bd version and OS platform. That's how we decide + | what to polish next. We never collect your issues, paths, remotes, identity, or + | any user-supplied text. + | + | bd metrics show the current status and what is collected + | bd metrics on turn metrics on + | bd metrics off turn metrics off + | bd metrics example show real examples of the events bd sends + +bd metrics example + use: "example" + short: "Show real examples of the anonymous metrics bd sends" + hidden: false + runnable: true + +bd metrics off + use: "off" + short: "Turn anonymous usage metrics off" + hidden: false + runnable: true + +bd metrics on + use: "on" + short: "Turn anonymous usage metrics on" + hidden: false + runnable: true + +bd migrate + use: "migrate" + short: "Database migration commands" + group: "maint" + hidden: false + runnable: true + long: + | Database migration and data transformation commands. + | + | Without subcommand, checks and updates database metadata to current version. + | + | Subcommands: + | hooks Plan git hook migration to marker-managed format + | issues Move issues between repositories + | schema Apply pending schema migrations (idempotent) + | sync Set up sync.branch workflow for multi-clone setups + | from-server-to-proxied-server [EXPERIMENTAL] Switch server mode to proxied-server mode + | from-proxied-server-to-server [EXPERIMENTAL] Switch proxied-server mode to server mode + | from-shared-server-to-proxied-server [EXPERIMENTAL] Switch shared-server mode to proxied-server mode + | from-proxied-server-to-shared-server [EXPERIMENTAL] Switch proxied-server mode to shared-server mode + | + | On a remote-backed database with pending schema migrations bd refuses to + | migrate in place (#4259): migrating two clones independently forks the schema + | so bd dolt pull can no longer merge — the break is silent and unrecoverable. + | Use --force to confirm you are the single designated migrator, after which you + | should publish the migrated schema with 'bd dolt push'. The env-var equivalent + | BD_ALLOW_REMOTE_MIGRATE=1 remains supported for scripted/CI use. + | + flag --dry-run type=bool default="false" usage="Show what would be done without making changes" + flag --force type=bool default="false" usage="Bypass the remote-migrate gate as the single designated migrator (equivalent to BD_ALLOW_REMOTE_MIGRATE=1)" + flag --inspect type=bool default="false" usage="Show migration plan and database state for AI agent analysis" + flag --update-repo-id type=bool default="false" usage="Update repository ID (use after changing git remote)" + flag --yes type=bool default="false" usage="Auto-confirm prompts" + +bd migrate from-proxied-server-to-server + use: "from-proxied-server-to-server" + short: "[EXPERIMENTAL] Switch a proxied-server repo to server mode" + hidden: false + runnable: true + long: + | Switch a repo from proxied-server mode to server mode (bd init --server). + | + | Both modes root their dolt sql-server at the same .beads/dolt directory, so this + | only rewrites .beads/metadata.json (dolt_mode) and removes the proxied-server + | sidecar — no Dolt data is copied or moved. Stop the running proxy first with + | 'bd dolt stop'. + | + | Note: dolt_mode lives in the committed metadata.json, so this change propagates + | to clones on the next push. + flag --dry-run type=bool default="false" usage="Show what would be done without making changes" + +bd migrate from-proxied-server-to-shared-server + use: "from-proxied-server-to-shared-server" + short: "[EXPERIMENTAL] Switch a proxied-server repo back to shared-server mode" + hidden: false + runnable: true + long: + | Switch a repo from proxied-server mode back to shared-server mode. + | + | Only applies to a proxied-server repo rooted at the shared dolt directory + | (~/.beads/shared-server/dolt) — the reverse of from-shared-server-to-proxied-server. + | This rewrites .beads/metadata.json (dolt_mode), re-enables dolt.shared-server, and + | removes the proxied-server sidecar; no Dolt data is copied or moved. Stop the + | running proxy first with 'bd dolt stop'. + flag --dry-run type=bool default="false" usage="Show what would be done without making changes" + +bd migrate from-server-to-proxied-server + use: "from-server-to-proxied-server" + short: "[EXPERIMENTAL] Switch a server-mode repo to proxied-server mode" + hidden: false + runnable: true + long: + | Switch a repo from server mode (bd init --server) to proxied-server mode. + | + | Both modes root their dolt sql-server at the same .beads/dolt directory, so this + | only rewrites .beads/metadata.json (dolt_mode) and writes the proxied-server + | sidecar — no Dolt data is copied or moved. Stop the running server first with + | 'bd dolt stop'. + | + | Note: dolt_mode lives in the committed metadata.json, so this change propagates + | to clones on the next push. + flag --dry-run type=bool default="false" usage="Show what would be done without making changes" + flag --idle-timeout type=duration default="0s" usage="Proxy idle timeout; omit for the 30s default, 0 for indefinite uptime" + +bd migrate from-shared-server-to-proxied-server + use: "from-shared-server-to-proxied-server" + short: "[EXPERIMENTAL] Switch a shared-server repo to proxied-server mode" + hidden: false + runnable: true + long: + | Switch a repo from shared-server mode to proxied-server mode. + | + | The proxied server is rooted at the shared dolt directory + | (~/.beads/shared-server/dolt), so no Dolt data is copied or moved; this rewrites + | .beads/metadata.json (dolt_mode), turns off dolt.shared-server for this repo, and + | writes the proxied-server sidecar. Stop the running shared server first with + | 'bd dolt stop' — note that stops it for every project sharing it. + flag --dry-run type=bool default="false" usage="Show what would be done without making changes" + flag --idle-timeout type=duration default="0s" usage="Proxy idle timeout; omit for the 30s default, 0 for indefinite uptime" + +bd migrate hooks + use: "hooks [path]" + short: "Plan or apply git hook migration to marker-managed format" + hidden: false + runnable: true + long: + | Analyze git hook files and sidecar artifacts for migration to marker-managed format. + | + | Modes: + | --dry-run Preview migration operations without changing files + | --apply Apply migration operations + | + | Examples: + | bd migrate hooks --dry-run + | bd migrate hooks --apply + | bd migrate hooks --apply --yes + | bd migrate hooks --dry-run --json + flag --apply type=bool default="false" usage="Apply planned hook migration changes" + flag --dry-run type=bool default="false" usage="Show what would be done without making changes" + flag --yes type=bool default="false" usage="Skip confirmation prompt for --apply" + +bd migrate issues + use: "issues" + short: "Move issues between repositories" + hidden: false + runnable: true + long: + | Move issues from one source repository to another with filtering and dependency preservation. + | + | This command updates the source_repo field for selected issues, allowing you to: + | - Move contributor planning issues to upstream repository + | - Reorganize issues across multi-phase repositories + | - Consolidate issues from multiple repos + | + | Examples: + | # Preview migration from planning repo to current repo + | bd migrate-issues --from ~/.beads-planning --to . --dry-run + | + | # Move all open P1 bugs + | bd migrate-issues --from ~/repo1 --to ~/repo2 --priority 1 --type bug --status open + | + | # Move specific issues with their dependencies + | bd migrate-issues --from . --to ~/archive --id bd-abc --id bd-xyz --include closure + | + | # Move issues with label filter + | bd migrate-issues --from . --to ~/feature-work --label frontend --label urgent + flag --dry-run type=bool default="false" usage="Show plan without making changes" + flag --from type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="Source repository (required)" + flag --id type=stringSlice default="[]" usage="Specific issue IDs to migrate (can specify multiple)" + flag --ids-file type=string default="" usage="File containing issue IDs (one per line)" + flag --include type=string default="none" usage="Include dependencies: none/upstream/downstream/closure" + flag --label type=stringSlice default="[]" usage="Filter by labels (can specify multiple)" + flag --priority type=int default="-1" usage="Filter by priority (0-4)" + flag --status type=string default="" usage="Filter by status (open/closed/all)" + flag --strict type=bool default="false" usage="Fail on orphaned dependencies or missing repos" + flag --to type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="Destination repository (required)" + flag --type type=string default="" usage="Filter by issue type (bug/feature/task/epic/chore/decision)" + flag --within-from-only type=bool default="true" usage="Only include dependencies from source repo" + flag --yes type=bool default="false" usage="Skip confirmation prompt" + +bd migrate legacy-sqlite + use: "legacy-sqlite --source-db PATH --output PATH|-" + short: "Read an authenticated legacy SQLite database as JSONL" + hidden: false + runnable: true + annotations: {bd:skip_store="1"} + flag --output type=string default="" usage="JSONL output path, or - for stdout" + flag --source-db type=string default="" usage="Legacy SQLite database (read-only)" + +bd migrate schema + use: "schema" + short: "Apply pending schema migrations (idempotent)" + hidden: false + runnable: true + long: + | Apply pending schema migrations idempotently. + | + | bd does not apply pending schema migrations to an existing database without + | the operator's consent, and running this command is that consent + | (BD_ALLOW_MIGRATE=1 is the scripted form). On a remote-backed database the + | remote-migrate gate still applies: --force confirms you are the single + | designated migrator. Migrating is one-way: bd releases built for an older + | schema refuse the migrated database. + | + | On an up-to-date database this is a no-op, so it is safe to run in CI, + | release gates, and recovery scenarios. + | + | Example: + | bd migrate schema + | bd migrate schema --json + flag --force type=bool default="false" usage="Bypass the remote-migrate gate as the single designated migrator (equivalent to BD_ALLOW_REMOTE_MIGRATE=1)" + +bd migrate sync + use: "sync " + short: "Set up sync.branch workflow for multi-clone setups" + hidden: false + runnable: true + long: + | Configure separate branch workflow for multi-clone setups. + | + | This sets the sync.branch config value so that issue data is committed + | to a dedicated branch, keeping your main branch clean. + | + | Example: + | bd migrate sync beads-sync + flag --dry-run type=bool default="false" usage="Show what would be done without making changes" + +bd migrate-issues + use: "migrate-issues" + short: "Move issues between repositories" + hidden: true + deprecated: "use 'bd migrate issues' instead (will be removed in v1.0.0)" + runnable: true + long: + | Move issues from one source repository to another with filtering and dependency preservation. + | + | This command updates the source_repo field for selected issues, allowing you to: + | - Move contributor planning issues to upstream repository + | - Reorganize issues across multi-phase repositories + | - Consolidate issues from multiple repos + | + | Examples: + | # Preview migration from planning repo to current repo + | bd migrate-issues --from ~/.beads-planning --to . --dry-run + | + | # Move all open P1 bugs + | bd migrate-issues --from ~/repo1 --to ~/repo2 --priority 1 --type bug --status open + | + | # Move specific issues with their dependencies + | bd migrate-issues --from . --to ~/archive --id bd-abc --id bd-xyz --include closure + | + | # Move issues with label filter + | bd migrate-issues --from . --to ~/feature-work --label frontend --label urgent + flag --dry-run type=bool default="false" usage="Show plan without making changes" + flag --from type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="Source repository (required)" + flag --id type=stringSlice default="[]" usage="Specific issue IDs to migrate (can specify multiple)" + flag --ids-file type=string default="" usage="File containing issue IDs (one per line)" + flag --include type=string default="none" usage="Include dependencies: none/upstream/downstream/closure" + flag --label type=stringSlice default="[]" usage="Filter by labels (can specify multiple)" + flag --priority type=int default="-1" usage="Filter by priority (0-4)" + flag --status type=string default="" usage="Filter by status (open/closed/all)" + flag --strict type=bool default="false" usage="Fail on orphaned dependencies or missing repos" + flag --to type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="Destination repository (required)" + flag --type type=string default="" usage="Filter by issue type (bug/feature/task/epic/chore/decision)" + flag --within-from-only type=bool default="true" usage="Only include dependencies from source repo" + flag --yes type=bool default="false" usage="Skip confirmation prompt" + +bd migrate-personal + use: "migrate-personal" + short: "Move personal planning issues from the project database to your planning repo" + group: "setup" + hidden: false + runnable: true + long: + | Identify issues you created in the project database and move them to your + | personal planning repository (~/.beads-planning by default). + | + | This is a one-time migration for contributors who created personal planning + | issues before contributor routing was configured. + | + | The command: + | 1. Finds all issues in the project database created by your git identity + | 2. Shows you the list and asks for confirmation + | 3. Moves them to the planning repo configured in routing.contributor + | + | EXAMPLES: + | bd migrate-personal # Interactive: show list and prompt + | bd migrate-personal -y # Non-interactive: skip confirmation + flag --yes -y type=bool default="false" usage="Skip confirmation prompt" + +bd mol + use: "mol" + aliases: ["protomolecule"] + short: "Molecule commands (work templates)" + hidden: false + runnable: false + long: + | Manage molecules - work templates for agent workflows. + | + | Protos are template epics with the "template" label. They define a DAG of work + | that can be spawned to create real issues (molecules). + | + | The molecule metaphor: + | - A proto is an uninstantiated template (reusable work pattern) + | - Spawning creates a molecule (real issues) from the proto + | - Variables ({{key}}) are substituted during spawning + | - Bonding combines protos or molecules into compounds + | - Distilling extracts a proto from an ad-hoc epic + | + | Commands: + | show Show proto/molecule structure and variables + | pour Instantiate proto as persistent mol (liquid phase) + | wisp Instantiate proto as ephemeral wisp (vapor phase) + | bond Polymorphic combine: proto+proto, proto+mol, mol+mol + | squash Condense molecule to digest + | burn Discard wisp + | distill Extract proto from ad-hoc epic + | + | Use "bd formula list" to list available formulas. + +bd mol bond + use: "bond " + aliases: ["fart"] + short: "Bond two protos or molecules together" + hidden: false + runnable: true + long: + | Bond two protos or molecules to create a compound. + | + | The bond command is polymorphic - it handles different operand types: + | + | formula + formula → cook both, compound proto + | formula + proto → cook formula, compound proto + | formula + mol → cook formula, spawn and attach + | proto + proto → compound proto (reusable template) + | proto + mol → spawn proto, attach to molecule + | mol + proto → spawn proto, attach to molecule + | mol + mol → join into compound molecule + | + | Formula names (e.g., mol-polecat-arm) are cooked inline as ephemeral protos. + | This avoids needing pre-cooked proto beads in the database. + | + | Bond types: + | sequential (default) - B runs after A completes + | parallel - B runs alongside A + | conditional - B runs only if A fails + | + | With --ref, the spawned proto arm is nested inside the molecule operand + | rather than ordered against it - whichever side of the bond that is - so a + | sequential --ref arm carries no ordering: it is ready as soon as it is + | spawned. Ordering a nested arm against its own container is unsatisfiable + | in both directions (the arm waits for the molecule to close; the molecule + | cannot close while it holds an open arm). --ref is refused with --type + | conditional, which has no safe degradation - omit --ref to bond a + | conditional arm as a sibling. + | + | Phase control: + | By default, spawned protos follow the target's phase: + | - Attaching to mol (Ephemeral=false) → spawns as persistent (Ephemeral=false) + | - Attaching to ephemeral issue (Ephemeral=true) → spawns as ephemeral (Ephemeral=true) + | + | Override with: + | --pour Force spawn as liquid (persistent, Ephemeral=false) + | --ephemeral Force spawn as vapor (ephemeral, Ephemeral=true, excluded from Dolt sync via dolt_ignore) + | + | Dynamic bonding (Christmas Ornament pattern): + | Use --ref to specify a custom child reference with variable substitution. + | This creates IDs like "parent.child-ref" instead of random hashes. + | + | Example: + | bd mol bond mol-worker-arm bd-patrol --ref arm-{{worker_name}} --var worker_name=ace + | # Creates: bd-patrol.arm-ace (and children like bd-patrol.arm-ace.capture) + | + | Use cases: + | - Found important bug during patrol? Use --pour to persist it + | - Need ephemeral diagnostic on persistent feature? Use --ephemeral + | - Spawning per-worker arms on a patrol? Use --ref for readable IDs + | + | Examples: + | bd mol bond mol-feature mol-deploy # Compound proto + | bd mol bond mol-feature mol-deploy --type parallel # Run in parallel + | bd mol bond mol-feature bd-abc123 # Attach proto to molecule + | bd mol bond bd-abc123 bd-def456 # Join two molecules + | bd mol bond mol-critical-bug wisp-patrol --pour # Persist found bug + | bd mol bond mol-temp-check bd-feature --ephemeral # Ephemeral diagnostic + | bd mol bond mol-arm bd-patrol --ref arm-{{name}} --var name=ace # Dynamic child ID + flag --as type=string default="" usage="Custom title for compound proto (proto+proto only)" + flag --dry-run type=bool default="false" usage="Preview what would be created" + flag --ephemeral type=bool default="false" usage="Force spawn as vapor (ephemeral, Ephemeral=true)" + flag --pour type=bool default="false" usage="Force spawn as liquid (persistent, Ephemeral=false)" + flag --ref type=string default="" usage="Custom child reference with {{var}} substitution (e.g., arm-{{polecat_name}})" + flag --type type=string default="sequential" usage="Bond type: sequential, parallel, or conditional" + flag --var type=stringArray default="[]" usage="Variable substitution for spawned protos (key=value)" + +bd mol burn + use: "burn [molecule-id...]" + short: "Delete a molecule without creating a digest" + hidden: false + runnable: true + long: + | Burn a molecule, deleting it without creating a digest. + | + | Unlike squash (which creates a permanent digest before deletion), burn + | completely removes the molecule with no trace. Use this for: + | - Abandoned patrol cycles + | - Crashed or failed workflows + | - Test/debug molecules you don't want to preserve + | + | The burn operation differs based on molecule phase: + | - Wisp (ephemeral): Direct delete + | - Mol (persistent): Cascade delete (syncs to remotes) + | + | CAUTION: This is a destructive operation. The molecule's data will be + | permanently lost. If you want to preserve a summary, use 'bd mol squash'. + | + | Example: + | bd mol burn bd-abc123 # Delete molecule with no trace + | bd mol burn bd-abc123 --dry-run # Preview what would be deleted + | bd mol burn bd-abc123 --force # Skip confirmation + | bd mol burn bd-a1 bd-b2 bd-c3 # Batch delete multiple wisps + flag --dry-run type=bool default="false" usage="Preview what would be deleted" + flag --force type=bool default="false" usage="Skip confirmation prompt" + flag --yes -y type=bool default="false" hidden usage="Alias for --force (skip confirmation)" + +bd mol current + use: "current [molecule-id]" + short: "Show current position in molecule workflow" + hidden: false + runnable: true + long: + | Show where you are in a molecule workflow. + | + | If molecule-id is given, show status for that molecule. + | If not given, infer from in_progress issues assigned to current agent. + | + | The output shows all steps with status indicators: + | [done] - Step is complete (closed) + | [current] - Step is in_progress (you are here) + | [ready] - Step is ready to start (unblocked) + | [blocked] - Step is blocked by dependencies + | [pending] - Step is waiting + | + | For large molecules (>100 steps), a summary is shown instead. + | Use --limit or --range to view specific steps: + | bd mol current --limit 50 # Show first 50 steps + | bd mol current --range 100-150 # Show steps 100-150 + flag --for type=string default="" usage="Show molecules for a specific agent/assignee" + flag --limit type=int default="0" usage="Maximum number of steps to display (0 = auto, use 'all' threshold)" + flag --range type=string default="" usage="Display specific step range (e.g., '1-50', '100-150')" + +bd mol distill + use: "distill [formula-name]" + short: "Extract a formula from an existing epic" + hidden: false + runnable: true + long: + | Distill a molecule by extracting a reusable formula from an existing epic. + | + | This is the reverse of pour: instead of formula → molecule, it's molecule → formula. + | + | The distill command: + | 1. Loads the existing epic and all its children + | 2. Converts the structure to a .formula.json file + | 3. Replaces concrete values with {{variable}} placeholders (via --var flags) + | + | Use cases: + | - Team develops good workflow organically, wants to reuse it + | - Capture tribal knowledge as executable templates + | - Create starting point for similar future work + | + | Variable syntax (both work - we detect which side is the concrete value): + | --var branch=feature-auth Spawn-style: variable=value (recommended) + | --var feature-auth=branch Substitution-style: value=variable + | + | Output locations (first writable wins): + | 1. /formulas/ (project-level, default) + | 2. /.beads/formulas/ (repo-local formulas) + | 3. ~/.beads/formulas/ (user-level, if project not writable) + | + | Examples: + | bd mol distill bd-o5xe my-workflow + | bd mol distill bd-abc release-workflow --var feature_name=auth-refactor + flag --dry-run type=bool default="false" usage="Preview what would be created" + flag --output type=string default="" usage="Output directory for formula file" + flag --var type=stringArray default="[]" usage="Replace value with {{variable}} placeholder (variable=value)" + +bd mol last-activity + use: "last-activity " + short: "Show last activity timestamp for a molecule" + hidden: false + runnable: true + long: + | Show the most recent activity timestamp for a molecule. + | + | Returns the timestamp of the most recent change to any step in the molecule, + | making it easy to detect stale or stuck molecules. + | + | Activity sources: + | step_closed - A step was closed + | step_updated - A step was updated (claimed, edited, etc.) + | molecule_updated - The molecule root itself was updated + | + | Examples: + | bd mol last-activity hq-wisp-0laki + | bd mol last-activity hq-wisp-0laki --json + +bd mol pour + use: "pour " + short: "Instantiate a proto as a persistent mol (solid -> liquid)" + hidden: false + runnable: true + long: + | Pour a proto into a persistent mol - like pouring molten metal into a mold. + | + | This is the chemistry-inspired command for creating PERSISTENT work from templates. + | The resulting mol is stored as persistent beads in the issue database and + | syncs like any other bead (bd dolt push / pull). + | + | Phase transition: Proto (solid) -> pour -> Mol (liquid) + | + | WHEN TO USE POUR vs WISP: + | pour (liquid): Persistent work that needs audit trail + | - Feature implementations spanning multiple sessions + | - Work you may need to reference later + | - Anything worth preserving in git history + | + | wisp (vapor): Ephemeral work that auto-cleans up + | - Release workflows (one-time execution) + | - Operational loops and recurring cycles + | - Health checks and diagnostics + | - Any operational workflow without audit value + | + | TIP: Formulas can specify phase:"vapor" to recommend wisp usage. + | If you pour a vapor-phase formula, you'll get a warning. + | + | Examples: + | bd mol pour mol-feature --var name=auth # Persistent feature work + | bd mol pour mol-review --var pr=123 # Persistent code review + flag --assignee type=string default="" usage="Assign the root issue to this agent/user" + flag --attach type=stringSlice default="[]" usage="Proto to attach after spawning (repeatable)" + flag --attach-type type=string default="sequential" usage="Bond type for attachments: sequential, parallel, or conditional" + flag --dry-run type=bool default="false" usage="Preview what would be created" + flag --var type=stringArray default="[]" usage="Variable substitution (key=value); a name the proto cannot consume is an error" + +bd mol progress + use: "progress [molecule-id]" + short: "Show molecule progress summary" + hidden: false + runnable: true + long: + | Show efficient progress summary for a molecule. + | + | This command uses indexed queries to count progress without loading all steps, + | making it suitable for very large molecules (millions of steps). + | + | If no molecule-id is given, shows progress for any molecule you're working on. + | + | Output includes: + | - Progress: completed / total (percentage) + | - Current step: the in-progress step (if any) + | - Rate: steps/hour based on closure times + | - ETA: estimated time to completion + | + | Example: + | bd mol progress bd-hanoi-xyz + +bd mol ready + use: "ready --gated" + short: "Find molecules ready for gate-resume dispatch" + hidden: false + runnable: true + long: + | Find molecules where a gate has closed and the workflow is ready to resume. + | + | This command discovers molecules waiting at a gate step where: + | 1. The molecule has a gate bead that blocks a step + | 2. The gate bead is now closed (condition satisfied) + | 3. The blocked step is now ready to proceed + | 4. No agent currently has this molecule hooked + | + | This enables discovery-based resume without explicit waiter tracking. + | The patrol system uses this to find and dispatch gate-ready molecules. + | + | Examples: + | bd mol ready --gated # Find all gate-ready molecules + | bd mol ready --gated --json # JSON output for automation + flag --gated type=bool default="false" usage="Find molecules ready for gate-resume dispatch (always on for this subcommand)" + +bd mol seed + use: "seed " + short: "Verify formula accessibility" + hidden: false + runnable: true + long: + | Verify that a formula is accessible and can be cooked. + | + | The seed command checks formula search paths to ensure a formula exists + | and can be loaded. This is useful for verifying system health before + | attempting to spawn work from a formula. + | + | Formula search paths (checked in order): + | 1. /formulas/ (active project) + | 2. /.beads/formulas/ (repo-local formulas) + | 3. ~/.beads/formulas/ (user level) + | 4. $GT_ROOT/.beads/formulas/ (shared workspace root, if GT_ROOT set) + | + | Examples: + | bd mol seed mol-feature # Verify specific formula + | bd mol seed mol-review --var name=test # Verify with variable substitution + flag --var type=stringArray default="[]" usage="Variable substitution for condition filtering (key=value)" + +bd mol show + use: "show " + short: "Show molecule details" + hidden: false + runnable: true + long: + | Show molecule structure and details. + | + | A molecule's steps are the issues linked to its root by parent-child + | dependencies (bd create --parent, bd mol pour); an id that merely looks + | hierarchical, like bd-xyz.4, does not make an issue a step. + | + | The --parallel flag highlights parallelizable steps: + | - Steps with no blocking dependencies can run in parallel + | - Shows which steps are ready to start now + | - Identifies parallel groups (steps that can run concurrently) + | + | Example: + | bd mol show bd-patrol --parallel + flag --parallel -p type=bool default="false" usage="Show parallel step analysis" + +bd mol squash + use: "squash " + short: "Compress molecule execution into a digest" + hidden: false + runnable: true + long: + | Squash a molecule's ephemeral children into a single digest issue. + | + | This command collects all ephemeral child issues of a molecule (Ephemeral=true), + | generates a summary digest, and promotes the wisps to persistent by + | clearing their Wisp flag (or optionally deletes them). + | + | The squash operation: + | 1. Loads the molecule and all its children + | 2. Filters to only wisps (ephemeral issues with Ephemeral=true) + | 3. Generates a digest (summary of work done) + | 4. Creates a permanent digest issue (Ephemeral=false) + | 5. Clears Wisp flag on children (promotes to persistent) + | OR keeps them with --keep-children (default: delete) + | + | AGENT INTEGRATION: + | Use --summary to provide an AI-generated summary. This keeps bd as a pure + | tool - the calling agent (orchestrator worker, Claude Code, etc.) is responsible + | for generating intelligent summaries. Without --summary, a basic concatenation + | of child issue content is used. + | + | This is part of the wisp workflow: spawn creates wisps, + | execution happens, squash compresses the trace into an outcome (digest). + | + | Example: + | bd mol squash bd-abc123 # Squash and promote children + | bd mol squash bd-abc123 --dry-run # Preview what would be squashed + | bd mol squash bd-abc123 --keep-children # Keep wisps after digest + | bd mol squash bd-abc123 --summary "Agent-generated summary of work done" + flag --dry-run type=bool default="false" usage="Preview what would be squashed" + flag --keep-children type=bool default="false" usage="Don't delete ephemeral children after squash" + flag --summary type=string default="" usage="Agent-provided summary (bypasses auto-generation)" + +bd mol stale + use: "stale" + short: "Detect molecules with all children closed but root still open" + hidden: false + runnable: true + long: + | Detect molecules (epics with children) whose children are all closed + | with completing reasons, but the root is still open. + | + | A molecule is considered stale if: + | 1. Every child is closed with a completing reason (not duplicate/wontfix/superseded) + | 2. Root issue is still open + | 3. Not assigned to anyone (optional, use --unassigned) + | 4. Is blocking other work (optional, use --blocking) + | + | Review whether epic scope is actually finished before closing — child + | closure alone does not prove the epic's stated work is done (GH#5026). + | + | Examples: + | bd mol stale # List all stale molecules + | bd mol stale --json # Machine-readable output + | bd mol stale --blocking # Only show those blocking other work + | bd mol stale --unassigned # Only show unassigned molecules + | bd mol stale --all # Include molecules with 0 children + flag --all type=bool default="false" usage="Include molecules with 0 children" + flag --blocking type=bool default="false" usage="Only show molecules blocking other work" + flag --unassigned type=bool default="false" usage="Only show unassigned molecules" + +bd mol wisp + use: "wisp [proto-id]" + short: "Create or manage wisps (ephemeral molecules)" + hidden: false + runnable: true + long: + | Create or manage wisps - EPHEMERAL molecules for operational workflows. + | + | When called with a proto-id argument, creates a wisp from that proto. + | When called with a subcommand (list, gc), manages existing wisps. + | + | Wisps are issues with Ephemeral=true in the main database. They're stored + | locally but NOT synced via git. + | + | WHEN TO USE WISP vs POUR: + | wisp (vapor): Ephemeral work that auto-cleans up + | - Release workflows (one-time execution) + | - Operational loops and recurring cycles + | - Health checks and diagnostics + | - Any operational workflow without audit value + | + | pour (liquid): Persistent work that needs audit trail + | - Feature implementations spanning multiple sessions + | - Work you may need to reference later + | - Anything worth preserving in git history + | + | TIP: Formulas can specify phase:"vapor" to recommend wisp usage. + | If you use pour on a vapor-phase formula, you'll get a warning. + | + | The wisp lifecycle: + | 1. Create: bd mol wisp or bd create --ephemeral + | 2. Execute: Normal bd operations work on wisp issues + | 3. Squash: bd mol squash (clears Ephemeral flag, promotes to persistent) + | 4. Or burn: bd mol burn (deletes without creating digest) + | + | Examples: + | bd mol wisp beads-release --var version=1.0 # Release workflow + | bd mol wisp mol-my-workflow # Ephemeral operational cycle + | bd mol wisp list # List all wisps + | bd mol wisp gc # Garbage collect old wisps + | + | Subcommands: + | list List all wisps in current context + | gc Garbage collect orphaned wisps + flag --dry-run type=bool default="false" usage="Preview what would be created" + flag --root-only type=bool default="false" usage="Create only the root issue (no child step issues)" + flag --var type=stringArray default="[]" usage="Variable substitution (key=value); a name the proto cannot consume is an error" + +bd mol wisp create + use: "create " + short: "Instantiate a proto as a wisp (solid -> vapor)" + hidden: false + runnable: true + long: + | Create a wisp from a proto - sublimation from solid to vapor. + | + | This is the chemistry-inspired command for creating ephemeral work from templates. + | The resulting wisp is stored in the main database with Ephemeral=true and NOT synced via git. + | + | Phase transition: Proto (solid) -> Wisp (vapor) + | + | Use wisp for: + | - Operational loops and recurring cycles + | - Health checks and monitoring + | - One-shot orchestration runs + | - Routine operations with no audit value + | + | The wisp will: + | - Be stored in main database with Ephemeral=true flag + | - NOT be synced via git + | - Either evaporate (burn) or condense to digest (squash) + | + | Examples: + | bd mol wisp create mol-patrol # Ephemeral patrol cycle + | bd mol wisp create mol-health-check # One-time health check + | bd mol wisp create mol-diagnostics --var target=db # Diagnostic run + flag --dry-run type=bool default="false" usage="Preview what would be created" + flag --root-only type=bool default="false" usage="Create only the root issue (no child step issues)" + flag --var type=stringArray default="[]" usage="Variable substitution (key=value); a name the proto cannot consume is an error" + +bd mol wisp gc + use: "gc" + short: "Garbage collect old/abandoned wisps" + hidden: false + runnable: true + long: + | Garbage collect old or abandoned wisps from the database. + | + | A wisp is considered abandoned if: + | - It hasn't been updated in --age duration and is not closed + | - AND it is not live work: blocked steps (waiting on a dependency), pinned + | beads, and any step whose status category is wip (in_progress, blocked, + | hooked) or frozen (deferred, pinned) are never reclaimed by age, no matter + | how long they have been waiting (GH#4394). Custom statuses count by their + | configured category, so only plain open (active) and closed (done) steps + | are age-reclaimable. If the blocked set or the custom-status list cannot be + | read, the GC aborts rather than risk reclaiming live steps. + | + | Wisps that depend on an abandoned wisp are deleted with it, closed ones + | included even without --all, but only if they too haven't been updated in + | --age duration and are not live work. A dependent that is too young is + | spared and loses its edge to the deleted wisp: an open one is collected once + | it ages out, a closed one only by --all, --closed or 'bd purge'. + | + | Abandoned wisps are deleted without creating a digest. Use 'bd mol squash' + | if you want to preserve a summary before garbage collection. + | + | Use --closed to purge ALL closed wisps (regardless of age). This is the + | fastest way to reclaim space from accumulated wisp bloat. Safe by default: + | requires --force to actually delete. + | + | Note: This uses time-based cleanup, appropriate for ephemeral wisps. + | For graph-pressure staleness detection (blocking other work), see 'bd mol stale'. + | + | Examples: + | bd mol wisp gc # Clean abandoned wisps (default: 1h threshold) + | bd mol wisp gc --dry-run # Preview what would be cleaned + | bd mol wisp gc --age 24h # Custom age threshold + | bd mol wisp gc --all # Also clean closed wisps older than threshold + | bd mol wisp gc --closed # Preview closed wisp deletion + | bd mol wisp gc --closed --force # Delete all closed wisps + | bd mol wisp gc --closed --dry-run # Explicit dry-run (same as no --force) + | bd mol wisp gc --exclude-type agent,rig # Protect agent and rig wisps from GC + | bd mol wisp gc --closed --force --exclude-type mol # Delete closed wisps except mol type + flag --age type=string default="1h" usage="Age threshold for abandoned wisp detection" + flag --all type=bool default="false" usage="Also clean closed wisps older than threshold" + flag --closed type=bool default="false" usage="Delete all closed wisps (ignores --age threshold)" + flag --dry-run type=bool default="false" usage="Preview what would be cleaned" + flag --exclude-type type=stringSlice default="[]" usage="Exclude wisps of these types from GC (comma-separated, e.g., agent,rig)" + flag --force -f type=bool default="false" usage="Actually delete (default: preview only)" + +bd mol wisp list + use: "list" + short: "List all wisps in current context" + hidden: false + runnable: true + long: + | List all wisps (ephemeral molecules) in the current context. + | + | Wisps are issues with Ephemeral=true in the main database. They are stored + | locally but not synced via git. + | + | The list shows: + | - ID: Issue ID of the wisp + | - Title: Wisp title + | - Status: Current status (open, in_progress, closed) + | - Started: When the wisp was created + | - Updated: Last modification time + | + | Old wisp detection: + | - Old wisps haven't been updated in 24+ hours + | - Use 'bd mol wisp gc' to clean up old/abandoned wisps + | + | Examples: + | bd mol wisp list # List all wisps + | bd mol wisp list --json # JSON output for programmatic use + | bd mol wisp list --all # Include closed wisps + flag --all type=bool default="false" usage="Include closed wisps" + flag --type type=string default="" usage="Filter by issue type (e.g., agent, task, patrol)" + +bd note + use: "note [text...]" + short: "Append a note to an issue" + group: "issues" + hidden: false + runnable: true + long: + | Append a note to an issue's notes field. + | + | Shorthand for 'bd update --append-notes "text"'. + | + | Examples: + | bd note gt-abc "Fixed the flaky test" + | bd note gt-abc Fixed the flaky test + | echo "note from pipe" | bd note gt-abc --stdin + | bd note gt-abc --file notes.txt + | + | Note: "note" has NO subcommands — it only appends. + | To read notes on an issue, use: bd show + flag --file type=string default="" annotations={cobra_annotation_mutually_exclusive=["stdin file"]} usage="Read note text from file" + flag --stdin type=bool default="false" annotations={cobra_annotation_mutually_exclusive=["stdin file"]} usage="Read note text from stdin" + +bd notion + use: "notion" + short: "Notion integration commands" + hidden: false + runnable: false + long: + | Commands for syncing issues between beads and Notion. + +bd notion connect + use: "connect" + short: "Connect bd to an existing Notion database or data source" + hidden: false + runnable: true + flag --url type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="Existing Notion database or data source URL" + +bd notion init + use: "init" + short: "Create a dedicated Beads database in Notion" + hidden: false + runnable: true + flag --parent type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="Parent page ID" + flag --title type=string default="Beads Issues" usage="Database title" + +bd notion pull + use: "pull [refs...]" + short: "Pull specific items from Notion" + hidden: false + runnable: true + long: + | Pull one or more items from Notion. + | + | Accepts bead IDs or external references as positional arguments. + | Equivalent to: bd notion sync --pull --issues + flag --dry-run type=bool default="false" usage="Preview pull without making changes" + +bd notion push + use: "push [bead-ids...]" + short: "Push specific beads to Notion" + hidden: false + runnable: true + long: + | Push one or more beads issues to Notion. + | + | Accepts bead IDs as positional arguments. + | Equivalent to: bd notion sync --push --issues + flag --dry-run type=bool default="false" usage="Preview push without making changes" + +bd notion status + use: "status" + short: "Show Notion sync status" + hidden: false + runnable: true + +bd notion sync + use: "sync" + short: "Sync issues with Notion" + hidden: false + runnable: true + long: + | Synchronize issues between beads and Notion. + | + | By default this performs bidirectional sync. Use --pull or --push to limit direction. + flag --create-only type=bool default="false" usage="Only create missing remote pages, do not update existing ones" + flag --dry-run type=bool default="false" usage="Preview changes without making mutations" + flag --issues type=string default="" usage="Comma-separated bead IDs to sync selectively (e.g., bd-abc,bd-def). Mutually exclusive with --parent." + flag --parent type=string default="" usage="Limit push to this bead and its descendants (push only). Mutually exclusive with --issues." + flag --prefer-local type=bool default="false" usage="On conflict, keep the local beads version" + flag --prefer-notion type=bool default="false" usage="On conflict, use the Notion version" + flag --pull type=bool default="false" usage="Only pull issues from Notion" + flag --push type=bool default="false" usage="Only push issues to Notion" + flag --state type=string default="all" usage="Issue state to sync: open, closed, or all" + +bd onboard + use: "onboard" + short: "Display minimal snippet for agent instructions file" + group: "setup" + hidden: false + runnable: true + long: + | Display a minimal snippet to add to your agent instructions file for bd integration. + | + | By default, the agent instructions file is AGENTS.md. Use 'bd init --agents-file' + | to configure a different filename (e.g. BEADS.md). + | + | This outputs a small (~10 line) snippet that points to 'bd prime' for full + | workflow context. This is the same minimal profile that 'bd init' generates + | by default. This approach: + | + | • Keeps your agent file lean (doesn't bloat with instructions) + | • bd prime provides dynamic, always-current workflow details + | • Hooks auto-inject bd prime at session start + | + | For agents or environments that do not auto-inject hook output, use + | 'bd init --agents-profile=full' to embed the complete command reference. + +bd orphans + use: "orphans" + short: "Identify orphaned issues (referenced in commits but still open)" + hidden: false + runnable: true + long: + | Identify orphaned issues - issues that are referenced in commit messages but remain open or in_progress in the database. + | + | This helps identify work that has been implemented but not formally closed. + | + | Examples: + | bd orphans # Show orphaned issues + | bd orphans --json # Machine-readable output + | bd orphans --details # Show full commit information + | bd orphans --fix # Close orphaned issues with confirmation + | bd orphans --label theme:personal # Only orphans with this label + | bd orphans --label-any theme:personal,theme:ventures # Orphans with either label + flag --details type=bool default="false" usage="Show full commit information" + flag --fix -f type=bool default="false" usage="Close orphaned issues with confirmation" + flag --label -l type=stringSlice default="[]" usage="Filter by labels (AND: must have ALL). Can combine with --label-any" + flag --label-any type=stringSlice default="[]" usage="Filter by labels (OR: must have AT LEAST ONE). Can combine with --label" + +bd ping + use: "ping" + short: "Check database connectivity" + group: "maint" + hidden: false + runnable: true + long: + | Lightweight health check that confirms bd can reach its database. + | + | Steps: + | 1. Resolve the .beads workspace + | 2. Open the store (embedded or server) + | 3. Run a trivial query (one issue row; a remote backend reads its + | authenticated server context instead) + | 4. Report timing + | + | Exit 0 on success, exit 1 on failure. + | + | Examples: + | bd ping # Quick connectivity check + | bd ping --json # Structured output for automation + +bd preflight + use: "preflight" + short: "Show PR readiness checklist" + group: "maint" + hidden: false + runnable: true + long: + | Display a checklist of common pre-PR checks for contributors. + | + | This command helps catch common issues before pushing to CI: + | - Tests not run locally + | - Lint errors + | - Unformatted Go files + | - .beads/issues.jsonl pollution + | - Stale nix vendorHash + | - Version mismatches + | + | Examples: + | bd preflight # Show checklist + | bd preflight --check # Run checks automatically + | bd preflight --check --json # JSON output for programmatic use + | bd preflight --check --skip-lint # Explicitly skip lint check + | + flag --check type=bool default="false" usage="Run checks automatically" + flag --fix type=bool default="false" usage="Auto-fix issues where possible (vendorHash)" + flag --skip-lint type=bool default="false" usage="Skip lint check explicitly" + +bd prime + use: "prime" + short: "Output AI-optimized workflow context" + group: "setup" + hidden: false + runnable: true + long: + | Output essential Beads workflow context in AI-optimized markdown format. + | + | Automatically detects if MCP server is active and adapts output: + | - MCP mode: Brief workflow reminders (~50 tokens) + | - CLI mode: Full command reference (~1-2k tokens) + | + | Designed for Claude Code, Gemini CLI, and Codex SessionStart hooks to prevent + | agents from forgetting bd workflow after context compaction. + | + | Config options: + | - no-git-ops: When true, outputs stealth mode (no git commands in session close protocol). + | Set via: bd config set no-git-ops true + | Useful when you want to control when commits happen manually. + | - agent.profile: Explicit policy profile for git/commit authority wording + | (conservative | minimal | team-maintainer; default conservative). + | Set via: bd config set agent.profile team-maintainer + | Or per-session: BD_AGENT_PROFILE=team-maintainer (env var takes precedence). + | See docs/getting-started/ide-setup.md#policy-profiles for what each profile means. + | - prime.disabled: When true, bd prime prints nothing, as it does outside a + | workspace (an empty envelope under --hook-json), whatever its other flags. + | For sessions whose host briefs the agent itself but that inherit a + | project's bd prime hook, such as an orchestrator's agent sessions. + | Set per-session: BD_PRIME_DISABLED=true. Or: bd config set prime.disabled true + | + | Workflow customization: + | - Place a PRIME.md file to override the default workflow text. Checked in this + | order, once bd resolves a workspace (outside one, prime emits no content): + | (1) .beads/PRIME.md relative to the current directory (the -C target + | when -C is set); + | (2) PRIME.md in the .beads directory bd resolves for this workspace + | (honors $BEADS_DIR, which -C overrides; a redirected .beads is + | followed); + | (3) the global PRIME.md in bd's user config dir: + | ~/.config/beads/ on Linux ($XDG_CONFIG_HOME/beads/ if set), + | ~/Library/Application Support/beads/ on macOS, + | %AppData%\beads\ on Windows. + | - Persistent memories (from bd remember) are still appended so memory + | injection keeps working under a custom template. + | - Use --export to dump the default content for customization. + | - Use --memories-only for hook contexts that should inject only persistent memories; this returns only the memories section even when a custom PRIME.md is present. + | - Use --no-memories to omit the persistent memories section (useful when the memories section is large and would dominate a context budget). --memories-only takes precedence if both are set. + | + | Memory injection caps: + | Large memory sets can exceed what a session-start hook host will ingest, + | and hosts truncate silently. Cap what prime injects with --max-memories N + | and/or --max-memory-chars N (or the prime.max-memories / + | prime.max-memory-chars config keys; an explicit flag wins, and an explicit + | 0 forces unlimited). Caps apply at whole-memory boundaries, at least one + | memory is always emitted, and a banner ahead of the entries reports how + | many were elided and how to browse the rest with bd memories. + | --max-memory-chars caps the total bytes of the injected memory entries; + | the section header and elision banner are excluded from the budget. + flag --export type=bool default="false" usage="Output default content (ignores PRIME.md override)" + flag --full type=bool default="false" usage="Force full CLI output (ignore MCP detection)" + flag --hook-json type=bool default="false" usage="Wrap output in the SessionStart hook JSON envelope (Claude Code, Gemini CLI, Codex)" + flag --max-memories type=int default="0" usage="Cap injected persistent memories to N entries (0 = unlimited; falls back to the prime.max-memories config key)" + flag --max-memory-chars type=int default="0" usage="Cap the total bytes of injected memory entries, at whole-memory boundaries; section header and banner are not counted (0 = unlimited; falls back to the prime.max-memory-chars config key)" + flag --mcp type=bool default="false" usage="Force MCP mode (minimal output)" + flag --memories-only type=bool default="false" usage="Output only persistent memories for compact hook contexts" + flag --no-memories type=bool default="false" usage="Omit the persistent memories section (ignored when --memories-only is set, which wins)" + flag --stealth type=bool default="false" usage="Stealth mode (no git operations, flush only)" + +bd priority + use: "priority " + short: "Set the priority of an issue" + group: "issues" + hidden: false + runnable: true + long: + | Set the priority of an issue. + | + | Shorthand for 'bd update --priority '. + | + | Priority levels: + | 0 - Critical (security, data loss, broken builds) + | 1 - High (major features, important bugs) + | 2 - Medium (default) + | 3 - Low (polish, optimization) + | 4 - Backlog (future ideas) + | + | Examples: + | bd priority bd-123 0 # Critical + | bd priority bd-123 2 # Medium + +bd promote + use: "promote " + short: "Promote a wisp to a permanent bead" + group: "issues" + hidden: false + runnable: true + long: + | Promote a wisp (ephemeral issue) to a permanent bead. + | + | This copies the issue from the wisps table (dolt_ignored) to the permanent + | issues table (Dolt-versioned), preserving labels, dependencies, events, and + | comments. The original ID is preserved so all links keep working. + | + | A comment is added recording the promotion and optional reason. + | + | Examples: + | bd promote bd-wisp-abc123 + | bd promote bd-wisp-abc123 --reason "Worth tracking long-term" + flag --reason -r type=string default="" usage="Reason for promotion" + +bd provenance + use: "provenance" + short: "Append-only provenance event log" + group: "issues" + hidden: false + runnable: false + long: + | Record and read provenance events: typed bindings from an issue to an + | opaque external artifact (a git SHA, PR, work-id, transcript, or branch). + | + | The log is append-only — there is no update or delete. bd never interprets the + | actor or ref; only kind and ref-kind are structurally validated. Recording is + | idempotent on a deterministic id, so a producer firing twice is harmless. + +bd provenance by-ref + use: "by-ref " + short: "List provenance events bound to a ref" + hidden: false + runnable: true + +bd provenance log + use: "log " + short: "List provenance events for an issue" + hidden: false + runnable: true + flag --kind type=string default="" usage="filter by kind (optional)" + +bd provenance record + use: "record --issue --kind --source " + short: "Record a provenance event (idempotent)" + hidden: false + runnable: true + long: + | Record a provenance event. The event is appended idempotently: a + | deterministic id is computed from source:issue:kind:(ref or --at), so re-running + | the same record is a no-op. + | + | An event recorded without --ref requires --at so the id is caller-owned. + flag --actor type=string default="" usage="opaque actor identifier (optional)" + flag --at type=string default="" usage="event-time as RFC3339 (required for ref-less kinds)" + flag --issue type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="issue id (required)" + flag --kind type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="event kind: cut|claim|suspend|resume|handoff|commit|land|used (required)" + flag --payload type=string default="" usage="opaque payload, e.g. JSON (optional)" + flag --ref type=string default="" usage="opaque external reference, e.g. a SHA or PR url (optional)" + flag --ref-kind type=string default="" usage="ref kind: git-sha|pr|work-id|transcript|branch (optional)" + flag --source type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="producer of the event, e.g. git-hook, orchestrator (required)" + +bd prune + use: "prune" + short: "Delete old closed beads to reclaim space and shrink exports" + group: "maint" + hidden: false + runnable: true + long: + | Permanently delete closed non-ephemeral beads and their associated data. + | + | Use this to trim closed regular beads (tasks, features, bugs, chores, etc.) + | that are no longer useful. The common case is a long-lived repo where + | closed work has piled up and is bloating auto-export or slowing queries. + | + | Requires --older-than or --pattern. The flag is a safety gate — without + | it, a muscle-memory `--force` could wipe every closed bead in the repo. + | Use `--pattern '*'` if you really do want to sweep everything closed. + | + | Deletes: issues, dependencies, labels, events, and comments for matching beads. + | Skips: pinned beads (protected), open/in-progress beads, and ephemeral beads. + | + | Also skips closed beads whose ID appears in the description, notes, or + | comments of any open / in-progress bead. This protects ADR / decision / + | verification trails that downstream beads still cite. Use + | --ignore-references to override (e.g., when bulk-decommissioning a + | retired label across the rig). + | + | To delete closed ephemeral beads (wisps, transient molecules) use + | `bd purge` instead. + | + | For full Dolt storage reclaim after deleting many rows, follow with `bd flatten` + | so history can be collapsed and old chunks can be garbage-collected. + | + | EXAMPLES: + | bd prune --older-than 30d # Preview closed beads >30d old + | bd prune --older-than 30d --force # Delete them + | bd prune --older-than 90d --dry-run # Detailed preview with stats + | bd prune --pattern "*" --force # Delete all closed regular beads + | bd prune --pattern "gm-temp-*" --force # Scope to a pattern + flag --dry-run type=bool default="false" usage="Preview what would be pruned with stats" + flag --force -f type=bool default="false" usage="Actually prune (without this, shows preview)" + flag --ignore-references type=bool default="false" usage="Delete closed beads even when referenced by open beads (use with care; see --help for details)" + flag --older-than type=string default="" usage="Only prune beads closed more than N ago (e.g., 30d, 2w, 60, 36h)" + flag --pattern type=string default="" usage="Only prune beads matching ID glob pattern (e.g., 'gm-old-*')" + +bd purge + use: "purge" + short: "Delete closed ephemeral beads to reclaim space" + group: "maint" + hidden: false + runnable: true + long: + | Permanently delete closed ephemeral beads and their associated data. + | + | Closed ephemeral beads (wisps, transient molecules) accumulate rapidly and + | have no value once closed. This command removes them to reclaim storage. + | + | Deletes: issues, dependencies, labels, events, and comments for matching beads. + | Skips: pinned beads, and closed beads a live bead still depends on through a + | parent-child, tracks or blocks edge (a closed molecule root whose step is open, + | a closed bead a live convoy tracks). Live means any status that is not done. + | + | --wisps-plane selects by storage plane instead: every closed row stored in the + | wisps table, including --no-history beads, which the default ephemeral + | selection leaves to `bd prune`. It reaches durable-tier rows, so it requires + | --older-than or --pattern, like `bd prune`. + | + | --limit N caps one run at N beads, oldest-closed first, so a large backlog can + | be drained in bounded transactions; --json then reports "remaining" and + | "has_more". Loop while has_more is true. + | + | --older-than takes days (7, 7d), weeks (2w) or a duration with hour precision + | (36h, 168h, 90m). + | + | To delete closed non-ephemeral beads (regular tasks, features, bugs, etc.) + | use `bd prune` instead. + | + | For full Dolt storage reclaim after deleting many rows, follow with `bd flatten` + | so history can be collapsed and old chunks can be garbage-collected. + | + | EXAMPLES: + | bd purge # Preview what would be purged + | bd purge --force # Delete all closed ephemeral beads + | bd purge --older-than 7d --force # Only purge items closed 7+ days ago + | bd purge --pattern "*-wisp-*" # Only purge matching ID pattern + | bd purge --older-than 36h --force # Hour precision: closed 36+ hours ago + | bd purge --wisps-plane --older-than 168h --force + | # Every closed wisps-table row, incl. no-history + | bd purge --dry-run # Detailed preview with stats + flag --dry-run type=bool default="false" usage="Preview what would be purged with stats" + flag --force -f type=bool default="false" usage="Actually purge (without this, shows preview)" + flag --limit type=int default="0" usage="Purge at most N beads this run, oldest-closed first (0 = no limit); --json reports remaining/has_more" + flag --older-than type=string default="" usage="Only purge beads closed more than N ago (e.g., 7d, 2w, 30, 36h)" + flag --pattern type=string default="" usage="Only purge beads matching ID glob pattern (e.g., *-wisp-*)" + flag --wisps-plane type=bool default="false" usage="Select every closed row in the wisps table, including --no-history beads (requires --older-than or --pattern)" + +bd q + use: "q [title]" + short: "Quick capture: create issue and output only ID" + group: "issues" + hidden: false + runnable: true + long: + | Quick capture creates an issue and outputs only the issue ID. + | Designed for scripting and AI agent integration. + | + | Example: + | bd q "Fix login bug" # Outputs: bd-a1b2 + | ISSUE=$(bd q "New feature") # Capture ID in variable + | bd q "Task" | xargs bd show # Pipe to other commands + | bd q "Subtask" --parent=bd-a1b2 # Hierarchical child (outputs: bd-a1b2.1) + flag --labels -l type=stringSlice default="[]" usage="Labels" + flag --parent type=string default="" usage="Parent issue ID for hierarchical child (e.g., 'bd-a3f8e9')" + flag --priority -p type=string default="2" usage="Priority (0-4 or P0-P4)" + flag --type -t type=string default="task" usage="Issue type" + +bd query + use: "query [expression]" + short: "Query issues using a simple query language" + group: "issues" + hidden: false + runnable: true + long: + | Query issues using a simple query language that supports compound filters, + | boolean operators, and date-relative expressions. + | + | The query language enables complex filtering that would otherwise require + | multiple flags or piping through jq. + | + | Syntax: + | field=value Equality comparison + | field!=value Inequality comparison + | field>value Greater than + | field>=value Greater than or equal + | field1" + | bd query "status=open AND priority<=2 AND updated>7d" + | bd query "(status=open OR status=blocked) AND priority<2" + | bd query "type=bug AND label=urgent" + | bd query "NOT status=closed" + | bd query "assignee=none AND type=task" + | bd query "created>30d AND status!=closed" + | bd query "label=frontend OR label=backend" + | bd query "title=authentication AND priority=0" + flag --all -a type=bool default="false" usage="Include closed issues (default: exclude closed)" + flag --limit -n type=int default="50" usage="Limit results (default: 50, 0 = unlimited)" + flag --long type=bool default="false" usage="Show detailed multi-line output for each issue" + flag --offset type=int default="0" usage="Skip the first N matching results (0-based). Only supported under --proxied-server." + flag --parse-only type=bool default="false" usage="Only parse the query and show the AST (for debugging)" + flag --reverse -r type=bool default="false" usage="Invert the sort field's default direction (created/updated/closed default to newest-first, so --sort updated --reverse is oldest-first)" + flag --sort type=string default="" usage="Sort by field: priority, created, updated, closed, status, id, title, type, assignee" + +bd quickstart + use: "quickstart" + short: "Quick start guide for bd" + group: "setup" + hidden: false + runnable: true + long: + | Display a quick start guide showing common bd workflows and patterns. + +bd ready + use: "ready" + short: "Show ready work (open, no active blockers)" + hidden: false + runnable: true + long: + | Show ready work (open issues with no active blockers). + | + | Excludes in_progress, blocked, deferred, and hooked issues. This uses the + | GetReadyWork API which applies blocker-aware semantics to find truly claimable work. + | + | Note: 'bd list --ready' uses the same blocker-aware ready-work semantics. + | + | Use --mol to filter to a specific molecule's steps: + | bd ready --mol bd-patrol # Show ready steps within molecule + | + | Use --gated to find molecules ready for gate-resume dispatch: + | bd ready --gated # Find molecules where a gate closed + | + | Use --claim to atomically claim the first ready issue matching the filters: + | bd ready --claim --json + | + | This is useful for agents executing molecules to see which steps can run next. + flag --assignee -a type=string default="" usage="Filter by assignee" + flag --brief type=bool default="false" usage="Omit the free-form text (description, design, acceptance criteria, notes, payload, waiters) from each row. Filters that read those fields still select on them. An omitted field is indistinguishable from an empty one; fetch a whole issue with bd show. Requires --json, and cannot be combined with --claim, --gated, --mol or --explain." + flag --claim type=bool default="false" usage="Atomically claim the first ready issue matching the filters" + flag --exclude-label type=stringSlice default="[]" usage="Exclude issues that have ANY of these labels" + flag --exclude-type type=stringSlice default="[]" usage="Exclude issue types from results (comma-separated or repeatable, e.g., --exclude-type=convoy,epic)" + flag --explain type=bool default="false" usage="Show dependency-aware reasoning for why issues are ready or blocked" + flag --flat type=bool default="false" usage="Alias for --plain, spelled the way bd list spells it" + flag --gated type=bool default="false" usage="Find molecules ready for gate-resume dispatch" + flag --has-metadata-key type=string default="" usage="Filter issues that have this metadata key set" + flag --include-deferred type=bool default="false" usage="Include issues with future defer_until timestamps" + flag --include-ephemeral type=bool default="false" usage="Include ephemeral issues (wisps) in results" + flag --label -l type=stringSlice default="[]" usage="Filter by labels (AND: must have ALL). Can combine with --label-any" + flag --label-any type=stringSlice default="[]" usage="Filter by labels (OR: must have AT LEAST ONE). Can combine with --label" + flag --label-pattern type=string default="" usage="Filter by label glob pattern (e.g., 'tech-*' matches tech-debt, tech-legacy)" + flag --label-regex type=string default="" usage="Filter by label regex pattern (e.g., 'tech-(debt|legacy)')" + flag --limit -n type=int default="100" usage="Maximum issues to show (use 0 for unlimited)" + flag --max-rows type=int default="0" usage="Hard upper bound on rows fetched from storage. Returns a non-zero exit (code 2) and an error to stderr if exceeded. 0 disables (the default). Overrides BEADS_MAX_ROWS for this invocation. Useful in CI/agent rigs that want a circuit breaker against pathological queries. Not supported under --proxied-server: an explicit --max-rows or BEADS_MAX_ROWS cap errors out rather than silently going unenforced." + flag --metadata-field type=stringArray default="[]" usage="Filter by metadata field (key=value, repeatable)" + flag --mol type=string default="" usage="Filter to steps within a specific molecule" + flag --mol-type type=string default="" usage="Filter by molecule type: swarm, patrol, or work" + flag --offset type=int default="0" usage="Skip the first N matching results (0-based). Only supported under --proxied-server." + flag --parent type=string default="" usage="Filter to descendants of this bead/epic" + flag --plain type=bool default="false" usage="Display issues as a plain numbered list" + flag --pretty type=bool default="true" usage="Display issues in a tree format with status/priority symbols" + flag --priority -p type=int default="0" usage="Filter by priority" + flag --sort -s type=string default="priority" usage="Sort policy: priority (default), hybrid, oldest" + flag --type -t type=string default="" usage="Filter by issue type (task, bug, feature, epic, decision, merge-request). Aliases: mr→merge-request, feat→feature, mol→molecule, dec/adr→decision" + flag --unassigned -u type=bool default="false" usage="Show only unassigned issues" + +bd recall + use: "recall " + short: "Retrieve a specific memory" + group: "setup" + hidden: false + runnable: true + long: + | Retrieve the full content of a memory by its key. + | + | Examples: + | bd recall dolt-phantoms + | bd recall auth-jwt + +bd reclaim + use: "reclaim" + short: "Revert stale-lease in_progress issues back to ready (dead-worker recovery)" + group: "issues" + hidden: false + runnable: true + long: + | Revert in_progress issues whose lease has gone stale back to ready. + | + | When a worker claims an issue it takes a lease that expires after a TTL, kept + | alive by 'bd heartbeat'. A worker that dies stops heartbeating, so its lease + | expires and its issue would otherwise stay in_progress forever. reclaim is the + | reaper: it finds in_progress issues whose lease expired more than --older-than + | ago, clears the assignee, and sets them back to open so another worker can + | claim them. The previous owner's stale lease is recorded as a recovery event. + | + | --older-than is a grace window past lease expiry: only leases that expired at + | least this long ago are reclaimed, so a worker briefly paused (GC, clock skew) + | is not robbed of live work. Run it from a supervisor on a timer with a window + | of roughly 2× the claim TTL. + | + | By default reclaim covers every stale lease THIS replica granted. The scope + | filters below narrow it further, using the same label surface claiming is + | scoped by (--label / --label-any / --exclude-label), plus --assignee and --id. + | Filters AND-combine and never widen the set: a reclaimed lease must still be + | stale. + | + | Replicas and leases (federated deployments) + | ------------------------------------------- + | A lease is only meaningful on the replica that granted it. Every other + | replica's view of the holder's liveness is stale by up to one sync interval, + | so a reaper elsewhere can revert a unit that is very much alive over there. + | reclaim therefore records the granting replica on each lease and SKIPS a lease + | another replica granted, summarizing what it declined on stderr (one line per + | run; 'bd -v' expands it to the first 20 leases individually). Reap it where it + | was granted; use --any-replica only when that replica is permanently gone (or + | when this node was renamed and its own old leases now look foreign — an + | ordinary heartbeat keeps a lease alive but does not re-home it to the node + | heartbeating it). Prefer the narrow form '--any-replica --id ': bare + | --any-replica reverts EVERY foreign stale lease, live peers included. + | + | Two invariants the guard cannot enforce for you: + | + | grace window > sync interval, and lease TTL > sync interval. + | + | A TTL or grace shorter than the cadence at which replicas exchange state is + | meaningless across the bridge — the remote view is a full interval old by + | construction. Raise the TTL/grace above the sync interval, never the reverse. + | The guard is opt-in: set BEADS_NODE_ID, or run 'bd config set node_id ' + | (which writes the per-machine ~/.config/bd/config.yaml — never commit a node_id + | to the git-tracked .beads/config.yaml, or every clone reads the same name and + | the guard goes armed-but-inert). One id per STORE, not per host: machines that + | are clients of the same dolt sql-server are ONE replica and must share one value + | or leave it unset. There is no hostname fallback — the hostname names the client + | process's machine, not the store — so an unnamed deployment keeps the old, + | unguarded behavior instead of stranding its own work. + | + | Examples: + | bd reclaim # default grace window (2× the lease TTL) + | bd reclaim --older-than 10m # reclaim leases expired >10m ago + | bd reclaim --older-than 0s # reclaim every currently-expired lease + | bd reclaim --label lane-a # only this machine's claim partition + | bd reclaim --label-any lane-a,lane-b --exclude-label pinned + | bd reclaim --assignee zelda --assignee epona # only these workers' leases + | bd reclaim --id wy-abc --id wy-def # exactly these issues + | bd reclaim --any-replica # also reap leases granted by a departed replica + flag --any-replica type=bool default="false" usage="Also reclaim leases granted by ANOTHER replica (unsafe unless that replica is gone; see 'Replicas and leases')" + flag --assignee -a type=stringSlice default="[]" usage="Only reclaim leases held by these assignees (repeatable)" + flag --exclude-label type=stringSlice default="[]" usage="Never reclaim issues carrying ANY of these labels" + flag --id type=stringSlice default="[]" usage="Only reclaim these issue IDs (repeatable)" + flag --label -l type=stringSlice default="[]" usage="Only reclaim issues with ALL these labels (AND). Can combine with --label-any" + flag --label-any type=stringSlice default="[]" usage="Only reclaim issues with AT LEAST ONE of these labels (OR). Can combine with --label" + flag --older-than type=duration default="10m0s" usage="Only reclaim leases that expired at least this long ago (grace window)" + +bd recompute-blocked + use: "recompute-blocked" + short: "Recompute is_blocked for all issues (repairs stale flags after a pull)" + group: "maint" + hidden: false + runnable: true + long: + | Recompute the denormalized is_blocked flag for every issue and wisp. + | + | is_blocked is derived from the dependency graph and maintained automatically by + | local writes and by a post-pull recompute scoped to what the merge changed. If + | that scoped recompute is skipped — a recompute that failed after its merge + | committed, or a conflicted pull resolved by hand — the flag can go stale, and a + | later pull that merges nothing will not refresh it (bd-6dnrw.37). 'bd ready' + | trusts the flag, so stale values silently hide ready work or surface blocked + | work. + | + | This command runs the full recompute unconditionally and commits the result. + | It is idempotent: on a consistent database it changes nothing. Works in every + | storage mode — embedded, server, and proxied-server (unlike 'bd doctor', which + | is server-mode only). + | + | On a workspace connected to a bd serve backend ('bd connect') it is a no-op + | that exits 0: the server maintains is_blocked on every write, and the column + | lives in the server's database, so the repair is run there, by the server's + | operator. --json then reports {"rows_corrected": 0, "maintained_by": "server"}. + | + | Examples: + | bd recompute-blocked # Repair stale is_blocked flags + | bd recompute-blocked --json # Machine-parseable {"rows_corrected": N} + +bd remember + use: "remember \"\"" + short: "Store a persistent memory" + group: "setup" + hidden: false + runnable: true + long: + | Store a memory that persists across sessions and account rotations. + | + | Memories are injected at prime time (bd prime) so you have them + | in every session without manual loading. + | + | The positional arg is the memory CONTENT (the key is auto-generated from it + | unless --key is given). As a convenience, if the arg is a bare key naming an + | existing memory, it is RECALLED instead of stored (same as 'bd recall'); + | a bare key naming nothing is refused. Use --key to store slug-like content. + | + | Corpus budget (opt-in, off by default): set the memories.budget-chars config + | key to a byte ceiling for the whole memory corpus (sum of len(key)+len(content)). + | A write that would cross it is refused, and --force overrides; a write that + | lands at or above 80% of the budget warns. With the key unset nothing changes. + | bd prime's --max-memory-chars also counts bytes, but of each memory as + | injected, formatting included, so a budget equal to that cap does not + | guarantee prime injects every memory; set the budget below it for that. + | + | Examples: + | bd remember "always run tests with -race flag" + | bd remember "Dolt phantom DBs hide in three places" --key dolt-phantoms + | bd remember "auth module uses JWT not sessions" --key auth-jwt + | bd remember dolt-phantoms # bare existing key: reads it (= bd recall) + flag --force type=bool default="false" usage="Store even when the write would cross the memories.budget-chars corpus budget (no effect when the budget is unset)" + flag --key type=string default="" usage="Explicit key for the memory (auto-generated from content if not set). If a memory with this key already exists, it will be updated in place" + +bd rename + use: "rename " + short: "Rename an issue ID" + hidden: false + runnable: true + long: + | Rename an issue from one ID to another. + | + | This updates: + | - The issue's primary ID + | - All references in other issues (descriptions, titles, notes, etc.) + | - Dependencies pointing to/from this issue + | - Labels, comments, and events + | + | Examples: + | bd rename bd-w382l bd-dolt # Rename to memorable ID + | bd rename gt-abc123 gt-auth # Use descriptive ID + | + | Note: The new ID must use a valid prefix for this database. + +bd rename-prefix + use: "rename-prefix " + short: "Rename the issue prefix for all issues in the database" + group: "maint" + hidden: false + runnable: true + long: + | Rename the issue prefix for all issues in the database. + | This will update all issue IDs and all text references across all fields. + | + | USE CASES: + | - Shortening long prefixes (e.g., 'knowledge-work-' → 'kw-') + | - Rebranding project naming conventions + | - Consolidating multiple prefixes after database corruption + | - Migrating to team naming standards + | + | Prefix validation rules: + | - Allowed characters: lowercase letters, numbers, hyphens + | - Must start with a letter + | - Must end with a hyphen (e.g., 'kw-', 'work-') + | - Cannot be empty or just a hyphen + | + | Multiple prefix detection and repair: + | If issues have multiple prefixes (corrupted database), use --repair to consolidate them. + | The --repair flag will rename all issues with incorrect prefixes to the new prefix, + | preserving issues that already have the correct prefix. + | + | EXAMPLES: + | bd rename-prefix kw- # Rename from 'knowledge-work-' to 'kw-' + | bd rename-prefix mtg- --repair # Consolidate multiple prefixes into 'mtg-' + | bd rename-prefix team- --dry-run # Preview changes without applying + | + | NOTE: This is a rare operation. Most users never need this command. + flag --dry-run type=bool default="false" usage="Preview changes without applying them" + flag --repair type=bool default="false" usage="Repair database with multiple prefixes by consolidating them" + +bd reopen + use: "reopen [id...]" + short: "Reopen one or more closed issues" + group: "issues" + hidden: false + runnable: true + long: + | Reopen closed issues by setting status to 'open' and clearing the closed_at timestamp. + | This is more explicit than 'bd update --status open' and emits a Reopened event. + | + | Exit codes: 1 for general failures; 13 when the failure is a stale + | --if-revision guard (the precondition no longer held, nothing was written — + | another actor won the race, so retrying the same guard is pointless). + flag --if-revision type=string default="" usage="Apply the write only if the issue's current revision equals this value (the decimal int64 from `bd show --json`'s \"revision\" field). One id only: refused before any write when combined with more than one issue id. A mismatch writes nothing and exits 13 (vs 1 for other failures)." + flag --reason -r type=string default="" usage="Reason for reopening" + +bd repo + use: "repo" + short: "Manage multiple repository configuration" + group: "advanced" + hidden: false + runnable: false + long: + | Configure and manage multiple repository support for multi-repo hydration. + | + | Multi-repo support allows hydrating issues from multiple beads repositories + | into a single database for unified cross-repo issue tracking. + | + | Configuration is stored in .beads/config.yaml under the 'repos' section: + | + | repos: + | primary: "." + | additional: + | - ~/beads-planning + | - ~/work-repo + | + | Examples: + | bd repo add ~/beads-planning # Add planning repo + | bd repo add ../other-repo # Add relative path repo + | bd repo list # Show all configured repos + | bd repo remove ~/beads-planning # Remove by path + | bd repo sync # Sync from all configured repos + +bd repo add + use: "add " + short: "Add an additional repository to sync" + hidden: false + runnable: true + long: + | Add a repository path to the repos.additional list in config.yaml. + | + | The path should point to a directory containing a .beads folder. + | Paths can be absolute or relative (they are stored as-is). + | + | This modifies .beads/config.yaml, which is version-controlled and + | shared across all clones of this repository. + +bd repo list + use: "list" + short: "List all configured repositories" + hidden: false + runnable: true + long: + | List all repositories configured in .beads/config.yaml. + | + | Shows the primary repository (always ".") and any additional + | repositories configured for hydration. + +bd repo remove + use: "remove " + short: "Remove a repository from sync configuration" + hidden: false + runnable: true + long: + | Remove a repository path from the repos.additional list in config.yaml. + | + | The path must exactly match what was added (e.g., if you added "~/foo", + | you must remove "~/foo", not "/home/user/foo"). + | + | This command also removes any previously-hydrated issues from the database + | that came from the removed repository. + +bd repo sync + use: "sync" + short: "Manually trigger multi-repo sync" + hidden: false + runnable: true + long: + | Synchronize issues from all configured additional repositories. + | + | Reads issues.jsonl from each additional repository and imports them into + | the primary database with their original prefixes and source_repo set. + | Uses mtime caching to skip repos whose JSONL hasn't changed. + | + | Also triggers Dolt push/pull if a remote is configured. + flag --verbose type=bool default="false" usage="Show detailed sync progress" + +bd restore + use: "restore " + short: "Restore the pre-compaction content of a compacted issue" + group: "sync" + hidden: false + runnable: true + long: + | Restore the pre-compaction content of a compacted issue. + | + | When an issue is compacted, its description/design/notes/acceptance criteria + | are summarized and the originals are archived to a compaction snapshot. This + | command recovers that original content. + | + | By default it is read-only: it displays the archived content without modifying + | the database. Pass --apply to write the original content back into the issue + | and step its compaction level back down. + | + | If no archived snapshot exists (e.g. the issue was compacted by an older bd + | before snapshot archiving), restore falls back to a best-effort reconstruction + | from Dolt version history, which can only be displayed, not applied. + flag --apply type=bool default="false" usage="Write the restored content back into the issue (default: display only)" + +bd rules + use: "rules" + short: "Audit and compact Claude rules" + group: "maint" + hidden: false + runnable: false + +bd rules audit + use: "audit" + short: "Scan rules for contradictions and merge opportunities" + hidden: false + runnable: true + flag --path type=string default=".claude/rules/" usage="Path to rules directory" + flag --threshold type=float64 default="0.6" usage="Jaccard similarity threshold" + +bd rules compact + use: "compact" + short: "Merge related rules into composites" + hidden: false + runnable: true + flag --auto type=bool default="false" usage="Apply audit suggestions" + flag --dry-run type=bool default="false" usage="Preview without applying" + flag --group type=stringSlice default="[]" usage="Rule names to merge" + flag --path type=string default=".claude/rules/" usage="Path to rules directory" + +bd schema + use: "schema" + short: "Print the JSON Schema for bd's --json / export output" + hidden: false + runnable: true + annotations: {bd:skip_store="1"} + long: + | Print the JSON Schema for bd's canonical output record types. + | + | The schema is reflected from the same Go structs bd serializes, so it stays in + | lockstep with the actual --json / export output. Use it to generate typed + | consumer models (datamodel-code-generator, quicktype, ...) rather than + | hand-maintaining them. + | + | bd schema | jq '.types.issue' # the issue record schema + | bd schema | jq '.types.dependency' # the dependency record schema + +bd search + use: "search [query]" + short: "Search issues by text query" + group: "issues" + hidden: false + runnable: true + long: + | Search issues across title and ID (all statuses, including closed). + | + | ID-like queries (e.g., "bd-123", "hq-319") use fast exact/prefix matching. + | Text queries search titles. Use --desc-contains for description search. + | Use --status open (etc.) to narrow; closed issues are included by default + | so "was this already filed/fixed?" cannot silently answer no. Matches + | beyond --limit are dropped status-blind, so when hunting live work in a + | large DB, narrow with --status open or raise --limit. + | + | Examples: + | bd search "authentication bug" + | bd search "login" --status open + | bd search "database" --label backend --limit 10 + | bd search --query "performance" --assignee alice + | bd search "bd-5q" # Search by partial ID (fast prefix match) + | bd search "security" --priority-min 0 --priority-max 2 + | bd search "bug" --created-after 2025-01-01 + | bd search "refactor" --status open # Only open issues + | bd search "bug" --sort priority + | bd search "task" --sort created --reverse + | bd search "api" --desc-contains "endpoint" + | bd search "cleanup" --no-assignee --no-labels + flag --assignee -a type=string default="" usage="Filter by assignee" + flag --closed-after type=string default="" usage="Filter issues closed after date (YYYY-MM-DD or RFC3339)" + flag --closed-before type=string default="" usage="Filter issues closed before date (YYYY-MM-DD or RFC3339)" + flag --created-after type=string default="" usage="Filter issues created after date (YYYY-MM-DD or RFC3339)" + flag --created-before type=string default="" usage="Filter issues created before date (YYYY-MM-DD or RFC3339)" + flag --desc-contains type=string default="" usage="Filter by description substring (case-insensitive)" + flag --empty-description type=bool default="false" usage="Filter issues with empty or missing description" + flag --external-contains type=string default="" usage="Filter by external ref substring (case-insensitive)" + flag --has-metadata-key type=string default="" usage="Filter issues that have this metadata key set" + flag --label -l type=stringSlice default="[]" usage="Filter by labels (AND: must have ALL)" + flag --label-any type=stringSlice default="[]" usage="Filter by labels (OR: must have AT LEAST ONE)" + flag --limit -n type=int default="50" usage="Limit results (default: 50)" + flag --long type=bool default="false" usage="Show detailed multi-line output for each issue" + flag --metadata-field type=stringArray default="[]" usage="Filter by metadata field (key=value, repeatable)" + flag --no-assignee type=bool default="false" usage="Filter issues with no assignee" + flag --no-labels type=bool default="false" usage="Filter issues with no labels" + flag --notes-contains type=string default="" usage="Filter by notes substring (case-insensitive)" + flag --priority-max type=string default="" usage="Filter by maximum priority (inclusive, 0-4 or P0-P4)" + flag --priority-min type=string default="" usage="Filter by minimum priority (inclusive, 0-4 or P0-P4)" + flag --query type=string default="" usage="Search query (alternative to positional argument)" + flag --reverse -r type=bool default="false" usage="Invert the sort field's default direction (created/updated/closed default to newest-first, so --sort updated --reverse is oldest-first)" + flag --sort type=string default="" usage="Sort by field: priority, created, updated, closed, status, id, title, type, assignee" + flag --status -s type=string default="" usage="Filter by stored status (comma-separated for OR; open, in_progress, blocked, deferred, closed, all). Default searches all statuses including closed. Note: dependency-blocked issues use 'bd blocked'" + flag --type -t type=string default="" usage="Filter by type (bug, feature, task, epic, chore, decision, message, molecule, gate, spike, story, milestone); custom types require types.custom config, and an unknown type is an error. Aliases: feat/enhancement→feature, mol→molecule, dec/adr→decision, mr→merge-request" + flag --updated-after type=string default="" usage="Filter issues updated after date (YYYY-MM-DD or RFC3339)" + flag --updated-before type=string default="" usage="Filter issues updated before date (YYYY-MM-DD or RFC3339)" + +bd send-metrics + use: "send-metrics" + short: "Internal: flush queued telemetry events (spawned by bd)" + hidden: true + runnable: true + +bd serve + use: "serve" + short: "Serve the beads HTTP API over loopback" + hidden: false + runnable: true + long: + | Serve the beads HTTP API — the same work surface the CLI answers, for + | automation clients that would otherwise fork a bd subprocess per call. + | + | The wire contract is described by an OpenAPI document (/v0); GET + | /v0/beads/context reports which operations this build actually implements. + | + | DEPLOYMENT + | + | Pass an explicit port. The default 127.0.0.1:0 takes an ephemeral one, which + | is right for ad-hoc and test use — where the bound address printed on stdout + | is read immediately — but carries no mutual exclusion: two serves against one + | workspace then run side by side on different ports with no way to enumerate + | them. On a fixed port the second one fails to bind, which is the intended + | behavior. Concurrent serves are data-safe either way; claims are arbitrated + | in the SQL server. + | + | Run it under a supervisor. bd shuts down gracefully on SIGHUP as well as + | SIGINT and SIGTERM, so closing the terminal of a foreground bd serve stops it. + | + | PROBES + | + | GET /healthz is LIVENESS only: it answers from the process and never touches + | the database, so it stays green while the database is unreachable. For + | readiness use GET /v0/beads/ready?limit=1 — a real query, where 200 means + | ready and 503 means live but not ready. + | + | AUTHENTICATION + | + | Optional, and off by default on loopback — where the trust model is the + | loopback boundary itself, the same one the database behind it already relies + | on. --auth-token-file turns it on: every operation except GET /healthz then + | requires an "Authorization: Bearer " header, GET /v0/beads/context + | included, because it reports the repo root, beads directory and database name. + | + | The file holds ONE TOKEN PER LINE and every line is accepted. That is the + | rotation mechanism: write the new token alongside the old, roll the clients + | over, then delete the old line. The file is re-read while the server runs, so + | both the addition and the removal take effect within about a second and + | neither needs a restart. Write it atomically (a temp file plus rename; a + | Kubernetes secret mount already does this). + | + | There is deliberately no --auth-token flag. A credential passed as an + | argument is readable by every local user in the process listing. + | + | --allow-non-loopback REQUIRES a token file. Beyond loopback, reaching the + | address would otherwise be the whole authorization: any peer could read every + | issue and claim work as any actor. --insecure-no-auth is the explicit, + | auditable way to say you meant that anyway. + | + | The Host allowlist is what a service deployment usually trips over first. The + | DNS-rebinding check answers only to loopback spellings and the bind address, + | so a client dialing a service name gets 400 on every request; enumerate the + | names it dials with --allowed-host (repeatable). Matching is exact — no + | wildcards — and the startup log line prints the effective allowlist. + | + | WHAT THIS DOES NOT DO + | + | No TLS. Even with a token, the credential and every issue body travel in + | plaintext, so a deployment beyond loopback has to supply confidentiality + | itself — a service mesh, or a network boundary you already trust. + | + | Hooks do not fire. A hook is a user-controlled subprocess per mutation: in a + | concurrent server that is an unbounded latency multiplier and an orphaned + | child at shutdown, and its working-directory-derived hook lookup is + | meaningless in a server process. A CLI claim runs on_update; an HTTP claim + | does not. + | + | The per-command auto-commit machinery does not run. Durability is per request: + | a successful claim commits inside its own transaction, exactly as a proxied + | CLI claim does today. + | + | An actor on an HTTP request is caller-asserted provenance for the audit trail, + | not authenticated identity — the same thing it has always been on the CLI, + | where any local process can pass any --actor. + | + | It does not run under --readonly, and refuses to start rather than binding. + | Every server it binds publishes the issue-claim operation, and the capability + | set it advertises is a property of the build rather than of the flags on the + | process that started it — so a read-only server would advertise a write it + | could never land. + | + | DESTRUCTIVE OPERATIONS + | + | POST /v0/beads/issues:sweep deletes closed beads in bulk — the operation + | behind bd purge and bd prune — and nothing it deletes comes back. It shares + | the library surface those commands call, so it inherits their guards: pinned + | beads are never swept, and a durable sweep with neither a cutoff nor an id + | pattern is refused rather than clearing every closed bead in the workspace. + | Combined with the trust model above, that means anyone who can reach this + | address can erase closed work; bind it accordingly. + flag --addr type=string default="127.0.0.1:0" usage="Address to bind as IP:PORT; the host must be a numeric IP literal, and port 0 takes an ephemeral port" + flag --allow-non-loopback type=bool default="false" usage="Permit a bind beyond loopback. Requires --auth-token-file, since reaching the address would otherwise be the whole authorization" + flag --allowed-host type=stringArray default="[]" usage="Additional Host header value to answer to, e.g. a service DNS name. Repeatable; matched exactly, with no wildcards" + flag --auth-token-file type=string default="" usage="Require an Authorization: Bearer token from this file, one token per line, all accepted. Re-read while running, so rewriting it rotates tokens without a restart (env BEADS_SERVE_TOKEN_FILE)" + flag --insecure-no-auth type=bool default="false" usage="Serve a non-loopback bind with NO authentication. Every peer that can reach the address gets full read and claim access" + flag --large-apply-ceiling type=duration default="5m0s" usage="Whole-run budget for a POST issues:batchApply request over 100 items (a ceiling, not a target; requests at or under 100 items are unaffected). Must be positive. The orchestrator's stop grace must be at least this long plus 5s, or an external SIGKILL can cut off an in-flight large apply a graceful drain would otherwise have waited out; see engdocs/SERVE_RUNBOOK.md" + +bd set-state + use: "set-state =" + short: "Set operational state (creates event + updates label)" + group: "issues" + hidden: false + runnable: true + long: + | Atomically set operational state on an issue. + | + | This command: + | 1. Creates an event bead recording the state change (source of truth) + | 2. Removes any existing label for the dimension + | 3. Adds the new dimension:value label (fast lookup cache) + | + | State labels follow the convention :, for example: + | patrol:active, patrol:muted + | mode:normal, mode:degraded + | health:healthy, health:failing + | + | Examples: + | bd set-state agent-abc patrol=muted --reason "Investigating stuck worker" + | bd set-state agent-abc mode=degraded --reason "High error rate detected" + | bd set-state agent-abc health=healthy + | + | The --reason flag provides context for the event bead (recommended). + flag --reason type=string default="" usage="Reason for the state change (recorded in event)" + +bd setup + use: "setup [recipe]" + short: "Setup integration with AI editors" + group: "setup" + hidden: false + runnable: true + long: + | Setup integration files for AI editors and coding assistants. + | + | Recipes define where beads workflow instructions are written. Built-in recipes + | include cursor, claude, copilot, gemini, aider, factory, codex, mux, opencode, junie, kiro, windsurf, cody, and kilocode. + | + | Examples: + | bd setup cursor # Install Cursor IDE integration (rules + agent hooks) + | bd setup cursor --global # Install global Cursor hooks (~/.cursor/hooks.json) + | bd setup kiro # Install Kiro steering guidance + | bd setup codex # Install Codex skill + AGENTS.md guidance + native hooks + | bd setup codex --global # Install global Codex skill + guidance + native hooks + | bd setup copilot # Install Copilot CLI plugin + repository instructions + | bd setup mux --project # Install Mux workspace layer (.mux/AGENTS.md) + | bd setup mux --global # Install Mux global layer (~/.mux/AGENTS.md) + | bd setup mux --project --global # Install both Mux layers + | bd setup --list # Show all available recipes + | bd setup --print # Print the template to stdout + | bd setup -o rules.md # Write template to custom path + | bd setup --add myeditor .myeditor/rules.md # Add custom recipe + | + | Use 'bd setup --check' to verify installation status. + | Use 'bd setup --remove' to uninstall. + flag --add type=string default="" usage="Add a custom recipe with given name" + flag --check type=bool default="false" usage="Check if integration is installed" + flag --global type=bool default="false" usage="Install globally (claude/codex/cursor/mux; writes to ~/.claude/settings.json, $CODEX_HOME/AGENTS.md or ~/.codex/AGENTS.md, ~/.cursor/hooks.json, or ~/.mux/AGENTS.md)" + flag --list type=bool default="false" usage="List all available recipes" + flag --output -o type=string default="" usage="Write template to custom path" + flag --print type=bool default="false" usage="Print the template to stdout" + flag --project type=bool default="false" usage="Install for this project only (gemini/mux)" + flag --remove type=bool default="false" usage="Remove the integration" + flag --stealth type=bool default="false" usage="Use stealth mode (claude/gemini)" + +bd ship + use: "ship " + short: "Publish a capability for cross-project dependencies" + hidden: false + runnable: true + long: + | Ship a capability to satisfy cross-project dependencies. + | + | This command: + | 1. Finds issue with export: label + | 2. Validates issue is closed (or --force to override) + | 3. Adds provides: label + | + | External projects can depend on this capability using: + | bd dep add external:: + | + | The capability is resolved when the external project has a closed issue + | with the provides: label. + | + | Examples: + | bd ship mol-run-assignee # Ship the mol-run-assignee capability + | bd ship mol-run-assignee --force # Ship even if issue is not closed + | bd ship mol-run-assignee --dry-run # Preview without making changes + flag --dry-run type=bool default="false" usage="Preview without making changes" + flag --force type=bool default="false" usage="Ship even if issue is not closed" + +bd show + use: "show [id...] [--id=...] [--current]" + aliases: ["view"] + short: "Show issue details" + group: "issues" + hidden: false + runnable: true + flag --as-of type=string default="" usage="Show issue as it existed at a specific commit hash or branch (requires Dolt; commit hashes must be the full 32-character hash from 'bd history', not a prefix)" + flag --brief-deps type=bool default="false" usage="Reduce each dependency to its identity fields in JSON output (--json only; drops description, design, notes and acceptance criteria)" + flag --children type=bool default="false" usage="Show only the children of this issue" + flag --comments-tail type=int default="0" usage="Render only the last N comments in text output, preceded by one elision line naming how many older ones were hidden (text output only; 0, the default, renders every comment unchanged; N must be >= 0)" + flag --current type=bool default="false" usage="Show the currently active issue (in-progress, hooked, or last touched)" + flag --id type=stringArray default="[]" usage="Issue ID (use for IDs that look like flags, e.g., --id=gt--xyz)" + flag --include-comments type=bool default="false" usage="Stream full comment bodies in JSON output (--json only; may be slow on issues with many comments). Without it the comments field is absent entirely, not truncated: an issue with comments carries comment_count and comments_omitted=true, so check those before reading an absent comments field as none." + flag --include-dependents type=bool default="false" usage="Stream full dependent issues in JSON output (--json only; may be slow on hub beads)" + flag --local-time type=bool default="false" usage="Show timestamps in local time instead of UTC (date-only fields are always local)" + flag --long type=bool default="false" usage="Show all available fields (extended metadata, agent identity, gate fields, etc.)" + flag --refs type=bool default="false" usage="Show issues that reference this issue (reverse lookup)" + flag --short type=bool default="false" usage="Show compact one-line output per issue" + flag --thread type=bool default="false" usage="Show full conversation thread (for messages)" + flag --watch -w type=bool default="false" usage="Watch for changes and auto-refresh display" + +bd sql + use: "sql " + short: "Execute raw SQL against the beads database" + group: "maint" + hidden: false + runnable: true + long: + | Execute a raw SQL query against the underlying database (Dolt). + | + | Useful for debugging, maintenance, and working around bugs in higher-level commands. + | + | Examples: + | bd sql 'SELECT COUNT(*) FROM issues' + | bd sql 'SELECT id, title FROM issues WHERE status = "open" LIMIT 5' + | bd sql 'DELETE FROM dirty_issues WHERE issue_id = "bd-abc123"' + | bd sql --csv 'SELECT id, title, status FROM issues' + | + | The query is passed directly to the database. Reads (SELECT, WITH ... SELECT, + | SHOW, EXPLAIN) return results as a table (or JSON/CSV with --json/--csv). Writes + | (INSERT, UPDATE, DELETE, DDL) report the number of rows affected. Statements + | that may both write and return rows, such as CALL, are committed and print any + | rows they return (or "OK"). + | + | In proxied-server mode, multiple statements separated by ';' run as a single + | committed batch and report "OK", and --database runs the query against a + | different server database (equivalent to a session USE) without changing the + | project's configured database. + | + | WARNING: Direct database access bypasses the storage layer. Use with caution. + flag --csv type=bool default="false" usage="Output results in CSV format" + +bd stale + use: "stale" + short: "Show stale issues (not updated recently)" + group: "views" + hidden: false + runnable: true + long: + | Show issues that haven't been updated recently and may need attention. + | This helps identify: + | - In-progress issues with no recent activity (may be abandoned) + | - Open issues that have been forgotten + | - Issues that might be outdated or no longer relevant + flag --days -d type=int default="30" usage="Issues not updated in this many days" + flag --exclude-label type=stringSlice default="[]" usage="Exclude issues that have ANY of these labels" + flag --label -l type=stringSlice default="[]" usage="Filter by labels (AND: must have ALL). Can combine with --label-any" + flag --label-any type=stringSlice default="[]" usage="Filter by labels (OR: must have AT LEAST ONE). Can combine with --label" + flag --limit -n type=int default="50" usage="Maximum issues to show" + flag --status -s type=string default="" usage="Filter by status (open|in_progress|blocked|deferred)" + +bd state + use: "state " + short: "Query the current value of a state dimension" + group: "issues" + hidden: false + runnable: true + long: + | Query the current value of a state dimension from an issue's labels. + | + | State labels follow the convention :, for example: + | patrol:active + | mode:degraded + | health:healthy + | + | This command extracts the value for a given dimension. + | + | Examples: + | bd state witness-abc patrol # Output: active + | bd state witness-abc mode # Output: normal + | bd state witness-abc health # Output: healthy + +bd state list + use: "list " + short: "List all state dimensions on an issue" + hidden: false + runnable: true + long: + | List all state labels (dimension:value format) on an issue. + | + | This filters labels to only show those following the state convention. + | + | Example: + | bd state list witness-abc + | # Output: + | # patrol: active + | # mode: normal + | # health: healthy + +bd status + use: "status" + aliases: ["stats"] + short: "Show issue database overview and statistics" + group: "views" + hidden: false + runnable: true + long: + | Show a quick snapshot of the issue database state and statistics. + | + | This command provides a summary of issue counts by state (open, in_progress, + | blocked, closed), ready work, extended statistics (pinned issues, + | average lead time), and recent activity over the last 24 hours from git history. + | + | Similar to how 'git status' shows working tree state, 'bd status' gives you + | a quick overview of your issue database without needing multiple queries. + | + | Use cases: + | - Quick project health check + | - Onboarding for new contributors + | - Integration with shell prompts or CI/CD + | - Daily standup reference + | - Fast CI status checks that don't need blocked-count accuracy + | + | Examples: + | bd status # Show summary with activity + | bd status --no-activity # Skip git activity (faster) + | bd status --no-blocked # Skip slow blocked-count scan (faster) + | bd stats --no-blocked --json # JSON output without blocked count + | bd status --json # JSON format output + | bd status --assigned # Show issues assigned to current user + | bd stats # Alias for bd status + flag --all type=bool default="false" usage="Show all issues (default behavior)" + flag --assigned type=bool default="false" usage="Show issues assigned to current user" + flag --no-activity type=bool default="false" usage="Skip git activity summary (faster)" + flag --no-blocked type=bool default="false" usage="Skip blocked-count computation (faster on large rigs; not supported in proxied-server mode)" + +bd statuses + use: "statuses" + short: "List valid issue statuses" + group: "views" + hidden: false + runnable: true + long: + | List all valid issue statuses and their categories. + | + | Built-in statuses (open, in_progress, blocked, etc.) are always valid. + | Additional statuses can be configured via status.custom: + | + | bd config set status.custom "in_review:active,qa_testing:wip,on_hold:frozen" + | + | Categories control behavior: + | active — appears in 'bd ready' and default 'bd list' + | wip — excluded from 'bd ready', visible in default 'bd list' + | done — excluded from 'bd ready' and default 'bd list' + | frozen — excluded from 'bd ready' and default 'bd list' + | + | Statuses without a category (legacy format) are valid but excluded from 'bd ready'. + | + | Examples: + | bd statuses # List all statuses with icons and categories + | bd statuses --json # Output as JSON + | + +bd supersede + use: "supersede --with " + short: "Mark an issue as superseded by a replacement issue" + group: "deps" + hidden: false + runnable: true + long: + | Mark an issue as superseded by a different (replacement) issue. + | + | The superseded issue is automatically closed with a reference to the replacement. + | Useful for design docs, specs, and evolving artifacts. + | + | Examples: + | bd supersede bd-old --with bd-new # Mark bd-old as superseded by bd-new + flag --with type=string default="" annotations={cobra_annotation_bash_completion_one_required_flag=["true"]} usage="Replacement issue ID (required)" + +bd swarm + use: "swarm" + short: "Swarm management for structured epics" + group: "deps" + hidden: false + runnable: false + long: + | Swarm management commands for coordinating parallel work on epics. + | + | A swarm is a structured body of work defined by an epic and its children, + | with dependencies forming a DAG (directed acyclic graph) of work. + +bd swarm create + use: "create [epic-id]" + short: "Create a swarm molecule from an epic" + hidden: false + runnable: true + long: + | Create a swarm molecule to orchestrate parallel work on an epic. + | + | The swarm molecule: + | - Links to the epic it orchestrates + | - Has mol_type=swarm for discovery + | - Specifies a coordinator (optional) + | - Can be picked up by any coordinator agent + | + | If given a single issue (not an epic), it will be auto-wrapped: + | - Creates an epic with that issue as its only child + | - Then creates the swarm molecule for that epic + | + | Examples: + | bd swarm create bd-epic-123 # Create swarm for epic + | bd swarm create bd-epic-123 --coordinator=observer/ # With specific coordinator + | bd swarm create bd-task-456 # Auto-wrap single issue + flag --coordinator type=string default="" usage="Coordinator address (e.g., my-project/witness)" + flag --force type=bool default="false" usage="Create new swarm even if one already exists" + +bd swarm list + use: "list" + short: "List all swarm molecules" + hidden: false + runnable: true + long: + | List all swarm molecules with their status. + | + | Shows each swarm molecule with: + | - Progress (completed/total issues) + | - Active workers + | - Epic ID and title + | + | Examples: + | bd swarm list # List all swarms + | bd swarm list --json # Machine-readable output + +bd swarm status + use: "status [epic-or-swarm-id]" + short: "Show current swarm status" + hidden: false + runnable: true + long: + | Show the current status of a swarm, computed from beads. + | + | Accepts either: + | - An epic ID (shows status for that epic's children) + | - A swarm molecule ID (follows the link to find the epic) + | + | Displays issues grouped by state: + | - Completed: Closed issues + | - Active: Issues currently in_progress (with assignee) + | - Ready: Open issues with all dependencies satisfied + | - Blocked: Open issues waiting on dependencies + | + | The status is COMPUTED from beads, not stored separately. + | If beads changes, status changes. + | + | Examples: + | bd swarm status gt-epic-123 # Show swarm status by epic + | bd swarm status gt-swarm-456 # Show status via swarm molecule + | bd swarm status gt-epic-123 --json # Machine-readable output + +bd swarm validate + use: "validate [epic-id]" + short: "Validate epic structure for swarming" + hidden: false + runnable: true + long: + | Validate an epic's structure to ensure it's ready for swarm execution. + | + | Checks for: + | - Correct dependency direction (requirement-based, not temporal) + | - Orphaned issues (roots with no dependents) + | - Missing dependencies (leaves that should depend on something) + | - Cycles (impossible to resolve) + | - Disconnected subgraphs + | + | Reports: + | - Ready fronts (waves of parallel work) + | - Estimated worker-sessions + | - Maximum parallelism + | - Warnings for potential issues + | + | Examples: + | bd swarm validate gt-epic-123 # Validate epic structure + | bd swarm validate gt-epic-123 --verbose # Include detailed issue graph + flag --verbose type=bool default="false" usage="Include detailed issue graph in output" + +bd sync + use: "sync" + short: "Pull, check for conflicts, repair is_blocked, and push (the federation loop)" + group: "sync" + hidden: false + runnable: true + long: + | Run one full synchronization cycle against the Dolt remote. + | + | This is the loop every multi-machine beads deployment otherwise hand-rolls in + | shell: + | + | 1. pull from the remote + | 2. check for merge conflicts POSITIVELY, from the merge's own conflict rows + | and from Dolt's conflict tables — never inferred from the pull's exit + | status, which is not a trustworthy conflict signal in either direction + | 3. recompute the denormalized is_blocked flag, so dependency edges merged in + | from another replica do not leave 'bd ready' stale + | 4. push, retrying a bounded number of times when another replica wins the + | push race + | + | The repair in step 3 refuses to run while another writer has uncommitted changes + | to issues/dependencies. That is transient and not this sync's doing, so it is + | retried on the same budget as a push race rather than failing the run. A working + | set that is NOT transient exits 4 instead, because no amount of retrying will + | ever publish and only an operator can clear it. Two kinds of evidence say so: + | constraint violations on the dirty tables are detected positively and escalate + | on the very attempt that finds them; an abandoned uncommitted edit has no such + | positive signal, so it is only inferred once the same pending graph edits have + | blocked every attempt of several consecutive runs. + | + | Conflicts sync cannot resolve safely are NEVER auto-resolved: it halts before + | recomputing or pushing and exits 2, and repeated runs keep halting the same way + | until an operator resolves the divergence. (The pull underneath does auto-settle + | the conflict classes it can settle convergently — machine-local metadata, + | audit-only dependency rows, and last-write-wins on issue cells. Anything beyond + | those halts here.) Whether the halted merge was aborted or left live in the + | working set depends on the pull route, so the halt message reports which. + | + | Exit codes (a sync timer can branch on these without parsing output): + | + | 0 synced, or nothing to do + | 1 error (transport, auth, storage) + | 2 merge conflict — halted, nothing pushed, resolve it by hand + | 3 retries exhausted (push race, or a concurrent writer's dirty working set) + | — transient, nothing pushed, retry on the next tick + | 4 the dirty working set is stuck, not busy: identical pending graph edits + | blocked every attempt of several consecutive runs — nothing pushed, and no + | later tick will publish until an operator clears it + | + | On the default-remote path, a rig with no Dolt remote yet but a git origin + | configured adopts that origin as its Dolt remote first, exactly as 'bd dolt push' + | does — so 'bd sync' works as a first-time federation bring-up step instead of + | reporting 'no remote' and doing nothing. Passing --remote never adopts anything. + | + | This is not 'bd federation sync', which syncs with named peer towns and takes a + | --strategy ours|theirs to resolve whatever conflicts it meets. 'bd sync' targets + | the configured remote and has no such switch: what it cannot settle, it halts on. + | + | Examples: + | bd sync # sync with the default remote + | bd sync --remote mini # sync with a specific remote + | bd sync --attempts 5 # allow more push-race retries + | bd sync --json # machine-parseable outcome + flag --attempts type=int default="3" usage="Maximum pull/push attempts before reporting a transient retry exhaustion (exit 3)" + flag --no-adopt type=bool default="false" usage="Never derive a Dolt remote from git origin (also BD_NO_REMOTE_ADOPT=1)" + flag --remote type=string default="" usage="Sync with a specific named remote instead of the default" + flag --yes -y type=bool default="false" usage="Consent to adopting a Dolt remote derived from git origin when none is configured" + +bd tag + use: "tag