Commit graph

9 commits

Author SHA1 Message Date
Julian Knutsen
7e7dce8177
fix(storage): stop reporting a broken wisp plane as an empty one in search (re-land of #5805) (#5932)
* fix(storage): stop reporting a broken wisp plane as an empty one in search

A wisp search folded any table-not-exist error into "there are no wisps". The
tolerance is there for pre-migration databases, which have no wisp plane at
all, but the wisp query reads more tables than the wisp plane owns: its FROM
clause carries sqlbuild.LeaseJoin and its hydration reads wisp_labels. So a
database that has wisps but has lost wisp_labels is reported as having no wisps
-- with a nil error, from an ordinary list.

Reproduced end-to-end against embedded Dolt, with the row present in the same
transaction that is told it is not:

  RENAME TABLE wisp_labels TO wisp_labels_renamed_away
  SearchIssues(ephemeral)  before: 1 row   after: 0 rows, nil error
  SearchIssues(no filter)  before: 2 rows  after: 1 row,  nil error

GetIssueInTx already surfaces a missing wisp_labels as an error (pinned by the
existing embedded-Dolt wisp_label_table_error_propagates case), so the two read
paths disagreed today about whether that database is broken or empty.

Narrow the tolerance to the tables the wisp plane may legitimately not have.
sqlbuild.OptionalWispTable becomes the one place that set is written down --
issueops.optionalBlockedTable now reads it, so the classic and domain/db stacks
cannot drift on what a missing table means, which is the same reason the query
text itself lives in sqlbuild. The check is gated on the existing
IsTableNotExist classification via dberrors.MissingTableName, so nothing that
was previously an error becomes tolerated.

The leases half of the same over-broad check is included for the same reason
but is not, on its own, a reachable wrong answer today: the ephemeral branch
falls through to an issues-plane query over the same join, which re-raises the
missing table, so the caller does get an error. That was verified rather than
assumed, and the tests pin it at the guard rather than claiming a user-visible
repro.

Tests are red-first: sqlmock cases on both stacks and an end-to-end
RENAME TABLE against real embedded Dolt. Both directions have controls -- a
pre-migration database missing wisps must still search to an empty result, and
a missing wisp_dependencies must stay invisible -- because over-tightening to
"never tolerate" passes every must-error assertion and only those controls
catch it.

* test(storage): cover the union hydrate-wisps guard, which nothing reached

The review council mutated all four guarded call sites and found one that
no test in the repository reaches: the domain/db searchUnion hydrate-wisps
check. Reverting it to the old blanket table-not-exist test survived the
full domain tree and the nine-minute uow contract suite with embedded Dolt
enabled.

It is the site that matters most. The ephemeral branch needs an explicit
filter; the union is what a plain `bd list` runs. And the union is exactly
where the two stacks diverge: the leg's FROM clause is plan.FromSQL, with
no lease join and no label read, so a database missing leases or
wisp_labels answers the union query successfully and returns wisp ids --
the breakage surfaces only in fetchIssuesByIDs, which adds
sqlbuild.LeaseJoin and hydrates labels. Tolerating it there sends every
wisp through reassembleBySrc as a miss and drops it behind a nil error.

Also adds the fail-open control the domain/db file was missing. The
council's other surviving mutant made the guard return true for any error
it could not parse a table name out of; issueops caught it, domain/db was
green across the whole package.

Red before green (real edits, restored by trap):

  union guard -> blanket IsTableNotExist   TestSearchUnionBrokenWispPlane
                                           IsAnError, both subtests; every
                                           other test stays green
  fail open on unparseable errors          both UnrelatedErrorPropagates
                                           tests, ephemeral and union

The e2e wisp_dependencies control keeps its name but loses its overclaim:
a search that never asks for dependencies cannot see that table missing,
so it stays green whatever the tolerance set is. The sqlmock control is
what pins that entry; this one proves the query still answers end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 16:58:43 -07:00
Stephanie Jarmak
7e825441b4
feat(workapi,storage): opt-in Brief projection for the list and ready pages (#5554)
STORAGE LAYER ONLY, per CONTRIBUTING_PR_GUIDELINES.md. issueops.ListRequest
and issueops.ReadyRequest gain Brief; no CLI flag and no HTTP parameter sets
it yet, so no front door's payload changes. The wiring stacks on top.

WHAT IT DROPS is the free-form text: description, design, acceptance_criteria,
notes, payload and waiters are not selected, and the row comes back carrying
types.Issue.IsLitePartial. #5546 measured a 16,051-issue store where
`bd ready --json` returned 454,021 bytes and notes plus description were 76% of
it, on the call an agent makes to decide what to work on. Measured through
issueops.Reader on 100 rows sized from those per-row ratios: 362,301 bytes
full, 24,001 brief, a 93.4% reduction.

THE PRIMITIVE WAS HALF-BUILT AND UNREACHABLE. #3906 landed
types.IssueFilter.Lite, IssueSelectColumnsLite, HeavyDropList,
ScanIssueLiteFrom and Issue.IsLitePartial, then wired filter.Lite into exactly
one place: SearchIssuesInTx, the UNCOUNTED store-backed search. Every read that
answers with IssueWithCounts, which is every --json route of both commands,
both implementations of issueops.Reader, and both HTTP reads, runs the counts
mega-query instead and ignored the field. A request that set it would have come
back correct, fully hydrated, and with nothing on it to say the projection had
not happened. That silent no-op is what this closes.

THE APPLICATION POINT IS THE COUNTS MEGA-QUERY, and it is one point rather than
four because sqlbuild.SearchCountsSQL and issueops.ScanReadyWorkRowWithCounts
are already the pair every counted read passes through: the store seam's list
(search_counts.go) and ready (ready_work_counts.go), and the unit-of-work
seam's list (issue_search_counts.go) and ready (ready_work_union.go, via
fetchCountsByIDs). sqlbuild.CountsHydration is already the value that carries
hydration opt-outs across that pair on both seams, so Lite is a third field on
it rather than a new channel.

IT CHANGES THE PROJECTION'S SHAPE, which SkipLabels and SkipCounts do not, and
that is why ScanReadyWorkRowWithCounts now takes the whole hydration instead of
a bool: the value that chose the SELECT list is the value that drives the scan,
so the two cannot be given different answers. Substituting empty strings in the
heavy positions would have kept one scan path and was rejected, because the row
would carry blank text with IsLitePartial=false, which is indistinguishable
from a genuinely textless row, the ambiguity ga-clgh and CommentsOmitted
already record.

WHERE IS INDEPENDENT OF SELECT, so this bounds the size of a row and never
which rows match: DescContains, NotesContains and EmptyDesc keep selecting
exactly what they select today, and no sort key, keyset cursor or wisp-merge
dedup reads a dropped column (CompareIssuesBy orders on nine fields, none of
them heavy).

THE CLAIM REFUSES IT, which is not the same call the page makes and is the
sharper case of the rule Limit and Offset already state. A claim does not read
its row through the page's query at all: ExecuteClaimNext refetches the winning
row whole with GetIssueInTx and hydrates its counts itself, so the projection
has nothing to apply to. Accepting the field would answer a MUTATING request
with a fully hydrated row carrying IsLitePartial=false, the caller's one signal
that it did not get what it asked for, reporting that it did. The check lives
in issueops.ValidateClaimNextRequest, which is the shared rule both ready
claimers and BatchCloser's claim already run, so all four backends refuse it on
the same line.

BuildReadyCountFilter CARRIES it, which is the opposite treatment and the
opposite reason. A count reads no field of any row, so the projection cannot
make the number wrong, and the count is not always cheap enough for that to be
the end of it: the unit-of-work seam has no COUNT(*) over the ready predicate
and sizes the set by running the unbounded page and taking its length
(uow/ready_counter.go), so dropping the field there would hydrate every heavy
column of the whole ready set to answer `bd ready --brief`, a larger read than
the page it was asked for. Carrying it also keeps that builder what its doc
says it is: the listing's filter with the PAGE removed and nothing else.

TWO ASYMMETRIES on the page path, both deliberate and both stated in code:

  - Brief IS carried onto the --ready arm (ReadyFilterFromIssueFilter) where
    SkipLabels and SkipCounts are dropped. Those two hide a number the ready
    renderings print; this drops a body no listing prints, so carrying it is
    what makes `bd list --ready` and `bd ready` answer one request one way.
  - types.WorkFilter carries Lite and neither of the other two, so the ready
    paths get readyHydrationFor rather than hydrationFor. Before this, both
    uow ready call sites passed a hardcoded CountsHydration{}, a literal that
    could not carry a filter's answer at all, which is how that path would have
    been missed.

CONFORMANCE at all four registered sites, one Test per case, for the two page
cases and for the claim refusal. Every field the cases assert stripped is
seeded heavy first and the premise is checked, so a backend that never
round-trips one of the six fails loudly instead of turning that clause into one
that cannot fail. Identity is asserted too (ID, Title, Status, IssueType,
Priority), or a body that stripped the row to its id would satisfy every
emptiness clause. IsLitePartial is asserted in both directions.

THE LITE COLUMN LIST IS PINNED BY ORDER, not only by membership. Both lists are
scanned positionally, so two columns of the same SQL type can be transposed
with no membership change at all: the pre-existing set-based parity test stays
green, the scan succeeds, and every row silently carries one column's value in
another's field. TestIssueSelectColumnsLite_IsFullMinusHeavyInOrder derives the
expected list by deleting HeavyDropList from IssueSelectColumns in place, which
makes the lite list a derivation rather than a second hand-maintained copy.

RED BEFORE GREEN, five sabotages, each verified and reverted:

  - deleting the lite branch from CountsHydration.IssueColumns fails both page
    cases with "expected 58 destination arguments in Scan, not 52";
  - accepting Brief without plumbing it to either filter, the silent no-op this
    change exists to close, fails both page cases on all six text fields and on
    IsLitePartial, which is the run that proves the assertions are
    load-bearing;
  - transposing created_by and owner in IssueSelectColumnsLite fails only the
    new order test, on IssueSelectColumnsLite[9] = owner, want created_by,
    while the set-based test stays green, which is the gap it was added for;
  - dropping the claim's refusal fails the claimer contract with "ClaimNext
    with brief: error = <nil>, want ErrValidation"; and
  - clearing Brief in BuildReadyCountFilter fails
    TestBuildReadyCountFilterIsTheUnboundedReadyFilter, which already existed to
    catch a ReadyRequest field the count builder forgets to carry and gains
    Brief in its fixture here.

The recorded ready-filter golden gains "Lite": false in 62 cases because
types.WorkFilter gained the field and the test compares marshaled JSON
byte-for-byte. False is what a request that did not ask for the projection
resolves that field to, so the replay stays exact; the test comment says so.

Gates: go build ./..., go vet ./..., gofmt, internal/storage/... (dolt,
embeddeddolt with BEADS_TEST_EMBEDDED_DOLT=1, uow, issueops, sqlbuild,
domain/db), internal/workapi/..., backend/conformance/..., issueops/...,
make api-check.

Refs: #5546, #3906

Co-authored-by: sjarmak <t@t.co>
2026-08-10 10:45:23 -07:00
Julian Knutsen
b92442d1a3
feat(storage): one guarded issue-lifecycle surface for the CLI and linked consumers (#5191)
Adds issueops.Lifecycle — Create, Update, Close, Reopen — reached from the storage handle via
store.IssueLifecycle(), implemented once and shared by the CLI and linked library consumers. Three backends
implement it and a shared conformance suite asserts them against one spec. The bd CLI's create, update,
close and reopen now route through it.

Fixes several defects that predate this change and became visible once the write paths were forced to
agree: swallowed metadata-edit failures, a claim applied before field validation, missing value validation
on title/priority/estimate, silently dropped create edges to missing targets, infra-typed creates landing in
the wrong table, a dropped SourceRepo, closed_by_session written but never read, and a compound update
writing twice.

Behavior changes needing release notes: create with an occupied explicit id now fails instead of silently
overwriting; a compound update is one atomic operation with one hook and one version commit; reparenting
replaces all parent edges; reopening something not closed reports nothing to do.
2026-07-31 11:26:38 -07:00
Steve Yegge
b0c2286bff
feat(storage): storage-class plumbing — schema marker, --storage-class flag, per-type config defaults (bd-8rifr) (#5149)
* feat(storage): storage_class marker column + types plumbing (bd-8rifr)

Protocol v0.1 §C (bd-7vb13): types.StorageClass (versioned|unversioned|
ephemeral, empty = unset), Issue.StorageClass with C1.2 defaulting via
EffectiveStorageClass, Validate coherence with the wisp-plane flags,
migration 0060 (nullable marker on issues+wisps, no row rewrite), and the
column threaded through IssueBaseColumns/Lite, both scans, and both
insert stacks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxHUGXk944MDbtQ9xCNygJ

* feat(create,config): --storage-class flag + storage-class.<type> defaults (bd-8rifr)

Create-time resolution per C1.3: explicit flag > per-type config default >
unset; versioned normalizes to the omitted/NULL marker form (C2.4);
--storage-class ephemeral routes to the wisp plane as the spelled-out
--ephemeral (C1.4). storage-class.<type> is a yaml-plane key (dolt.auto-commit
precedent) validated at set time in both config set and set-many. Unit tests
plus an embedded end-to-end contract test (flag, config default, override,
at-rest NULL rule, wisp routing, set-time validation).

Docs: deferred to the cli-docs.pin bump per docs policy (docs describe the
pinned release; #5052/#5089 precedent) — flag help text regenerates into the
CLI reference then.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LxHUGXk944MDbtQ9xCNygJ

* fix(graph): per-node storage_class in graph plans (bd-8rifr)

TestGraphApplyNodeCoversCreateIssueParams caught the gap on CI:
createIssueParams grew StorageClass but graph plans could not address it.
Wire a per-node storage_class with single-issue create's exact C1.3
precedence (explicit node value > storage-class.<type> config > unset),
including the ephemeral spelling routing to the wisp plane with the same
mutual-exclusion errors, and extend the graph_full_fields round-trip to
pin the at-rest value.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(config): validate the storage-class.<type> key suffix at set time (bd-8rifr)

Lion's #5149 should-fix: the value was validated but the key suffix was
not, so an alias (storage-class.feat) or typo (storage-class.taks) passed
set-time validation and silently never matched — create-time lookup keys
on the Normalize()d type. Reject non-canonical suffixes with the canonical
key as the hint, and unknown types with a types.custom pointer. Also the
ephemeral-constant doc nit: class-ephemeral is typically TTL-bounded, not
definitionally (no-history rows are TTL-exempt).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-29 03:20:07 -07:00
Steve Yegge
966ddac5af feat(leases): replica-aware leases — record the granting node, refuse cross-replica reclaim (wy-jpd3.7)
A claim lease is only meaningful on the replica that granted it. Every other
replica's view of the holder's liveness is stale by up to one sync interval,
so a reaper over there can revert a unit that is very much alive here. Until
now the only defense was operator discipline: partition every supervisor's
reclaim by label and hope nobody runs an unscoped sweep.

Leases now record WHICH replica granted them (leases.granted_node, ignored
migration 0016, stamped from config.NodeID(): BEADS_NODE_ID / node_id config,
falling back to the hostname). bd reclaim skips a lease that positively names
another replica — on both the snapshot SELECT and the per-row DELETE
re-check — and names each skip on stderr, since silence there is
indistinguishable from "nothing to recover" while a genuinely dead remote
worker's unit sits in_progress forever. bd reclaim --any-replica is the
operator escape hatch for a replica that is gone for good.

The guard is deliberately fail-open on unknown provenance: granted_node ''
(a row predating the column, or a deployment whose NodeID is empty) is
treated as local, so the upgrade itself can never strand a stale lease the
reaper could previously recover. A heartbeat backfills an unknown row with
the local node and never overwrites a known one.

The one path that can materialize a FOREIGN lease row locally is
RestoreLeaseOnImportInTx — lease fields round-trip the JSONL interchange — so
the granting replica rides types.Issue.LeaseGrantedNode and a restored lease
keeps the replica the snapshot names, never this node. A snapshot from an
older binary carries none, restores as unknown, and behaves exactly as before.

What the guard cannot enforce is documented instead, in both bd reclaim --help
and docs/multi-agent/federation.md: grace window > sync interval, and lease
TTL > sync interval. A TTL shorter than the cadence at which replicas exchange
state is meaningless across the bridge — the remote view is a full interval
old by construction.

doc/beads-upstream-asks.md section 5.
2026-07-24 23:53:34 -07:00
Julian Knutsen
607df586d1
feat(engine): guarded write ops — atomic metadata merge, row-version CAS, typed dep errors + events, bd close delegation (#4911)
* feat(metadata): MergeMetadata — atomic single-key metadata merge

SlotSet did a read-modify-write across TWO transactions: GetIssue (tx 1),
metadata[key]=value in memory, then UpdateIssue (tx 2). Two concurrent SlotSet
calls — even on DIFFERENT keys — both read the same base metadata and each
wrote back its whole blob, so the second clobbered the first's key. SlotClear
had the same bug. Add MergeMetadata, which does the read-modify-write inside
ONE transaction (so a concurrent conflict is retried and re-reads), stores a
JSON value (nested objects/arrays, not just strings), and reimplement SlotSet
and SlotClear on top of it so they inherit the atomicity.

- issueops.MergeMetadataInTx / DeleteMetadataInTx: read the metadata (routed
  issues/wisps; missing issue -> ErrNotFound), merge/delete the one key, and
  write the whole object back THROUGH UpdateIssueInTx — so the operation keeps
  everything the old SlotSet got from UpdateIssue (the EventUpdated history
  event with actor attribution, the configured metadata-schema validation, and
  updated_at), now atomic. The read and write share the caller's transaction,
  so a concurrent merge of a different key is retried and re-read, never
  clobbered.
- Storage.MergeMetadata implemented on DoltStore (perm withRetryTx +
  DOLT_ADD(issues,events) + DOLT_COMMIT; wisp path no commit) and
  EmbeddedDoltStore (withConn); forwarded on InstrumentedStorage; auto-promoted
  on HookFiringStore; no-op on the configStore mock. SlotSet/SlotClear rebuilt
  on it (a string value marshals to a JSON string, byte-compatible with the
  historical rewrite).
- Cross-backend conformance covers MergeMetadata + atomic SlotClear; the
  8-goroutine concurrent-no-clobber test proves the B1 fix on real Dolt, and
  a test asserts the EventUpdated event and schema validation are preserved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(concurrency): expose a per-row version on Issue reads (RowVersion)

The engine's internal row_lock column — a random non-zero int64 the engine
rewrites on every status/ownership-mutating write and on the generic update
path — was not surfaced on the public types.Issue. Consumers that wanted a
fine-grained optimistic-concurrency token had only updated_at, which is stored
at second granularity, so two same-second writes were indistinguishable. Expose
row_lock read-only as types.Issue.RowVersion so a caller can tell same-second
writes apart.

- types.Issue gains RowVersion int64 (json:"-": a Go-only concurrency token,
  never on the CLI/export JSON wire — it is opaque and equality-only). Added
  row_lock to the canonical sqlbuild.IssueBaseColumns and a matching scan target
  in issueops.ScanIssueFrom; both engine write stacks, wisps, and the counts /
  dependents parallel scans share those, so the field hydrates everywhere from a
  single-point change. Read-only — no CAS/write behavior (that is a follow-up).
- Doc is precise about coverage: RowVersion changes on claim/close/unclaim and
  the generic update path; it does NOT change on some direct-UPDATE paths
  (restore, compaction text), which bump updated_at — so a complete change key
  combines RowVersion with updated_at, status, and labels. A created row is
  already non-zero (create stamps row_lock); 0 only appears on legacy rows
  backfilled by migration 0054's DEFAULT 0.
- Tests: RowVersion hydrates on read and on the list path, changes on a mutating
  write, distinguishes two same-second writes (identical updated_at, distinct
  RowVersion), and never appears on any JSON surface.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(deps): return a typed error on dependency type-conflict

Adding a dependency between a pair that already has an edge of a different type
is a deterministic rejection, but the two engine write paths disagreed on how
they surfaced it: one returned a typed *DependencyTypeConflictError, the other a
plain fmt.Errorf string. So a caller could errors.As the conflict on one path
but had to string-match the message on the other. Make both paths return the
same typed error, and re-export the two dependency-conflict types on the public
beads package so `bd` (and the library's own call sites) classify by errors.As
instead of parsing message text.

- issueops.AddDependencyInTx now returns *domain.DependencyTypeConflictError for
  the type-conflict case (byte-identical message — the struct's Error() is the
  exact former string, so no string-matcher changes). The hierarchy/cross-type
  case already returned *domain.DependencyHierarchyConflictError on both paths;
  self-dependency and cycle are already typed sentinels.
- beads.go re-exports DependencyTypeConflictError and
  DependencyHierarchyConflictError as type aliases, so errors.As against the
  public type matches the value the engine returns.
- Tests assert errors.As + the four fields + the byte-identical message on both
  write paths (permanent issues->issues and the wisp-source BeginTx/Commit seam),
  and lock the re-exports; the deterministic conflict is returned immediately
  (never retried as a transient serialization error).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(deps): emit dependency_added / dependency_removed events

`bd dep add` and `bd dep remove` recorded no event, so the events feed never
surfaced dependency changes even though the event types (types.EventDependencyAdded
/ EventDependencyRemoved) have existed since the original SQLite backend, which
emitted them on every add/remove. The emission was dropped when that backend was
removed for the Dolt-only migration and never ported to the current write paths.
Restore it on both Dolt write plumbings, matching the original shape (an events
row on the source issue: "Added dependency: <src> <type> <target>" /
"Removed dependency on <target>").

- issueops.AddDependencyInTx / RemoveDependencyInTx record the event via
  RecordEventInTable on the source's event table (wisp-routed), on the genuine
  add/remove only — the idempotent same-type re-add and the no-op remove of a
  missing edge record nothing.
- The event is committed with the edge: DoltStore.AddDependency/RemoveDependency
  DOLT_ADD `events`, and the transaction-path (RunInTransaction / batch /
  graph-apply) methods mark the source's event table dirty (events or wisp_events)
  so StageAndCommit commits it — without this the event row would dangle in the
  working set.
- Emission is gated to the explicit dep-add/remove verb on both plumbings: the
  proxied-server repo only records when the use-case sets DepInsertOpts.EmitEvent,
  which create-with-deps (implicit parent-child / --deps / waits-for, via a direct
  repo Insert) does not — so `bd create --parent` produces the same history on
  either backend (the embedded create path via PersistDependencies likewise emits
  nothing).
- Tests on both plumbings: add/remove emit exactly once and commit (verified via
  `events AS OF 'HEAD'`), idempotent re-add and no-op remove emit nothing, wisp
  sources route to wisp_events, and create-with-deps emits no dependency_added.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(close): optional ExpectedVersion CAS on CloseIssueChecked

CloseIssueChecked guards a close on is_blocked in-transaction, but a caller that
read an issue and then closes it has no way to say "only if it hasn't changed
since I read it" — a lost-update window. Add an optional compare-and-swap on the
row's version (the RowVersion / row_lock token): when the caller supplies an
ExpectedVersion, the close proceeds only if the row still has that version, else
it refuses with a typed ErrVersionMismatch. The version read and the close share
one transaction, so it is a true CAS with no read-then-write gap.

- storage.ErrVersionMismatch (re-exported as beads.ErrVersionMismatch) and
  CloseIssueOptions.ExpectedVersion *int64 (nil disables the check; a pointer so
  nil "no check" is distinct from "require version 0").
- issueops.CheckVersionInTx reads row_lock (wisp-routed; ErrNotFound on a missing
  row) and returns ErrVersionMismatch on divergence. CloseIssueCheckedInTx runs
  it FIRST — before the is_blocked guard and before the Force short-circuit, so
  Force bypasses only the guard, never the CAS (the version check is an
  orthogonal precondition). A mismatch returns before any write, so the tx rolls
  back leaving the issue open with no `closed` event (atomic refuse).
- The CAS has two limbs: the read-side check catches a writer that committed
  before the close began, and on the retry-wrapped permanent path a commit-time
  row_lock conflict is replayed by withRetryTx so the re-read refuses.
- Tests on both stores: match closes, stale refuses atomically (still open, zero
  closed events), a committed concurrent write invalidates a captured version,
  nil is unchanged behavior, Force does not bypass, missing id → ErrNotFound,
  wisp sources route to the wisps table, and an already-closed re-close with the
  post-close version stays idempotent (Unchanged).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(close): document RowVersion CAS coverage boundary on ExpectedVersion

Both adoption reviewers flagged that the public ExpectedVersion godoc and the
shared CloseIssueCheckedInTx doc could be misread as a full-row "unchanged"
check. RowVersion (row_lock) only tracks lifecycle/ownership writes
(status/assignee/started_at), so concurrent label, dependency, rename, or
is_blocked writes leave it untouched and are intentionally outside this CAS
boundary. Document that on both sites, grounded in the freshRowLock invariant.
Doc-only; no behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(cli): bd close delegates to the engine's guarded close

The embedded `bd close` path ran its own is_blocked pre-check and then closed in
a separate call — a read-then-write TOCTOU that duplicated a guard the engine
provides atomically. The verb now builds CloseIssueOptions and delegates to
CloseIssueChecked, which runs the guard and the close in ONE transaction; the
CLI's duplicated guard is deleted. --force maps to Options.Force; a blocked
close refuses with the blockers named and the --force hint.

Behavior-preserving on purpose (verified against the previous binary):

- The engine guard now uses the exact historical CLI predicate — refuse only on
  a LIVE direct blocker (blocked && len(blockers) > 0), not on the bare
  denormalized is_blocked column. A transitively-blocked child (is_blocked
  inherited from a blocked parent, zero direct blockers) still closes without
  --force, and a stale is_blocked with since-closed blockers self-heals, exactly
  as before.
- An already-closed issue stays an idempotent success: it still appears in the
  --json array and the text report (same output shape), still exits 0, but no
  longer produces a spurious closed->closed audit entry, a no-op commit, or the
  real-close side effects (molecule auto-close, newly-unblocked, claim-next).

Tests cover: direct blocker refuses atomically (still open) and closes with
--force; transitively-blocked and stale-is_blocked beads close without --force;
already-closed emits in --json alone and in a mixed batch; idempotent exit 0 —
on the engine (both stores) and through the bd binary.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(update): UpdateIssueChecked — optional ExpectedVersion CAS on the update

UpdateIssue applies a field-map update with no optimistic-concurrency option, so
a caller that wants "update only if unchanged since I read it" had to
read-then-write — a lost-update window. Add UpdateIssueChecked: when the caller
supplies an ExpectedVersion (the RowVersion / row_lock token), the version read
and the update share ONE transaction, refusing with the typed ErrVersionMismatch
if the row has moved — a true compare-and-swap. Nil disables the check and is
byte-identical to UpdateIssue, which is left untouched on the hot path.

- storage.UpdateIssueOptions{ExpectedVersion *int64} (aliased as
  beads.UpdateIssueOptions) + Storage.UpdateIssueChecked, mirroring the
  CloseIssueChecked pattern: CheckVersionInTx runs first inside the same
  transaction on every route — the permanent withRetryTx path, the wisp path,
  AND the demote route (an update carrying no_history/wisp routes through the
  demote flow; a pure demoteToWispInTx extraction lets the check compose
  atomically with the row move — verified byte-identical to the old
  DemoteToWisp).
- Implemented on DoltStore and EmbeddedDoltStore; InstrumentedStorage wraps it;
  HookFiringStore fires on_update only on success (never on a refused update).
- Tests on both stores: match updates and bumps RowVersion; stale refuses
  atomically (field unchanged, zero updated events); a committed concurrent
  write invalidates a captured version; nil behaves exactly like UpdateIssue;
  wisp match/stale; demote-route match (migrates + applies) and stale (still in
  issues, unchanged); missing id → ErrNotFound; the hook decorator fires once on
  success and never on refusal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(cli): proxied bd close delegates to a library checked close

The proxied-server `bd close` path duplicated the close guard in the CLI:
it queried blockers via the dependency use-case, refused with its own message,
and then closed in a separate use-case call. Move the guard into the library —
IssueUseCase gains CloseIssueChecked / CloseWispChecked, which run the same
live-direct-blocker predicate (refuse only on blocked && len(blockers) > 0)
and then the untouched unchecked close — and thin the CLI verb to a single
delegated call. Both `bd close` paths now enforce the guard in the library and
emit the byte-identical refusal: the same storage.ErrCloseBlocked sentinel
("cannot close blocked issue: <id> is blocked by [...]") plus the
"(use --force to override)" hint.

- Guard and close run on the one pinned unit-of-work connection (the whole
  close batch is a single transaction), and the guard is read-only and refuses
  before any write — a refused id leaves the shared batch transaction clean,
  and an all-refused batch skips the commit entirely, exactly as before.
- The unchecked CloseIssue / CloseWisp are untouched: internal closes (molecule
  auto-close, gate closes, bd todo done) intentionally stay unguarded.
- Behavior-parity preserved: guard ordering (validate → epic-child → gate →
  blocked), exit codes, JSON/batch shape, already-closed handling, audit fields,
  and commit-message contents are unchanged; a transitively-blocked bead
  (is_blocked with no direct open blocker) still closes without --force, and
  --force still bypasses. The guard-failure diagnostic now uses the embedded
  path's generic wording (further convergence; nothing matched the old text).
- Tests: use-case level on real Dolt (direct blocker refuses with blockers
  named + still open, transitive-only closes, force closes, already-closed
  parity, wisp refuse/force) plus the proxied CLI integration suite.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(review): cross-backend dependency-event + RowVersion parity

Maintainer review fixups for PR #4911 (adopt-pr review loop):

- RowVersion: the proxied/domain create (insertIssueRow) and generic Update
  paths never stamped or rewrote row_lock, so an issue written through the
  proxied-server backend read back RowVersion 0 on create and unchanged on
  update -- a stale CAS token that violated the types.Issue.RowVersion
  contract on that backend. Stamp a fresh non-zero row_lock on insert (and the
  duplicate-key rewrite) and append RowLockClause() to the generic update,
  mirroring the classic issueops insert/update invariant. Adds domain/db
  regression tests proving CreateIssue/CreateWisp are non-zero and
  UpdateIssue/UpdateWisp change it.

- Dependency events: embedded issueops AddDependencyInTx/RemoveDependencyInTx
  recorded history unconditionally, so structural create-with-deps
  (bd create --parent/--deps/--waits-for) and reparent (bd update --parent)
  emitted dependency_added/removed on embedded but were silent on the proxied
  backend, diverging issue history for identical commands. Gate the embedded
  emit on EmitEvent and thread it through AddDependencyWithOptions /
  RemoveDependencyWithOptions on both the store and transaction interfaces:
  the plain AddDependency/RemoveDependency are the no-event structural default
  (create-with-deps, reparent), while the explicit dep verbs (bd dep add/remove,
  bd link, bd relate/unrelate) pass EmitEvent -- matching the proxied
  DepInsertOpts.EmitEvent gate. DoltStore stages the events table only when an
  event was actually written (GH#2455). Corrects the false
  "parity with PersistDependencies" comments and adds symmetric embedded/proxied
  dep-event coverage for create-with-parent/deps and structural removal.

- Retain iteration-1 fixups: heal molecules on already-closed re-close,
  idempotent proxied checked-close, and documented public Storage growth.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(storage): drop always-constant event param from fireDependencyHookByID (unparam)

golangci-lint (latest, v2.12.2 in CI) unparam flags fireDependencyHookByID:
its `event` parameter always receives hooks.EventUpdate. The parameter was
already effectively constant before this PR (both base call sites passed
EventUpdate); this PR adds two more call sites, all EventUpdate. Remove the
dead parameter and hardcode hooks.EventUpdate at the single runner.Run call.
All four call sites pass hooks.EventUpdate, so behavior is unchanged; go
build, go vet, and a scoped golangci-lint run pass clean.

Maintainer fixup during adoption finalize (PR #4911).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Test User <test@test.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <ci@beads.test>
2026-07-21 18:13:03 -07:00
Steve Yegge
59f8d6ae97 feat(leases): move leases to ephemeral dolt_ignored leases table (bd-lrgn1)
Claims/heartbeats no longer mint dolt commits: lease state (holder,
granted_at, lease_expires_at, heartbeat_at) moves off the issues table
into a leases table registered in dolt_ignore. Migration 0055 +
ignored/0012 move live leases over; all full-issue query sites hydrate
via LEFT JOIN; import never clobbers a live local lease.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 01:02:29 -07:00
Steve Yegge
e97839a2e1
feat(lease): land the leasing stack + schema v54 (claim-TTL, heartbeat, reclaim) (#4537)
* fix(dolt): retry mutate+commit methods on serialization conflict (1213/1205)

ClaimIssue, ClaimReadyIssue, UpdateIssue and CloseIssue each hand-rolled
their own tx (BeginTx + DOLT_ADD/DOLT_COMMIT + Commit) outside
withRetryTx, so a concurrent writer that lost Dolt's optimistic
commit-time merge surfaced MySQL 1213/1205 as a hard failure instead of
retrying. Dolt has no real row locking — FOR UPDATE / SKIP LOCKED are
parse-only no-ops (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/)
— so retry is the only safety net under N concurrent workers. These four
are the work-queue hot paths: `bd ready --claim`, claim, update, close.

Move each body into a withRetryTx closure (which owns BeginTx and the
final Commit and retries serialization failures with backoff), preserving
the single-tx CAS/mutation + DOLT_COMMIT atomicity exactly. On retry,
ClaimReadyIssue re-scans the ready front from a fresh snapshot and claims
the next available issue.

Tested: TestEmbeddedReadyClaimConcurrent, TestEmbeddedReadyConcurrent,
TestEmbeddedCloseConcurrent, TestEmbeddedUpdateConcurrent and the
non-concurrent embedded Update/Close/Ready tests pass; build + vet clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(dolt): add concurrent work-queue drain test (Gas Station claim invariant)

N workers concurrently dequeue one shared ready-front via ClaimReadyIssue
until empty; asserts every issue claimed exactly once, none stranded, no
double-claims, and no serialization error ever surfaces (withRetryTx absorbs
the 1213/1205 from the claim-CAS cell collision). This is the regression guard
for the multi-agent claim queue that #4491 makes safe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(lease): claim-TTL + heartbeat + reclaim for dead-worker recovery

Gas Station v1.1 (wy-5r9j). A claim was previously permanent: a worker that
died mid-task stranded its issue in_progress forever. This adds a lease so a
claim can expire and be recovered.

Schema (migration 0054): add lease_expires_at, heartbeat_at (DATETIME) and
row_lock (BIGINT) to issues (and wisps, for uniform issueops routing), plus
idx_issues_lease for the reaper scan. assignee doubles as the lease owner.

Lifecycle (shared issueops, so both server-mode DoltStore and EmbeddedDoltStore
get it):
  - Claim stamps lease_expires_at = now + TTL (default 5m, WithLeaseTTL to
    override), heartbeat_at = now, and a fresh row_lock.
  - bd heartbeat <id> — owner-only; pushes the lease forward and rewrites
    row_lock. Fails (ErrNotClaimable/ErrAlreadyClaimed) once the lease is gone.
  - bd reclaim --older-than <dur> — reverts in_progress issues whose lease
    expired more than <dur> ago back to ready (clears assignee/started_at,
    records a lease_reclaimed event). Default grace 2×TTL.

The row_lock trick: Dolt has no row locking and merges concurrent commits
cell-by-cell, so a heartbeat (writes heartbeat_at) racing a reclaim (writes
status) would silently cell-merge into a zombie — an open/unassigned issue that
still carries the worker's fresh heartbeat. Every mutating path (claim, close,
update, heartbeat, reclaim) now rewrites the shared row_lock cell, forcing the
loser to a 1213/1205 serialization conflict that withRetryTx replays. All lease
paths are withRetryTx/withConn wrapped.

Tests (real Dolt): TestRowLockForcesConflictOnDisjointCellWrites proves the bug
without row_lock (silent merge to zombie) and correctness with it (conflict);
TestConcurrentHeartbeatReclaimClose is the integration race (live workers
heartbeat+close vs a continuous reaper vs dead workers) asserting exactly-once,
no lost close, no zombie; plus claim/heartbeat/reclaim unit coverage on both
backends.

Note: lease columns are second-granular DATETIME (Dolt rounds), so sub-second
TTLs are meaningless — production TTL is minutes. Heartbeat writes a Dolt
commit, so cadence must stay ≪ claim cadence to avoid history bloat.

Fixes the test coupling in TestMigration0053PromotesRigWisps, which assumed the
rig-wisp repair was always LatestVersion().

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci+docs: allowlist lease stress tests; scope the row_lock invariant comment

- Add TestConcurrentWorkQueueDrain and TestConcurrentHeartbeatReclaimClose
  to the check-testing-short.sh allowlist; both are real-Dolt runtime/stress
  skips and were failing ci-pr-policy.
- freshRowLock doc said "EVERY mutating path" rewrites row_lock, which
  overstates it. Scope it to status/ownership/lease-mutating paths (the set
  the reclaim/heartbeat races care about) and document the invariant + why
  orthogonal-cell writers (is_blocked, compaction, deps, rename, reopen) are
  intentionally exempt, so a future writer does not silently reintroduce the
  zombie-merge bug.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(show): surface lease_expires_at/heartbeat_at in bd show (--json + text) (wy-9cdw)

Migration 0054 added lease_expires_at/heartbeat_at (and internal row_lock) to
the issues/wisps tables for the claim/heartbeat/reclaim leasing feature, but
bd show could not observe lease state — you had to drop to raw SQL.

- types.Issue: add LeaseExpiresAt/HeartbeatAt (*time.Time, omitempty), matching
  the StartedAt nullable-timestamp convention. row_lock stays unexported.
- sqlbuild.IssueSelectColumns: append lease_expires_at, heartbeat_at to the
  canonical hydration list (flows to every stack via the shared constant).
- issueops.ScanIssueFrom: scan + map the two nullable columns (before the
  trailing extra dests, keeping positional parity across all scan sites).
- bd show text view: emit a muted 'Lease: expires <rel> (heartbeat <rel> ago)'
  line only for in_progress issues holding an active lease; add formatTimeUntil
  as the forward-looking mirror of formatTimeAgo.
- types_test: assert lease keys appear when set, are omitted when nil, and that
  row_lock is never surfaced.

* fix(lease): settle v54 review blockers

Address #4537 review blockers before landing v54: keep heartbeat off wisps in both backends, keep direct status/assignee updates from leaving stale leases, guard the down migration, and emit reclaim JSON as snake_case.

Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie

* fix(lease): cover lease fields in tests and hooks

Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
2026-07-03 23:28:55 -07:00
Steve Yegge
1f8f1fff31 refactor(storage): extract shared SQL builders into internal/storage/sqlbuild (bd-6dnrw.46)
Move all pure SQL-text builders shared by issueops (classic) and
domain/db (proxied) into one package; both stacks now delegate, so
ready/filter/counts query semantics can no longer drift (Seam A).
Execution orchestration stays per-stack per the extract-don't-unify
decision. Canonical SQL text is classic's; the only proxied-side text
deltas are IN-list whitespace.

Adversarial re-review (bd-6dnrw.47) verified all 10 attack items with
no production divergence, and adds an import-boundary test pinning the
layering (domain must never import issueops/sqlbuild) plus a single-use
constraint note on LabelSearchPlan.MergeInto. Follow-ups filed:
bd-emrs2 (+1-overfetch convention), bd-kzeme (IN-helper consolidation).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 15:56:23 -04:00