From a45199a546f959044426b98716975c70b7c77a16 Mon Sep 17 00:00:00 2001 From: Steve Yegge Date: Fri, 24 Jul 2026 07:21:24 -0700 Subject: [PATCH] fix(claim): verify claim-family writes by re-read under a degraded server (bd-zccb9) (#5006) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(claim): verify claim-family writes by re-read under a degraded server (bd-zccb9) From wyvern incident wy-ejph3: under a degraded dolt sql-server the exit status of a claim is not truth in either direction — a reported success can roll back with the abandoned connection (phantom claim), and a printed connection error can mask an applied write (wy-x543k). New verifiedClaimWrite resolves claim-family outcomes against the DB instead of trusting exit status: reported success is verified by re-read and fails loudly on mismatch; an indeterminate commit-phase loss (errCommitPhase) is settled by re-read — applied becomes success, verified-rolled-back is replayed once. Wired into ClaimIssue, UnclaimIssue, UnclaimIssueIfAssignee, and ClaimReadyIssue (bespoke: claimed ID only known post-body; a replay may legitimately claim a different issue). Wisps and non-server mode exempt. Metrics: bd.claim_verify_lost_total, bd.claim_verify_recovered_total (op, outcome=applied|replayed). Co-Authored-By: Claude Fable 5 * docs(proposal): CAS conditional update design for bd-wsqvw (from fly) Design handoff from wyvern's wheelhouse hostile-review (epic wy-mdi5h): generalize the claim-side CAS into --if-assignee/--if-status guards on 'bd update', covering the X->Y reassign and status-guarded claim-on-behalf transitions that --claim and unclaim --if-assignee cannot express. Authored by fly; committed so the untracked doc survives workspace cleanup. Depends on bd-zccb9 (verify-after-write). Co-Authored-By: Claude Fable 5 * fix(claim): ready-claim replay leg keeps verify semantics (bd-zccb9, lion review) The bespoke ClaimReadyIssue path's replay returned raw on any error, so a replayed write dying at commit phase skipped the verify pass — reintroducing the wy-x543k false-failure on attempt two, and potentially orphaning an applied claim on a DIFFERENT ready issue than the first attempt selected. The replay now falls through to verify on errCommitPhase-with-selection, exactly like verifiedClaimWrite's leg: applied -> recovered success, verified rollback -> honest 'did not land' (budget already spent). Extracted the resolution loop as verifiedReadyClaim so injection tests can drive the write seam; sixth test mirrors ConvertsAppliedIndeterminate on the replay leg. Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Fable 5 --- CHANGELOG.md | 19 ++ PROPOSAL-cas-conditional-update.md | 167 ++++++++++++++ internal/storage/dolt/claim_verify.go | 138 ++++++++++++ internal/storage/dolt/claim_verify_test.go | 244 +++++++++++++++++++++ internal/storage/dolt/issues.go | 213 ++++++++++++------ internal/storage/dolt/store.go | 28 ++- 6 files changed, 735 insertions(+), 74 deletions(-) create mode 100644 PROPOSAL-cas-conditional-update.md create mode 100644 internal/storage/dolt/claim_verify.go create mode 100644 internal/storage/dolt/claim_verify_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index f6bef420b..80cc90e54 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -200,6 +200,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 every remote-ahead case, as before ([#4259](https://github.com/gastownhall/beads/issues/4259)). +### Fixed + +- **Claim-family writes are now verified by re-read in Dolt server mode** + (bd-zccb9, from wyvern incident wy-ejph3). Under a degraded sql-server the + exit status of `bd update --claim` / `bd claim` / unclaim was not truth in + either direction: a claim could report success while the server-side + transaction died with the abandoned connection and rolled back (a phantom + claim that later cost a duplicate implementation), and conversely a + connection error could print with the write actually applied. After the + claim transaction, bd now re-reads the issue's assignee and status on a + fresh connection and resolves the outcome against the database: a reported + success that did not land fails loudly ("treat the claim as NOT applied"); + an ambiguous commit-phase loss is settled by the re-read — verified applied + becomes an accurate success, verified rolled back is replayed once (safe: + nothing landed). Applies to claim, ready-claim, and unclaim in server mode; + wisps and embedded mode are unchanged. New metrics + `bd.claim_verify_lost_total` and `bd.claim_verify_recovered_total` count + loud failures and converted outcomes. + ## [1.1.0] - 2026-07-04 First stable release of the 1.1.0 line. It consolidates everything from diff --git a/PROPOSAL-cas-conditional-update.md b/PROPOSAL-cas-conditional-update.md new file mode 100644 index 000000000..8ca5c2115 --- /dev/null +++ b/PROPOSAL-cas-conditional-update.md @@ -0,0 +1,167 @@ +# PROPOSAL: general conditional (compare-and-set) update for `bd` + +**Status:** design, ready for implementation as a beads PR +**Author:** design handoff (wyvern hostile-review epic wy-mdi5h) +**Repo:** `github.com/steveyegge/beads` (`~/src/beads`, branch `main` @ `3a6dcff96`) +**Motivating incidents:** wyvern wheelhouse review findings wy-mdi5h.3, .6, .7, .4 (TOCTOU park/restore/take/bulldog); upstream lineage bd-zccb9, #4727, wy-ejph3, wy-x543k. + +--- + +## 1. Problem + +A hostile review of wyvern's wheelhouse fleet scripts found that its single most +common bug class is **check-then-act on `bd` assignee/status**: park, interactive +restore, queue-take, and bulldog-claim each read a bead's state, then issue a +*blind* `bd update -a X` / `-s Y`, and lose to a racing writer in the gap. The +scripts paper over it with re-reads (`wh-take.sh`), but re-reads only narrow the +window; they cannot close it. The root cause is not a wheelhouse bug — it is that +**`bd` exposes no general compare-and-set for the coordination fields.** + +`bd` already recognizes this need and has been closing it verb by verb: + +| primitive | CAS semantics | added | +|---|---|---| +| `bd update --claim` | `assignee ∈ {'', pool-alias} → me`, `status → in_progress` | (existing) | +| `bd unclaim --if-assignee X` | `assignee == X → ''`, `status → open` | #4727 | +| `verifiedClaimWrite` (WIP) | verify-after-write for the above under a degraded server | bd-zccb9 (uncommitted in tree) | + +The pattern is a `UPDATE … WHERE id=? AND status IN(…) AND assignee=?` with +`RowsAffected` as the verdict and a `row_lock` rewrite so a racing reclaim/close +conflicts instead of silently merging (`issueops.UnclaimIssueIfAssigneeInTx`). +This proposal **generalizes that idiom once** instead of minting a third and +fourth special-case verb, and covers the two transitions no existing verb can +express. + +## 2. The gap + +Two wheelhouse transitions have no primitive: + +- **Reassign `X → Y`** (park): "set assignee to `mayor` only if it is still the + worker I just reclaimed." `--claim` only does `'' → me`; `unclaim --if-assignee` + only does `X → ''`. Nothing does `X → Y`. +- **Claim-on-behalf with a status guard** (restore): "set assignee to the crew + owner *and* status to `in_progress`, only if the bead is still `open`/unassigned." + `--claim` always claims for `$BEADS_ACTOR`, never for a third party, and takes + no status precondition. + +Queue-take (`wh-take.sh`) and bulldog-claim are **not** new-primitive gaps — they +are adoption gaps: `--claim` already handles pool aliases (`claim.pools`, e.g. +`fable-crew`) atomically, so both should simply call `--claim` and honor its +non-zero exit instead of a blind `-a` + swallowed refusal. This proposal fixes +the primitive; those two are follow-up wheelhouse patches (wy-mdi5h.7, .4). + +## 3. Proposal: `--if-assignee` / `--if-status` guards on `bd update` + +Add two precondition flags to the existing `update` command. When either is +present, the mutation becomes an atomic CAS; when neither is present, `update` is +unchanged. + +``` +bd update --if-assignee [--if-status ] [ -a ] [ -s ] [ …other field flags ] +``` + +Semantics: + +- **Precondition** = the conjunction of every `--if-*` guard present. The row must + currently match *all* of them or the write does not apply. +- **`--if-assignee ''`** means "expected unassigned." Presence is detected with + `cmd.Flags().Changed("if-assignee")`, exactly as `unclaim` distinguishes an + explicit empty value (unclaim.go:59–61) — so `--if-assignee ""` is a real + guard, not "no guard." +- **Atomicity**: one `UPDATE … SET WHERE id=? AND assignee=? [AND status=?]` + in a single tx, `RowsAffected==1` is the verdict, `row_lock` rewritten to + `freshRowLock()` (same invariant as unclaim — a racing reclaim/close on the row + must conflict, not merge). The guard read and the UPDATE run in the same + transaction, so there is no internal TOCTOU (mirrors `UnclaimIssueIfAssigneeInTx`). +- **Precondition failure** (`RowsAffected==0`): return a typed + `storage.ErrAssigneeMismatch` / new `ErrStatusMismatch`, surfaced as a **loud, + non-zero exit** with actual-vs-expected in the message. It must **never** be + swallowed or collapse to exit 0 — this is the whole point; the CLI already has + the "requested-but-lost claim ⇒ SilentExit()/non-zero" plumbing for `--claim` + (update_proxied_server.go, "beads audit finding #10") and this reuses it. +- **Verify-after-write**: route through the WIP `verifiedClaimWrite` (a guarded + reassign is coordination-critical and idempotent for the winner — safe to + replay when a re-read proves the commit rolled back). Add a `reassigned(actor)` + / `matched(post)` `claimPostcondition` alongside `claimedBy`/`unclaimed`. +- **Wisps exempt**, closed issues rejected — same as the sibling verbs. +- **Batch/JSON**: same mixed-batch exit-code rule as `--claim` (one lost + precondition in a batch ⇒ non-zero overall), same JSON shape. + +### Why guards on `update` rather than a new `bd reassign` verb + +- The transitions we need are *arbitrary* field CAS (assignee and/or status), not + a single named operation; a verb per transition is how we got here. +- `update` already owns multi-field mutation, batching, JSON, proxied-server + routing, and the finding-#10 exit-code contract — guards compose with all of it + for free. +- `--claim` and `unclaim --if-assignee` remain the ergonomic shorthands for their + two hot paths; the general guards are the escape hatch the wrappers currently + fake with `bd sql`. + +Alternative considered — **dedicated `bd reassign --from --to`**: more +discoverable for the park case, but doesn't express restore's status-guarded +claim-on-behalf without yet more flags, and duplicates update's plumbing. Rejected +in favor of the general guards. (Flag it for maintainer preference — it is a CLI +surface call, not a correctness call.) + +## 4. How each wheelhouse finding collapses to it + +```sh +# park (wy-mdi5h.3) — reassign only if the worker still holds it: +bd update "$id" --if-assignee "$reclaimed_worker" -a "$PARK_ASSIGNEE" + +# interactive restore (wy-mdi5h.6) — claim-on-behalf, only while still open: +bd update "$id" --if-assignee '' --if-status open -a "$owner" -s in_progress + +# queue-take (wy-mdi5h.7) — ADOPTION, no new primitive: use pool-aware claim +bd update "$id" --claim # fable-crew is a claim.pools alias + +# bulldog (wy-mdi5h.4) — ADOPTION: claim + honor the non-zero exit, don't swallow +bd update "$BEAD" --claim || { echo "foreign/lost; not mutating"; exit 1; } +``` + +The first two are the primitive this proposal adds; the last two are wheelhouse +patches that consume the *existing* `--claim`. + +## 5. Implementation sketch (files) + +- `internal/storage/issueops/update_cas.go` (new): `UpdateIssueIfMatchInTx(ctx, + tx, id, guards, updates)` — build the `WHERE` from present guards, reuse the + unclaim.go CAS body (row_lock rewrite, `status IN ('open','in_progress')` + liveness clause, RowsAffected verdict, typed mismatch error incl. actual state). +- `internal/storage/storage.go`: add `ErrStatusMismatch`; add + `UpdateIssueIfMatch(ctx, id, actor, guards, updates)` to the `Storage` interface. +- `internal/storage/dolt/issues.go`: `DoltStore.UpdateIssueIfMatch` wrapping + `verifiedClaimWrite` with a `matched(post)` postcondition + DOLT_ADD/COMMIT. +- `cmd/bd/update.go`: register `--if-assignee` (String) and `--if-status` + (String); when either `Changed()`, route to the CAS path; wire the + precondition-failed → non-zero exit through the existing `claimFailed` machinery + (both `update.go` and `update_proxied_server.go`). Reject `--if-*` combined with + `--claim` (mutually exclusive; `--claim` is its own CAS). +- `cmd/bd/prime.go`: document the new guards next to `--claim`. + +## 6. Tests + +Beads side (their conventions): +- `internal/storage/issueops` unit tests: match applies, assignee-mismatch no-op, + status-mismatch no-op, `--if-assignee ''` matches only unassigned, closed-issue + reject, row_lock rewritten on success. +- A concurrency test in the spirit of the existing claim CAS tests: two goroutines + reassign the same row, exactly one gets `RowsAffected==1`. +- `verifiedClaimWrite` path: reported-success-but-rolled-back ⇒ replay; applied ⇒ + success; genuine mismatch ⇒ loud non-zero. +- CLI: non-zero exit on precondition failure, mixed-batch exit code, JSON shape. + +Wyvern side (house rule — each consuming fix gets a `test-*.sh` pin): +- `test-park-cas.sh`, `test-restore-cas.sh` asserting a racing claim in the window + is refused (non-zero) rather than clobbered. + +## 7. Scope / non-goals + +- **In:** assignee + status guards on `update`; storage + CLI + verify integration; + the two wheelhouse consumers that are genuine primitive gaps. +- **Out:** label/other-field guards (`--if-label`) — no current consumer; add later + if one appears. The wheelhouse *adoption* patches (queue-take, bulldog switching + to `--claim`) are separate wy-mdi5h children, not part of this beads PR. +- **Coordinate with the uncommitted `claim_verify.go` WIP** already in the tree — + this proposal depends on `verifiedClaimWrite` landing (or lands alongside it). diff --git a/internal/storage/dolt/claim_verify.go b/internal/storage/dolt/claim_verify.go new file mode 100644 index 000000000..e4b57a52b --- /dev/null +++ b/internal/storage/dolt/claim_verify.go @@ -0,0 +1,138 @@ +package dolt + +import ( + "context" + "database/sql" + "errors" + "fmt" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + + "github.com/steveyegge/beads/internal/types" +) + +// Claim-family verify-after-write (bd-zccb9, from wyvern incident wy-ejph3). +// +// Under a degraded sql-server the write's exit status is not truth in either +// direction: a claim can report success while the server-side transaction dies +// with the abandoned connection and rolls back (wy-ejph3: the lost claim cost a +// duplicate full implementation), and the inverse — an error printed with the +// write actually applied — is on record from wy-x543k. Fleet wrappers solved +// this with verify-by-re-read (wyvern's wh-take.sh); this file folds that +// protocol into bd itself for the coordination-critical writes: claim, ready +// claim, and unclaim. Ordinary field updates are out of scope — a lost label +// edit is an annoyance, a phantom claim is a duplicated implementation. +// +// The protocol: run the write, then resolve its outcome against the database +// instead of trusting the exit status. +// +// - reported success -> verify; a mismatch fails LOUDLY (the caller +// must know it does not hold the claim) +// - ambiguous commit loss -> (errCommitPhase, surfaced by withRetryTx as +// indeterminate) verify; applied -> success, verified rolled back -> replay +// the write once (safe: nothing landed) +// - any other error -> honest failure (CAS lost, not-claimable, or +// pre-commit errors withRetryTx already retried); no verify needed +// +// Wisps are exempt: they are ephemeral, never leased, and not reclaimable +// coordination state. + +// claimPostcondition describes the row state a claim-family write must leave +// behind to count as applied, plus the words to use when it didn't. +type claimPostcondition struct { + op string // "claim" | "unclaim", for messages and metrics + want func(assignee string, status types.Status) bool + desc string // expected state, human-readable, for mismatch messages +} + +func claimedBy(actor string) claimPostcondition { + return claimPostcondition{ + op: "claim", + want: func(assignee string, status types.Status) bool { + return assignee == actor && status == types.StatusInProgress + }, + desc: fmt.Sprintf("assignee=%q status=%q", actor, types.StatusInProgress), + } +} + +func unclaimed() claimPostcondition { + return claimPostcondition{ + op: "unclaim", + want: func(assignee string, status types.Status) bool { + return assignee == "" && status == types.StatusOpen + }, + desc: fmt.Sprintf("assignee=%q status=%q", "", types.StatusOpen), + } +} + +// readClaimState re-reads an issue's assignee and status on a fresh +// transaction. withReadTx retries transient connection errors itself, so a +// failure here means the server is genuinely unreachable, not a blip. +func (s *DoltStore) readClaimState(ctx context.Context, id string) (string, types.Status, error) { + var assignee sql.NullString + var status types.Status + err := s.withReadTx(ctx, func(tx *sql.Tx) error { + return tx.QueryRowContext(ctx, + `SELECT assignee, status FROM issues WHERE id = ?`, id, + ).Scan(&assignee, &status) + }) + if err != nil { + return "", "", err + } + if !assignee.Valid { + return "", status, nil + } + return assignee.String, status, nil +} + +// verifiedClaimWrite runs write and resolves its outcome against the database +// state per the protocol above. write must be safe to run twice when its first +// run verifiably did not land (all claim-family writes are: they are CAS +// updates, idempotent for the winning actor). +// +// A verify that contradicts a reported success can in principle also be a +// legitimate concurrent mutation (a forced unclaim landing within the +// verification window). That reads as a lost write and fails loudly too — +// acceptable: the caller must re-establish its view either way. +func (s *DoltStore) verifiedClaimWrite(ctx context.Context, id string, post claimPostcondition, write func() error) error { + if !s.serverMode || s.isActiveWisp(ctx, id) { + return write() + } + const maxReplays = 1 + for attempt := 0; ; attempt++ { + err := write() + if err != nil && !errors.Is(err, errCommitPhase) { + return err + } + assignee, status, verr := s.readClaimState(ctx, id) + if verr != nil { + if err != nil { + return err // the honest indeterminate error from withRetryTx + } + return fmt.Errorf("%s of %s reported success but could not be verified (server degraded?): %w — re-read the issue before trusting the %s", + post.op, id, verr, post.op) + } + if post.want(assignee, status) { + if err != nil { + doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes( + attribute.String("op", post.op), attribute.String("outcome", "applied"))) + } + return nil + } + if err != nil { + if attempt < maxReplays { + // Verified rolled back: nothing landed, so one replay is safe. + doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes( + attribute.String("op", post.op), attribute.String("outcome", "replayed"))) + continue + } + return fmt.Errorf("%s of %s did not land (connection lost during commit; rollback verified by re-read): %w", + post.op, id, err) + } + doltMetrics.claimVerifyLost.Add(ctx, 1, metric.WithAttributes( + attribute.String("op", post.op))) + return fmt.Errorf("%s of %s reported success but did not land (found assignee=%q status=%q, want %s) — server likely degraded; treat the %s as NOT applied", + post.op, id, assignee, status, post.desc, post.op) + } +} diff --git a/internal/storage/dolt/claim_verify_test.go b/internal/storage/dolt/claim_verify_test.go new file mode 100644 index 000000000..f9aa8c0c8 --- /dev/null +++ b/internal/storage/dolt/claim_verify_test.go @@ -0,0 +1,244 @@ +package dolt + +import ( + "database/sql" + "fmt" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// Tests for claim-family verify-after-write (bd-zccb9, wyvern incident +// wy-ejph3): under a degraded server the write's exit status is not truth in +// either direction, so claim outcomes are resolved against the database by +// re-read. Failure injection happens at the write-func seam — the write funcs +// here lie in exactly the ways the incident documented. + +func claimVerifyTestIssue(t *testing.T, s *DoltStore) string { + t.Helper() + ctx, cancel := testContext(t) + defer cancel() + issue := &types.Issue{ + Title: "claim verify target", + IssueType: types.TypeTask, + Priority: 2, + Status: types.StatusOpen, + } + if err := s.CreateIssue(ctx, issue, "tester"); err != nil { + t.Fatalf("create issue: %v", err) + } + return issue.ID +} + +// rawClaim performs the real claim write without the verify wrapper — the +// inner body of ClaimIssue — so tests can control exactly which attempt lands. +func rawClaim(t *testing.T, s *DoltStore, id, actor string) error { + t.Helper() + ctx, cancel := testContext(t) + defer cancel() + return s.withRetryTx(ctx, func(tx *sql.Tx) error { + _, err := issueops.ClaimIssueInTx(ctx, tx, id, actor) + return err + }) +} + +// TestVerifiedClaimWriteConvertsAppliedIndeterminate: the wy-x543k direction — +// the write actually landed but the connection died during commit and bd +// printed an error. Verify-by-re-read must convert it into an accurate success. +func TestVerifiedClaimWriteConvertsAppliedIndeterminate(t *testing.T) { + s, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + id := claimVerifyTestIssue(t, s) + if err := rawClaim(t, s, id, "alice"); err != nil { + t.Fatalf("raw claim: %v", err) + } + + calls := 0 + err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error { + calls++ + return fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase) + }) + if err != nil { + t.Fatalf("expected applied-indeterminate to resolve to success, got: %v", err) + } + if calls != 1 { + t.Fatalf("write must not be replayed when the re-read shows it applied; ran %d times", calls) + } +} + +// TestVerifiedClaimWriteReplaysVerifiedRollback: the wy-ejph3 direction — the +// connection died during commit AND the transaction rolled back. The re-read +// proves nothing landed, so one replay is safe and must proceed. +func TestVerifiedClaimWriteReplaysVerifiedRollback(t *testing.T) { + s, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + id := claimVerifyTestIssue(t, s) + + calls := 0 + err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error { + calls++ + if calls == 1 { + // First attempt: commit-phase connection loss, nothing landed. + return fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase) + } + return rawClaim(t, s, id, "alice") + }) + if err != nil { + t.Fatalf("expected verified-rollback replay to succeed, got: %v", err) + } + if calls != 2 { + t.Fatalf("expected exactly one replay (2 write runs), got %d", calls) + } + + assignee, status, err := s.readClaimState(ctx, id) + if err != nil { + t.Fatalf("read claim state: %v", err) + } + if assignee != "alice" || status != types.StatusInProgress { + t.Fatalf("replayed claim not in effect: assignee=%q status=%q", assignee, status) + } +} + +// TestVerifiedClaimWriteFailsLoudlyOnLostWrite: the false-success direction — +// the write reported success but the claim is not in the database. The caller +// must get a loud non-nil error, never a silent phantom claim. +func TestVerifiedClaimWriteFailsLoudlyOnLostWrite(t *testing.T) { + s, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + id := claimVerifyTestIssue(t, s) + + err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error { + return nil // lie: report success without writing anything + }) + if err == nil { + t.Fatal("expected loud failure for a success-reported-but-lost claim, got nil") + } + if !strings.Contains(err.Error(), "did not land") { + t.Fatalf("error should say the claim did not land, got: %v", err) + } +} + +// TestVerifiedClaimWriteIndeterminateStaysIndeterminateTwice: a replay that +// itself dies at commit phase with nothing landed must not loop forever — the +// one-replay budget exhausts and the caller gets the verified-rollback error. +func TestVerifiedClaimWriteIndeterminateStaysIndeterminateTwice(t *testing.T) { + s, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + id := claimVerifyTestIssue(t, s) + + calls := 0 + err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error { + calls++ + return fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase) + }) + if err == nil { + t.Fatal("expected failure after replay budget exhausted, got nil") + } + if calls != 2 { + t.Fatalf("expected exactly 2 write runs (original + one replay), got %d", calls) + } + if !strings.Contains(err.Error(), "did not land") { + t.Fatalf("error should report the verified rollback, got: %v", err) + } +} + +// TestVerifiedReadyClaimReplayConvertsAppliedIndeterminate: the replay leg of +// the bespoke ready-claim path must keep the wrapper's verify semantics (lion +// review on PR #5006). First attempt: commit-phase loss, verified rolled back +// (nothing landed) -> replay. The replay actually lands the claim but ALSO +// reports commit-phase loss (the wy-x543k direction, now on attempt two). The +// verify pass must convert that into an accurate success instead of returning +// the raw indeterminate error — which would leave an applied claim orphaned +// on an issue the caller was told it failed to claim. +func TestVerifiedReadyClaimReplayConvertsAppliedIndeterminate(t *testing.T) { + s, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + id := claimVerifyTestIssue(t, s) + + calls := 0 + got, err := s.verifiedReadyClaim(ctx, "alice", func() (*types.Issue, error) { + calls++ + if calls == 1 { + // Commit-phase loss, transaction rolled back: nothing landed. + return &types.Issue{ID: id}, fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase) + } + // Replay: the claim lands, but the connection dies during commit again. + if err := rawClaim(t, s, id, "alice"); err != nil { + return nil, err + } + return &types.Issue{ID: id}, fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase) + }) + if err != nil { + t.Fatalf("expected applied-indeterminate replay to resolve to success, got: %v", err) + } + if got == nil || got.ID != id { + t.Fatalf("recovered success must return the claimed issue %s, got %+v", id, got) + } + if calls != 2 { + t.Fatalf("expected exactly one replay (2 write runs), got %d", calls) + } + + assignee, status, verr := s.readClaimState(ctx, id) + if verr != nil { + t.Fatalf("read claim state: %v", verr) + } + if assignee != "alice" || status != types.StatusInProgress { + t.Fatalf("claim not in effect after recovered replay: assignee=%q status=%q", assignee, status) + } +} + +// TestClaimUnclaimVerifiedEndToEnd: the public paths still work with the +// verify layer in place — a healthy claim and unclaim pass their +// postconditions and leave the expected states. +func TestClaimUnclaimVerifiedEndToEnd(t *testing.T) { + s, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + id := claimVerifyTestIssue(t, s) + + if err := s.ClaimIssue(ctx, id, "alice"); err != nil { + t.Fatalf("claim: %v", err) + } + assignee, status, err := s.readClaimState(ctx, id) + if err != nil { + t.Fatalf("read claim state: %v", err) + } + if assignee != "alice" || status != types.StatusInProgress { + t.Fatalf("post-claim state: assignee=%q status=%q", assignee, status) + } + + // Idempotent re-claim by the same actor still succeeds through verify. + if err := s.ClaimIssue(ctx, id, "alice"); err != nil { + t.Fatalf("idempotent re-claim: %v", err) + } + + if err := s.UnclaimIssue(ctx, id, "alice", false); err != nil { + t.Fatalf("unclaim: %v", err) + } + assignee, status, err = s.readClaimState(ctx, id) + if err != nil { + t.Fatalf("read claim state: %v", err) + } + if assignee != "" || status != types.StatusOpen { + t.Fatalf("post-unclaim state: assignee=%q status=%q", assignee, status) + } +} diff --git a/internal/storage/dolt/issues.go b/internal/storage/dolt/issues.go index d19b38ebb..4fb313a2d 100644 --- a/internal/storage/dolt/issues.go +++ b/internal/storage/dolt/issues.go @@ -4,11 +4,15 @@ import ( "context" "database/sql" "encoding/json" + "errors" "fmt" "sort" "strings" "time" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + "github.com/steveyegge/beads/internal/idgen" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/issueops" @@ -304,22 +308,26 @@ func (s *DoltStore) ClaimIssue(ctx context.Context, id string, actor string) err // (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/) — so retry is the // only safety net under concurrent claimants. The body stays a single tx // (CAS + DOLT_COMMIT); withRetryTx owns BeginTx and the final Commit. - return s.withRetryTx(ctx, func(tx *sql.Tx) error { - if _, err := issueops.ClaimIssueInTx(ctx, tx, id, actor); err != nil { - return err - } + // The whole write is then resolved by verify-by-re-read (bd-zccb9): under a + // degraded server the exit status is not truth in either direction. + return s.verifiedClaimWrite(ctx, id, claimedBy(actor), func() error { + return s.withRetryTx(ctx, func(tx *sql.Tx) error { + if _, err := issueops.ClaimIssueInTx(ctx, tx, id, actor); err != nil { + return err + } - // Dolt versioning for permanent issues. - // GH#2455: Stage only the tables we modified, then commit without -A. - for _, table := range []string{"issues", "events"} { - _, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table) - } - commitMsg := fmt.Sprintf("bd: claim %s", id) - if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)", - commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) { - return fmt.Errorf("dolt commit: %w", err) - } - return nil + // Dolt versioning for permanent issues. + // GH#2455: Stage only the tables we modified, then commit without -A. + for _, table := range []string{"issues", "events"} { + _, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table) + } + commitMsg := fmt.Sprintf("bd: claim %s", id) + if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)", + commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) { + return fmt.Errorf("dolt commit: %w", err) + } + return nil + }) }) } @@ -332,31 +340,99 @@ func (s *DoltStore) ClaimReadyIssue(ctx context.Context, filter types.WorkFilter // no real row locking — FOR UPDATE / SKIP LOCKED are parse-only no-ops // (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/) — so retry is the // safety net. withRetryTx owns BeginTx and the final Commit. - var claimed *types.Issue - err := s.withRetryTx(ctx, func(tx *sql.Tx) error { - var err error - claimed, err = issueops.ClaimReadyIssueInTx(ctx, tx, filter, actor) - if err != nil { - return err - } - if claimed == nil { - return nil - } + write := func() (*types.Issue, error) { + var claimed *types.Issue + err := s.withRetryTx(ctx, func(tx *sql.Tx) error { + var err error + claimed, err = issueops.ClaimReadyIssueInTx(ctx, tx, filter, actor) + if err != nil { + return err + } + if claimed == nil { + return nil + } - for _, table := range []string{"issues", "events"} { - _, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table) - } - commitMsg := fmt.Sprintf("bd: claim ready %s", claimed.ID) - if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)", - commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) { - return fmt.Errorf("dolt commit: %w", err) - } - return nil - }) - if err != nil { + for _, table := range []string{"issues", "events"} { + _, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table) + } + commitMsg := fmt.Sprintf("bd: claim ready %s", claimed.ID) + if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)", + commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) { + return fmt.Errorf("dolt commit: %w", err) + } + return nil + }) + return claimed, err + } + return s.verifiedReadyClaim(ctx, actor, write) +} + +// verifiedReadyClaim resolves a ready-claim write by verify-by-re-read +// (bd-zccb9): the claimed ID is only known once the write body has run, so +// this cannot ride verifiedClaimWrite's id parameter — but the resolution +// protocol is the same. A replay after a verified rollback re-scans the ready +// front and may legitimately claim a different issue. Split from +// ClaimReadyIssue so injection tests can drive the write seam directly, the +// same way the verifiedClaimWrite tests do. +func (s *DoltStore) verifiedReadyClaim(ctx context.Context, actor string, write func() (*types.Issue, error)) (*types.Issue, error) { + claimed, err := write() + if err != nil && !(errors.Is(err, errCommitPhase) && claimed != nil && s.serverMode) { return nil, err } - return claimed, nil + if claimed == nil || !s.serverMode { + return claimed, err + } + for attempt := 0; ; attempt++ { + assignee, status, verr := s.readClaimState(ctx, claimed.ID) + if verr != nil { + if err != nil { + return nil, err // the honest indeterminate error from withRetryTx + } + return nil, fmt.Errorf("ready claim of %s reported success but could not be verified (server degraded?): %w — re-read the issue before trusting the claim", + claimed.ID, verr) + } + post := claimedBy(actor) + if post.want(assignee, status) { + if err != nil { + doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes( + attribute.String("op", "ready-claim"), attribute.String("outcome", "applied"))) + } + return claimed, nil + } + if err != nil && attempt < 1 { + // Verified rolled back: nothing landed, safe to replay once. + doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes( + attribute.String("op", "ready-claim"), attribute.String("outcome", "replayed"))) + claimed, err = write() + // The replay carries the same commit-phase ambiguity as the first + // attempt: an errCommitPhase with a selection made must fall + // through to the verify pass — attempt is past the replay budget + // now, so a second verified rollback still exhausts with the + // honest "did not land" error — never surface the raw + // indeterminate error, which would reintroduce the wy-x543k + // false-failure and could orphan an applied claim on a DIFFERENT + // ready issue than the first attempt selected. + if err != nil && !(errors.Is(err, errCommitPhase) && claimed != nil) { + return nil, err + } + if claimed == nil { + if err != nil { + // Commit-phase loss before anything was selected: nothing + // to verify, honest indeterminate error. + return nil, err + } + return nil, nil + } + continue + } + if err != nil { + return nil, fmt.Errorf("ready claim of %s did not land (connection lost during commit; rollback verified by re-read): %w", claimed.ID, err) + } + doltMetrics.claimVerifyLost.Add(ctx, 1, metric.WithAttributes( + attribute.String("op", "ready-claim"))) + return nil, fmt.Errorf("ready claim of %s reported success but did not land (found assignee=%q status=%q, want %s) — server likely degraded; treat the claim as NOT applied", + claimed.ID, assignee, status, post.desc) + } } // HeartbeatIssue refreshes the lease on an issue actor holds in_progress, @@ -419,21 +495,25 @@ func (s *DoltStore) ReclaimExpiredLeases(ctx context.Context, olderThan time.Dur // writer that loses Dolt's optimistic commit-time merge (1213/1205) is retried // rather than surfaced as a hard failure. func (s *DoltStore) UnclaimIssue(ctx context.Context, id string, actor string, force bool) error { - return s.withRetryTx(ctx, func(tx *sql.Tx) error { - if err := issueops.UnclaimIssueInTx(ctx, tx, id, actor, force); err != nil { - return err - } + // verify-by-re-read (bd-zccb9): a phantom unclaim leaves the caller + // believing the issue is released while it still holds the claim. + return s.verifiedClaimWrite(ctx, id, unclaimed(), func() error { + return s.withRetryTx(ctx, func(tx *sql.Tx) error { + if err := issueops.UnclaimIssueInTx(ctx, tx, id, actor, force); err != nil { + return err + } - // Dolt versioning for permanent issues. - for _, table := range []string{"issues", "events"} { - _, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table) - } - commitMsg := fmt.Sprintf("bd: unclaim %s", id) - if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)", - commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) { - return fmt.Errorf("dolt commit: %w", err) - } - return nil + // Dolt versioning for permanent issues. + for _, table := range []string{"issues", "events"} { + _, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table) + } + commitMsg := fmt.Sprintf("bd: unclaim %s", id) + if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)", + commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) { + return fmt.Errorf("dolt commit: %w", err) + } + return nil + }) }) } @@ -445,21 +525,24 @@ func (s *DoltStore) UnclaimIssue(ctx context.Context, id string, actor string, f // UnclaimIssue so a concurrent writer that loses Dolt's optimistic commit-time // merge is retried rather than surfaced as a hard failure. func (s *DoltStore) UnclaimIssueIfAssignee(ctx context.Context, id string, actor string, expectedAssignee string) error { - return s.withRetryTx(ctx, func(tx *sql.Tx) error { - if err := issueops.UnclaimIssueIfAssigneeInTx(ctx, tx, id, actor, expectedAssignee); err != nil { - return err - } + // verify-by-re-read (bd-zccb9), same reasoning as UnclaimIssue. + return s.verifiedClaimWrite(ctx, id, unclaimed(), func() error { + return s.withRetryTx(ctx, func(tx *sql.Tx) error { + if err := issueops.UnclaimIssueIfAssigneeInTx(ctx, tx, id, actor, expectedAssignee); err != nil { + return err + } - // Dolt versioning for permanent issues. - for _, table := range []string{"issues", "events"} { - _, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table) - } - commitMsg := fmt.Sprintf("bd: unclaim %s", id) - if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)", - commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) { - return fmt.Errorf("dolt commit: %w", err) - } - return nil + // Dolt versioning for permanent issues. + for _, table := range []string{"issues", "events"} { + _, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table) + } + commitMsg := fmt.Sprintf("bd: unclaim %s", id) + if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)", + commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) { + return fmt.Errorf("dolt commit: %w", err) + } + return nil + }) }) } diff --git a/internal/storage/dolt/store.go b/internal/storage/dolt/store.go index ed1f627a4..6f415c602 100644 --- a/internal/storage/dolt/store.go +++ b/internal/storage/dolt/store.go @@ -668,15 +668,17 @@ var doltTracer = otel.Tracer("github.com/steveyegge/beads/storage/dolt") // Instruments are registered against the global delegating provider at init time, // so they automatically forward to the real provider once telemetry.Init() runs. var doltMetrics struct { - retryCount metric.Int64Counter - lockWaitMs metric.Float64Histogram - circuitTrips metric.Int64Counter - circuitRejected metric.Int64Counter - serializationErrors metric.Int64Counter - writeRetries metric.Int64Counter - connAcquireMs metric.Float64Histogram - poolWaitCount metric.Int64Counter - poolWaitMs metric.Float64Histogram + retryCount metric.Int64Counter + lockWaitMs metric.Float64Histogram + circuitTrips metric.Int64Counter + circuitRejected metric.Int64Counter + serializationErrors metric.Int64Counter + writeRetries metric.Int64Counter + connAcquireMs metric.Float64Histogram + poolWaitCount metric.Int64Counter + poolWaitMs metric.Float64Histogram + claimVerifyLost metric.Int64Counter + claimVerifyRecovered metric.Int64Counter } func init() { @@ -717,6 +719,14 @@ func init() { metric.WithDescription("Total time connections spent waiting due to pool exhaustion"), metric.WithUnit("ms"), ) + doltMetrics.claimVerifyLost, _ = m.Int64Counter("bd.claim_verify_lost_total", + metric.WithDescription("Claim-family writes that reported success but failed verify-by-re-read (label: op=claim|unclaim)"), + metric.WithUnit("{write}"), + ) + doltMetrics.claimVerifyRecovered, _ = m.Int64Counter("bd.claim_verify_recovered_total", + metric.WithDescription("Indeterminate claim-family commits resolved by re-read (label: op, outcome=applied|replayed)"), + metric.WithUnit("{write}"), + ) } // registerPoolGauges registers observable gauges that report sql.DB pool stats