fix(claim): verify claim-family writes by re-read under a degraded server (bd-zccb9) (#5006)

* fix(claim): verify claim-family writes by re-read under a degraded server (bd-zccb9)

From wyvern incident wy-ejph3: under a degraded dolt sql-server the exit
status of a claim is not truth in either direction — a reported success
can roll back with the abandoned connection (phantom claim), and a
printed connection error can mask an applied write (wy-x543k).

New verifiedClaimWrite resolves claim-family outcomes against the DB
instead of trusting exit status: reported success is verified by
re-read and fails loudly on mismatch; an indeterminate commit-phase
loss (errCommitPhase) is settled by re-read — applied becomes success,
verified-rolled-back is replayed once. Wired into ClaimIssue,
UnclaimIssue, UnclaimIssueIfAssignee, and ClaimReadyIssue (bespoke:
claimed ID only known post-body; a replay may legitimately claim a
different issue). Wisps and non-server mode exempt.

Metrics: bd.claim_verify_lost_total, bd.claim_verify_recovered_total
(op, outcome=applied|replayed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(proposal): CAS conditional update design for bd-wsqvw (from fly)

Design handoff from wyvern's wheelhouse hostile-review (epic wy-mdi5h):
generalize the claim-side CAS into --if-assignee/--if-status guards on
'bd update', covering the X->Y reassign and status-guarded
claim-on-behalf transitions that --claim and unclaim --if-assignee
cannot express. Authored by fly; committed so the untracked doc
survives workspace cleanup. Depends on bd-zccb9 (verify-after-write).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(claim): ready-claim replay leg keeps verify semantics (bd-zccb9, lion review)

The bespoke ClaimReadyIssue path's replay returned raw on any error, so a
replayed write dying at commit phase skipped the verify pass — reintroducing
the wy-x543k false-failure on attempt two, and potentially orphaning an
applied claim on a DIFFERENT ready issue than the first attempt selected.
The replay now falls through to verify on errCommitPhase-with-selection,
exactly like verifiedClaimWrite's leg: applied -> recovered success,
verified rollback -> honest 'did not land' (budget already spent).

Extracted the resolution loop as verifiedReadyClaim so injection tests can
drive the write seam; sixth test mirrors ConvertsAppliedIndeterminate on
the replay leg.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Steve Yegge 2026-07-24 07:21:24 -07:00 • committed by GitHub
commit a45199a546
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 733 additions and 72 deletions

View file

@ -200,6 +200,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
every remote-ahead case, as before
([#4259](https://github.com/gastownhall/beads/issues/4259)).
### Fixed
- **Claim-family writes are now verified by re-read in Dolt server mode**
(bd-zccb9, from wyvern incident wy-ejph3). Under a degraded sql-server the
exit status of `bd update --claim` / `bd claim` / unclaim was not truth in
either direction: a claim could report success while the server-side
transaction died with the abandoned connection and rolled back (a phantom
claim that later cost a duplicate implementation), and conversely a
connection error could print with the write actually applied. After the
claim transaction, bd now re-reads the issue's assignee and status on a
fresh connection and resolves the outcome against the database: a reported
success that did not land fails loudly ("treat the claim as NOT applied");
an ambiguous commit-phase loss is settled by the re-read — verified applied
becomes an accurate success, verified rolled back is replayed once (safe:
nothing landed). Applies to claim, ready-claim, and unclaim in server mode;
wisps and embedded mode are unchanged. New metrics
`bd.claim_verify_lost_total` and `bd.claim_verify_recovered_total` count
loud failures and converted outcomes.
## [1.1.0] - 2026-07-04
First stable release of the 1.1.0 line. It consolidates everything from

View file

@ -0,0 +1,167 @@
# PROPOSAL: general conditional (compare-and-set) update for `bd`
**Status:** design, ready for implementation as a beads PR
**Author:** design handoff (wyvern hostile-review epic wy-mdi5h)
**Repo:** `github.com/steveyegge/beads` (`~/src/beads`, branch `main` @ `3a6dcff96`)
**Motivating incidents:** wyvern wheelhouse review findings wy-mdi5h.3, .6, .7, .4 (TOCTOU park/restore/take/bulldog); upstream lineage bd-zccb9, #4727, wy-ejph3, wy-x543k.
---
## 1. Problem
A hostile review of wyvern's wheelhouse fleet scripts found that its single most
common bug class is **check-then-act on `bd` assignee/status**: park, interactive
restore, queue-take, and bulldog-claim each read a bead's state, then issue a
*blind* `bd update -a X` / `-s Y`, and lose to a racing writer in the gap. The
scripts paper over it with re-reads (`wh-take.sh`), but re-reads only narrow the
window; they cannot close it. The root cause is not a wheelhouse bug — it is that
**`bd` exposes no general compare-and-set for the coordination fields.**
`bd` already recognizes this need and has been closing it verb by verb:
| primitive | CAS semantics | added |
|---|---|---|
| `bd update --claim` | `assignee ∈ {'', pool-alias} → me`, `status → in_progress` | (existing) |
| `bd unclaim --if-assignee X` | `assignee == X → ''`, `status → open` | #4727 |
| `verifiedClaimWrite` (WIP) | verify-after-write for the above under a degraded server | bd-zccb9 (uncommitted in tree) |
The pattern is a `UPDATE … WHERE id=? AND status IN(…) AND assignee=?` with
`RowsAffected` as the verdict and a `row_lock` rewrite so a racing reclaim/close
conflicts instead of silently merging (`issueops.UnclaimIssueIfAssigneeInTx`).
This proposal **generalizes that idiom once** instead of minting a third and
fourth special-case verb, and covers the two transitions no existing verb can
express.
## 2. The gap
Two wheelhouse transitions have no primitive:
- **Reassign `X → Y`** (park): "set assignee to `mayor` only if it is still the
worker I just reclaimed." `--claim` only does `'' → me`; `unclaim --if-assignee`
only does `X → ''`. Nothing does `X → Y`.
- **Claim-on-behalf with a status guard** (restore): "set assignee to the crew
owner *and* status to `in_progress`, only if the bead is still `open`/unassigned."
`--claim` always claims for `$BEADS_ACTOR`, never for a third party, and takes
no status precondition.
Queue-take (`wh-take.sh`) and bulldog-claim are **not** new-primitive gaps — they
are adoption gaps: `--claim` already handles pool aliases (`claim.pools`, e.g.
`fable-crew`) atomically, so both should simply call `--claim` and honor its
non-zero exit instead of a blind `-a` + swallowed refusal. This proposal fixes
the primitive; those two are follow-up wheelhouse patches (wy-mdi5h.7, .4).
## 3. Proposal: `--if-assignee` / `--if-status` guards on `bd update`
Add two precondition flags to the existing `update` command. When either is
present, the mutation becomes an atomic CAS; when neither is present, `update` is
unchanged.
```
bd update <id> --if-assignee <expected> [--if-status <expected>] [ -a <new> ] [ -s <new> ] [ …other field flags ]
```
Semantics:
- **Precondition** = the conjunction of every `--if-*` guard present. The row must
currently match *all* of them or the write does not apply.
- **`--if-assignee ''`** means "expected unassigned." Presence is detected with
`cmd.Flags().Changed("if-assignee")`, exactly as `unclaim` distinguishes an
explicit empty value (unclaim.go:59–61) — so `--if-assignee ""` is a real
guard, not "no guard."
- **Atomicity**: one `UPDATE … SET <fields> WHERE id=? AND assignee=? [AND status=?]`
in a single tx, `RowsAffected==1` is the verdict, `row_lock` rewritten to
`freshRowLock()` (same invariant as unclaim — a racing reclaim/close on the row
must conflict, not merge). The guard read and the UPDATE run in the same
transaction, so there is no internal TOCTOU (mirrors `UnclaimIssueIfAssigneeInTx`).
- **Precondition failure** (`RowsAffected==0`): return a typed
`storage.ErrAssigneeMismatch` / new `ErrStatusMismatch`, surfaced as a **loud,
non-zero exit** with actual-vs-expected in the message. It must **never** be
swallowed or collapse to exit 0 — this is the whole point; the CLI already has
the "requested-but-lost claim ⇒ SilentExit()/non-zero" plumbing for `--claim`
(update_proxied_server.go, "beads audit finding #10") and this reuses it.
- **Verify-after-write**: route through the WIP `verifiedClaimWrite` (a guarded
reassign is coordination-critical and idempotent for the winner — safe to
replay when a re-read proves the commit rolled back). Add a `reassigned(actor)`
/ `matched(post)` `claimPostcondition` alongside `claimedBy`/`unclaimed`.
- **Wisps exempt**, closed issues rejected — same as the sibling verbs.
- **Batch/JSON**: same mixed-batch exit-code rule as `--claim` (one lost
precondition in a batch ⇒ non-zero overall), same JSON shape.
### Why guards on `update` rather than a new `bd reassign` verb
- The transitions we need are *arbitrary* field CAS (assignee and/or status), not
a single named operation; a verb per transition is how we got here.
- `update` already owns multi-field mutation, batching, JSON, proxied-server
routing, and the finding-#10 exit-code contract — guards compose with all of it
for free.
- `--claim` and `unclaim --if-assignee` remain the ergonomic shorthands for their
two hot paths; the general guards are the escape hatch the wrappers currently
fake with `bd sql`.
Alternative considered — **dedicated `bd reassign --from --to`**: more
discoverable for the park case, but doesn't express restore's status-guarded
claim-on-behalf without yet more flags, and duplicates update's plumbing. Rejected
in favor of the general guards. (Flag it for maintainer preference — it is a CLI
surface call, not a correctness call.)
## 4. How each wheelhouse finding collapses to it
```sh
# park (wy-mdi5h.3) — reassign only if the worker still holds it:
bd update "$id" --if-assignee "$reclaimed_worker" -a "$PARK_ASSIGNEE"
# interactive restore (wy-mdi5h.6) — claim-on-behalf, only while still open:
bd update "$id" --if-assignee '' --if-status open -a "$owner" -s in_progress
# queue-take (wy-mdi5h.7) — ADOPTION, no new primitive: use pool-aware claim
bd update "$id" --claim # fable-crew is a claim.pools alias
# bulldog (wy-mdi5h.4) — ADOPTION: claim + honor the non-zero exit, don't swallow
bd update "$BEAD" --claim || { echo "foreign/lost; not mutating"; exit 1; }
```
The first two are the primitive this proposal adds; the last two are wheelhouse
patches that consume the *existing* `--claim`.
## 5. Implementation sketch (files)
- `internal/storage/issueops/update_cas.go` (new): `UpdateIssueIfMatchInTx(ctx,
tx, id, guards, updates)` — build the `WHERE` from present guards, reuse the
unclaim.go CAS body (row_lock rewrite, `status IN ('open','in_progress')`
liveness clause, RowsAffected verdict, typed mismatch error incl. actual state).
- `internal/storage/storage.go`: add `ErrStatusMismatch`; add
`UpdateIssueIfMatch(ctx, id, actor, guards, updates)` to the `Storage` interface.
- `internal/storage/dolt/issues.go`: `DoltStore.UpdateIssueIfMatch` wrapping
`verifiedClaimWrite` with a `matched(post)` postcondition + DOLT_ADD/COMMIT.
- `cmd/bd/update.go`: register `--if-assignee` (String) and `--if-status`
(String); when either `Changed()`, route to the CAS path; wire the
precondition-failed → non-zero exit through the existing `claimFailed` machinery
(both `update.go` and `update_proxied_server.go`). Reject `--if-*` combined with
`--claim` (mutually exclusive; `--claim` is its own CAS).
- `cmd/bd/prime.go`: document the new guards next to `--claim`.
## 6. Tests
Beads side (their conventions):
- `internal/storage/issueops` unit tests: match applies, assignee-mismatch no-op,
status-mismatch no-op, `--if-assignee ''` matches only unassigned, closed-issue
reject, row_lock rewritten on success.
- A concurrency test in the spirit of the existing claim CAS tests: two goroutines
reassign the same row, exactly one gets `RowsAffected==1`.
- `verifiedClaimWrite` path: reported-success-but-rolled-back ⇒ replay; applied ⇒
success; genuine mismatch ⇒ loud non-zero.
- CLI: non-zero exit on precondition failure, mixed-batch exit code, JSON shape.
Wyvern side (house rule — each consuming fix gets a `test-*.sh` pin):
- `test-park-cas.sh`, `test-restore-cas.sh` asserting a racing claim in the window
is refused (non-zero) rather than clobbered.
## 7. Scope / non-goals
- **In:** assignee + status guards on `update`; storage + CLI + verify integration;
the two wheelhouse consumers that are genuine primitive gaps.
- **Out:** label/other-field guards (`--if-label`) — no current consumer; add later
if one appears. The wheelhouse *adoption* patches (queue-take, bulldog switching
to `--claim`) are separate wy-mdi5h children, not part of this beads PR.
- **Coordinate with the uncommitted `claim_verify.go` WIP** already in the tree —
this proposal depends on `verifiedClaimWrite` landing (or lands alongside it).

View file

@ -0,0 +1,138 @@
package dolt
import (
"context"
"database/sql"
"errors"
"fmt"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
"github.com/steveyegge/beads/internal/types"
)
// Claim-family verify-after-write (bd-zccb9, from wyvern incident wy-ejph3).
//
// Under a degraded sql-server the write's exit status is not truth in either
// direction: a claim can report success while the server-side transaction dies
// with the abandoned connection and rolls back (wy-ejph3: the lost claim cost a
// duplicate full implementation), and the inverse — an error printed with the
// write actually applied — is on record from wy-x543k. Fleet wrappers solved
// this with verify-by-re-read (wyvern's wh-take.sh); this file folds that
// protocol into bd itself for the coordination-critical writes: claim, ready
// claim, and unclaim. Ordinary field updates are out of scope — a lost label
// edit is an annoyance, a phantom claim is a duplicated implementation.
//
// The protocol: run the write, then resolve its outcome against the database
// instead of trusting the exit status.
//
// - reported success -> verify; a mismatch fails LOUDLY (the caller
// must know it does not hold the claim)
// - ambiguous commit loss -> (errCommitPhase, surfaced by withRetryTx as
// indeterminate) verify; applied -> success, verified rolled back -> replay
// the write once (safe: nothing landed)
// - any other error -> honest failure (CAS lost, not-claimable, or
// pre-commit errors withRetryTx already retried); no verify needed
//
// Wisps are exempt: they are ephemeral, never leased, and not reclaimable
// coordination state.
// claimPostcondition describes the row state a claim-family write must leave
// behind to count as applied, plus the words to use when it didn't.
type claimPostcondition struct {
op string // "claim" | "unclaim", for messages and metrics
want func(assignee string, status types.Status) bool
desc string // expected state, human-readable, for mismatch messages
}
func claimedBy(actor string) claimPostcondition {
return claimPostcondition{
op: "claim",
want: func(assignee string, status types.Status) bool {
return assignee == actor && status == types.StatusInProgress
},
desc: fmt.Sprintf("assignee=%q status=%q", actor, types.StatusInProgress),
}
}
func unclaimed() claimPostcondition {
return claimPostcondition{
op: "unclaim",
want: func(assignee string, status types.Status) bool {
return assignee == "" && status == types.StatusOpen
},
desc: fmt.Sprintf("assignee=%q status=%q", "", types.StatusOpen),
}
}
// readClaimState re-reads an issue's assignee and status on a fresh
// transaction. withReadTx retries transient connection errors itself, so a
// failure here means the server is genuinely unreachable, not a blip.
func (s *DoltStore) readClaimState(ctx context.Context, id string) (string, types.Status, error) {
var assignee sql.NullString
var status types.Status
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
return tx.QueryRowContext(ctx,
`SELECT assignee, status FROM issues WHERE id = ?`, id,
).Scan(&assignee, &status)
})
if err != nil {
return "", "", err
}
if !assignee.Valid {
return "", status, nil
}
return assignee.String, status, nil
}
// verifiedClaimWrite runs write and resolves its outcome against the database
// state per the protocol above. write must be safe to run twice when its first
// run verifiably did not land (all claim-family writes are: they are CAS
// updates, idempotent for the winning actor).
//
// A verify that contradicts a reported success can in principle also be a
// legitimate concurrent mutation (a forced unclaim landing within the
// verification window). That reads as a lost write and fails loudly too —
// acceptable: the caller must re-establish its view either way.
func (s *DoltStore) verifiedClaimWrite(ctx context.Context, id string, post claimPostcondition, write func() error) error {
if !s.serverMode || s.isActiveWisp(ctx, id) {
return write()
}
const maxReplays = 1
for attempt := 0; ; attempt++ {
err := write()
if err != nil && !errors.Is(err, errCommitPhase) {
return err
}
assignee, status, verr := s.readClaimState(ctx, id)
if verr != nil {
if err != nil {
return err // the honest indeterminate error from withRetryTx
}
return fmt.Errorf("%s of %s reported success but could not be verified (server degraded?): %w — re-read the issue before trusting the %s",
post.op, id, verr, post.op)
}
if post.want(assignee, status) {
if err != nil {
doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes(
attribute.String("op", post.op), attribute.String("outcome", "applied")))
}
return nil
}
if err != nil {
if attempt < maxReplays {
// Verified rolled back: nothing landed, so one replay is safe.
doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes(
attribute.String("op", post.op), attribute.String("outcome", "replayed")))
continue
}
return fmt.Errorf("%s of %s did not land (connection lost during commit; rollback verified by re-read): %w",
post.op, id, err)
}
doltMetrics.claimVerifyLost.Add(ctx, 1, metric.WithAttributes(
attribute.String("op", post.op)))
return fmt.Errorf("%s of %s reported success but did not land (found assignee=%q status=%q, want %s) — server likely degraded; treat the %s as NOT applied",
post.op, id, assignee, status, post.desc, post.op)
}
}

View file

@ -0,0 +1,244 @@
package dolt
import (
"database/sql"
"fmt"
"strings"
"testing"
"github.com/steveyegge/beads/internal/storage/issueops"
"github.com/steveyegge/beads/internal/types"
)
// Tests for claim-family verify-after-write (bd-zccb9, wyvern incident
// wy-ejph3): under a degraded server the write's exit status is not truth in
// either direction, so claim outcomes are resolved against the database by
// re-read. Failure injection happens at the write-func seam — the write funcs
// here lie in exactly the ways the incident documented.
func claimVerifyTestIssue(t *testing.T, s *DoltStore) string {
t.Helper()
ctx, cancel := testContext(t)
defer cancel()
issue := &types.Issue{
Title: "claim verify target",
IssueType: types.TypeTask,
Priority: 2,
Status: types.StatusOpen,
}
if err := s.CreateIssue(ctx, issue, "tester"); err != nil {
t.Fatalf("create issue: %v", err)
}
return issue.ID
}
// rawClaim performs the real claim write without the verify wrapper — the
// inner body of ClaimIssue — so tests can control exactly which attempt lands.
func rawClaim(t *testing.T, s *DoltStore, id, actor string) error {
t.Helper()
ctx, cancel := testContext(t)
defer cancel()
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
_, err := issueops.ClaimIssueInTx(ctx, tx, id, actor)
return err
})
}
// TestVerifiedClaimWriteConvertsAppliedIndeterminate: the wy-x543k direction —
// the write actually landed but the connection died during commit and bd
// printed an error. Verify-by-re-read must convert it into an accurate success.
func TestVerifiedClaimWriteConvertsAppliedIndeterminate(t *testing.T) {
s, cleanup := setupTestStore(t)
defer cleanup()
ctx, cancel := testContext(t)
defer cancel()
id := claimVerifyTestIssue(t, s)
if err := rawClaim(t, s, id, "alice"); err != nil {
t.Fatalf("raw claim: %v", err)
}
calls := 0
err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error {
calls++
return fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
})
if err != nil {
t.Fatalf("expected applied-indeterminate to resolve to success, got: %v", err)
}
if calls != 1 {
t.Fatalf("write must not be replayed when the re-read shows it applied; ran %d times", calls)
}
}
// TestVerifiedClaimWriteReplaysVerifiedRollback: the wy-ejph3 direction — the
// connection died during commit AND the transaction rolled back. The re-read
// proves nothing landed, so one replay is safe and must proceed.
func TestVerifiedClaimWriteReplaysVerifiedRollback(t *testing.T) {
s, cleanup := setupTestStore(t)
defer cleanup()
ctx, cancel := testContext(t)
defer cancel()
id := claimVerifyTestIssue(t, s)
calls := 0
err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error {
calls++
if calls == 1 {
// First attempt: commit-phase connection loss, nothing landed.
return fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
}
return rawClaim(t, s, id, "alice")
})
if err != nil {
t.Fatalf("expected verified-rollback replay to succeed, got: %v", err)
}
if calls != 2 {
t.Fatalf("expected exactly one replay (2 write runs), got %d", calls)
}
assignee, status, err := s.readClaimState(ctx, id)
if err != nil {
t.Fatalf("read claim state: %v", err)
}
if assignee != "alice" || status != types.StatusInProgress {
t.Fatalf("replayed claim not in effect: assignee=%q status=%q", assignee, status)
}
}
// TestVerifiedClaimWriteFailsLoudlyOnLostWrite: the false-success direction —
// the write reported success but the claim is not in the database. The caller
// must get a loud non-nil error, never a silent phantom claim.
func TestVerifiedClaimWriteFailsLoudlyOnLostWrite(t *testing.T) {
s, cleanup := setupTestStore(t)
defer cleanup()
ctx, cancel := testContext(t)
defer cancel()
id := claimVerifyTestIssue(t, s)
err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error {
return nil // lie: report success without writing anything
})
if err == nil {
t.Fatal("expected loud failure for a success-reported-but-lost claim, got nil")
}
if !strings.Contains(err.Error(), "did not land") {
t.Fatalf("error should say the claim did not land, got: %v", err)
}
}
// TestVerifiedClaimWriteIndeterminateStaysIndeterminateTwice: a replay that
// itself dies at commit phase with nothing landed must not loop forever — the
// one-replay budget exhausts and the caller gets the verified-rollback error.
func TestVerifiedClaimWriteIndeterminateStaysIndeterminateTwice(t *testing.T) {
s, cleanup := setupTestStore(t)
defer cleanup()
ctx, cancel := testContext(t)
defer cancel()
id := claimVerifyTestIssue(t, s)
calls := 0
err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error {
calls++
return fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
})
if err == nil {
t.Fatal("expected failure after replay budget exhausted, got nil")
}
if calls != 2 {
t.Fatalf("expected exactly 2 write runs (original + one replay), got %d", calls)
}
if !strings.Contains(err.Error(), "did not land") {
t.Fatalf("error should report the verified rollback, got: %v", err)
}
}
// TestVerifiedReadyClaimReplayConvertsAppliedIndeterminate: the replay leg of
// the bespoke ready-claim path must keep the wrapper's verify semantics (lion
// review on PR #5006). First attempt: commit-phase loss, verified rolled back
// (nothing landed) -> replay. The replay actually lands the claim but ALSO
// reports commit-phase loss (the wy-x543k direction, now on attempt two). The
// verify pass must convert that into an accurate success instead of returning
// the raw indeterminate error — which would leave an applied claim orphaned
// on an issue the caller was told it failed to claim.
func TestVerifiedReadyClaimReplayConvertsAppliedIndeterminate(t *testing.T) {
s, cleanup := setupTestStore(t)
defer cleanup()
ctx, cancel := testContext(t)
defer cancel()
id := claimVerifyTestIssue(t, s)
calls := 0
got, err := s.verifiedReadyClaim(ctx, "alice", func() (*types.Issue, error) {
calls++
if calls == 1 {
// Commit-phase loss, transaction rolled back: nothing landed.
return &types.Issue{ID: id}, fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
}
// Replay: the claim lands, but the connection dies during commit again.
if err := rawClaim(t, s, id, "alice"); err != nil {
return nil, err
}
return &types.Issue{ID: id}, fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
})
if err != nil {
t.Fatalf("expected applied-indeterminate replay to resolve to success, got: %v", err)
}
if got == nil || got.ID != id {
t.Fatalf("recovered success must return the claimed issue %s, got %+v", id, got)
}
if calls != 2 {
t.Fatalf("expected exactly one replay (2 write runs), got %d", calls)
}
assignee, status, verr := s.readClaimState(ctx, id)
if verr != nil {
t.Fatalf("read claim state: %v", verr)
}
if assignee != "alice" || status != types.StatusInProgress {
t.Fatalf("claim not in effect after recovered replay: assignee=%q status=%q", assignee, status)
}
}
// TestClaimUnclaimVerifiedEndToEnd: the public paths still work with the
// verify layer in place — a healthy claim and unclaim pass their
// postconditions and leave the expected states.
func TestClaimUnclaimVerifiedEndToEnd(t *testing.T) {
s, cleanup := setupTestStore(t)
defer cleanup()
ctx, cancel := testContext(t)
defer cancel()
id := claimVerifyTestIssue(t, s)
if err := s.ClaimIssue(ctx, id, "alice"); err != nil {
t.Fatalf("claim: %v", err)
}
assignee, status, err := s.readClaimState(ctx, id)
if err != nil {
t.Fatalf("read claim state: %v", err)
}
if assignee != "alice" || status != types.StatusInProgress {
t.Fatalf("post-claim state: assignee=%q status=%q", assignee, status)
}
// Idempotent re-claim by the same actor still succeeds through verify.
if err := s.ClaimIssue(ctx, id, "alice"); err != nil {
t.Fatalf("idempotent re-claim: %v", err)
}
if err := s.UnclaimIssue(ctx, id, "alice", false); err != nil {
t.Fatalf("unclaim: %v", err)
}
assignee, status, err = s.readClaimState(ctx, id)
if err != nil {
t.Fatalf("read claim state: %v", err)
}
if assignee != "" || status != types.StatusOpen {
t.Fatalf("post-unclaim state: assignee=%q status=%q", assignee, status)
}
}

View file

@ -4,11 +4,15 @@ import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"sort"
"strings"
"time"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
"github.com/steveyegge/beads/internal/idgen"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/issueops"
@ -304,22 +308,26 @@ func (s *DoltStore) ClaimIssue(ctx context.Context, id string, actor string) err
// (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/) — so retry is the
// only safety net under concurrent claimants. The body stays a single tx
// (CAS + DOLT_COMMIT); withRetryTx owns BeginTx and the final Commit.
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
if _, err := issueops.ClaimIssueInTx(ctx, tx, id, actor); err != nil {
return err
}
// The whole write is then resolved by verify-by-re-read (bd-zccb9): under a
// degraded server the exit status is not truth in either direction.
return s.verifiedClaimWrite(ctx, id, claimedBy(actor), func() error {
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
if _, err := issueops.ClaimIssueInTx(ctx, tx, id, actor); err != nil {
return err
}
// Dolt versioning for permanent issues.
// GH#2455: Stage only the tables we modified, then commit without -A.
for _, table := range []string{"issues", "events"} {
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
}
commitMsg := fmt.Sprintf("bd: claim %s", id)
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
return fmt.Errorf("dolt commit: %w", err)
}
return nil
// Dolt versioning for permanent issues.
// GH#2455: Stage only the tables we modified, then commit without -A.
for _, table := range []string{"issues", "events"} {
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
}
commitMsg := fmt.Sprintf("bd: claim %s", id)
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
return fmt.Errorf("dolt commit: %w", err)
}
return nil
})
})
}
@ -332,31 +340,99 @@ func (s *DoltStore) ClaimReadyIssue(ctx context.Context, filter types.WorkFilter
// no real row locking — FOR UPDATE / SKIP LOCKED are parse-only no-ops
// (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/) — so retry is the
// safety net. withRetryTx owns BeginTx and the final Commit.
var claimed *types.Issue
err := s.withRetryTx(ctx, func(tx *sql.Tx) error {
var err error
claimed, err = issueops.ClaimReadyIssueInTx(ctx, tx, filter, actor)
if err != nil {
return err
}
if claimed == nil {
return nil
}
write := func() (*types.Issue, error) {
var claimed *types.Issue
err := s.withRetryTx(ctx, func(tx *sql.Tx) error {
var err error
claimed, err = issueops.ClaimReadyIssueInTx(ctx, tx, filter, actor)
if err != nil {
return err
}
if claimed == nil {
return nil
}
for _, table := range []string{"issues", "events"} {
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
}
commitMsg := fmt.Sprintf("bd: claim ready %s", claimed.ID)
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
return fmt.Errorf("dolt commit: %w", err)
}
return nil
})
if err != nil {
for _, table := range []string{"issues", "events"} {
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
}
commitMsg := fmt.Sprintf("bd: claim ready %s", claimed.ID)
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
return fmt.Errorf("dolt commit: %w", err)
}
return nil
})
return claimed, err
}
return s.verifiedReadyClaim(ctx, actor, write)
}
// verifiedReadyClaim resolves a ready-claim write by verify-by-re-read
// (bd-zccb9): the claimed ID is only known once the write body has run, so
// this cannot ride verifiedClaimWrite's id parameter — but the resolution
// protocol is the same. A replay after a verified rollback re-scans the ready
// front and may legitimately claim a different issue. Split from
// ClaimReadyIssue so injection tests can drive the write seam directly, the
// same way the verifiedClaimWrite tests do.
func (s *DoltStore) verifiedReadyClaim(ctx context.Context, actor string, write func() (*types.Issue, error)) (*types.Issue, error) {
claimed, err := write()
if err != nil && !(errors.Is(err, errCommitPhase) && claimed != nil && s.serverMode) {
return nil, err
}
return claimed, nil
if claimed == nil || !s.serverMode {
return claimed, err
}
for attempt := 0; ; attempt++ {
assignee, status, verr := s.readClaimState(ctx, claimed.ID)
if verr != nil {
if err != nil {
return nil, err // the honest indeterminate error from withRetryTx
}
return nil, fmt.Errorf("ready claim of %s reported success but could not be verified (server degraded?): %w — re-read the issue before trusting the claim",
claimed.ID, verr)
}
post := claimedBy(actor)
if post.want(assignee, status) {
if err != nil {
doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes(
attribute.String("op", "ready-claim"), attribute.String("outcome", "applied")))
}
return claimed, nil
}
if err != nil && attempt < 1 {
// Verified rolled back: nothing landed, safe to replay once.
doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes(
attribute.String("op", "ready-claim"), attribute.String("outcome", "replayed")))
claimed, err = write()
// The replay carries the same commit-phase ambiguity as the first
// attempt: an errCommitPhase with a selection made must fall
// through to the verify pass — attempt is past the replay budget
// now, so a second verified rollback still exhausts with the
// honest "did not land" error — never surface the raw
// indeterminate error, which would reintroduce the wy-x543k
// false-failure and could orphan an applied claim on a DIFFERENT
// ready issue than the first attempt selected.
if err != nil && !(errors.Is(err, errCommitPhase) && claimed != nil) {
return nil, err
}
if claimed == nil {
if err != nil {
// Commit-phase loss before anything was selected: nothing
// to verify, honest indeterminate error.
return nil, err
}
return nil, nil
}
continue
}
if err != nil {
return nil, fmt.Errorf("ready claim of %s did not land (connection lost during commit; rollback verified by re-read): %w", claimed.ID, err)
}
doltMetrics.claimVerifyLost.Add(ctx, 1, metric.WithAttributes(
attribute.String("op", "ready-claim")))
return nil, fmt.Errorf("ready claim of %s reported success but did not land (found assignee=%q status=%q, want %s) — server likely degraded; treat the claim as NOT applied",
claimed.ID, assignee, status, post.desc)
}
}
// HeartbeatIssue refreshes the lease on an issue actor holds in_progress,
@ -419,21 +495,25 @@ func (s *DoltStore) ReclaimExpiredLeases(ctx context.Context, olderThan time.Dur
// writer that loses Dolt's optimistic commit-time merge (1213/1205) is retried
// rather than surfaced as a hard failure.
func (s *DoltStore) UnclaimIssue(ctx context.Context, id string, actor string, force bool) error {
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
if err := issueops.UnclaimIssueInTx(ctx, tx, id, actor, force); err != nil {
return err
}
// verify-by-re-read (bd-zccb9): a phantom unclaim leaves the caller
// believing the issue is released while it still holds the claim.
return s.verifiedClaimWrite(ctx, id, unclaimed(), func() error {
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
if err := issueops.UnclaimIssueInTx(ctx, tx, id, actor, force); err != nil {
return err
}
// Dolt versioning for permanent issues.
for _, table := range []string{"issues", "events"} {
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
}
commitMsg := fmt.Sprintf("bd: unclaim %s", id)
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
return fmt.Errorf("dolt commit: %w", err)
}
return nil
// Dolt versioning for permanent issues.
for _, table := range []string{"issues", "events"} {
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
}
commitMsg := fmt.Sprintf("bd: unclaim %s", id)
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
return fmt.Errorf("dolt commit: %w", err)
}
return nil
})
})
}
@ -445,21 +525,24 @@ func (s *DoltStore) UnclaimIssue(ctx context.Context, id string, actor string, f
// UnclaimIssue so a concurrent writer that loses Dolt's optimistic commit-time
// merge is retried rather than surfaced as a hard failure.
func (s *DoltStore) UnclaimIssueIfAssignee(ctx context.Context, id string, actor string, expectedAssignee string) error {
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
if err := issueops.UnclaimIssueIfAssigneeInTx(ctx, tx, id, actor, expectedAssignee); err != nil {
return err
}
// verify-by-re-read (bd-zccb9), same reasoning as UnclaimIssue.
return s.verifiedClaimWrite(ctx, id, unclaimed(), func() error {
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
if err := issueops.UnclaimIssueIfAssigneeInTx(ctx, tx, id, actor, expectedAssignee); err != nil {
return err
}
// Dolt versioning for permanent issues.
for _, table := range []string{"issues", "events"} {
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
}
commitMsg := fmt.Sprintf("bd: unclaim %s", id)
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
return fmt.Errorf("dolt commit: %w", err)
}
return nil
// Dolt versioning for permanent issues.
for _, table := range []string{"issues", "events"} {
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
}
commitMsg := fmt.Sprintf("bd: unclaim %s", id)
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
return fmt.Errorf("dolt commit: %w", err)
}
return nil
})
})
}

View file

@ -668,15 +668,17 @@ var doltTracer = otel.Tracer("github.com/steveyegge/beads/storage/dolt")
// Instruments are registered against the global delegating provider at init time,
// so they automatically forward to the real provider once telemetry.Init() runs.
var doltMetrics struct {
retryCount metric.Int64Counter
lockWaitMs metric.Float64Histogram
circuitTrips metric.Int64Counter
circuitRejected metric.Int64Counter
serializationErrors metric.Int64Counter
writeRetries metric.Int64Counter
connAcquireMs metric.Float64Histogram
poolWaitCount metric.Int64Counter
poolWaitMs metric.Float64Histogram
retryCount metric.Int64Counter
lockWaitMs metric.Float64Histogram
circuitTrips metric.Int64Counter
circuitRejected metric.Int64Counter
serializationErrors metric.Int64Counter
writeRetries metric.Int64Counter
connAcquireMs metric.Float64Histogram
poolWaitCount metric.Int64Counter
poolWaitMs metric.Float64Histogram
claimVerifyLost metric.Int64Counter
claimVerifyRecovered metric.Int64Counter
}
func init() {
@ -717,6 +719,14 @@ func init() {
metric.WithDescription("Total time connections spent waiting due to pool exhaustion"),
metric.WithUnit("ms"),
)
doltMetrics.claimVerifyLost, _ = m.Int64Counter("bd.claim_verify_lost_total",
metric.WithDescription("Claim-family writes that reported success but failed verify-by-re-read (label: op=claim|unclaim)"),
metric.WithUnit("{write}"),
)
doltMetrics.claimVerifyRecovered, _ = m.Int64Counter("bd.claim_verify_recovered_total",
metric.WithDescription("Indeterminate claim-family commits resolved by re-read (label: op, outcome=applied|replayed)"),
metric.WithUnit("{write}"),
)
}
// registerPoolGauges registers observable gauges that report sql.DB pool stats