fix(claim): verify claim-family writes by re-read under a degraded server (bd-zccb9) (#5006)
* fix(claim): verify claim-family writes by re-read under a degraded server (bd-zccb9) From wyvern incident wy-ejph3: under a degraded dolt sql-server the exit status of a claim is not truth in either direction — a reported success can roll back with the abandoned connection (phantom claim), and a printed connection error can mask an applied write (wy-x543k). New verifiedClaimWrite resolves claim-family outcomes against the DB instead of trusting exit status: reported success is verified by re-read and fails loudly on mismatch; an indeterminate commit-phase loss (errCommitPhase) is settled by re-read — applied becomes success, verified-rolled-back is replayed once. Wired into ClaimIssue, UnclaimIssue, UnclaimIssueIfAssignee, and ClaimReadyIssue (bespoke: claimed ID only known post-body; a replay may legitimately claim a different issue). Wisps and non-server mode exempt. Metrics: bd.claim_verify_lost_total, bd.claim_verify_recovered_total (op, outcome=applied|replayed). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(proposal): CAS conditional update design for bd-wsqvw (from fly) Design handoff from wyvern's wheelhouse hostile-review (epic wy-mdi5h): generalize the claim-side CAS into --if-assignee/--if-status guards on 'bd update', covering the X->Y reassign and status-guarded claim-on-behalf transitions that --claim and unclaim --if-assignee cannot express. Authored by fly; committed so the untracked doc survives workspace cleanup. Depends on bd-zccb9 (verify-after-write). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(claim): ready-claim replay leg keeps verify semantics (bd-zccb9, lion review) The bespoke ClaimReadyIssue path's replay returned raw on any error, so a replayed write dying at commit phase skipped the verify pass — reintroducing the wy-x543k false-failure on attempt two, and potentially orphaning an applied claim on a DIFFERENT ready issue than the first attempt selected. The replay now falls through to verify on errCommitPhase-with-selection, exactly like verifiedClaimWrite's leg: applied -> recovered success, verified rollback -> honest 'did not land' (budget already spent). Extracted the resolution loop as verifiedReadyClaim so injection tests can drive the write seam; sixth test mirrors ConvertsAppliedIndeterminate on the replay leg. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
45a7a9599c
commit
a45199a546
6 changed files with 733 additions and 72 deletions
19
CHANGELOG.md
19
CHANGELOG.md
|
|
@ -200,6 +200,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
every remote-ahead case, as before
|
||||
([#4259](https://github.com/gastownhall/beads/issues/4259)).
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Claim-family writes are now verified by re-read in Dolt server mode**
|
||||
(bd-zccb9, from wyvern incident wy-ejph3). Under a degraded sql-server the
|
||||
exit status of `bd update --claim` / `bd claim` / unclaim was not truth in
|
||||
either direction: a claim could report success while the server-side
|
||||
transaction died with the abandoned connection and rolled back (a phantom
|
||||
claim that later cost a duplicate implementation), and conversely a
|
||||
connection error could print with the write actually applied. After the
|
||||
claim transaction, bd now re-reads the issue's assignee and status on a
|
||||
fresh connection and resolves the outcome against the database: a reported
|
||||
success that did not land fails loudly ("treat the claim as NOT applied");
|
||||
an ambiguous commit-phase loss is settled by the re-read — verified applied
|
||||
becomes an accurate success, verified rolled back is replayed once (safe:
|
||||
nothing landed). Applies to claim, ready-claim, and unclaim in server mode;
|
||||
wisps and embedded mode are unchanged. New metrics
|
||||
`bd.claim_verify_lost_total` and `bd.claim_verify_recovered_total` count
|
||||
loud failures and converted outcomes.
|
||||
|
||||
## [1.1.0] - 2026-07-04
|
||||
|
||||
First stable release of the 1.1.0 line. It consolidates everything from
|
||||
|
|
|
|||
167
PROPOSAL-cas-conditional-update.md
Normal file
167
PROPOSAL-cas-conditional-update.md
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
# PROPOSAL: general conditional (compare-and-set) update for `bd`
|
||||
|
||||
**Status:** design, ready for implementation as a beads PR
|
||||
**Author:** design handoff (wyvern hostile-review epic wy-mdi5h)
|
||||
**Repo:** `github.com/steveyegge/beads` (`~/src/beads`, branch `main` @ `3a6dcff96`)
|
||||
**Motivating incidents:** wyvern wheelhouse review findings wy-mdi5h.3, .6, .7, .4 (TOCTOU park/restore/take/bulldog); upstream lineage bd-zccb9, #4727, wy-ejph3, wy-x543k.
|
||||
|
||||
---
|
||||
|
||||
## 1. Problem
|
||||
|
||||
A hostile review of wyvern's wheelhouse fleet scripts found that its single most
|
||||
common bug class is **check-then-act on `bd` assignee/status**: park, interactive
|
||||
restore, queue-take, and bulldog-claim each read a bead's state, then issue a
|
||||
*blind* `bd update -a X` / `-s Y`, and lose to a racing writer in the gap. The
|
||||
scripts paper over it with re-reads (`wh-take.sh`), but re-reads only narrow the
|
||||
window; they cannot close it. The root cause is not a wheelhouse bug — it is that
|
||||
**`bd` exposes no general compare-and-set for the coordination fields.**
|
||||
|
||||
`bd` already recognizes this need and has been closing it verb by verb:
|
||||
|
||||
| primitive | CAS semantics | added |
|
||||
|---|---|---|
|
||||
| `bd update --claim` | `assignee ∈ {'', pool-alias} → me`, `status → in_progress` | (existing) |
|
||||
| `bd unclaim --if-assignee X` | `assignee == X → ''`, `status → open` | #4727 |
|
||||
| `verifiedClaimWrite` (WIP) | verify-after-write for the above under a degraded server | bd-zccb9 (uncommitted in tree) |
|
||||
|
||||
The pattern is a `UPDATE … WHERE id=? AND status IN(…) AND assignee=?` with
|
||||
`RowsAffected` as the verdict and a `row_lock` rewrite so a racing reclaim/close
|
||||
conflicts instead of silently merging (`issueops.UnclaimIssueIfAssigneeInTx`).
|
||||
This proposal **generalizes that idiom once** instead of minting a third and
|
||||
fourth special-case verb, and covers the two transitions no existing verb can
|
||||
express.
|
||||
|
||||
## 2. The gap
|
||||
|
||||
Two wheelhouse transitions have no primitive:
|
||||
|
||||
- **Reassign `X → Y`** (park): "set assignee to `mayor` only if it is still the
|
||||
worker I just reclaimed." `--claim` only does `'' → me`; `unclaim --if-assignee`
|
||||
only does `X → ''`. Nothing does `X → Y`.
|
||||
- **Claim-on-behalf with a status guard** (restore): "set assignee to the crew
|
||||
owner *and* status to `in_progress`, only if the bead is still `open`/unassigned."
|
||||
`--claim` always claims for `$BEADS_ACTOR`, never for a third party, and takes
|
||||
no status precondition.
|
||||
|
||||
Queue-take (`wh-take.sh`) and bulldog-claim are **not** new-primitive gaps — they
|
||||
are adoption gaps: `--claim` already handles pool aliases (`claim.pools`, e.g.
|
||||
`fable-crew`) atomically, so both should simply call `--claim` and honor its
|
||||
non-zero exit instead of a blind `-a` + swallowed refusal. This proposal fixes
|
||||
the primitive; those two are follow-up wheelhouse patches (wy-mdi5h.7, .4).
|
||||
|
||||
## 3. Proposal: `--if-assignee` / `--if-status` guards on `bd update`
|
||||
|
||||
Add two precondition flags to the existing `update` command. When either is
|
||||
present, the mutation becomes an atomic CAS; when neither is present, `update` is
|
||||
unchanged.
|
||||
|
||||
```
|
||||
bd update <id> --if-assignee <expected> [--if-status <expected>] [ -a <new> ] [ -s <new> ] [ …other field flags ]
|
||||
```
|
||||
|
||||
Semantics:
|
||||
|
||||
- **Precondition** = the conjunction of every `--if-*` guard present. The row must
|
||||
currently match *all* of them or the write does not apply.
|
||||
- **`--if-assignee ''`** means "expected unassigned." Presence is detected with
|
||||
`cmd.Flags().Changed("if-assignee")`, exactly as `unclaim` distinguishes an
|
||||
explicit empty value (unclaim.go:59–61) — so `--if-assignee ""` is a real
|
||||
guard, not "no guard."
|
||||
- **Atomicity**: one `UPDATE … SET <fields> WHERE id=? AND assignee=? [AND status=?]`
|
||||
in a single tx, `RowsAffected==1` is the verdict, `row_lock` rewritten to
|
||||
`freshRowLock()` (same invariant as unclaim — a racing reclaim/close on the row
|
||||
must conflict, not merge). The guard read and the UPDATE run in the same
|
||||
transaction, so there is no internal TOCTOU (mirrors `UnclaimIssueIfAssigneeInTx`).
|
||||
- **Precondition failure** (`RowsAffected==0`): return a typed
|
||||
`storage.ErrAssigneeMismatch` / new `ErrStatusMismatch`, surfaced as a **loud,
|
||||
non-zero exit** with actual-vs-expected in the message. It must **never** be
|
||||
swallowed or collapse to exit 0 — this is the whole point; the CLI already has
|
||||
the "requested-but-lost claim ⇒ SilentExit()/non-zero" plumbing for `--claim`
|
||||
(update_proxied_server.go, "beads audit finding #10") and this reuses it.
|
||||
- **Verify-after-write**: route through the WIP `verifiedClaimWrite` (a guarded
|
||||
reassign is coordination-critical and idempotent for the winner — safe to
|
||||
replay when a re-read proves the commit rolled back). Add a `reassigned(actor)`
|
||||
/ `matched(post)` `claimPostcondition` alongside `claimedBy`/`unclaimed`.
|
||||
- **Wisps exempt**, closed issues rejected — same as the sibling verbs.
|
||||
- **Batch/JSON**: same mixed-batch exit-code rule as `--claim` (one lost
|
||||
precondition in a batch ⇒ non-zero overall), same JSON shape.
|
||||
|
||||
### Why guards on `update` rather than a new `bd reassign` verb
|
||||
|
||||
- The transitions we need are *arbitrary* field CAS (assignee and/or status), not
|
||||
a single named operation; a verb per transition is how we got here.
|
||||
- `update` already owns multi-field mutation, batching, JSON, proxied-server
|
||||
routing, and the finding-#10 exit-code contract — guards compose with all of it
|
||||
for free.
|
||||
- `--claim` and `unclaim --if-assignee` remain the ergonomic shorthands for their
|
||||
two hot paths; the general guards are the escape hatch the wrappers currently
|
||||
fake with `bd sql`.
|
||||
|
||||
Alternative considered — **dedicated `bd reassign --from --to`**: more
|
||||
discoverable for the park case, but doesn't express restore's status-guarded
|
||||
claim-on-behalf without yet more flags, and duplicates update's plumbing. Rejected
|
||||
in favor of the general guards. (Flag it for maintainer preference — it is a CLI
|
||||
surface call, not a correctness call.)
|
||||
|
||||
## 4. How each wheelhouse finding collapses to it
|
||||
|
||||
```sh
|
||||
# park (wy-mdi5h.3) — reassign only if the worker still holds it:
|
||||
bd update "$id" --if-assignee "$reclaimed_worker" -a "$PARK_ASSIGNEE"
|
||||
|
||||
# interactive restore (wy-mdi5h.6) — claim-on-behalf, only while still open:
|
||||
bd update "$id" --if-assignee '' --if-status open -a "$owner" -s in_progress
|
||||
|
||||
# queue-take (wy-mdi5h.7) — ADOPTION, no new primitive: use pool-aware claim
|
||||
bd update "$id" --claim # fable-crew is a claim.pools alias
|
||||
|
||||
# bulldog (wy-mdi5h.4) — ADOPTION: claim + honor the non-zero exit, don't swallow
|
||||
bd update "$BEAD" --claim || { echo "foreign/lost; not mutating"; exit 1; }
|
||||
```
|
||||
|
||||
The first two are the primitive this proposal adds; the last two are wheelhouse
|
||||
patches that consume the *existing* `--claim`.
|
||||
|
||||
## 5. Implementation sketch (files)
|
||||
|
||||
- `internal/storage/issueops/update_cas.go` (new): `UpdateIssueIfMatchInTx(ctx,
|
||||
tx, id, guards, updates)` — build the `WHERE` from present guards, reuse the
|
||||
unclaim.go CAS body (row_lock rewrite, `status IN ('open','in_progress')`
|
||||
liveness clause, RowsAffected verdict, typed mismatch error incl. actual state).
|
||||
- `internal/storage/storage.go`: add `ErrStatusMismatch`; add
|
||||
`UpdateIssueIfMatch(ctx, id, actor, guards, updates)` to the `Storage` interface.
|
||||
- `internal/storage/dolt/issues.go`: `DoltStore.UpdateIssueIfMatch` wrapping
|
||||
`verifiedClaimWrite` with a `matched(post)` postcondition + DOLT_ADD/COMMIT.
|
||||
- `cmd/bd/update.go`: register `--if-assignee` (String) and `--if-status`
|
||||
(String); when either `Changed()`, route to the CAS path; wire the
|
||||
precondition-failed → non-zero exit through the existing `claimFailed` machinery
|
||||
(both `update.go` and `update_proxied_server.go`). Reject `--if-*` combined with
|
||||
`--claim` (mutually exclusive; `--claim` is its own CAS).
|
||||
- `cmd/bd/prime.go`: document the new guards next to `--claim`.
|
||||
|
||||
## 6. Tests
|
||||
|
||||
Beads side (their conventions):
|
||||
- `internal/storage/issueops` unit tests: match applies, assignee-mismatch no-op,
|
||||
status-mismatch no-op, `--if-assignee ''` matches only unassigned, closed-issue
|
||||
reject, row_lock rewritten on success.
|
||||
- A concurrency test in the spirit of the existing claim CAS tests: two goroutines
|
||||
reassign the same row, exactly one gets `RowsAffected==1`.
|
||||
- `verifiedClaimWrite` path: reported-success-but-rolled-back ⇒ replay; applied ⇒
|
||||
success; genuine mismatch ⇒ loud non-zero.
|
||||
- CLI: non-zero exit on precondition failure, mixed-batch exit code, JSON shape.
|
||||
|
||||
Wyvern side (house rule — each consuming fix gets a `test-*.sh` pin):
|
||||
- `test-park-cas.sh`, `test-restore-cas.sh` asserting a racing claim in the window
|
||||
is refused (non-zero) rather than clobbered.
|
||||
|
||||
## 7. Scope / non-goals
|
||||
|
||||
- **In:** assignee + status guards on `update`; storage + CLI + verify integration;
|
||||
the two wheelhouse consumers that are genuine primitive gaps.
|
||||
- **Out:** label/other-field guards (`--if-label`) — no current consumer; add later
|
||||
if one appears. The wheelhouse *adoption* patches (queue-take, bulldog switching
|
||||
to `--claim`) are separate wy-mdi5h children, not part of this beads PR.
|
||||
- **Coordinate with the uncommitted `claim_verify.go` WIP** already in the tree —
|
||||
this proposal depends on `verifiedClaimWrite` landing (or lands alongside it).
|
||||
138
internal/storage/dolt/claim_verify.go
Normal file
138
internal/storage/dolt/claim_verify.go
Normal file
|
|
@ -0,0 +1,138 @@
|
|||
package dolt
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/metric"
|
||||
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// Claim-family verify-after-write (bd-zccb9, from wyvern incident wy-ejph3).
|
||||
//
|
||||
// Under a degraded sql-server the write's exit status is not truth in either
|
||||
// direction: a claim can report success while the server-side transaction dies
|
||||
// with the abandoned connection and rolls back (wy-ejph3: the lost claim cost a
|
||||
// duplicate full implementation), and the inverse — an error printed with the
|
||||
// write actually applied — is on record from wy-x543k. Fleet wrappers solved
|
||||
// this with verify-by-re-read (wyvern's wh-take.sh); this file folds that
|
||||
// protocol into bd itself for the coordination-critical writes: claim, ready
|
||||
// claim, and unclaim. Ordinary field updates are out of scope — a lost label
|
||||
// edit is an annoyance, a phantom claim is a duplicated implementation.
|
||||
//
|
||||
// The protocol: run the write, then resolve its outcome against the database
|
||||
// instead of trusting the exit status.
|
||||
//
|
||||
// - reported success -> verify; a mismatch fails LOUDLY (the caller
|
||||
// must know it does not hold the claim)
|
||||
// - ambiguous commit loss -> (errCommitPhase, surfaced by withRetryTx as
|
||||
// indeterminate) verify; applied -> success, verified rolled back -> replay
|
||||
// the write once (safe: nothing landed)
|
||||
// - any other error -> honest failure (CAS lost, not-claimable, or
|
||||
// pre-commit errors withRetryTx already retried); no verify needed
|
||||
//
|
||||
// Wisps are exempt: they are ephemeral, never leased, and not reclaimable
|
||||
// coordination state.
|
||||
|
||||
// claimPostcondition describes the row state a claim-family write must leave
|
||||
// behind to count as applied, plus the words to use when it didn't.
|
||||
type claimPostcondition struct {
|
||||
op string // "claim" | "unclaim", for messages and metrics
|
||||
want func(assignee string, status types.Status) bool
|
||||
desc string // expected state, human-readable, for mismatch messages
|
||||
}
|
||||
|
||||
func claimedBy(actor string) claimPostcondition {
|
||||
return claimPostcondition{
|
||||
op: "claim",
|
||||
want: func(assignee string, status types.Status) bool {
|
||||
return assignee == actor && status == types.StatusInProgress
|
||||
},
|
||||
desc: fmt.Sprintf("assignee=%q status=%q", actor, types.StatusInProgress),
|
||||
}
|
||||
}
|
||||
|
||||
func unclaimed() claimPostcondition {
|
||||
return claimPostcondition{
|
||||
op: "unclaim",
|
||||
want: func(assignee string, status types.Status) bool {
|
||||
return assignee == "" && status == types.StatusOpen
|
||||
},
|
||||
desc: fmt.Sprintf("assignee=%q status=%q", "", types.StatusOpen),
|
||||
}
|
||||
}
|
||||
|
||||
// readClaimState re-reads an issue's assignee and status on a fresh
|
||||
// transaction. withReadTx retries transient connection errors itself, so a
|
||||
// failure here means the server is genuinely unreachable, not a blip.
|
||||
func (s *DoltStore) readClaimState(ctx context.Context, id string) (string, types.Status, error) {
|
||||
var assignee sql.NullString
|
||||
var status types.Status
|
||||
err := s.withReadTx(ctx, func(tx *sql.Tx) error {
|
||||
return tx.QueryRowContext(ctx,
|
||||
`SELECT assignee, status FROM issues WHERE id = ?`, id,
|
||||
).Scan(&assignee, &status)
|
||||
})
|
||||
if err != nil {
|
||||
return "", "", err
|
||||
}
|
||||
if !assignee.Valid {
|
||||
return "", status, nil
|
||||
}
|
||||
return assignee.String, status, nil
|
||||
}
|
||||
|
||||
// verifiedClaimWrite runs write and resolves its outcome against the database
|
||||
// state per the protocol above. write must be safe to run twice when its first
|
||||
// run verifiably did not land (all claim-family writes are: they are CAS
|
||||
// updates, idempotent for the winning actor).
|
||||
//
|
||||
// A verify that contradicts a reported success can in principle also be a
|
||||
// legitimate concurrent mutation (a forced unclaim landing within the
|
||||
// verification window). That reads as a lost write and fails loudly too —
|
||||
// acceptable: the caller must re-establish its view either way.
|
||||
func (s *DoltStore) verifiedClaimWrite(ctx context.Context, id string, post claimPostcondition, write func() error) error {
|
||||
if !s.serverMode || s.isActiveWisp(ctx, id) {
|
||||
return write()
|
||||
}
|
||||
const maxReplays = 1
|
||||
for attempt := 0; ; attempt++ {
|
||||
err := write()
|
||||
if err != nil && !errors.Is(err, errCommitPhase) {
|
||||
return err
|
||||
}
|
||||
assignee, status, verr := s.readClaimState(ctx, id)
|
||||
if verr != nil {
|
||||
if err != nil {
|
||||
return err // the honest indeterminate error from withRetryTx
|
||||
}
|
||||
return fmt.Errorf("%s of %s reported success but could not be verified (server degraded?): %w — re-read the issue before trusting the %s",
|
||||
post.op, id, verr, post.op)
|
||||
}
|
||||
if post.want(assignee, status) {
|
||||
if err != nil {
|
||||
doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes(
|
||||
attribute.String("op", post.op), attribute.String("outcome", "applied")))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
if attempt < maxReplays {
|
||||
// Verified rolled back: nothing landed, so one replay is safe.
|
||||
doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes(
|
||||
attribute.String("op", post.op), attribute.String("outcome", "replayed")))
|
||||
continue
|
||||
}
|
||||
return fmt.Errorf("%s of %s did not land (connection lost during commit; rollback verified by re-read): %w",
|
||||
post.op, id, err)
|
||||
}
|
||||
doltMetrics.claimVerifyLost.Add(ctx, 1, metric.WithAttributes(
|
||||
attribute.String("op", post.op)))
|
||||
return fmt.Errorf("%s of %s reported success but did not land (found assignee=%q status=%q, want %s) — server likely degraded; treat the %s as NOT applied",
|
||||
post.op, id, assignee, status, post.desc, post.op)
|
||||
}
|
||||
}
|
||||
244
internal/storage/dolt/claim_verify_test.go
Normal file
244
internal/storage/dolt/claim_verify_test.go
Normal file
|
|
@ -0,0 +1,244 @@
|
|||
package dolt
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
"github.com/steveyegge/beads/internal/types"
|
||||
)
|
||||
|
||||
// Tests for claim-family verify-after-write (bd-zccb9, wyvern incident
|
||||
// wy-ejph3): under a degraded server the write's exit status is not truth in
|
||||
// either direction, so claim outcomes are resolved against the database by
|
||||
// re-read. Failure injection happens at the write-func seam — the write funcs
|
||||
// here lie in exactly the ways the incident documented.
|
||||
|
||||
func claimVerifyTestIssue(t *testing.T, s *DoltStore) string {
|
||||
t.Helper()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
issue := &types.Issue{
|
||||
Title: "claim verify target",
|
||||
IssueType: types.TypeTask,
|
||||
Priority: 2,
|
||||
Status: types.StatusOpen,
|
||||
}
|
||||
if err := s.CreateIssue(ctx, issue, "tester"); err != nil {
|
||||
t.Fatalf("create issue: %v", err)
|
||||
}
|
||||
return issue.ID
|
||||
}
|
||||
|
||||
// rawClaim performs the real claim write without the verify wrapper — the
|
||||
// inner body of ClaimIssue — so tests can control exactly which attempt lands.
|
||||
func rawClaim(t *testing.T, s *DoltStore, id, actor string) error {
|
||||
t.Helper()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
_, err := issueops.ClaimIssueInTx(ctx, tx, id, actor)
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
// TestVerifiedClaimWriteConvertsAppliedIndeterminate: the wy-x543k direction —
|
||||
// the write actually landed but the connection died during commit and bd
|
||||
// printed an error. Verify-by-re-read must convert it into an accurate success.
|
||||
func TestVerifiedClaimWriteConvertsAppliedIndeterminate(t *testing.T) {
|
||||
s, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
id := claimVerifyTestIssue(t, s)
|
||||
if err := rawClaim(t, s, id, "alice"); err != nil {
|
||||
t.Fatalf("raw claim: %v", err)
|
||||
}
|
||||
|
||||
calls := 0
|
||||
err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error {
|
||||
calls++
|
||||
return fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected applied-indeterminate to resolve to success, got: %v", err)
|
||||
}
|
||||
if calls != 1 {
|
||||
t.Fatalf("write must not be replayed when the re-read shows it applied; ran %d times", calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestVerifiedClaimWriteReplaysVerifiedRollback: the wy-ejph3 direction — the
|
||||
// connection died during commit AND the transaction rolled back. The re-read
|
||||
// proves nothing landed, so one replay is safe and must proceed.
|
||||
func TestVerifiedClaimWriteReplaysVerifiedRollback(t *testing.T) {
|
||||
s, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
id := claimVerifyTestIssue(t, s)
|
||||
|
||||
calls := 0
|
||||
err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error {
|
||||
calls++
|
||||
if calls == 1 {
|
||||
// First attempt: commit-phase connection loss, nothing landed.
|
||||
return fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
|
||||
}
|
||||
return rawClaim(t, s, id, "alice")
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected verified-rollback replay to succeed, got: %v", err)
|
||||
}
|
||||
if calls != 2 {
|
||||
t.Fatalf("expected exactly one replay (2 write runs), got %d", calls)
|
||||
}
|
||||
|
||||
assignee, status, err := s.readClaimState(ctx, id)
|
||||
if err != nil {
|
||||
t.Fatalf("read claim state: %v", err)
|
||||
}
|
||||
if assignee != "alice" || status != types.StatusInProgress {
|
||||
t.Fatalf("replayed claim not in effect: assignee=%q status=%q", assignee, status)
|
||||
}
|
||||
}
|
||||
|
||||
// TestVerifiedClaimWriteFailsLoudlyOnLostWrite: the false-success direction —
|
||||
// the write reported success but the claim is not in the database. The caller
|
||||
// must get a loud non-nil error, never a silent phantom claim.
|
||||
func TestVerifiedClaimWriteFailsLoudlyOnLostWrite(t *testing.T) {
|
||||
s, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
id := claimVerifyTestIssue(t, s)
|
||||
|
||||
err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error {
|
||||
return nil // lie: report success without writing anything
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("expected loud failure for a success-reported-but-lost claim, got nil")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "did not land") {
|
||||
t.Fatalf("error should say the claim did not land, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestVerifiedClaimWriteIndeterminateStaysIndeterminateTwice: a replay that
|
||||
// itself dies at commit phase with nothing landed must not loop forever — the
|
||||
// one-replay budget exhausts and the caller gets the verified-rollback error.
|
||||
func TestVerifiedClaimWriteIndeterminateStaysIndeterminateTwice(t *testing.T) {
|
||||
s, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
id := claimVerifyTestIssue(t, s)
|
||||
|
||||
calls := 0
|
||||
err := s.verifiedClaimWrite(ctx, id, claimedBy("alice"), func() error {
|
||||
calls++
|
||||
return fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("expected failure after replay budget exhausted, got nil")
|
||||
}
|
||||
if calls != 2 {
|
||||
t.Fatalf("expected exactly 2 write runs (original + one replay), got %d", calls)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "did not land") {
|
||||
t.Fatalf("error should report the verified rollback, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestVerifiedReadyClaimReplayConvertsAppliedIndeterminate: the replay leg of
|
||||
// the bespoke ready-claim path must keep the wrapper's verify semantics (lion
|
||||
// review on PR #5006). First attempt: commit-phase loss, verified rolled back
|
||||
// (nothing landed) -> replay. The replay actually lands the claim but ALSO
|
||||
// reports commit-phase loss (the wy-x543k direction, now on attempt two). The
|
||||
// verify pass must convert that into an accurate success instead of returning
|
||||
// the raw indeterminate error — which would leave an applied claim orphaned
|
||||
// on an issue the caller was told it failed to claim.
|
||||
func TestVerifiedReadyClaimReplayConvertsAppliedIndeterminate(t *testing.T) {
|
||||
s, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
id := claimVerifyTestIssue(t, s)
|
||||
|
||||
calls := 0
|
||||
got, err := s.verifiedReadyClaim(ctx, "alice", func() (*types.Issue, error) {
|
||||
calls++
|
||||
if calls == 1 {
|
||||
// Commit-phase loss, transaction rolled back: nothing landed.
|
||||
return &types.Issue{ID: id}, fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
|
||||
}
|
||||
// Replay: the claim lands, but the connection dies during commit again.
|
||||
if err := rawClaim(t, s, id, "alice"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &types.Issue{ID: id}, fmt.Errorf("write commit result indeterminate after connection loss: i/o timeout (%w)", errCommitPhase)
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("expected applied-indeterminate replay to resolve to success, got: %v", err)
|
||||
}
|
||||
if got == nil || got.ID != id {
|
||||
t.Fatalf("recovered success must return the claimed issue %s, got %+v", id, got)
|
||||
}
|
||||
if calls != 2 {
|
||||
t.Fatalf("expected exactly one replay (2 write runs), got %d", calls)
|
||||
}
|
||||
|
||||
assignee, status, verr := s.readClaimState(ctx, id)
|
||||
if verr != nil {
|
||||
t.Fatalf("read claim state: %v", verr)
|
||||
}
|
||||
if assignee != "alice" || status != types.StatusInProgress {
|
||||
t.Fatalf("claim not in effect after recovered replay: assignee=%q status=%q", assignee, status)
|
||||
}
|
||||
}
|
||||
|
||||
// TestClaimUnclaimVerifiedEndToEnd: the public paths still work with the
|
||||
// verify layer in place — a healthy claim and unclaim pass their
|
||||
// postconditions and leave the expected states.
|
||||
func TestClaimUnclaimVerifiedEndToEnd(t *testing.T) {
|
||||
s, cleanup := setupTestStore(t)
|
||||
defer cleanup()
|
||||
ctx, cancel := testContext(t)
|
||||
defer cancel()
|
||||
|
||||
id := claimVerifyTestIssue(t, s)
|
||||
|
||||
if err := s.ClaimIssue(ctx, id, "alice"); err != nil {
|
||||
t.Fatalf("claim: %v", err)
|
||||
}
|
||||
assignee, status, err := s.readClaimState(ctx, id)
|
||||
if err != nil {
|
||||
t.Fatalf("read claim state: %v", err)
|
||||
}
|
||||
if assignee != "alice" || status != types.StatusInProgress {
|
||||
t.Fatalf("post-claim state: assignee=%q status=%q", assignee, status)
|
||||
}
|
||||
|
||||
// Idempotent re-claim by the same actor still succeeds through verify.
|
||||
if err := s.ClaimIssue(ctx, id, "alice"); err != nil {
|
||||
t.Fatalf("idempotent re-claim: %v", err)
|
||||
}
|
||||
|
||||
if err := s.UnclaimIssue(ctx, id, "alice", false); err != nil {
|
||||
t.Fatalf("unclaim: %v", err)
|
||||
}
|
||||
assignee, status, err = s.readClaimState(ctx, id)
|
||||
if err != nil {
|
||||
t.Fatalf("read claim state: %v", err)
|
||||
}
|
||||
if assignee != "" || status != types.StatusOpen {
|
||||
t.Fatalf("post-unclaim state: assignee=%q status=%q", assignee, status)
|
||||
}
|
||||
}
|
||||
|
|
@ -4,11 +4,15 @@ import (
|
|||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/metric"
|
||||
|
||||
"github.com/steveyegge/beads/internal/idgen"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
|
|
@ -304,22 +308,26 @@ func (s *DoltStore) ClaimIssue(ctx context.Context, id string, actor string) err
|
|||
// (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/) — so retry is the
|
||||
// only safety net under concurrent claimants. The body stays a single tx
|
||||
// (CAS + DOLT_COMMIT); withRetryTx owns BeginTx and the final Commit.
|
||||
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
if _, err := issueops.ClaimIssueInTx(ctx, tx, id, actor); err != nil {
|
||||
return err
|
||||
}
|
||||
// The whole write is then resolved by verify-by-re-read (bd-zccb9): under a
|
||||
// degraded server the exit status is not truth in either direction.
|
||||
return s.verifiedClaimWrite(ctx, id, claimedBy(actor), func() error {
|
||||
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
if _, err := issueops.ClaimIssueInTx(ctx, tx, id, actor); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Dolt versioning for permanent issues.
|
||||
// GH#2455: Stage only the tables we modified, then commit without -A.
|
||||
for _, table := range []string{"issues", "events"} {
|
||||
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
|
||||
}
|
||||
commitMsg := fmt.Sprintf("bd: claim %s", id)
|
||||
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
|
||||
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
|
||||
return fmt.Errorf("dolt commit: %w", err)
|
||||
}
|
||||
return nil
|
||||
// Dolt versioning for permanent issues.
|
||||
// GH#2455: Stage only the tables we modified, then commit without -A.
|
||||
for _, table := range []string{"issues", "events"} {
|
||||
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
|
||||
}
|
||||
commitMsg := fmt.Sprintf("bd: claim %s", id)
|
||||
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
|
||||
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
|
||||
return fmt.Errorf("dolt commit: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -332,31 +340,99 @@ func (s *DoltStore) ClaimReadyIssue(ctx context.Context, filter types.WorkFilter
|
|||
// no real row locking — FOR UPDATE / SKIP LOCKED are parse-only no-ops
|
||||
// (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/) — so retry is the
|
||||
// safety net. withRetryTx owns BeginTx and the final Commit.
|
||||
var claimed *types.Issue
|
||||
err := s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
var err error
|
||||
claimed, err = issueops.ClaimReadyIssueInTx(ctx, tx, filter, actor)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if claimed == nil {
|
||||
return nil
|
||||
}
|
||||
write := func() (*types.Issue, error) {
|
||||
var claimed *types.Issue
|
||||
err := s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
var err error
|
||||
claimed, err = issueops.ClaimReadyIssueInTx(ctx, tx, filter, actor)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if claimed == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, table := range []string{"issues", "events"} {
|
||||
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
|
||||
}
|
||||
commitMsg := fmt.Sprintf("bd: claim ready %s", claimed.ID)
|
||||
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
|
||||
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
|
||||
return fmt.Errorf("dolt commit: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
for _, table := range []string{"issues", "events"} {
|
||||
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
|
||||
}
|
||||
commitMsg := fmt.Sprintf("bd: claim ready %s", claimed.ID)
|
||||
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
|
||||
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
|
||||
return fmt.Errorf("dolt commit: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return claimed, err
|
||||
}
|
||||
return s.verifiedReadyClaim(ctx, actor, write)
|
||||
}
|
||||
|
||||
// verifiedReadyClaim resolves a ready-claim write by verify-by-re-read
|
||||
// (bd-zccb9): the claimed ID is only known once the write body has run, so
|
||||
// this cannot ride verifiedClaimWrite's id parameter — but the resolution
|
||||
// protocol is the same. A replay after a verified rollback re-scans the ready
|
||||
// front and may legitimately claim a different issue. Split from
|
||||
// ClaimReadyIssue so injection tests can drive the write seam directly, the
|
||||
// same way the verifiedClaimWrite tests do.
|
||||
func (s *DoltStore) verifiedReadyClaim(ctx context.Context, actor string, write func() (*types.Issue, error)) (*types.Issue, error) {
|
||||
claimed, err := write()
|
||||
if err != nil && !(errors.Is(err, errCommitPhase) && claimed != nil && s.serverMode) {
|
||||
return nil, err
|
||||
}
|
||||
return claimed, nil
|
||||
if claimed == nil || !s.serverMode {
|
||||
return claimed, err
|
||||
}
|
||||
for attempt := 0; ; attempt++ {
|
||||
assignee, status, verr := s.readClaimState(ctx, claimed.ID)
|
||||
if verr != nil {
|
||||
if err != nil {
|
||||
return nil, err // the honest indeterminate error from withRetryTx
|
||||
}
|
||||
return nil, fmt.Errorf("ready claim of %s reported success but could not be verified (server degraded?): %w — re-read the issue before trusting the claim",
|
||||
claimed.ID, verr)
|
||||
}
|
||||
post := claimedBy(actor)
|
||||
if post.want(assignee, status) {
|
||||
if err != nil {
|
||||
doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes(
|
||||
attribute.String("op", "ready-claim"), attribute.String("outcome", "applied")))
|
||||
}
|
||||
return claimed, nil
|
||||
}
|
||||
if err != nil && attempt < 1 {
|
||||
// Verified rolled back: nothing landed, safe to replay once.
|
||||
doltMetrics.claimVerifyRecovered.Add(ctx, 1, metric.WithAttributes(
|
||||
attribute.String("op", "ready-claim"), attribute.String("outcome", "replayed")))
|
||||
claimed, err = write()
|
||||
// The replay carries the same commit-phase ambiguity as the first
|
||||
// attempt: an errCommitPhase with a selection made must fall
|
||||
// through to the verify pass — attempt is past the replay budget
|
||||
// now, so a second verified rollback still exhausts with the
|
||||
// honest "did not land" error — never surface the raw
|
||||
// indeterminate error, which would reintroduce the wy-x543k
|
||||
// false-failure and could orphan an applied claim on a DIFFERENT
|
||||
// ready issue than the first attempt selected.
|
||||
if err != nil && !(errors.Is(err, errCommitPhase) && claimed != nil) {
|
||||
return nil, err
|
||||
}
|
||||
if claimed == nil {
|
||||
if err != nil {
|
||||
// Commit-phase loss before anything was selected: nothing
|
||||
// to verify, honest indeterminate error.
|
||||
return nil, err
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("ready claim of %s did not land (connection lost during commit; rollback verified by re-read): %w", claimed.ID, err)
|
||||
}
|
||||
doltMetrics.claimVerifyLost.Add(ctx, 1, metric.WithAttributes(
|
||||
attribute.String("op", "ready-claim")))
|
||||
return nil, fmt.Errorf("ready claim of %s reported success but did not land (found assignee=%q status=%q, want %s) — server likely degraded; treat the claim as NOT applied",
|
||||
claimed.ID, assignee, status, post.desc)
|
||||
}
|
||||
}
|
||||
|
||||
// HeartbeatIssue refreshes the lease on an issue actor holds in_progress,
|
||||
|
|
@ -419,21 +495,25 @@ func (s *DoltStore) ReclaimExpiredLeases(ctx context.Context, olderThan time.Dur
|
|||
// writer that loses Dolt's optimistic commit-time merge (1213/1205) is retried
|
||||
// rather than surfaced as a hard failure.
|
||||
func (s *DoltStore) UnclaimIssue(ctx context.Context, id string, actor string, force bool) error {
|
||||
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
if err := issueops.UnclaimIssueInTx(ctx, tx, id, actor, force); err != nil {
|
||||
return err
|
||||
}
|
||||
// verify-by-re-read (bd-zccb9): a phantom unclaim leaves the caller
|
||||
// believing the issue is released while it still holds the claim.
|
||||
return s.verifiedClaimWrite(ctx, id, unclaimed(), func() error {
|
||||
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
if err := issueops.UnclaimIssueInTx(ctx, tx, id, actor, force); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Dolt versioning for permanent issues.
|
||||
for _, table := range []string{"issues", "events"} {
|
||||
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
|
||||
}
|
||||
commitMsg := fmt.Sprintf("bd: unclaim %s", id)
|
||||
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
|
||||
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
|
||||
return fmt.Errorf("dolt commit: %w", err)
|
||||
}
|
||||
return nil
|
||||
// Dolt versioning for permanent issues.
|
||||
for _, table := range []string{"issues", "events"} {
|
||||
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
|
||||
}
|
||||
commitMsg := fmt.Sprintf("bd: unclaim %s", id)
|
||||
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
|
||||
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
|
||||
return fmt.Errorf("dolt commit: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
|
|
@ -445,21 +525,24 @@ func (s *DoltStore) UnclaimIssue(ctx context.Context, id string, actor string, f
|
|||
// UnclaimIssue so a concurrent writer that loses Dolt's optimistic commit-time
|
||||
// merge is retried rather than surfaced as a hard failure.
|
||||
func (s *DoltStore) UnclaimIssueIfAssignee(ctx context.Context, id string, actor string, expectedAssignee string) error {
|
||||
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
if err := issueops.UnclaimIssueIfAssigneeInTx(ctx, tx, id, actor, expectedAssignee); err != nil {
|
||||
return err
|
||||
}
|
||||
// verify-by-re-read (bd-zccb9), same reasoning as UnclaimIssue.
|
||||
return s.verifiedClaimWrite(ctx, id, unclaimed(), func() error {
|
||||
return s.withRetryTx(ctx, func(tx *sql.Tx) error {
|
||||
if err := issueops.UnclaimIssueIfAssigneeInTx(ctx, tx, id, actor, expectedAssignee); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Dolt versioning for permanent issues.
|
||||
for _, table := range []string{"issues", "events"} {
|
||||
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
|
||||
}
|
||||
commitMsg := fmt.Sprintf("bd: unclaim %s", id)
|
||||
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
|
||||
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
|
||||
return fmt.Errorf("dolt commit: %w", err)
|
||||
}
|
||||
return nil
|
||||
// Dolt versioning for permanent issues.
|
||||
for _, table := range []string{"issues", "events"} {
|
||||
_, _ = tx.ExecContext(ctx, "CALL DOLT_ADD(?)", table)
|
||||
}
|
||||
commitMsg := fmt.Sprintf("bd: unclaim %s", id)
|
||||
if _, err := tx.ExecContext(ctx, "CALL DOLT_COMMIT('-m', ?, '--author', ?)",
|
||||
commitMsg, s.commitAuthorString()); err != nil && !isDoltNothingToCommit(err) {
|
||||
return fmt.Errorf("dolt commit: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -668,15 +668,17 @@ var doltTracer = otel.Tracer("github.com/steveyegge/beads/storage/dolt")
|
|||
// Instruments are registered against the global delegating provider at init time,
|
||||
// so they automatically forward to the real provider once telemetry.Init() runs.
|
||||
var doltMetrics struct {
|
||||
retryCount metric.Int64Counter
|
||||
lockWaitMs metric.Float64Histogram
|
||||
circuitTrips metric.Int64Counter
|
||||
circuitRejected metric.Int64Counter
|
||||
serializationErrors metric.Int64Counter
|
||||
writeRetries metric.Int64Counter
|
||||
connAcquireMs metric.Float64Histogram
|
||||
poolWaitCount metric.Int64Counter
|
||||
poolWaitMs metric.Float64Histogram
|
||||
retryCount metric.Int64Counter
|
||||
lockWaitMs metric.Float64Histogram
|
||||
circuitTrips metric.Int64Counter
|
||||
circuitRejected metric.Int64Counter
|
||||
serializationErrors metric.Int64Counter
|
||||
writeRetries metric.Int64Counter
|
||||
connAcquireMs metric.Float64Histogram
|
||||
poolWaitCount metric.Int64Counter
|
||||
poolWaitMs metric.Float64Histogram
|
||||
claimVerifyLost metric.Int64Counter
|
||||
claimVerifyRecovered metric.Int64Counter
|
||||
}
|
||||
|
||||
func init() {
|
||||
|
|
@ -717,6 +719,14 @@ func init() {
|
|||
metric.WithDescription("Total time connections spent waiting due to pool exhaustion"),
|
||||
metric.WithUnit("ms"),
|
||||
)
|
||||
doltMetrics.claimVerifyLost, _ = m.Int64Counter("bd.claim_verify_lost_total",
|
||||
metric.WithDescription("Claim-family writes that reported success but failed verify-by-re-read (label: op=claim|unclaim)"),
|
||||
metric.WithUnit("{write}"),
|
||||
)
|
||||
doltMetrics.claimVerifyRecovered, _ = m.Int64Counter("bd.claim_verify_recovered_total",
|
||||
metric.WithDescription("Indeterminate claim-family commits resolved by re-read (label: op, outcome=applied|replayed)"),
|
||||
metric.WithUnit("{write}"),
|
||||
)
|
||||
}
|
||||
|
||||
// registerPoolGauges registers observable gauges that report sql.DB pool stats
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue