* fix(dolt): retry mutate+commit methods on serialization conflict (1213/1205)
ClaimIssue, ClaimReadyIssue, UpdateIssue and CloseIssue each hand-rolled
their own tx (BeginTx + DOLT_ADD/DOLT_COMMIT + Commit) outside
withRetryTx, so a concurrent writer that lost Dolt's optimistic
commit-time merge surfaced MySQL 1213/1205 as a hard failure instead of
retrying. Dolt has no real row locking — FOR UPDATE / SKIP LOCKED are
parse-only no-ops (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/)
— so retry is the only safety net under N concurrent workers. These four
are the work-queue hot paths: `bd ready --claim`, claim, update, close.
Move each body into a withRetryTx closure (which owns BeginTx and the
final Commit and retries serialization failures with backoff), preserving
the single-tx CAS/mutation + DOLT_COMMIT atomicity exactly. On retry,
ClaimReadyIssue re-scans the ready front from a fresh snapshot and claims
the next available issue.
Tested: TestEmbeddedReadyClaimConcurrent, TestEmbeddedReadyConcurrent,
TestEmbeddedCloseConcurrent, TestEmbeddedUpdateConcurrent and the
non-concurrent embedded Update/Close/Ready tests pass; build + vet clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(dolt): add concurrent work-queue drain test (Gas Station claim invariant)
N workers concurrently dequeue one shared ready-front via ClaimReadyIssue
until empty; asserts every issue claimed exactly once, none stranded, no
double-claims, and no serialization error ever surfaces (withRetryTx absorbs
the 1213/1205 from the claim-CAS cell collision). This is the regression guard
for the multi-agent claim queue that #4491 makes safe.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(lease): claim-TTL + heartbeat + reclaim for dead-worker recovery
Gas Station v1.1 (wy-5r9j). A claim was previously permanent: a worker that
died mid-task stranded its issue in_progress forever. This adds a lease so a
claim can expire and be recovered.
Schema (migration 0054): add lease_expires_at, heartbeat_at (DATETIME) and
row_lock (BIGINT) to issues (and wisps, for uniform issueops routing), plus
idx_issues_lease for the reaper scan. assignee doubles as the lease owner.
Lifecycle (shared issueops, so both server-mode DoltStore and EmbeddedDoltStore
get it):
- Claim stamps lease_expires_at = now + TTL (default 5m, WithLeaseTTL to
override), heartbeat_at = now, and a fresh row_lock.
- bd heartbeat <id> — owner-only; pushes the lease forward and rewrites
row_lock. Fails (ErrNotClaimable/ErrAlreadyClaimed) once the lease is gone.
- bd reclaim --older-than <dur> — reverts in_progress issues whose lease
expired more than <dur> ago back to ready (clears assignee/started_at,
records a lease_reclaimed event). Default grace 2×TTL.
The row_lock trick: Dolt has no row locking and merges concurrent commits
cell-by-cell, so a heartbeat (writes heartbeat_at) racing a reclaim (writes
status) would silently cell-merge into a zombie — an open/unassigned issue that
still carries the worker's fresh heartbeat. Every mutating path (claim, close,
update, heartbeat, reclaim) now rewrites the shared row_lock cell, forcing the
loser to a 1213/1205 serialization conflict that withRetryTx replays. All lease
paths are withRetryTx/withConn wrapped.
Tests (real Dolt): TestRowLockForcesConflictOnDisjointCellWrites proves the bug
without row_lock (silent merge to zombie) and correctness with it (conflict);
TestConcurrentHeartbeatReclaimClose is the integration race (live workers
heartbeat+close vs a continuous reaper vs dead workers) asserting exactly-once,
no lost close, no zombie; plus claim/heartbeat/reclaim unit coverage on both
backends.
Note: lease columns are second-granular DATETIME (Dolt rounds), so sub-second
TTLs are meaningless — production TTL is minutes. Heartbeat writes a Dolt
commit, so cadence must stay ≪ claim cadence to avoid history bloat.
Fixes the test coupling in TestMigration0053PromotesRigWisps, which assumed the
rig-wisp repair was always LatestVersion().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ci+docs: allowlist lease stress tests; scope the row_lock invariant comment
- Add TestConcurrentWorkQueueDrain and TestConcurrentHeartbeatReclaimClose
to the check-testing-short.sh allowlist; both are real-Dolt runtime/stress
skips and were failing ci-pr-policy.
- freshRowLock doc said "EVERY mutating path" rewrites row_lock, which
overstates it. Scope it to status/ownership/lease-mutating paths (the set
the reclaim/heartbeat races care about) and document the invariant + why
orthogonal-cell writers (is_blocked, compaction, deps, rename, reopen) are
intentionally exempt, so a future writer does not silently reintroduce the
zombie-merge bug.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(show): surface lease_expires_at/heartbeat_at in bd show (--json + text) (wy-9cdw)
Migration 0054 added lease_expires_at/heartbeat_at (and internal row_lock) to
the issues/wisps tables for the claim/heartbeat/reclaim leasing feature, but
bd show could not observe lease state — you had to drop to raw SQL.
- types.Issue: add LeaseExpiresAt/HeartbeatAt (*time.Time, omitempty), matching
the StartedAt nullable-timestamp convention. row_lock stays unexported.
- sqlbuild.IssueSelectColumns: append lease_expires_at, heartbeat_at to the
canonical hydration list (flows to every stack via the shared constant).
- issueops.ScanIssueFrom: scan + map the two nullable columns (before the
trailing extra dests, keeping positional parity across all scan sites).
- bd show text view: emit a muted 'Lease: expires <rel> (heartbeat <rel> ago)'
line only for in_progress issues holding an active lease; add formatTimeUntil
as the forward-looking mirror of formatTimeAgo.
- types_test: assert lease keys appear when set, are omitted when nil, and that
row_lock is never surfaced.
* fix(lease): settle v54 review blockers
Address #4537 review blockers before landing v54: keep heartbeat off wisps in both backends, keep direct status/assignee updates from leaving stale leases, guard the down migration, and emit reclaim JSON as snake_case.
Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie
* fix(lease): cover lease fields in tests and hooks
Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
Move all pure SQL-text builders shared by issueops (classic) and
domain/db (proxied) into one package; both stacks now delegate, so
ready/filter/counts query semantics can no longer drift (Seam A).
Execution orchestration stays per-stack per the extract-don't-unify
decision. Canonical SQL text is classic's; the only proxied-side text
deltas are IN-list whitespace.
Adversarial re-review (bd-6dnrw.47) verified all 10 attack items with
no production divergence, and adds an import-boundary test pinning the
layering (domain must never import issueops/sqlbuild) plus a single-use
constraint note on LabelSearchPlan.MergeInto. Follow-ups filed:
bd-emrs2 (+1-overfetch convention), bd-kzeme (IN-helper consolidation).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>