beads/internal/storage/sqlbuild/sqlbuild.go
Steve Yegge e97839a2e1
feat(lease): land the leasing stack + schema v54 (claim-TTL, heartbeat, reclaim) (#4537)
* fix(dolt): retry mutate+commit methods on serialization conflict (1213/1205)

ClaimIssue, ClaimReadyIssue, UpdateIssue and CloseIssue each hand-rolled
their own tx (BeginTx + DOLT_ADD/DOLT_COMMIT + Commit) outside
withRetryTx, so a concurrent writer that lost Dolt's optimistic
commit-time merge surfaced MySQL 1213/1205 as a hard failure instead of
retrying. Dolt has no real row locking — FOR UPDATE / SKIP LOCKED are
parse-only no-ops (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/)
— so retry is the only safety net under N concurrent workers. These four
are the work-queue hot paths: `bd ready --claim`, claim, update, close.

Move each body into a withRetryTx closure (which owns BeginTx and the
final Commit and retries serialization failures with backoff), preserving
the single-tx CAS/mutation + DOLT_COMMIT atomicity exactly. On retry,
ClaimReadyIssue re-scans the ready front from a fresh snapshot and claims
the next available issue.

Tested: TestEmbeddedReadyClaimConcurrent, TestEmbeddedReadyConcurrent,
TestEmbeddedCloseConcurrent, TestEmbeddedUpdateConcurrent and the
non-concurrent embedded Update/Close/Ready tests pass; build + vet clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(dolt): add concurrent work-queue drain test (Gas Station claim invariant)

N workers concurrently dequeue one shared ready-front via ClaimReadyIssue
until empty; asserts every issue claimed exactly once, none stranded, no
double-claims, and no serialization error ever surfaces (withRetryTx absorbs
the 1213/1205 from the claim-CAS cell collision). This is the regression guard
for the multi-agent claim queue that #4491 makes safe.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(lease): claim-TTL + heartbeat + reclaim for dead-worker recovery

Gas Station v1.1 (wy-5r9j). A claim was previously permanent: a worker that
died mid-task stranded its issue in_progress forever. This adds a lease so a
claim can expire and be recovered.

Schema (migration 0054): add lease_expires_at, heartbeat_at (DATETIME) and
row_lock (BIGINT) to issues (and wisps, for uniform issueops routing), plus
idx_issues_lease for the reaper scan. assignee doubles as the lease owner.

Lifecycle (shared issueops, so both server-mode DoltStore and EmbeddedDoltStore
get it):
  - Claim stamps lease_expires_at = now + TTL (default 5m, WithLeaseTTL to
    override), heartbeat_at = now, and a fresh row_lock.
  - bd heartbeat <id> — owner-only; pushes the lease forward and rewrites
    row_lock. Fails (ErrNotClaimable/ErrAlreadyClaimed) once the lease is gone.
  - bd reclaim --older-than <dur> — reverts in_progress issues whose lease
    expired more than <dur> ago back to ready (clears assignee/started_at,
    records a lease_reclaimed event). Default grace 2×TTL.

The row_lock trick: Dolt has no row locking and merges concurrent commits
cell-by-cell, so a heartbeat (writes heartbeat_at) racing a reclaim (writes
status) would silently cell-merge into a zombie — an open/unassigned issue that
still carries the worker's fresh heartbeat. Every mutating path (claim, close,
update, heartbeat, reclaim) now rewrites the shared row_lock cell, forcing the
loser to a 1213/1205 serialization conflict that withRetryTx replays. All lease
paths are withRetryTx/withConn wrapped.

Tests (real Dolt): TestRowLockForcesConflictOnDisjointCellWrites proves the bug
without row_lock (silent merge to zombie) and correctness with it (conflict);
TestConcurrentHeartbeatReclaimClose is the integration race (live workers
heartbeat+close vs a continuous reaper vs dead workers) asserting exactly-once,
no lost close, no zombie; plus claim/heartbeat/reclaim unit coverage on both
backends.

Note: lease columns are second-granular DATETIME (Dolt rounds), so sub-second
TTLs are meaningless — production TTL is minutes. Heartbeat writes a Dolt
commit, so cadence must stay ≪ claim cadence to avoid history bloat.

Fixes the test coupling in TestMigration0053PromotesRigWisps, which assumed the
rig-wisp repair was always LatestVersion().

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ci+docs: allowlist lease stress tests; scope the row_lock invariant comment

- Add TestConcurrentWorkQueueDrain and TestConcurrentHeartbeatReclaimClose
  to the check-testing-short.sh allowlist; both are real-Dolt runtime/stress
  skips and were failing ci-pr-policy.
- freshRowLock doc said "EVERY mutating path" rewrites row_lock, which
  overstates it. Scope it to status/ownership/lease-mutating paths (the set
  the reclaim/heartbeat races care about) and document the invariant + why
  orthogonal-cell writers (is_blocked, compaction, deps, rename, reopen) are
  intentionally exempt, so a future writer does not silently reintroduce the
  zombie-merge bug.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(show): surface lease_expires_at/heartbeat_at in bd show (--json + text) (wy-9cdw)

Migration 0054 added lease_expires_at/heartbeat_at (and internal row_lock) to
the issues/wisps tables for the claim/heartbeat/reclaim leasing feature, but
bd show could not observe lease state — you had to drop to raw SQL.

- types.Issue: add LeaseExpiresAt/HeartbeatAt (*time.Time, omitempty), matching
  the StartedAt nullable-timestamp convention. row_lock stays unexported.
- sqlbuild.IssueSelectColumns: append lease_expires_at, heartbeat_at to the
  canonical hydration list (flows to every stack via the shared constant).
- issueops.ScanIssueFrom: scan + map the two nullable columns (before the
  trailing extra dests, keeping positional parity across all scan sites).
- bd show text view: emit a muted 'Lease: expires <rel> (heartbeat <rel> ago)'
  line only for in_progress issues holding an active lease; add formatTimeUntil
  as the forward-looking mirror of formatTimeAgo.
- types_test: assert lease keys appear when set, are omitted when nil, and that
  row_lock is never surfaced.

* fix(lease): settle v54 review blockers

Address #4537 review blockers before landing v54: keep heartbeat off wisps in both backends, keep direct status/assignee updates from leaving stale leases, guard the down migration, and emit reclaim JSON as snake_case.

Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie

* fix(lease): cover lease fields in tests and hooks

Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
2026-07-03 23:28:55 -07:00

89 lines
3.7 KiB
Go

// Package sqlbuild holds the pure SQL-text builders shared by the classic
// issueops stack (production, *sql.Tx) and the domain/db repository stack
// (proxied-server, Runner). Functions here map (filter, tables) -> (sql, args)
// and are execution-context-free: no database handles, no transactions.
//
// Both stacks must produce identical row sets for identical filters (pinned
// by the Seam A parity suite in internal/storage/domain/db). Keeping the
// query text in one place is what prevents the two implementations from
// drifting; see bd-6dnrw.46 for the unify-vs-duplicate decision. Execution
// orchestration (issues+wisps merge strategy, hydration, probes) is
// intentionally NOT unified and stays in each stack.
package sqlbuild
import "strings"
// FilterTables configures table names for the filter builders, allowing the
// same filter logic to target both the issues and wisps table families.
type FilterTables struct {
Main string // "issues" or "wisps"
Labels string // "labels" or "wisp_labels"
Dependencies string // "dependencies" or "wisp_dependencies"
Comments string // "comments" or "wisp_comments"
}
var (
IssuesFilterTables = FilterTables{Main: "issues", Labels: "labels", Dependencies: "dependencies", Comments: "comments"}
WispsFilterTables = FilterTables{Main: "wisps", Labels: "wisp_labels", Dependencies: "wisp_dependencies", Comments: "wisp_comments"}
)
// DepTargetExpr resolves a dependency row's target across the three
// mutually-exclusive target columns.
const DepTargetExpr = "COALESCE(depends_on_issue_id, depends_on_wisp_id, depends_on_external)"
// IssueSelectColumns is the canonical column list for full issue hydration.
// Every query that reads a complete types.Issue should use this constant;
// the scan side is issueops.ScanIssueFrom, which scans positionally and
// must stay in column-for-column agreement with this list.
const IssueSelectColumns = `id, content_hash, title, description, design, acceptance_criteria, notes,
status, priority, issue_type, assignee, estimated_minutes,
created_at, created_by, owner, updated_at, started_at, closed_at, external_ref, spec_id,
compaction_level, compacted_at, compacted_at_commit, original_size, source_repo, close_reason,
sender, ephemeral, no_history, wisp_type, pinned, is_template,
await_type, await_id, timeout_ns, waiters,
mol_type,
event_kind, actor, target, payload,
due_at, defer_until,
work_type, source_system, metadata,
lease_expires_at, heartbeat_at`
// QueryBatchSize bounds IN-clause sizes when long ID lists are folded into
// WHERE fragments.
const QueryBatchSize = 200
// QualifyColumns prefixes every column in a comma-separated list (e.g.
// IssueSelectColumns) with the given table qualifier such as "i.".
func QualifyColumns(columns, prefix string) string {
raw := strings.ReplaceAll(columns, "\n", " ")
raw = strings.ReplaceAll(raw, "\t", " ")
parts := strings.Split(raw, ",")
for i, p := range parts {
parts[i] = prefix + strings.TrimSpace(p)
}
return strings.Join(parts, ", ")
}
// InPlaceholders renders "?,?,..." and the matching arg slice for an IN clause.
func InPlaceholders[T ~string](values []T) (string, []any) {
ph := make([]string, len(values))
args := make([]any, len(values))
for i, v := range values {
ph[i] = "?"
args[i] = string(v)
}
return strings.Join(ph, ","), args
}
// CompactNonEmptyStrings drops empty entries, returning nil when none remain.
func CompactNonEmptyStrings(values []string) []string {
if len(values) == 0 {
return nil
}
out := make([]string, 0, len(values))
for _, value := range values {
if value != "" {
out = append(out, value)
}
}
return out
}