From 470ec832b50b8e452a43f56c47e87128b8d2ecb7 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 9 Oct 2026 14:56:32 +0000 Subject: [PATCH] feat(roles): LeaseReclaimer role, POST issues:reclaim, and bd reclaim on both routes (S9) (#7412) * feat(roles): LeaseReclaimer role and POST issues:reclaim (S9) Adds issueops.LeaseReclaimer.Reclaim(ReclaimRequest{Actor, OlderThan, Filter}) -> ReclaimResult{Reclaimed []ReclaimedLease{ID, PreviousOwner, Revision}}, the stale-lease sweep behind `bd reclaim`, on every leg: - dolt, embedded dolt and the unit-of-work provider all run one body, internal/storage/issueops.ExecuteReclaimInTx over ReclaimExpiredLeasesInTx, and record one version commit per sweep that reverted anything through the issue-operation funnel (so a deferred version commit on the context is honored); - each reverted row's fresh row_lock is published as its revision, so a former holder's guarded write fails with ErrVersionMismatch; - an id that is not a stale lease (absent, live, unheld) is left out of Reclaimed, never an error; the id scope is capped at 1000 (*TooManyReclaimIDsError, counted as sent); - the hook decorator fires on_update once per reverted row (and the proxied notifying provider records one update per row); telemetry wraps it like every role. bd serve publishes it as POST /v0/beads/issues:reclaim behind the new issues.reclaim capability (httpapi.Config.LeaseReclaimer, required like every role). Spec, apigen and the wire-shape golden are regenerated; the change is additive and wire_revision is unchanged. The HTTP client leg dials it (wire vocabulary, capability preflight, local validation in the role's order). Conformance: RunLeaseReclaimer* (12 cases: revert post-state, the revision race, live lease, grace window, id scope with absent/live ids, empty answer, assignee/label scopes re-homed from reclaim_scope.go, recovery event attribution, on_update once per row, refusals, the cap boundary, no request mutation) wired on dolt, embeddeddolt, uow and the served http leg, and so held by the leg lock. Handler tests cover the member mapping, the projection, every handler refusal by name and the role's refusals reaching the wire. Co-Authored-By: Claude Opus 5.5 * feat(bd): bd reclaim runs the LeaseReclaimer role on both routes (S9) `bd reclaim` takes issueops.LeaseReclaimer off the decorated store on the direct route (so a connected HTTP workspace dials issues:reclaim instead of refusing) and off the proxied provider's own accessor on the proxied route, replacing the raw ReclaimExpiredLeases calls and runReclaimProxiedServer. Visible changes: --json entries carry the reverted row's new `revision`, the workspace's on_update hook fires once per reverted row, and the version commit is the role's own ("bd: reclaim N expired lease(s)", honoring dolt.auto-commit through the context the way bd prune does). Tests: an embedded direct-route CLI test (scoped sweep with a live and an absent id, revision, one version commit, on_update once) and the proxied reclaim integration test now asserts the revision. Co-Authored-By: Claude Opus 5.5 * fix(roles): S9 review LOWs: one validator, field attribution, overflow bound - The library's ValidateReclaimRequest is the one definition of a valid reclaim: it now also refuses a blank assignee/label/label-any/exclude entry, and every refusal but the id cap is a *issueops.ReclaimFieldError naming its field. The httpapi handler no longer checks blank entries itself; failReclaimErr names the member the role's error is about (`ids` for the cap) instead of blaming `ids` for every validation. The HTTP client's restatement matches, pinned field by field. - older_than_seconds is refused at >= MaxInt64/1e9: that float rounds to 2^63 ns, which wrapped negative under the old > check. - New contract case RunLeaseReclaimerRefusesABlankScopeEntry on all four legs; handler tests for field attribution and the overflow boundary. - CHANGELOG names the 1000-id cap on bd reclaim --id and is rewrapped; the client's grace-window precision comment no longer claims exactness. Co-Authored-By: Claude Opus 5.5 * fix(roles): S9 review: regenerate the unsupported shell, cgo-tag the embedded contract, one actor spelling - go generate ./internal/httpclient: the new accessor moves the partial-shell trailer to 145 of 200. - lease_reclaimer_contract_test.go gets //go:build cgo like its siblings, so the pure-Go lane compiles. - The library trims the actor before attributing, matching the handler, so " alice " is attributed the same on every route. Co-Authored-By: Claude Opus 5.5 (1M context) * fix(roles): S9 review minors: bound the reclaim actor on every leg, pin any_replica, validate before the uow Review minors on #7412, each in code this PR adds. - The reclaim actor lands on every reverted row's recovery event, in a VARCHAR(255) column, but ValidateReclaimRequest never bounded it: a 256-character actor passed validation on the dolt, embedded-dolt and unit-of-work legs and failed the sweep with the backend's own "Error 1105: string '...' is too large for column 'actor'" on the first row it would have reverted (all three legs answer exactly that with the bound below removed). The shared validator now holds the trimmed actor to types.MaxFieldLen, as claimIssueInTx does, so it is a *ReclaimFieldError for `actor` on every leg. The HTTP client restates the rule, and the parity test covers both sides of it. bd serve already refused it through the claim's name rules. The role's Actor and REFUSALS docs name the bound, and the conformance case runs it on all four legs. - ValidateReclaimRequest's doc no longer says bd serve's handler leaves every rule to it: it names the three wire rules the handler adds of its own. - TestReclaimSendsAnyReplicaOnlyWhenAsked pins the one filter member that widens a sweep: the client sends any_replica only when the caller set it. - The unit-of-work leg validates before opening its unit of work, as the dolt and embedded-dolt legs do; TestLeaseReclaimerRefusesBeforeOpeningAUOW pins it. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- .github/scripts/embedded-cmd-test-shards.txt | 1 + .../scripts/embedded-storage-test-shards.txt | 1 + CHANGELOG.md | 15 + backend/conformance/BUILD.bazel | 2 + .../conformance/lease_reclaimer_contract.go | 625 ++++++++++++++++++ cmd/bd/BUILD.bazel | 1 + cmd/bd/reclaim.go | 76 ++- cmd/bd/reclaim_embedded_test.go | 133 ++++ cmd/bd/serve.go | 30 +- cmd/bd/serve_source_test.go | 27 + cmd/bd/serve_store_identity_test.go | 8 + cmd/bd/unclaim_proxied_integration_test.go | 6 + cmd/bd/unclaim_proxied_server.go | 31 - internal/httpapi/BUILD.bazel | 2 + internal/httpapi/apigen/types.gen.go | 43 +- internal/httpapi/claim.go | 7 + internal/httpapi/pinning.go | 2 + internal/httpapi/problem.go | 11 + internal/httpapi/reclaim.go | 249 +++++++ internal/httpapi/reclaim_test.go | 317 +++++++++ internal/httpapi/roles_test.go | 31 + internal/httpapi/routes.go | 11 + internal/httpapi/server.go | 25 +- internal/httpapi/spec/openapi.v0.yaml | 196 +++++- internal/httpapi/spec_parity_test.go | 11 +- .../httpapi/wireshape/testdata/golden.json | 87 +++ internal/httpclient/BUILD.bazel | 2 + internal/httpclient/accessors.go | 9 + internal/httpclient/encode/oracle_test.go | 2 + internal/httpclient/leasereclaimer.go | 135 ++++ internal/httpclient/served_harness_test.go | 2 + .../httpclient/served_lease_reclaimer_test.go | 111 ++++ internal/httpclient/served_test.go | 3 + internal/httpclient/unsupported_gen.go | 2 +- internal/httpclient/wave2b_roles_test.go | 86 +++ internal/httpclient/wire/handshake.go | 5 + internal/httpclient/wire/paths.go | 2 + .../wire/revision_tolerance_test.go | 4 + internal/httpclient/wire/skew_test.go | 11 +- internal/httpclient/wire/vocabulary_test.go | 1 + internal/storage/BUILD.bazel | 2 + internal/storage/dolt/BUILD.bazel | 2 + internal/storage/dolt/lease_reclaimer.go | 50 ++ .../dolt/lease_reclaimer_contract_test.go | 114 ++++ internal/storage/domain/db/issue.go | 7 + internal/storage/domain/issue.go | 17 + internal/storage/embeddeddolt/BUILD.bazel | 2 + .../storage/embeddeddolt/lease_reclaimer.go | 54 ++ .../lease_reclaimer_contract_test.go | 111 ++++ .../storage/externaldeps/uow_decorator.go | 6 +- internal/storage/hook_batch_applier_test.go | 4 + internal/storage/hook_decorator.go | 16 + internal/storage/hook_issue_operations.go | 1 + .../storage/hook_issue_operations_test.go | 3 + internal/storage/hook_lease_reclaimer.go | 42 ++ internal/storage/hook_lease_reclaimer_test.go | 83 +++ internal/storage/issueops/BUILD.bazel | 2 + internal/storage/issueops/lease.go | 9 +- internal/storage/issueops/lease_reclaimer.go | 123 ++++ .../storage/issueops/lease_reclaimer_test.go | 107 +++ .../storage/role_accessor_decorator_test.go | 130 ++-- internal/storage/storage.go | 9 + internal/storage/uow/BUILD.bazel | 2 + internal/storage/uow/lease_reclaimer.go | 60 ++ .../uow/lease_reclaimer_contract_test.go | 146 ++++ internal/storage/uow/notifying.go | 24 + internal/telemetry/BUILD.bazel | 1 + internal/telemetry/lease_reclaimer.go | 36 + .../telemetry/role_accessor_decorator_test.go | 12 +- internal/types/types.go | 7 + issue_roles_external_test.go | 33 + issueops/BUILD.bazel | 1 + issueops/leasereclaimer.go | 183 +++++ 73 files changed, 3582 insertions(+), 140 deletions(-) create mode 100644 backend/conformance/lease_reclaimer_contract.go create mode 100644 cmd/bd/reclaim_embedded_test.go create mode 100644 internal/httpapi/reclaim.go create mode 100644 internal/httpapi/reclaim_test.go create mode 100644 internal/httpclient/leasereclaimer.go create mode 100644 internal/httpclient/served_lease_reclaimer_test.go create mode 100644 internal/storage/dolt/lease_reclaimer.go create mode 100644 internal/storage/dolt/lease_reclaimer_contract_test.go create mode 100644 internal/storage/embeddeddolt/lease_reclaimer.go create mode 100644 internal/storage/embeddeddolt/lease_reclaimer_contract_test.go create mode 100644 internal/storage/hook_lease_reclaimer.go create mode 100644 internal/storage/hook_lease_reclaimer_test.go create mode 100644 internal/storage/issueops/lease_reclaimer.go create mode 100644 internal/storage/issueops/lease_reclaimer_test.go create mode 100644 internal/storage/uow/lease_reclaimer.go create mode 100644 internal/storage/uow/lease_reclaimer_contract_test.go create mode 100644 internal/telemetry/lease_reclaimer.go create mode 100644 issueops/leasereclaimer.go diff --git a/.github/scripts/embedded-cmd-test-shards.txt b/.github/scripts/embedded-cmd-test-shards.txt index 7a1c26452..a0ef80ddb 100644 --- a/.github/scripts/embedded-cmd-test-shards.txt +++ b/.github/scripts/embedded-cmd-test-shards.txt @@ -346,6 +346,7 @@ 50 24 TestEmbeddedCloseBasic 50 24 TestEmbeddedImportUpsertAndPrefixSync 50 24 TestEmbeddedQuery +50 24 TestEmbeddedReclaimRunsTheLeaseReclaimerRole 50 24 TestEmbeddedUpdateRoutedStoreCommitsTargetHead 50 25 TestEmbeddedBackupConcurrent diff --git a/.github/scripts/embedded-storage-test-shards.txt b/.github/scripts/embedded-storage-test-shards.txt index c92319f31..da78fa452 100644 --- a/.github/scripts/embedded-storage-test-shards.txt +++ b/.github/scripts/embedded-storage-test-shards.txt @@ -433,6 +433,7 @@ 20 15 TestExpectedRevisionContract 20 15 TestLargeBatchApplyStatementCounts356_Embedded 20 15 TestLargeGraphCreateTiming_Embedded +20 15 TestLeaseReclaimerContract 20 15 TestReadyWorkPageTotalFilterMatrix 20 15 TestReclaimExpiredLeaseSurvivesRestartEmbedded 20 15 TestRemoteEntryPointsUseVersionControlOps diff --git a/CHANGELOG.md b/CHANGELOG.md index 9831b832c..3497db5d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `issueops.LeaseReclaimer`, the stale-lease sweep behind `bd reclaim`, is a + role on every leg (dolt, embedded dolt, the unit-of-work provider and the + HTTP client), and `bd serve` publishes it as `POST /v0/beads/issues:reclaim` + behind the new `issues.reclaim` capability (additive; `wire_revision` is + unchanged). Every reverted row mints a new revision, reported per entry as + `revision`, so a holder that writes with its pre-reclaim token gets a version + mismatch. An id that is not a stale lease is left out of the answer, not + refused. `bd reclaim` now runs the role on both routes, which changes four + things a script can see: its `--json` entries gain `revision`; the + workspace's `on_update` hook fires once per reverted row (the raw path fired + none); `--id` now accepts at most 1000 ids per run (more is refused, never + truncated); and the role records its own version commit, honoring + `dolt.auto-commit` the way `bd prune` does, as + `bd: reclaim N expired lease(s)` on both routes. Over a connected HTTP + workspace `bd reclaim` now works instead of refusing. - `bd create --graph` now plans its batch through `issueops.BatchApplier` instead of the old `buildDomainGraphPlan` path, so a graph create gets the same atomic multi-row semantics as `bd batch apply`. A `waits-for` edge's diff --git a/backend/conformance/BUILD.bazel b/backend/conformance/BUILD.bazel index 1f9632e79..f5c8882cc 100644 --- a/backend/conformance/BUILD.bazel +++ b/backend/conformance/BUILD.bazel @@ -35,6 +35,7 @@ go_library( "issue_operations_contract.go", "issue_operations_staging.go", "journal_contract.go", + "lease_reclaimer_contract.go", "lifecycle_close_reopen_contract.go", "lifecycle_create_contract.go", "lifecycle_update_contract.go", @@ -62,6 +63,7 @@ go_library( importpath = "github.com/steveyegge/beads/backend/conformance", visibility = ["//visibility:public"], deps = [ + "//internal/hooks", "//internal/storage", "//internal/storage/issueops", "//internal/storage/kvkeys", diff --git a/backend/conformance/lease_reclaimer_contract.go b/backend/conformance/lease_reclaimer_contract.go new file mode 100644 index 000000000..eadc6c176 --- /dev/null +++ b/backend/conformance/lease_reclaimer_contract.go @@ -0,0 +1,625 @@ +package conformance + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + "runtime" + "slices" + "sort" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/hooks" + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// This file holds the contract every implementation of publicops.LeaseReclaimer +// must satisfy. Each case asserts what issueops/leasereclaimer.go PROMISES, +// cited by symbol. +// +// FOUR LEGS, ONE BODY. dolt, embedded dolt and the unit-of-work provider all +// reach internal/storage/issueops.ExecuteReclaimInTx, and the http leg reaches +// the same body on the far side of POST /v0/beads/issues:reclaim. So a per-leg +// failure here is a wrapper's failure — a lost scope, a revision that never +// reached the caller, a refusal that stopped matching errors.Is — and the +// cases are written to catch exactly those. +// +// EVERY SWEEP HERE IS SCOPED. The legs share one database per suite with every +// other role's cases, and an UNSCOPED reclaim would revert whatever stale lease +// another case left behind. Every call below names its ids, or an assignee or +// label this case alone uses, so a case can never reach outside what it seeded. +// The workspace-wide form is the same SQL with the scope clause absent, which +// sqlbuild.ReclaimScopeSQL's own tests pin. +// +// STALENESS IS MANUFACTURED, NOT WAITED FOR. The role refuses a negative +// OlderThan (the raw method's -time.Hour trick is exactly what it rules out), +// so a case claims through the leg's real Claimer — which writes the lease row +// every reclaim reads — and then moves that row's lease_expires_at into the +// past through Exec. +type LeaseReclaimerFixture struct { + // IssuePrefix namespaces the ids each assertion seeds, so several of them + // can share one database. + IssuePrefix string + // LeaseReclaimer is the surface under test. + LeaseReclaimer publicops.LeaseReclaimer + // CreateIssue seeds a durable issue in the issues plane. + CreateIssue func(context.Context, *types.Issue, string) error + // Claimer takes the claim, and with it the lease row, a reclaim reverts. It + // is the leg's own role rather than a raw insert so the lease is the shape + // a real claim leaves, granted_node included. + Claimer publicops.Claimer + // Lifecycle performs the guarded write the revision case races against the + // reclaim. A nil Lifecycle SKIPS that case loudly. + Lifecycle publicops.Lifecycle + // Exec runs raw statements against the backing database: the cases use it + // to age a lease and to attach labels. A nil Exec SKIPS every case that + // needs a stale lease, which is nearly all of them, so every leg supplies + // one. + Exec func(ctx context.Context, statements []SQLStatement) error + // QueryScalar runs a single-row query and scans it, so the post-state is + // read RAW rather than through the role being tested. + QueryScalar func(context.Context, string, []any, ...any) error + // HookedLeaseReclaimer is the same role taken off the leg's HOOK-FIRING + // chain (storage.HookFiringStore on the store legs and the http leg, + // uow.NewNotifyingProvider on the unit-of-work leg), built over the runner + // FiredUpdates reads. Nil, with FiredUpdates, SKIPS the hook case loudly. + HookedLeaseReclaimer publicops.LeaseReclaimer + // FiredUpdates reports, in firing order, the issue id of every on_update + // the hooked chain ran so far. It must not return before the hooks it is + // asked about have finished. + FiredUpdates func(context.Context) ([]string, error) +} + +const ( + leaseReclaimerReaper = "reaper" + leaseReclaimerHolder = "lr-holder" +) + +// RunLeaseReclaimerRevertsAStaleLease pins the post-state the role names +// (issueops/leasereclaimer.go, LeaseReclaimer: "each reclaimed row moves +// in_progress -> open and the assignee is cleared") and the three members of +// the ReclaimedLease it reports, each READ RAW. +func RunLeaseReclaimerRevertsAStaleLease(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-revert" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + + result := leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + }) + if len(result.Reclaimed) != 1 { + t.Fatalf("Reclaimed = %+v, want exactly %s", result.Reclaimed, id) + } + got := result.Reclaimed[0] + if got.ID != id || got.PreviousOwner != leaseReclaimerHolder { + t.Fatalf("Reclaimed[0] = %+v, want {ID:%s PreviousOwner:%s}", got, id, leaseReclaimerHolder) + } + + if status := leaseReclaimerScalar[string](t, ctx, fixture, "SELECT status FROM issues WHERE id = ?", id); status != string(types.StatusOpen) { + t.Fatalf("status after reclaim = %q, want %q", status, types.StatusOpen) + } + if held := leaseReclaimerScalar[int](t, ctx, fixture, + "SELECT COUNT(*) FROM issues WHERE id = ? AND (COALESCE(assignee, '') <> '' OR started_at IS NOT NULL)", id); held != 0 { + t.Fatal("reclaim left the assignee or started_at set; the row is still held by a dead worker") + } + if leases := leaseReclaimerScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); leases != 0 { + t.Fatalf("lease rows after reclaim = %d, want 0", leases) + } + stored := leaseReclaimerScalar[int64](t, ctx, fixture, "SELECT row_lock FROM issues WHERE id = ?", id) + if got.Revision != types.RevisionToken(stored) { + t.Fatalf("Reclaimed[0].Revision = %q, want the row's own token %q", got.Revision, types.RevisionToken(stored)) + } +} + +// RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder is the reason the +// revision is minted at all (LeaseReclaimer: "a racing holder's guarded write +// fails with ErrVersionMismatch"). A worker that lost its lease and wakes up +// holding the token its claim handed it must NOT be able to land a guarded +// write over the issue another worker may already have taken; the token the +// reclaim reported, and only it, lands. +func RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + if fixture.Lifecycle == nil { + t.Skip("LeaseReclaimerFixture.Lifecycle is nil: this backend cannot race a guarded write against the reclaim") + } + id := fixture.IssuePrefix + "-fence" + claimed := leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id)[id] + + result := leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + }) + if len(result.Reclaimed) != 1 { + t.Fatalf("Reclaimed = %+v, want exactly %s", result.Reclaimed, id) + } + minted, err := types.ParseRevisionToken(result.Reclaimed[0].Revision) + if err != nil { + t.Fatalf("Reclaimed[0].Revision %q is not a revision token: %v", result.Reclaimed[0].Revision, err) + } + if minted == claimed { + t.Fatalf("reclaim reported revision %d, the claim's own token: nothing was minted", minted) + } + + stale := claimed + _, err = fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{ + Actor: leaseReclaimerHolder, IssueID: id, ExpectedVersion: &stale, + Patch: publicops.IssuePatch{Notes: publicops.Field[string]{Set: true, Value: "late write"}}, + }) + if !errors.Is(err, publicops.ErrVersionMismatch) { + t.Fatalf("guarded write with the pre-reclaim token = %v, want ErrVersionMismatch", err) + } + + fresh := minted + if _, err := fixture.Lifecycle.Update(ctx, publicops.UpdateRequest{ + Actor: leaseReclaimerReaper, IssueID: id, ExpectedVersion: &fresh, + Patch: publicops.IssuePatch{Notes: publicops.Field[string]{Set: true, Value: "after reclaim"}}, + }); err != nil { + t.Fatalf("guarded write with the token the reclaim reported = %v, want it to land", err) + } +} + +// RunLeaseReclaimerLeavesALiveLeaseAlone pins staleness as the gate: a lease +// that has not expired is not eligible, even when its id is named, and naming +// it is not an error (ReclaimRequest.Filter: "FILTER.IDS IS NOT AN ERROR WHEN +// IT NAMES A NON-STALE OR UNKNOWN ISSUE"). +func RunLeaseReclaimerLeavesALiveLeaseAlone(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-live" + leaseReclaimerSeedClaimed(t, ctx, fixture, leaseReclaimerHolder, id) + + result := leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + }) + if len(result.Reclaimed) != 0 { + t.Fatalf("Reclaimed = %+v, want nothing: the lease has not expired", result.Reclaimed) + } + leaseReclaimerRequireHeld(t, ctx, fixture, id) +} + +// RunLeaseReclaimerHonorsTheGraceWindow pins ReclaimRequest.OlderThan: a lease +// expired less than OlderThan ago is left alone, and the same lease is +// reclaimed once the window is narrower than its age. +func RunLeaseReclaimerHonorsTheGraceWindow(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-grace" + leaseReclaimerSeedClaimed(t, ctx, fixture, leaseReclaimerHolder, id) + leaseReclaimerAge(t, ctx, fixture, 10*time.Minute, id) + + request := publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + OlderThan: time.Hour, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + } + if result := leaseReclaim(t, ctx, fixture, request); len(result.Reclaimed) != 0 { + t.Fatalf("OlderThan 1h over a lease 10m stale: Reclaimed = %+v, want nothing", result.Reclaimed) + } + leaseReclaimerRequireHeld(t, ctx, fixture, id) + + request.OlderThan = time.Minute + if got := leaseReclaimerIDs(leaseReclaim(t, ctx, fixture, request)); !slices.Equal(got, []string{id}) { + t.Fatalf("OlderThan 1m over a lease 10m stale: Reclaimed = %v, want [%s]", got, id) + } +} + +// RunLeaseReclaimerScopesToTheNamedIDs pins the IDs scope from both sides: an +// id that is not named is never reverted though it is stale, and an id that is +// named but absent or live is simply missing from Reclaimed — the call does +// not fail and does not report it. +func RunLeaseReclaimerScopesToTheNamedIDs(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + named := fixture.IssuePrefix + "-scope-named" + unnamed := fixture.IssuePrefix + "-scope-unnamed" + live := fixture.IssuePrefix + "-scope-live" + ghost := fixture.IssuePrefix + "-scope-ghost" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, named, unnamed) + leaseReclaimerSeedClaimed(t, ctx, fixture, leaseReclaimerHolder, live) + + result := leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{ghost, live, named}}, + }) + if got := leaseReclaimerIDs(result); !slices.Equal(got, []string{named}) { + t.Fatalf("Reclaimed = %v, want [%s]: only the named stale lease", got, named) + } + leaseReclaimerRequireHeld(t, ctx, fixture, unnamed) + leaseReclaimerRequireHeld(t, ctx, fixture, live) +} + +// RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult pins the empty answer: +// a sweep scoped only to ids that name no row is not a refusal, and its +// Reclaimed is empty and NON-NIL (ReclaimResult.Reclaimed: "Never nil for a +// successful call"). +func RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + result, err := fixture.LeaseReclaimer.Reclaim(ctx, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{fixture.IssuePrefix + "-absent-a", fixture.IssuePrefix + "-absent-b"}}, + }) + if err != nil { + t.Fatalf("Reclaim scoped to absent ids = %v, want an empty answer", err) + } + if result.Reclaimed == nil { + t.Fatal("Reclaimed is nil; the role promises an empty, non-nil slice") + } + if len(result.Reclaimed) != 0 { + t.Fatalf("Reclaimed = %+v, want nothing", result.Reclaimed) + } +} + +// RunLeaseReclaimerScopesToAssigneesAndLabels re-homes the raw method's scope +// cases (testReclaimScoped) onto the role: assignee, label (AND), label-any +// (OR) and exclude-label each narrow the sweep and never widen it. +func RunLeaseReclaimerScopesToAssigneesAndLabels(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + p := fixture.IssuePrefix + a1, a2, b1, c1 := p+"-lbl-a1", p+"-lbl-a2", p+"-lbl-b1", p+"-lbl-c1" + laneA, laneB, opus := p+"-lane-a", p+"-lane-b", p+"-tier-opus" + workerA, workerB, workerC := p+"-wa", p+"-wb", p+"-wc" + leaseReclaimerSeedStale(t, ctx, fixture, workerA, a1, a2) + leaseReclaimerSeedStale(t, ctx, fixture, workerB, b1) + leaseReclaimerSeedStale(t, ctx, fixture, workerC, c1) + leaseReclaimerLabel(t, ctx, fixture, a1, laneA) + leaseReclaimerLabel(t, ctx, fixture, a2, laneA) + leaseReclaimerLabel(t, ctx, fixture, a2, opus) + leaseReclaimerLabel(t, ctx, fixture, b1, laneB) + + reclaim := func(filter publicops.ReclaimFilter) []string { + t.Helper() + return leaseReclaimerIDs(leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{Actor: leaseReclaimerReaper, Filter: filter})) + } + + if got := reclaim(publicops.ReclaimFilter{Labels: []string{laneB}}); !slices.Equal(got, []string{b1}) { + t.Fatalf("reclaim(labels %s) = %v, want [%s]", laneB, got, b1) + } + if got := reclaim(publicops.ReclaimFilter{Assignees: []string{workerC}}); !slices.Equal(got, []string{c1}) { + t.Fatalf("reclaim(assignees %s) = %v, want [%s]", workerC, got, c1) + } + if got := reclaim(publicops.ReclaimFilter{Labels: []string{laneA}, ExcludeLabels: []string{opus}}); !slices.Equal(got, []string{a1}) { + t.Fatalf("reclaim(labels %s, exclude %s) = %v, want [%s]", laneA, opus, got, a1) + } + leaseReclaimerRequireHeld(t, ctx, fixture, a2) + if got := reclaim(publicops.ReclaimFilter{LabelsAny: []string{p + "-lane-z", opus}}); !slices.Equal(got, []string{a2}) { + t.Fatalf("reclaim(labels-any lane-z,%s) = %v, want [%s]", opus, got, a2) + } +} + +// RunLeaseReclaimerAttributesTheRecoveryEvent pins ReclaimRequest.Actor: the +// recovery event each reverted row records names the actor who ran the sweep, +// with the lease's former holder beside it. +func RunLeaseReclaimerAttributesTheRecoveryEvent(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-event" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + leaseReclaim(t, ctx, fixture, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id}}, + }) + if n := leaseReclaimerScalar[int](t, ctx, fixture, + "SELECT COUNT(*) FROM events WHERE issue_id = ? AND event_type = ? AND actor = ? AND old_value = ?", + id, string(types.EventLeaseReclaimed), leaseReclaimerReaper, leaseReclaimerHolder); n != 1 { + t.Fatalf("lease_reclaimed events by %s naming %s = %d, want 1", leaseReclaimerReaper, leaseReclaimerHolder, n) + } +} + +// RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow pins the hook the role +// names (LeaseReclaimer: "ITS HOOK IS on_update, once per reverted row"): each +// reverted row fires on_update exactly once, and a row the sweep did not +// revert — named but live — fires nothing. +func RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + if fixture.HookedLeaseReclaimer == nil || fixture.FiredUpdates == nil { + t.Skip("LeaseReclaimerFixture.HookedLeaseReclaimer/FiredUpdates is nil: this backend cannot observe its hooks") + } + first := fixture.IssuePrefix + "-hook-a" + second := fixture.IssuePrefix + "-hook-b" + live := fixture.IssuePrefix + "-hook-live" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, first, second) + leaseReclaimerSeedClaimed(t, ctx, fixture, leaseReclaimerHolder, live) + before, err := fixture.FiredUpdates(ctx) + if err != nil { + t.Fatalf("FiredUpdates before: %v", err) + } + + result, err := fixture.HookedLeaseReclaimer.Reclaim(ctx, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{first, second, live}}, + }) + if err != nil { + t.Fatalf("hooked Reclaim: %v", err) + } + if got := leaseReclaimerIDs(result); !slices.Equal(got, []string{first, second}) { + t.Fatalf("Reclaimed = %v, want [%s %s]", got, first, second) + } + + after, err := fixture.FiredUpdates(ctx) + if err != nil { + t.Fatalf("FiredUpdates after: %v", err) + } + fired := map[string]int{} + for _, id := range after[len(before):] { + fired[id]++ + } + for _, id := range []string{first, second} { + if fired[id] != 1 { + t.Fatalf("on_update fired %d time(s) for %s, want exactly 1 (fired: %v)", fired[id], id, after[len(before):]) + } + } + if fired[live] != 0 { + t.Fatalf("on_update fired for %s, which the sweep did not revert", live) + } +} + +// RunLeaseReclaimerRefusesAMalformedRequest pins every refusal the role lists, +// each as ErrValidation (the cap as *TooManyReclaimIDsError), and that a +// refusal changes nothing: the stale lease every request names survives all of +// them. +func RunLeaseReclaimerRefusesAMalformedRequest(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-malformed" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + + tooMany := make([]string, publicops.MaxReclaimIDs+1) + for i := range tooMany { + tooMany[i] = id + } + for _, test := range []struct { + name string + request publicops.ReclaimRequest + tooMany bool + }{ + {"an empty actor", publicops.ReclaimRequest{Filter: publicops.ReclaimFilter{IDs: []string{id}}}, false}, + // One past the column every reverted row's recovery event records it + // in: a typed refusal before the sweep, never the backend's own error + // from the first row it would have reverted. + {"an over-long actor", publicops.ReclaimRequest{Actor: strings.Repeat("r", types.MaxFieldLen+1), + Filter: publicops.ReclaimFilter{IDs: []string{id}}}, false}, + {"a negative grace window", publicops.ReclaimRequest{Actor: leaseReclaimerReaper, OlderThan: -time.Second, + Filter: publicops.ReclaimFilter{IDs: []string{id}}}, false}, + {"a blank scoped id", publicops.ReclaimRequest{Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: []string{id, ""}}}, false}, + {"more ids than the cap", publicops.ReclaimRequest{Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: tooMany}}, true}, + } { + _, err := fixture.LeaseReclaimer.Reclaim(ctx, test.request) + if !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("%s: Reclaim error = %v, want ErrValidation", test.name, err) + } + if test.tooMany { + var capErr *publicops.TooManyReclaimIDsError + if !errors.As(err, &capErr) || capErr.Requested != len(tooMany) || capErr.Cap != publicops.MaxReclaimIDs { + t.Fatalf("%s: Reclaim error = %v, want *TooManyReclaimIDsError{Requested:%d Cap:%d}", + test.name, err, len(tooMany), publicops.MaxReclaimIDs) + } + } + leaseReclaimerRequireHeld(t, ctx, fixture, id) + } +} + +// RunLeaseReclaimerRefusesABlankScopeEntry pins the blank-entry refusal on +// the scopes that are not ids: an empty or all-blank assignee or label is +// ErrValidation as a *ReclaimFieldError naming its field, and changes +// nothing — the stale lease every request names survives. +func RunLeaseReclaimerRefusesABlankScopeEntry(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-blankscope" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + + for _, test := range []struct { + field string + filter publicops.ReclaimFilter + }{ + {publicops.ReclaimFieldAssignees, publicops.ReclaimFilter{IDs: []string{id}, Assignees: []string{leaseReclaimerHolder, " "}}}, + {publicops.ReclaimFieldLabels, publicops.ReclaimFilter{IDs: []string{id}, Labels: []string{""}}}, + {publicops.ReclaimFieldLabelsAny, publicops.ReclaimFilter{IDs: []string{id}, LabelsAny: []string{"\t"}}}, + {publicops.ReclaimFieldExcludeLabels, publicops.ReclaimFilter{IDs: []string{id}, ExcludeLabels: []string{"x", " "}}}, + } { + _, err := fixture.LeaseReclaimer.Reclaim(ctx, publicops.ReclaimRequest{Actor: leaseReclaimerReaper, Filter: test.filter}) + var fieldErr *publicops.ReclaimFieldError + if !errors.Is(err, publicops.ErrValidation) || !errors.As(err, &fieldErr) || fieldErr.Field != test.field { + t.Fatalf("blank %s entry: Reclaim error = %v, want a *ReclaimFieldError naming %s", test.field, err, test.field) + } + leaseReclaimerRequireHeld(t, ctx, fixture, id) + } +} + +// RunLeaseReclaimerAcceptsExactlyTheCap pins the cap's boundary: MaxReclaimIDs +// entries is a request, one more is a refusal (the case above). +func RunLeaseReclaimerAcceptsExactlyTheCap(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + ids := make([]string, publicops.MaxReclaimIDs) + for i := range ids { + ids[i] = fmt.Sprintf("%s-cap-%04d", fixture.IssuePrefix, i) + } + result, err := fixture.LeaseReclaimer.Reclaim(ctx, publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, Filter: publicops.ReclaimFilter{IDs: ids}, + }) + if err != nil { + t.Fatalf("Reclaim with exactly MaxReclaimIDs ids = %v, want an answer", err) + } + if len(result.Reclaimed) != 0 { + t.Fatalf("Reclaimed = %+v, want nothing: none of the ids exist", result.Reclaimed) + } +} + +// RunLeaseReclaimerDoesNotMutateTheCallerRequest pins the role's "never mutate +// caller-owned request values": the scope slices come back as they were sent. +func RunLeaseReclaimerDoesNotMutateTheCallerRequest(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture) { + t.Helper() + requireLeaseReclaimerExec(t, fixture) + id := fixture.IssuePrefix + "-nomut" + leaseReclaimerSeedStale(t, ctx, fixture, leaseReclaimerHolder, id) + ids := []string{fixture.IssuePrefix + "-nomut-ghost", id} + assignees := []string{leaseReclaimerHolder} + request := publicops.ReclaimRequest{ + Actor: leaseReclaimerReaper, + Filter: publicops.ReclaimFilter{IDs: ids, Assignees: assignees}, + } + leaseReclaim(t, ctx, fixture, request) + if !slices.Equal(ids, []string{fixture.IssuePrefix + "-nomut-ghost", id}) || !slices.Equal(assignees, []string{leaseReclaimerHolder}) { + t.Fatalf("Reclaim rewrote the caller's scope: ids=%v assignees=%v", ids, assignees) + } +} + +// NewUpdateHookLog plants an on_update script that appends the id of every +// issue it fires for, and returns the runner a hook-firing chain is built over +// plus a reader for LeaseReclaimerFixture.FiredUpdates. The reader waits for +// in-flight hooks before reading, so it answers for every hook already fired. +// +// It answers (nil, nil) where a shell hook cannot run, so a fixture built from +// it leaves the hook case to skip rather than fail. +func NewUpdateHookLog(t *testing.T) (*hooks.Runner, func(context.Context) ([]string, error)) { + t.Helper() + if runtime.GOOS == "windows" { + return nil, nil + } + dir := t.TempDir() + log := filepath.Join(t.TempDir(), "on_update.log") + script := "#!/bin/sh\necho \"$1\" >> " + log + "\n" + if err := os.WriteFile(filepath.Join(dir, "on_update"), []byte(script), 0o755); err != nil { //nolint:gosec // G306: a hook must be executable to run at all + t.Fatalf("plant the on_update hook: %v", err) + } + runner := hooks.NewRunner(dir) + return runner, func(context.Context) ([]string, error) { + if !runner.Wait(30 * time.Second) { + return nil, errors.New("on_update hooks still running after 30s") + } + data, err := os.ReadFile(log) // #nosec G304 -- this helper's own temp file + if errors.Is(err, os.ErrNotExist) { + return []string{}, nil + } + if err != nil { + return nil, err + } + var ids []string + for _, line := range strings.Split(strings.TrimSpace(string(data)), "\n") { + if line = strings.TrimSpace(line); line != "" { + ids = append(ids, line) + } + } + return ids, nil + } +} + +func requireLeaseReclaimerExec(t *testing.T, fixture LeaseReclaimerFixture) { + t.Helper() + if fixture.Exec == nil || fixture.Claimer == nil || fixture.QueryScalar == nil { + t.Skip("LeaseReclaimerFixture.Exec/Claimer/QueryScalar is nil: this backend cannot manufacture a stale lease") + } +} + +// leaseReclaimerSeedClaimed creates each id and claims it as holder through the +// leg's Claimer, answering each claim's row version. +func leaseReclaimerSeedClaimed(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, holder string, ids ...string) map[string]int64 { + t.Helper() + versions := map[string]int64{} + for _, id := range ids { + if err := fixture.CreateIssue(ctx, &types.Issue{ + ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask, + }, "seed"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + claimed, err := fixture.Claimer.Claim(ctx, publicops.ClaimRequest{Actor: holder, IssueID: id}) + if err != nil { + t.Fatalf("claim %s as %s: %v", id, holder, err) + } + if claimed.Issue == nil { + t.Fatalf("claim %s answered no issue", id) + } + versions[id] = claimed.Issue.RowVersion + } + return versions +} + +// leaseReclaimerSeedStale claims each id and ages its lease an hour past +// expiry, far beyond any grace window a case below uses by default. +func leaseReclaimerSeedStale(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, holder string, ids ...string) map[string]int64 { + t.Helper() + versions := leaseReclaimerSeedClaimed(t, ctx, fixture, holder, ids...) + leaseReclaimerAge(t, ctx, fixture, time.Hour, ids...) + return versions +} + +// leaseReclaimerAge moves each id's lease expiry to age in the past. +func leaseReclaimerAge(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, age time.Duration, ids ...string) { + t.Helper() + expired := time.Now().UTC().Add(-age) + statements := make([]SQLStatement, 0, len(ids)) + for _, id := range ids { + statements = append(statements, SQLStatement{ + Query: "UPDATE leases SET lease_expires_at = ? WHERE issue_id = ?", + Args: []any{expired, id}, + }) + } + if err := fixture.Exec(ctx, statements); err != nil { + t.Fatalf("age leases %v: %v", ids, err) + } + for _, id := range ids { + if n := leaseReclaimerScalar[int](t, ctx, fixture, "SELECT COUNT(*) FROM leases WHERE issue_id = ?", id); n != 1 { + t.Fatalf("lease rows for %s = %d, want 1: the claim wrote no lease to reclaim", id, n) + } + } +} + +func leaseReclaimerLabel(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, id, label string) { + t.Helper() + if err := fixture.Exec(ctx, []SQLStatement{{ + Query: "INSERT INTO labels (issue_id, label) VALUES (?, ?)", + Args: []any{id, label}, + }}); err != nil { + t.Fatalf("label %s %s: %v", id, label, err) + } +} + +func leaseReclaimerRequireHeld(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, id string) { + t.Helper() + if status := leaseReclaimerScalar[string](t, ctx, fixture, "SELECT status FROM issues WHERE id = ?", id); status != string(types.StatusInProgress) { + t.Fatalf("%s status = %q, want it still %q", id, status, types.StatusInProgress) + } +} + +func leaseReclaim(t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, request publicops.ReclaimRequest) publicops.ReclaimResult { + t.Helper() + result, err := fixture.LeaseReclaimer.Reclaim(ctx, request) + if err != nil { + t.Fatalf("Reclaim(%+v): %v", request.Filter, err) + } + if result.Reclaimed == nil { + t.Fatal("Reclaimed is nil; the role promises a non-nil slice for a successful call") + } + return result +} + +func leaseReclaimerIDs(result publicops.ReclaimResult) []string { + ids := make([]string, 0, len(result.Reclaimed)) + for _, r := range result.Reclaimed { + ids = append(ids, r.ID) + } + sort.Strings(ids) + return ids +} + +func leaseReclaimerScalar[T any](t *testing.T, ctx context.Context, fixture LeaseReclaimerFixture, query string, args ...any) T { + t.Helper() + var value T + if err := fixture.QueryScalar(ctx, query, args, &value); err != nil { + t.Fatalf("%s %v: %v", query, args, err) + } + return value +} diff --git a/cmd/bd/BUILD.bazel b/cmd/bd/BUILD.bazel index b5e03704f..309ae4fc3 100644 --- a/cmd/bd/BUILD.bazel +++ b/cmd/bd/BUILD.bazel @@ -1048,6 +1048,7 @@ go_test( "ready_role_request_test.go", "ready_test.go", "ready_wisp_proxied_integration_test.go", + "reclaim_embedded_test.go", "reclaim_test.go", "recompute_blocked_proxied_integration_test.go", "registered_backend_workspace_paths_test.go", diff --git a/cmd/bd/reclaim.go b/cmd/bd/reclaim.go index 2a5262d99..7bde1f5b8 100644 --- a/cmd/bd/reclaim.go +++ b/cmd/bd/reclaim.go @@ -1,6 +1,7 @@ package main import ( + "errors" "fmt" "strings" @@ -8,9 +9,11 @@ import ( "github.com/spf13/pflag" "github.com/steveyegge/beads/internal/metrics" - "github.com/steveyegge/beads/internal/storage/issueops" + storeops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/storage/uow" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/ui" + "github.com/steveyegge/beads/issueops" ) var reclaimCmd = &cobra.Command{ @@ -99,31 +102,68 @@ Examples: CheckReadonly("reclaim") - if usesProxiedServer() { - return runReclaimProxiedServer(rootCtx, olderThan, filter) - } - - ctx := rootCtx - reclaimed, err := store.ReclaimExpiredLeases(ctx, olderThan, filter, currentActor()) + reclaimer, err := openLeaseReclaimer() if err != nil { return HandleErrorRespectJSON("reclaim: %v", err) } - - ids := make([]string, 0, len(reclaimed)) - for _, r := range reclaimed { - ids = append(ids, r.ID) + // The role mints the sweep's version commit itself, so batch and off + // modes have to be said on the CONTEXT, the way `bd prune` says them. + // The proxied route applied the policy to rootCtx in the root pre-run + // already, and issueOpsContext leaves a context that defers alone. + ctx := rootCtx + if !usesProxiedServer() { + if ctx, err = issueOpsContext(rootCtx); err != nil { + return HandleErrorRespectJSON("%v", err) + } } - if err := commitPendingIfEmbedded(ctx, store, currentActor(), doltAutoCommitParams{ - Command: "reclaim", - IssueIDs: ids, - }); err != nil { - return HandleErrorRespectJSON("failed to commit: %v", err) + result, err := reclaimer.Reclaim(ctx, issueops.ReclaimRequest{ + Actor: currentActor(), + OlderThan: olderThan, + Filter: filter, + }) + if err != nil { + return HandleErrorRespectJSON("reclaim: %v", err) + } + if len(result.Reclaimed) > 0 { + commandDidWrite.Store(true) } - return renderReclaim(reclaimed, !filter.IsEmpty()) + return renderReclaim(result.Reclaimed, !filter.IsEmpty()) }, } +// openLeaseReclaimer hands back the stale-lease sweep for whichever route this +// invocation is on: the store's own accessor (through every decorator the store +// was wired with, so the workspace's on_update hook fires once per reverted +// row, and over a registered HTTP backend the wire's reclaimIssues), or the +// proxied-server provider's. +func openLeaseReclaimer() (issueops.LeaseReclaimer, error) { + if usesProxiedServer() { + return proxiedLeaseReclaimer() + } + if store == nil { + if err := ensureStoreActive(); err != nil { + return nil, err + } + } + return store.LeaseReclaimer() +} + +// proxiedLeaseReclaimer takes the sweep through the proxied-server provider's +// OWN capability accessor, proxiedSweeper's two-step and for its reason: the +// accessor is where each layer (the notifying provider's hooks included) is +// added. +func proxiedLeaseReclaimer() (issueops.LeaseReclaimer, error) { + if uowProvider == nil { + return nil, errors.New("proxied-server UOW provider not initialized") + } + src, ok := uowProvider.(uow.LeaseReclaimerSource) + if !ok { + return nil, fmt.Errorf("proxied-server provider %T does not offer the lease-sweep surface", uowProvider) + } + return src.LeaseReclaimer() +} + func renderReclaim(reclaimed []types.ReclaimedLease, scoped bool) error { if jsonOutput { return outputJSON(map[string]interface{}{ @@ -224,7 +264,7 @@ func reclaimFilterFromFlags(cmd *cobra.Command) (types.ReclaimFilter, error) { } func init() { - reclaimCmd.Flags().Duration("older-than", 2*issueops.DefaultLeaseTTL, + reclaimCmd.Flags().Duration("older-than", 2*storeops.DefaultLeaseTTL, "Only reclaim leases that expired at least this long ago (grace window)") registerReclaimScopeFlags(reclaimCmd.Flags()) rootCmd.AddCommand(reclaimCmd) diff --git a/cmd/bd/reclaim_embedded_test.go b/cmd/bd/reclaim_embedded_test.go new file mode 100644 index 000000000..9a89f0961 --- /dev/null +++ b/cmd/bd/reclaim_embedded_test.go @@ -0,0 +1,133 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/types" +) + +// TestEmbeddedReclaimRunsTheLeaseReclaimerRole drives `bd reclaim` on the +// direct route, where it now runs issueops.LeaseReclaimer off the decorated +// store: the stale lease it names is reverted with a fresh revision, a live +// lease and an absent id it names are left out without an error, the sweep +// records its own version commit, and the workspace's on_update hook fires once +// for the reverted row. +func TestEmbeddedReclaimRunsTheLeaseReclaimerRole(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + + bd := buildEmbeddedBD(t) + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "rc") + + hookLog := filepath.Join(t.TempDir(), "on_update.log") + hooksDir := filepath.Join(beadsDir, "hooks") + if err := os.MkdirAll(hooksDir, 0o755); err != nil { + t.Fatalf("mkdir hooks: %v", err) + } + script := "#!/bin/sh\necho \"$1\" >> " + hookLog + "\n" + if err := os.WriteFile(filepath.Join(hooksDir, "on_update"), []byte(script), 0o755); err != nil { //nolint:gosec // G306: a hook must be executable to run at all + t.Fatalf("plant on_update: %v", err) + } + + stale := bdCreate(t, bd, dir, "Stale lease", "--type", "task") + live := bdCreate(t, bd, dir, "Live lease", "--type", "task") + bdUpdate(t, bd, dir, stale.ID, "--claim") + bdUpdate(t, bd, dir, live.ID, "--claim") + + cfg, _ := configfile.Load(beadsDir) + database := "" + if cfg != nil { + database = cfg.GetDoltDatabase() + } + dataDir := filepath.Join(beadsDir, "embeddeddolt") + withDB := func(fn func(exec func(string, ...any))) { + t.Helper() + db, cleanup, err := embeddeddolt.OpenSQL(t.Context(), dataDir, database, "main") + if err != nil { + t.Fatalf("OpenSQL: %v", err) + } + defer func() { _ = cleanup() }() + fn(func(query string, args ...any) { + t.Helper() + if _, err := db.ExecContext(t.Context(), query, args...); err != nil { + t.Fatalf("%s: %v", query, err) + } + }) + } + countCommits := func() int { + t.Helper() + db, cleanup, err := embeddeddolt.OpenSQL(t.Context(), dataDir, database, "main") + if err != nil { + t.Fatalf("OpenSQL: %v", err) + } + defer func() { _ = cleanup() }() + var n int + if err := db.QueryRowContext(t.Context(), "SELECT COUNT(*) FROM dolt_log WHERE message LIKE 'bd: reclaim%'").Scan(&n); err != nil { + t.Fatalf("count reclaim commits: %v", err) + } + return n + } + withDB(func(exec func(string, ...any)) { + exec("UPDATE leases SET lease_expires_at = ? WHERE issue_id = ?", time.Now().UTC().Add(-time.Hour), stale.ID) + }) + before := countCommits() + + out, err := bdRunWithFlockRetry(t, bd, dir, "reclaim", "--older-than", "0s", "--json", + "--id", stale.ID, "--id", live.ID, "--id", "rc-absent") + if err != nil { + t.Fatalf("bd reclaim: %v\n%s", err, out) + } + var got struct { + Count int `json:"count"` + Scoped bool `json:"scoped"` + Reclaimed []types.ReclaimedLease `json:"reclaimed"` + } + if err := json.Unmarshal(out, &got); err != nil { + t.Fatalf("unmarshal: %v\n%s", err, out) + } + if got.Count != 1 || len(got.Reclaimed) != 1 || got.Reclaimed[0].ID != stale.ID || !got.Scoped { + t.Fatalf("reclaim answered %+v, want exactly %s, scoped", got, stale.ID) + } + if _, err := types.ParseRevisionToken(got.Reclaimed[0].Revision); err != nil || got.Reclaimed[0].Revision == "" { + t.Fatalf("reclaimed revision %q is not a revision token: %v", got.Reclaimed[0].Revision, err) + } + + if after := countCommits(); after != before+1 { + t.Fatalf("reclaim version commits %d -> %d, want exactly one more", before, after) + } + if iss := bdShow(t, bd, dir, stale.ID); iss.Status != types.StatusOpen || iss.Assignee != "" { + t.Fatalf("%s after reclaim: status=%s assignee=%q, want open and unassigned", stale.ID, iss.Status, iss.Assignee) + } + if iss := bdShow(t, bd, dir, live.ID); iss.Status != types.StatusInProgress { + t.Fatalf("%s after reclaim: status=%s, want it still in_progress", live.ID, iss.Status) + } + + deadline := time.Now().Add(15 * time.Second) + for { + data, _ := os.ReadFile(hookLog) // #nosec G304 -- this test's own temp file + fired := map[string]int{} + for _, line := range strings.Fields(string(data)) { + fired[line]++ + } + // The claims fired on_update for both ids already; the reclaim adds + // one more for the reverted id and none for the live one. + if fired[stale.ID] == 2 && fired[live.ID] == 1 { + break + } + if time.Now().After(deadline) { + t.Fatalf("on_update fired %v, want %s twice (claim, reclaim) and %s once (claim)", fired, stale.ID, live.ID) + } + time.Sleep(50 * time.Millisecond) + } +} diff --git a/cmd/bd/serve.go b/cmd/bd/serve.go index fd5b1ad76..4a1592ccf 100644 --- a/cmd/bd/serve.go +++ b/cmd/bd/serve.go @@ -318,6 +318,7 @@ func runServe() error { EdgeReader: roles.edges, GraphCounter: roles.edgeCounter, BatchGetter: roles.batchGetter, + LeaseReclaimer: roles.leaseReclaimer, Relations: roles.relations, Commenter: roles.commenter, BlockingAnnotator: roles.blocking, @@ -674,6 +675,7 @@ type serveRoleSource interface { EdgeReader() (issueops.EdgeReader, error) GraphCounter() (issueops.GraphCounter, error) BatchGetter() (issueops.BatchGetter, error) + LeaseReclaimer() (issueops.LeaseReclaimer, error) IssueRelations() (issueops.Relations, error) Commenter() (issueops.Commenter, error) BlockingAnnotator() (issueops.BlockingAnnotator, error) @@ -741,6 +743,7 @@ func serveIssueRoles(src serveRoleSource, journalEnabled bool) (serveRoles, erro {"edge reader", func() (err error) { roles.edges, err = src.EdgeReader(); return }}, {"graph counter", func() (err error) { roles.edgeCounter, err = src.GraphCounter(); return }}, {"batch getter", func() (err error) { roles.batchGetter, err = src.BatchGetter(); return }}, + {"lease reclaimer", func() (err error) { roles.leaseReclaimer, err = src.LeaseReclaimer(); return }}, {"issue relations", func() (err error) { roles.relations, err = src.IssueRelations(); return }}, {"commenter", func() (err error) { roles.commenter, err = src.Commenter(); return }}, {"blocking annotator", func() (err error) { roles.blocking, err = src.BlockingAnnotator(); return }}, @@ -853,17 +856,22 @@ type serveRoles struct { // getter WITHOUT wrapping it — a read fires no completion hooks — so it is // taken off the peeled store with the rest for uniformity rather than out // of necessity. - batchGetter issueops.BatchGetter - relations issueops.Relations - commenter issueops.Commenter - blocking issueops.BlockingAnnotator - tree issueops.TreeWalker - readyCounter issueops.ReadyCounter - counter issueops.Counter - querier issueops.Querier - sweeper issueops.Sweeper - deleter issueops.Deleter - batchCreator issueops.BatchCreator + batchGetter issueops.BatchGetter + // leaseReclaimer is the stale-lease sweep behind + // POST /v0/beads/issues:reclaim. Its hook decorator WRAPS (a reclaim is an + // update per reverted row), which is why it comes off the peeled store with + // the releaser: bd serve runs no hooks. + leaseReclaimer issueops.LeaseReclaimer + relations issueops.Relations + commenter issueops.Commenter + blocking issueops.BlockingAnnotator + tree issueops.TreeWalker + readyCounter issueops.ReadyCounter + counter issueops.Counter + querier issueops.Querier + sweeper issueops.Sweeper + deleter issueops.Deleter + batchCreator issueops.BatchCreator // dependencyEditor is the second role here whose accessor recurses through // the hook decorator, so taking it off the peeled store is not optional: // HookFiringStore.DependencyEditor fires the workspace's update hook per diff --git a/cmd/bd/serve_source_test.go b/cmd/bd/serve_source_test.go index a0abf2235..1cb7ac105 100644 --- a/cmd/bd/serve_source_test.go +++ b/cmd/bd/serve_source_test.go @@ -100,6 +100,7 @@ func TestServeIssueRolesComeFromBeneathTheHookDecorator(t *testing.T) { batchCloser: &serveStubBatchCloser{}, readyClaimer: &serveStubReadyClaimer{}, releaser: &serveStubReleaser{}, + reclaimer: &serveStubLeaseReclaimer{}, lifecycle: &serveStubLifecycle{}, dependencies: &serveStubDependencyEditor{}, batchApplier: &serveStubBatchApplier{}, @@ -187,6 +188,15 @@ func TestServeIssueRolesComeFromBeneathTheHookDecorator(t *testing.T) { t.Fatal("the store's own accessor no longer returns a hook-firing releaser; this test proves nothing") } + // And for the lease reclaimer, which fires once per reverted row. + reclaimerFromTheStore, err := chained.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer: %v", err) + } + if !storage.RoleFiresHooks(reclaimerFromTheStore) { + t.Fatal("the store's own accessor no longer returns a hook-firing lease reclaimer; this test proves nothing") + } + // And for the commenter, the SEVENTH — and the one that was OUTSIDE the // RoleFiresHooks switch entirely until the add-comment operation went on the // wire. hook_commenter.go has fired the update hook for every comment it @@ -254,6 +264,9 @@ func TestServeIssueRolesComeFromBeneathTheHookDecorator(t *testing.T) { if storage.RoleFiresHooks(roles.releaser) { t.Error("bd serve would run this workspace's hooks on every HTTP release") } + if storage.RoleFiresHooks(roles.leaseReclaimer) || roles.leaseReclaimer != issueops.LeaseReclaimer(middle.reclaimer) { + t.Errorf("lease reclaimer came from %p, want the layer directly beneath the hooks (%p)", roles.leaseReclaimer, middle.reclaimer) + } reader, claimer := roles.reader, roles.claimer // The same predicate httpapi.Listen refuses on, so a regression here is a // server that refuses to boot rather than one that runs hooks silently. @@ -406,6 +419,7 @@ type serveRolesStore struct { batchCloser *serveStubBatchCloser readyClaimer *serveStubReadyClaimer releaser *serveStubReleaser + reclaimer *serveStubLeaseReclaimer lifecycle *serveStubLifecycle dependencies *serveStubDependencyEditor batchApplier *serveStubBatchApplier @@ -459,6 +473,13 @@ func (s *serveRolesStore) ReadyClaimer() (issueops.ReadyClaimer, error) { return // reaper draining abandoned work does in a tight loop. func (s *serveRolesStore) Releaser() (issueops.Releaser, error) { return s.releaser, nil } +// LeaseReclaimer carries an identifiable value for the releaser's reason, and +// for a sharper form of it: its decorator fires on_update once per reverted +// row, so an unpeeled reclaimer runs the workspace's hooks N times per sweep. +func (s *serveRolesStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return s.reclaimer, nil +} + // IssueLifecycle carries an identifiable value for the same reason the reader // and the claimer do: it is one of the roles the hook decorator wraps, so a peel // of the wrong depth would hand bd serve a lifecycle that runs the workspace's @@ -655,6 +676,12 @@ func (*serveStubReleaser) Release(context.Context, issueops.ReleaseRequest) (iss return issueops.ReleaseResult{}, errors.ErrUnsupported } +type serveStubLeaseReclaimer struct{} + +func (*serveStubLeaseReclaimer) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + return issueops.ReclaimResult{}, errors.ErrUnsupported +} + type serveStubLifecycle struct{} func (*serveStubLifecycle) Create(context.Context, issueops.CreateRequest) (issueops.CreateResult, error) { diff --git a/cmd/bd/serve_store_identity_test.go b/cmd/bd/serve_store_identity_test.go index 322adc4c3..4d3efc3f9 100644 --- a/cmd/bd/serve_store_identity_test.go +++ b/cmd/bd/serve_store_identity_test.go @@ -210,6 +210,13 @@ func (*serveIdentityStore) ReadyClaimer() (issueops.ReadyClaimer, error) { return serveIdentityRole{}, nil } func (*serveIdentityStore) Releaser() (issueops.Releaser, error) { return serveIdentityRole{}, nil } + +// LeaseReclaimer is declared at depth 1 for Releaser's reason: its hook +// decorator wraps, and the recursion would otherwise land on a nil-embedded +// promotion. +func (*serveIdentityStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return serveIdentityRole{}, nil +} func (*serveIdentityStore) IssueLifecycle() (issueops.Lifecycle, error) { return serveIdentityRole{}, nil } @@ -297,6 +304,7 @@ func (*serveIdentityStore) MetadataCAS() (issueops.MetadataCAS, error) { type serveIdentityRole struct { issueops.Lifecycle issueops.Releaser + issueops.LeaseReclaimer issueops.ReadyClaimer issueops.BatchCloser issueops.MetadataCAS diff --git a/cmd/bd/unclaim_proxied_integration_test.go b/cmd/bd/unclaim_proxied_integration_test.go index f7fcdfa06..7a64d548e 100644 --- a/cmd/bd/unclaim_proxied_integration_test.go +++ b/cmd/bd/unclaim_proxied_integration_test.go @@ -174,6 +174,7 @@ func TestProxiedServerReclaim(t *testing.T) { Reclaimed []struct { ID string `json:"id"` PreviousOwner string `json:"previous_owner"` + Revision string `json:"revision"` } `json:"reclaimed"` } if err := json.Unmarshal(out, &got); err != nil { @@ -186,6 +187,11 @@ func TestProxiedServerReclaim(t *testing.T) { if r.PreviousOwner == "" { t.Errorf("expected a previous owner for reclaimed %s", issue.ID) } + // The role mints a new revision per reverted row and reports + // it, on this route exactly as on the direct one. + if _, perr := types.ParseRevisionToken(r.Revision); perr != nil || r.Revision == "" { + t.Errorf("reclaimed %s carries revision %q, want a revision token: %v", issue.ID, r.Revision, perr) + } } } if !found { diff --git a/cmd/bd/unclaim_proxied_server.go b/cmd/bd/unclaim_proxied_server.go index 9269beb88..237d4e099 100644 --- a/cmd/bd/unclaim_proxied_server.go +++ b/cmd/bd/unclaim_proxied_server.go @@ -6,7 +6,6 @@ import ( "fmt" "os" "strings" - "time" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/uow" @@ -122,33 +121,3 @@ func runUnclaimProxiedServer(ctx context.Context, args []string, reason string, } return nil } - -func runReclaimProxiedServer(ctx context.Context, olderThan time.Duration, filter types.ReclaimFilter) error { - if uowProvider == nil { - return HandleError("proxied-server UOW provider not initialized") - } - - reclaimed, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) ([]types.ReclaimedLease, string, error) { - out, rerr := uw.IssueUseCase().ReclaimExpiredLeases(ctx, olderThan, filter, currentActor()) - if rerr != nil { - return nil, "", rerr - } - if len(out) == 0 { - return out, "", nil - } - ids := make([]string, 0, len(out)) - for _, r := range out { - ids = append(ids, r.ID) - } - return out, "bd: reclaim " + strings.Join(ids, ", "), nil - }) - if err != nil { - return HandleErrorRespectJSON("reclaim: %v", err) - } - - if len(reclaimed) > 0 { - commandDidWrite.Store(true) - } - - return renderReclaim(reclaimed, !filter.IsEmpty()) -} diff --git a/internal/httpapi/BUILD.bazel b/internal/httpapi/BUILD.bazel index fb7b25625..27efb854f 100644 --- a/internal/httpapi/BUILD.bazel +++ b/internal/httpapi/BUILD.bazel @@ -29,6 +29,7 @@ go_library( "problem.go", "query.go", "reads.go", + "reclaim.go", "related.go", "release.go", "reopen.go", @@ -93,6 +94,7 @@ go_test( "project_stamp_test.go", "reads_test.go", "ready_count_test.go", + "reclaim_test.go", "related_test.go", "release_test.go", "reopen_test.go", diff --git a/internal/httpapi/apigen/types.gen.go b/internal/httpapi/apigen/types.gen.go index 37e13b5f9..4fea37994 100644 --- a/internal/httpapi/apigen/types.gen.go +++ b/internal/httpapi/apigen/types.gen.go @@ -1047,7 +1047,7 @@ type ContextResponse struct { // OPTIONAL, and absent means only that this server does not disclose its filesystem layout — never that it has no workspace. A client MUST NOT require it, MUST NOT treat absence as an error, and has no use for the value beyond display: it is a path on the SERVER's filesystem, which the client cannot open. Identify the workspace by `project_id` and `database`, which are required. BeadsDir *string `json:"beads_dir,omitempty"` - // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`, and `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than a `400` naming the member unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. + // Capabilities The tokens this server advertises: the OPERATIONS it implements, derived from its route table, and the server-wide BEHAVIORS it enforces. v0's operation vocabulary is `ready.list`, `ready.count`, `issues.list`, `issues.query`, `issues.count`, `issues.get`, `issues.related`, `issues.create`, `issues.addComment`, `issues.batchClose`, `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, `issues.reclaim`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, `dependencies.list`, `dependencies.count`, `dependencies.blocking`, `dependencies.tree`, `dependencies.add`, `dependencies.remove`, `memories.list`, `memories.get`, `memories.remember`, `memories.forget`, `events.list`, `events.watch`, `issues.casMetadata`; the behavior tokens are `project.enforce`, which announces that a `Bd-Project-Id` stamp for the wrong workspace is refused here rather than silently ignored, `issues.batchApplyLarge`, which announces that `POST /v0/beads/issues:batchApply` accepts the RAISED envelope — up to 1000 items (`ApplyBatchRequest.items.maxItems`), a 16 MiB body, and (for a request over 100 items) a run budget that EXTENDS to an operator-configured ceiling (`bd serve --large-apply-ceiling`, 5 minutes by default) — rather than the original 100-item, 4 MiB bound. The ceiling is a flat whole-run budget, not a value scaled by item count: a 101-item request and a 1000-item request both get the same extended budget. A request at or under 100 items is unaffected either way and keeps the server's ordinary deadline exactly as before. A client that checks for this token before sending a plan over 100 items learns whether THIS server accepts it without paying for a round trip an older server would refuse anyway, and `issues.list.sort`, which announces that `GET /v0/beads/issues` accepts the `sort` query parameter (two closed, pageable total orders — see that parameter) rather than silently answering `unknown_parameter`, and `issues.count.scope`, which announces that `GET /v0/beads/issues:count` accepts `parent`, `no_parent`, `exclude_type`, and `exclude_status` (see those parameters) rather than silently answering `unknown_parameter` for all four, `issues.sweep.wispsPlane`, which announces that `POST /v0/beads/issues:sweep` accepts the `wisps-plane` value of `tier` rather than silently answering `invalid_value`, `issues.sweep.liveDependents`, which announces that the same operation accepts `protect_live_dependents` and answers `skipped.live_dependent` rather than silently answering `unknown_parameter`, `issues.sweep.limit`, which announces that it accepts `limit` and answers `remaining` rather than silently answering `unknown_parameter`, and `issues.batchApply.depAddLineage`, which announces that a `dep_add` item on `POST /v0/beads/issues:batchApply` accepts `has_spawner` and `thread_id` (see `ApplyDepAddItem`) rather than a `400` naming the member unknown. The list grows additively, and an operation never appears here unless it is fully implemented. This is how a client checks for an operation or a behavior — never the version string. // // THIS LIST IS BUILD-LEVEL, NOT WORKSPACE-LEVEL. It says which operations this binary serves, and for every entry but two that is the whole answer. `events.list` and `events.watch` are the exceptions: the durable events journal is a per-workspace setting that is OFF by default, so a server that advertises them may still refuse every request to both with 409 `events_journal_disabled` — correctly, because the operations exist and the workspace has no journal. A consumer of either MUST treat the capability as "this server speaks it" and the 409 as "not on this workspace", and must not read the capability as a promise that records will arrive. Capabilities []string `json:"capabilities"` @@ -1733,6 +1733,44 @@ type ReadyPage struct { Items []IssueWithCounts `json:"items"` } +// ReclaimIssuesRequest One sweep of stale leases. Every member but `actor` is optional, and every scope member NARROWS the sweep; see the operation. +// +// `additionalProperties: false`, so an unknown member is a `400` naming the member — a narrowing term this server silently ignored would widen what is reverted. +type ReclaimIssuesRequest struct { + // Actor Who ran the sweep. REQUIRED: each reverted row's recovery event and history entry are attributed to it. Validated by the same rules as the claim's `actor`. + Actor string `json:"actor"` + + // AnyReplica Also revert leases another replica granted. The one member that WIDENS the sweep; see the operation. Absent means `false`. + AnyReplica *bool `json:"any_replica,omitempty"` + + // Assignees Only leases held by one of these owners are eligible. + Assignees *[]string `json:"assignees,omitempty"` + + // ExcludeLabels An issue carrying ANY of these labels is never eligible. + ExcludeLabels *[]string `json:"exclude_labels,omitempty"` + + // Ids Only these issues are eligible. An id that is not currently a stale lease is absent from the answer rather than refused. The cap is on the REQUEST, counted before deduplication. A blank entry is refused. + Ids *[]string `json:"ids,omitempty"` + + // Labels Only issues carrying ALL of these labels are eligible. + Labels *[]string `json:"labels,omitempty"` + + // LabelsAny Only issues carrying AT LEAST ONE of these labels are eligible. + LabelsAny *[]string `json:"labels_any,omitempty"` + + // OlderThanSeconds The grace window past a lease's own expiry, in seconds (fractions allowed): only a lease that expired more than this long ago is eligible. Absent means `0`, every currently expired lease. A negative value is refused. + OlderThanSeconds *float64 `json:"older_than_seconds,omitempty"` +} + +// ReclaimIssuesResult Every row the sweep reverted, in the order the sweep found them. +type ReclaimIssuesResult struct { + // Reclaimed One entry per reverted row. Empty array (never null) when nothing was stale in scope. + Reclaimed []ReclaimedLease `json:"reclaimed"` +} + +// ReclaimedLease One row a lease sweep reverted. +type ReclaimedLease = types.ReclaimedLease + // Ref Names ONE issue, either by an id that already exists or by the `key` a create item earlier in the same request gave itself. // // EXACTLY ONE OF THE TWO IS SET, and both cases the schema cannot express are a `400`: both members set is a caller that cannot say which it meant, and neither set is a reference to nothing. (Spelling that as a schema alternation would need `oneOf`, which this document does not use — see `ApplyItem`.) @@ -2793,6 +2831,9 @@ type ClaimNextIssueJSONRequestBody = ClaimNextRequest // DeleteIssuesJSONRequestBody defines body for DeleteIssues for application/json ContentType. type DeleteIssuesJSONRequestBody = DeleteIssuesRequest +// ReclaimIssuesJSONRequestBody defines body for ReclaimIssues for application/json ContentType. +type ReclaimIssuesJSONRequestBody = ReclaimIssuesRequest + // SweepIssuesJSONRequestBody defines body for SweepIssues for application/json ContentType. type SweepIssuesJSONRequestBody = SweepRequest diff --git a/internal/httpapi/claim.go b/internal/httpapi/claim.go index 24d79093b..5b5db6326 100644 --- a/internal/httpapi/claim.go +++ b/internal/httpapi/claim.go @@ -350,6 +350,7 @@ var ( _ uow.EdgeReaderSource = timedProvider{} _ uow.GraphCounterSource = timedProvider{} _ uow.BatchGetterSource = timedProvider{} + _ uow.LeaseReclaimerSource = timedProvider{} _ uow.RelationsSource = timedProvider{} _ uow.CommenterSource = timedProvider{} _ uow.BlockingAnnotatorSource = timedProvider{} @@ -474,6 +475,12 @@ func (p timedProvider) BatchGetter() (issueops.BatchGetter, error) { return uow.NewBatchGetter(p) } +// LeaseReclaimer builds the stale-lease sweep OVER THIS WRAPPER, for the same +// reason and with the same hazard as IssueReader. +func (p timedProvider) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return uow.NewLeaseReclaimer(p) +} + // IssueRelations builds the single-anchor neighbor role OVER THIS WRAPPER, for // the same reason and with the same hazard as IssueReader. It is the one // accessor here whose name is not the role's: the seam spells it IssueRelations diff --git a/internal/httpapi/pinning.go b/internal/httpapi/pinning.go index 2144038fb..01435a28b 100644 --- a/internal/httpapi/pinning.go +++ b/internal/httpapi/pinning.go @@ -28,6 +28,7 @@ var ( _ apigen.IssueWithCounts = types.IssueWithCounts{} _ apigen.IssueDetails = types.IssueDetails{} _ apigen.BatchGetIssue = types.BatchGetIssue{} + _ apigen.ReclaimedLease = types.ReclaimedLease{} _ apigen.IssueWithDependencyMetadata = types.IssueWithDependencyMetadata{} _ apigen.Dependency = types.Dependency{} _ apigen.TreeNode = types.TreeNode{} @@ -73,6 +74,7 @@ var ( _ []issueops.IssueBlocking = apigen.BlockingAnnotations{}.Items _ []types.Issue = apigen.BatchCreateResponse{}.Items _ []types.BatchGetIssue = apigen.BatchGetIssuesResult{}.Issues + _ []types.ReclaimedLease = apigen.ReclaimIssuesResult{}.Reclaimed _ []eventsjournal.Record = apigen.EventsPage{}.Records ) diff --git a/internal/httpapi/problem.go b/internal/httpapi/problem.go index d3252ed9a..87618d3bd 100644 --- a/internal/httpapi/problem.go +++ b/internal/httpapi/problem.go @@ -466,6 +466,12 @@ const ( // answer EdgeCountRequest.IDs and EdgeReadRequest already give, applied to // the issues themselves rather than their edges. OpBatchGetIssues = "batchGetIssues" + // OpReclaimIssues reverts every stale lease a request scopes back to ready, + // behind issueops.LeaseReclaimer. It is NOT releaseIssue repeated: that + // operation gives up ONE claim on the holder's say-so and refuses an unheld + // row, and this one sweeps MANY leases on a clock's say-so and reports an id + // that was not a stale lease by leaving it out of `reclaimed`. + OpReclaimIssues = "reclaimIssues" // OpListRelatedIssues reads ONE issue's neighbors in a named direction, // behind issueops.Relations. It is NOT listDependencies narrowed to one // anchor: that operation answers the stored edge ROWS with their targets @@ -720,6 +726,11 @@ var operationCodes = map[string][]Code{ // `ids` (*issueops.TooManyIDsError) and a blank entry. An empty `ids` is // legal and answers with an empty result, so it earns no refusal at all. OpBatchGetIssues: {CodeInvalidArgument, CodeUnauthenticated, CodeBusy, CodeDBUnavailable, CodeInternal}, + // No not_found and no conflict: an id that is not a stale lease is left out + // of `reclaimed` rather than refused, and the sweep guards nothing a caller + // named a version of. Its only 400s are the role's own refusals and the + // handler's shape checks. + OpReclaimIssues: {CodeInvalidArgument, CodeUnauthenticated, CodeBusy, CodeDBUnavailable, CodeInternal}, // The same vocabulary as the stored-edge read beside it, and no not_found // for a stronger version of the same reason: this operation probes no id's // existence at all, so there is nothing it could 404 on. diff --git a/internal/httpapi/reclaim.go b/internal/httpapi/reclaim.go new file mode 100644 index 000000000..62f55a108 --- /dev/null +++ b/internal/httpapi/reclaim.go @@ -0,0 +1,249 @@ +package httpapi + +import ( + "encoding/json" + "errors" + "math" + "net/http" + "slices" + "strings" + "time" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/issueops" +) + +// The request body's member vocabulary. The schema is +// additionalProperties: false, so anything else is refused BY NAME, and on this +// operation for the sweep's sharper reason: a scope term the server silently +// ignored would widen what is reverted. +const ( + reclaimActorMember = "actor" + reclaimOlderThanMember = "older_than_seconds" + reclaimIDsMember = "ids" + reclaimAssigneesMember = "assignees" + reclaimLabelsMember = "labels" + reclaimLabelsAnyMember = "labels_any" + reclaimExcludeLabelsMember = "exclude_labels" + reclaimAnyReplicaMember = "any_replica" +) + +// reclaimMembers is the whole vocabulary, in one place, so the unknown-member +// refusal and the decoding below cannot come to disagree about what this +// operation accepts. +var reclaimMembers = []string{ + reclaimActorMember, + reclaimOlderThanMember, + reclaimIDsMember, + reclaimAssigneesMember, + reclaimLabelsMember, + reclaimLabelsAnyMember, + reclaimExcludeLabelsMember, + reclaimAnyReplicaMember, +} + +// maxReclaimOlderThanSeconds is the smallest grace window time.Duration can +// NOT hold: MaxInt64/1e9 as a float64 rounds up, so converting it back yields +// 2^63 nanoseconds, which wraps negative. The check is >=, so every accepted +// value converts in range; a refused one is not wrapped into a negative +// duration the role would refuse with a message about a value never sent. +var maxReclaimOlderThanSeconds = float64(math.MaxInt64) / float64(time.Second) + +// handleReclaimIssues answers POST /v0/beads/issues:reclaim. +// +// WHAT THIS HANDLER DOES NOT DO: it does not decide which lease is stale, does +// not apply the scope, does not honor the replica guard and does not mint the +// revisions. All of that is issueops.LeaseReclaimer, the same library surface +// `bd reclaim` calls, so a second front door cannot sweep differently from the +// first. Everything above the role here is argument validation and projection. +// +// THE ACTOR IS REQUIRED and validated by the claim's rules: every reverted row +// records a recovery event attributed to it, and the server's own identity +// would be meaningless to a remote reaper. +func (s *Server) handleReclaimIssues(w http.ResponseWriter, r *http.Request) { + if !s.requireNoQuery(w, r) { + return + } + if !s.requireJSONContent(w, r) { + return + } + request, ok := s.reclaimRequest(w, r) + if !ok { + return + } + + reclaimer, err := s.leaseReclaimer(r) + if err != nil { + s.failErr(w, r, err) + return + } + result, err := reclaimer.Reclaim(r.Context(), request) + if err != nil { + s.failReclaimErr(w, r, err) + return + } + writeJSON(w, reclaimResponse(result)) +} + +// reclaimRequest decodes the body into the role's request, member by member, +// so every refusal can NAME the member it is about. +func (s *Server) reclaimRequest(w http.ResponseWriter, r *http.Request) (issueops.ReclaimRequest, bool) { + members, res := decodeJSONObjectBody(w, r) + if res != nil { + s.fail(w, r, *res) + return issueops.ReclaimRequest{}, false + } + + var unknown []string + for name := range members { + if !slices.Contains(reclaimMembers, name) { + unknown = append(unknown, name) + } + } + if len(unknown) > 0 { + offender := slices.Min(unknown) + requestInfo(r.Context()).refuse(offender) + s.fail(w, r, InvalidArgument(offender, ReasonUnknownParameter, + "this operation's request body carries "+reclaimMemberList()+" and nothing else")) + return issueops.ReclaimRequest{}, false + } + + var request issueops.ReclaimRequest + + raw, ok := members[reclaimActorMember] + if !ok { + s.fail(w, r, InvalidArgument(reclaimActorMember, ReasonInvalidValue, + "`"+reclaimActorMember+"` is required")) + return issueops.ReclaimRequest{}, false + } + var actor *string + if err := json.Unmarshal(raw, &actor); err != nil || actor == nil { + s.fail(w, r, InvalidArgument(reclaimActorMember, ReasonInvalidValue, + "`"+reclaimActorMember+"` must be a string")) + return issueops.ReclaimRequest{}, false + } + trimmed, res := validateActor(*actor) + if res != nil { + s.fail(w, r, *res) + return issueops.ReclaimRequest{}, false + } + request.Actor = trimmed + + if raw, ok := members[reclaimOlderThanMember]; ok { + var seconds *float64 + if err := json.Unmarshal(raw, &seconds); err != nil || seconds == nil { + s.fail(w, r, InvalidArgument(reclaimOlderThanMember, ReasonInvalidValue, + "`"+reclaimOlderThanMember+"` must be a number")) + return issueops.ReclaimRequest{}, false + } + if *seconds < 0 || *seconds >= maxReclaimOlderThanSeconds { + s.fail(w, r, InvalidArgument(reclaimOlderThanMember, ReasonInvalidValue, + "`"+reclaimOlderThanMember+"` must be a non-negative number of seconds a duration can hold")) + return issueops.ReclaimRequest{}, false + } + request.OlderThan = time.Duration(*seconds * float64(time.Second)) + } + + for _, scope := range []struct { + member string + dest *[]string + }{ + {reclaimIDsMember, &request.Filter.IDs}, + {reclaimAssigneesMember, &request.Filter.Assignees}, + {reclaimLabelsMember, &request.Filter.Labels}, + {reclaimLabelsAnyMember, &request.Filter.LabelsAny}, + {reclaimExcludeLabelsMember, &request.Filter.ExcludeLabels}, + } { + raw, ok := members[scope.member] + if !ok { + continue + } + var values *[]string + if err := json.Unmarshal(raw, &values); err != nil || values == nil { + s.fail(w, r, InvalidArgument(scope.member, ReasonInvalidValue, + "`"+scope.member+"` must be an array of strings")) + return issueops.ReclaimRequest{}, false + } + // A PRESENT, EMPTY scope is refused rather than read as "no scope". + // The two would sweep the same rows, and that is the hazard: `bd + // reclaim --label "$LANE"` with LANE unset refuses for exactly this + // reason, so a caller that meant to narrow and sent nothing gets told + // rather than getting a workspace-wide sweep. + if len(*values) == 0 { + s.fail(w, r, InvalidArgument(scope.member, ReasonInvalidValue, + "`"+scope.member+"` is empty; an empty scope would reclaim everything, so omit the member to sweep without it")) + return issueops.ReclaimRequest{}, false + } + // The id cap and every blank entry are the ROLE's refusals + // (ValidateReclaimRequest, the one definition the CLI shares) and + // reach the wire through failReclaimErr, which names the field the + // role names. + *scope.dest = *values + } + + if raw, ok := members[reclaimAnyReplicaMember]; ok { + var value *bool + if err := json.Unmarshal(raw, &value); err != nil || value == nil { + s.fail(w, r, InvalidArgument(reclaimAnyReplicaMember, ReasonInvalidValue, + "`"+reclaimAnyReplicaMember+"` must be a boolean")) + return issueops.ReclaimRequest{}, false + } + request.Filter.AnyReplica = *value + } + + return request, true +} + +func reclaimMemberList() string { + quoted := make([]string, len(reclaimMembers)) + for i, name := range reclaimMembers { + quoted[i] = "`" + name + "`" + } + return strings.Join(quoted, ", ") +} + +// reclaimFieldMembers spells each field the role's *ReclaimFieldError can +// name as the wire member it arrived in. +var reclaimFieldMembers = map[string]string{ + issueops.ReclaimFieldActor: reclaimActorMember, + issueops.ReclaimFieldOlderThan: reclaimOlderThanMember, + issueops.ReclaimFieldIDs: reclaimIDsMember, + issueops.ReclaimFieldAssignees: reclaimAssigneesMember, + issueops.ReclaimFieldLabels: reclaimLabelsMember, + issueops.ReclaimFieldLabelsAny: reclaimLabelsAnyMember, + issueops.ReclaimFieldExcludeLabels: reclaimExcludeLabelsMember, +} + +// failReclaimErr answers a failed reclaim. issueops.ErrValidation is mapped to +// a 400 HERE, in the sweep's shape, because the role validates what the +// handler does not duplicate. The refusal names the member the role's error +// is about: `ids` for the cap, the *ReclaimFieldError's own field otherwise, +// and no member at all for a validation that names none. +func (s *Server) failReclaimErr(w http.ResponseWriter, r *http.Request, err error) { + if !errors.Is(err, issueops.ErrValidation) { + s.failErr(w, r, err) + return + } + member := "" + var capErr *issueops.TooManyReclaimIDsError + var fieldErr *issueops.ReclaimFieldError + switch { + case errors.As(err, &capErr): + member = reclaimIDsMember + case errors.As(err, &fieldErr): + member = reclaimFieldMembers[fieldErr.Field] + } + if member != "" { + requestInfo(r.Context()).refuse(member) + } + s.fail(w, r, InvalidArgument(member, ReasonInvalidValue, err.Error())) +} + +// reclaimResponse projects the role's result onto the wire envelope. +// apigen.ReclaimedLease IS types.ReclaimedLease (pinning.go), so this is a +// copy rather than a field list; `reclaimed` is never null. +func reclaimResponse(result issueops.ReclaimResult) apigen.ReclaimIssuesResult { + body := apigen.ReclaimIssuesResult{Reclaimed: []apigen.ReclaimedLease{}} + body.Reclaimed = append(body.Reclaimed, result.Reclaimed...) + return body +} diff --git a/internal/httpapi/reclaim_test.go b/internal/httpapi/reclaim_test.go new file mode 100644 index 000000000..4db65aeda --- /dev/null +++ b/internal/httpapi/reclaim_test.go @@ -0,0 +1,317 @@ +package httpapi + +import ( + "encoding/json" + "errors" + "fmt" + "net/http" + "reflect" + "strconv" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// These cover the transport half of POST /v0/beads/issues:reclaim: the body +// shape, the refusals the handler raises itself, how the role's own refusals +// reach the wire, and the projection of the role's result. Everything below +// the wire (staleness, scope, the replica guard, the minted revision, the hook) +// is issueops.LeaseReclaimer's, pinned by +// backend/conformance/lease_reclaimer_contract.go on all four legs. + +const reclaimPath = "/v0/beads/issues:reclaim" + +func newReclaimServer(t *testing.T, reclaimer *roleLeaseReclaimer) *testServer { + t.Helper() + return newTestServer(t, rolesConfig(Config{LeaseReclaimer: reclaimer})) +} + +// TestReclaimIssuesMapsEveryMemberOntoTheRole pins the request half: every +// member reaches the role's request, the actor trimmed by the claim's rules and +// the fractional grace window converted exactly. +func TestReclaimIssuesMapsEveryMemberOntoTheRole(t *testing.T) { + reclaimer := &roleLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{}}} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{ + "actor": " reaper ", + "older_than_seconds": 90.5, + "ids": ["bd-1", "bd-2"], + "assignees": ["w1"], + "labels": ["lane-a"], + "labels_any": ["x", "y"], + "exclude_labels": ["pinned"], + "any_replica": true + }`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + calls := reclaimer.reclaimRequests() + if len(calls) != 1 { + t.Fatalf("role calls = %d, want 1", len(calls)) + } + want := issueops.ReclaimRequest{ + Actor: "reaper", + OlderThan: 90*time.Second + 500*time.Millisecond, + Filter: issueops.ReclaimFilter{ + IDs: []string{"bd-1", "bd-2"}, + Assignees: []string{"w1"}, + Labels: []string{"lane-a"}, + LabelsAny: []string{"x", "y"}, + ExcludeLabels: []string{"pinned"}, + AnyReplica: true, + }, + } + if !reflect.DeepEqual(calls[0], want) { + t.Fatalf("role request = %+v, want %+v", calls[0], want) + } +} + +// TestReclaimIssuesDefaultsToAnUnscopedZeroGraceSweep pins the absent members: +// no grace window is zero, no scope is no scope, and any_replica is off. +func TestReclaimIssuesDefaultsToAnUnscopedZeroGraceSweep(t *testing.T) { + reclaimer := &roleLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{}}} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper"}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + calls := reclaimer.reclaimRequests() + if len(calls) != 1 || !reflect.DeepEqual(calls[0], issueops.ReclaimRequest{Actor: "reaper"}) { + t.Fatalf("role calls = %+v, want one bare request from reaper", calls) + } +} + +// TestReclaimIssuesAnswersEveryReclaimedLeaseWithItsRevision pins the response +// half: each entry carries id, previous_owner and the role's revision as the +// decimal string, past 2^53 so a float64 anywhere on the path would show. +func TestReclaimIssuesAnswersEveryReclaimedLeaseWithItsRevision(t *testing.T) { + reclaimer := &roleLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{ + {ID: "bd-1", PreviousOwner: "w1", Revision: types.RevisionToken(guardToken)}, + {ID: "bd-2", PreviousOwner: "", Revision: "7"}, + }}} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper"}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + raw := readAll(t, resp) + var body struct { + Reclaimed []map[string]any `json:"reclaimed"` + } + if err := json.Unmarshal([]byte(raw), &body); err != nil { + t.Fatalf("decode %q: %v", raw, err) + } + if len(body.Reclaimed) != 2 { + t.Fatalf("reclaimed = %v, want 2 entries", body.Reclaimed) + } + first := body.Reclaimed[0] + if first["id"] != "bd-1" || first["previous_owner"] != "w1" || first["revision"] != types.RevisionToken(guardToken) { + t.Errorf("reclaimed[0] = %v, want {bd-1 w1 %s}", first, types.RevisionToken(guardToken)) + } + if second := body.Reclaimed[1]; second["previous_owner"] != "" || second["revision"] != "7" { + t.Errorf("reclaimed[1] = %v, want an empty previous_owner and revision \"7\"", second) + } +} + +// TestReclaimIssuesAnswersAnEmptyArrayNotNull pins `reclaimed: []` when the +// sweep found nothing, whatever the role handed back. +func TestReclaimIssuesAnswersAnEmptyArrayNotNull(t *testing.T) { + for _, result := range []issueops.ReclaimResult{{}, {Reclaimed: []issueops.ReclaimedLease{}}} { + ts := newReclaimServer(t, &roleLeaseReclaimer{result: result}) + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","ids":["bd-gone"]}`) + if resp.StatusCode != http.StatusOK { + t.Fatalf("status = %d, want 200: %s", resp.StatusCode, readAll(t, resp)) + } + if raw := readAll(t, resp); !strings.Contains(raw, `"reclaimed":[]`) { + t.Errorf("body = %s, want `\"reclaimed\":[]`", raw) + } + } +} + +// TestReclaimIssuesRefusesAMalformedBodyByName pins every refusal the handler +// raises itself: each is a 400 naming its member, and none reaches the role. +func TestReclaimIssuesRefusesAMalformedBodyByName(t *testing.T) { + for _, tc := range []struct { + name, body, param, reason string + }{ + {"unknown member", `{"actor":"r","force":true}`, "force", string(ReasonUnknownParameter)}, + {"missing actor", `{"ids":["bd-1"]}`, "actor", string(ReasonInvalidValue)}, + {"null actor", `{"actor":null}`, "actor", string(ReasonInvalidValue)}, + {"blank actor", `{"actor":" "}`, "actor", ""}, + {"negative grace", `{"actor":"r","older_than_seconds":-1}`, "older_than_seconds", string(ReasonInvalidValue)}, + {"string grace", `{"actor":"r","older_than_seconds":"10m"}`, "older_than_seconds", string(ReasonInvalidValue)}, + {"grace past a duration", `{"actor":"r","older_than_seconds":1e300}`, "older_than_seconds", string(ReasonInvalidValue)}, + {"ids not an array", `{"actor":"r","ids":"bd-1"}`, "ids", string(ReasonInvalidValue)}, + {"empty ids", `{"actor":"r","ids":[]}`, "ids", string(ReasonInvalidValue)}, + {"empty labels", `{"actor":"r","labels":[]}`, "labels", string(ReasonInvalidValue)}, + {"empty assignees", `{"actor":"r","assignees":[]}`, "assignees", string(ReasonInvalidValue)}, + {"null exclude", `{"actor":"r","exclude_labels":null}`, "exclude_labels", string(ReasonInvalidValue)}, + {"any_replica not a bool", `{"actor":"r","any_replica":"yes"}`, "any_replica", string(ReasonInvalidValue)}, + } { + t.Run(tc.name, func(t *testing.T) { + reclaimer := &roleLeaseReclaimer{} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, tc.body) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["code"] != string(CodeInvalidArgument) { + t.Errorf("code = %v, want %q", body["code"], CodeInvalidArgument) + } + if body["param"] != tc.param { + t.Errorf("param = %v, want %s", body["param"], tc.param) + } + if tc.reason != "" && body["reason"] != tc.reason { + t.Errorf("reason = %v, want %s", body["reason"], tc.reason) + } + if calls := reclaimer.reclaimRequests(); len(calls) != 0 { + t.Errorf("the role was called %d times for a refused request", len(calls)) + } + }) + } +} + +// TestReclaimIssuesRefusesAQueryParameter pins the document-level +// unknown-parameter rule on this operation, which declares none. +func TestReclaimIssuesRefusesAQueryParameter(t *testing.T) { + reclaimer := &roleLeaseReclaimer{} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath+"?older_than=1h", `{"actor":"reaper"}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["param"] != "older_than" || body["reason"] != string(ReasonUnknownParameter) { + t.Errorf("param/reason = %v/%v, want older_than/unknown_parameter", body["param"], body["reason"]) + } + if calls := reclaimer.reclaimRequests(); len(calls) != 0 { + t.Errorf("the role was called %d times for a refused request", len(calls)) + } +} + +// TestReclaimIssuesLeavesTheIDsCapToTheRole pins that the handler neither +// counts nor trims `ids`: the whole oversized request reaches the role, and +// the role's *TooManyReclaimIDsError comes back as a 400 naming `ids`. +func TestReclaimIssuesLeavesTheIDsCapToTheRole(t *testing.T) { + const over = issueops.MaxReclaimIDs + 1 + reclaimer := &roleLeaseReclaimer{err: &issueops.TooManyReclaimIDsError{Requested: over, Cap: issueops.MaxReclaimIDs}} + ts := newReclaimServer(t, reclaimer) + + ids := make([]string, over) + for i := range ids { + ids[i] = fmt.Sprintf(`"bd-%d"`, i) + } + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","ids":[`+strings.Join(ids, ",")+`]}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + body := decodeBody(t, resp) + if body["code"] != string(CodeInvalidArgument) || body["param"] != "ids" || body["reason"] != string(ReasonInvalidValue) { + t.Errorf("problem = %v, want invalid_argument/ids/invalid_value", body) + } + calls := reclaimer.reclaimRequests() + if len(calls) != 1 || len(calls[0].Filter.IDs) != over { + t.Fatalf("role calls = %d, want one carrying all %d ids", len(calls), over) + } +} + +// TestReclaimIssuesAnswersABlankIDThroughTheRole pins the other role-side +// refusal: a blank entry in `ids` reaches the role (it is not pre-checked +// here), and its ErrValidation is a 400 naming `ids`. +func TestReclaimIssuesAnswersABlankIDThroughTheRole(t *testing.T) { + reclaimer := &roleLeaseReclaimer{err: &issueops.ReclaimFieldError{ + Field: issueops.ReclaimFieldIDs, Detail: "validation failed: reclaim scope id at position 1 is empty", + }} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","ids":["bd-1",""]}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + if body := decodeBody(t, resp); body["param"] != "ids" { + t.Errorf("param = %v, want ids", body["param"]) + } + if calls := reclaimer.reclaimRequests(); len(calls) != 1 { + t.Fatalf("role calls = %d, want 1", len(calls)) + } +} + +// TestReclaimIssuesNamesTheFieldTheRoleRefused pins that a role refusal is +// attributed to the member the role's *ReclaimFieldError names, not to `ids` +// by default, and that the handler leaves blank scope entries to the role: +// the request reaches it. +func TestReclaimIssuesNamesTheFieldTheRoleRefused(t *testing.T) { + for field, member := range reclaimFieldMembers { + t.Run(field, func(t *testing.T) { + reclaimer := &roleLeaseReclaimer{err: &issueops.ReclaimFieldError{Field: field, Detail: "validation failed: refused"}} + ts := newReclaimServer(t, reclaimer) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","labels_any":["a"," "]}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("status = %d, want 400: %s", resp.StatusCode, readAll(t, resp)) + } + if body := decodeBody(t, resp); body["param"] != member { + t.Errorf("param = %v, want %s", body["param"], member) + } + calls := reclaimer.reclaimRequests() + if len(calls) != 1 || !reflect.DeepEqual(calls[0].Filter.LabelsAny, []string{"a", " "}) { + t.Fatalf("role calls = %+v, want one carrying the blank entry unfiltered", calls) + } + }) + } +} + +// TestReclaimFieldMembersCoverEveryRoleField pins the attribution table +// against the role's whole field vocabulary, so a field added to the role +// cannot reach the wire with no member named. +func TestReclaimFieldMembersCoverEveryRoleField(t *testing.T) { + for _, field := range []string{ + issueops.ReclaimFieldActor, issueops.ReclaimFieldOlderThan, issueops.ReclaimFieldIDs, + issueops.ReclaimFieldAssignees, issueops.ReclaimFieldLabels, issueops.ReclaimFieldLabelsAny, + issueops.ReclaimFieldExcludeLabels, + } { + if reclaimFieldMembers[field] == "" { + t.Errorf("role field %q has no wire member", field) + } + } +} + +// TestReclaimIssuesRefusesTheFirstUnrepresentableGraceWindow pins the +// boundary: float64(MaxInt64)/1e9 converts back to 2^63 nanoseconds, which +// wraps negative, so that exact value is refused, not passed on. +func TestReclaimIssuesRefusesTheFirstUnrepresentableGraceWindow(t *testing.T) { + reclaimer := &roleLeaseReclaimer{} + ts := newReclaimServer(t, reclaimer) + + value := strconv.FormatFloat(maxReclaimOlderThanSeconds, 'g', -1, 64) + resp := ts.claim(t, reclaimPath, `{"actor":"reaper","older_than_seconds":`+value+`}`) + if resp.StatusCode != http.StatusBadRequest { + t.Fatalf("older_than_seconds=%s: status = %d, want 400: %s", value, resp.StatusCode, readAll(t, resp)) + } + if body := decodeBody(t, resp); body["param"] != "older_than_seconds" { + t.Errorf("param = %v, want older_than_seconds", body["param"]) + } + if calls := reclaimer.reclaimRequests(); len(calls) != 0 { + t.Fatalf("the role received %+v; an out-of-range window must not reach it", calls) + } +} + +// TestReclaimIssuesDoesNotDisguiseAStorageFailure pins that a failure that is +// not a validation is not answered as the caller's fault. +func TestReclaimIssuesDoesNotDisguiseAStorageFailure(t *testing.T) { + ts := newReclaimServer(t, &roleLeaseReclaimer{err: errors.New("disk on fire")}) + + resp := ts.claim(t, reclaimPath, `{"actor":"reaper"}`) + if resp.StatusCode < 500 { + t.Fatalf("status = %d, want a 5xx: %s", resp.StatusCode, readAll(t, resp)) + } +} diff --git a/internal/httpapi/roles_test.go b/internal/httpapi/roles_test.go index 083923e9c..6ada7952d 100644 --- a/internal/httpapi/roles_test.go +++ b/internal/httpapi/roles_test.go @@ -162,6 +162,34 @@ func (g *roleBatchGetter) getManyRequests() []issueops.GetManyRequest { return append([]issueops.GetManyRequest(nil), g.calls...) } +// roleLeaseReclaimer is the stale-lease sweep of the store-shaped source, its +// own fake beside roleReleaser for roleBatchGetter's reason: a separate role on +// a separate accessor, so one fake answering both would let a test pass on a +// server that had wired the reclaim handler to the wrong role. +type roleLeaseReclaimer struct { + result issueops.ReclaimResult + err error + + mu sync.Mutex + calls []issueops.ReclaimRequest +} + +func (l *roleLeaseReclaimer) Reclaim(_ context.Context, req issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + l.mu.Lock() + l.calls = append(l.calls, req) + l.mu.Unlock() + if l.err != nil { + return issueops.ReclaimResult{}, l.err + } + return l.result, nil +} + +func (l *roleLeaseReclaimer) reclaimRequests() []issueops.ReclaimRequest { + l.mu.Lock() + defer l.mu.Unlock() + return append([]issueops.ReclaimRequest(nil), l.calls...) +} + // roleRelations is the single-anchor NEIGHBOR role of the store-shaped source, // its own fake beside roleEdgeReader for roleGraphCounter's reason: the two sit // on adjacent accessors and answer about the same edges, so one fake serving @@ -960,6 +988,9 @@ func rolesConfig(cfg Config) Config { if cfg.BatchGetter == nil { cfg.BatchGetter = &roleBatchGetter{} } + if cfg.LeaseReclaimer == nil { + cfg.LeaseReclaimer = &roleLeaseReclaimer{} + } if cfg.Relations == nil { cfg.Relations = &roleRelations{} } diff --git a/internal/httpapi/routes.go b/internal/httpapi/routes.go index 4c426275a..1dc78fa94 100644 --- a/internal/httpapi/routes.go +++ b/internal/httpapi/routes.go @@ -783,6 +783,17 @@ var routeTable = []route{ implemented: true, handler: (*Server).handleBatchGetIssues, }, + { + op: OpReclaimIssues, + method: http.MethodPost, + // A literal collection-level custom method, registered and preferred + // over the claim's wildcard for the sweep row's reason: without it, + // a reclaim would parse as a claim of an issue called ":reclaim". + pattern: "/v0/beads/issues:reclaim", + capability: "issues.reclaim", + implemented: true, + handler: (*Server).handleReclaimIssues, + }, { op: OpAddDependencies, method: http.MethodPost, diff --git a/internal/httpapi/server.go b/internal/httpapi/server.go index a3b928390..3a27b520a 100644 --- a/internal/httpapi/server.go +++ b/internal/httpapi/server.go @@ -287,6 +287,12 @@ type Config struct { // rather than refusing the call. Required on the same terms as every field // here. BatchGetter issueops.BatchGetter + // LeaseReclaimer is the stale-lease sweep behind + // POST /v0/beads/issues:reclaim. It is a SEPARATE field from Releaser for + // the reason issueops.LeaseReclaimer's own doc gives: Releaser gives up ONE + // claim on its holder's say-so, and this reverts MANY on a clock's. + // Required on the same terms as every field here. + LeaseReclaimer issueops.LeaseReclaimer // Relations is the single-anchor neighbor read behind // GET /v0/beads/issues/{id}/related. It is a SEPARATE field from EdgeReader // for the reason issueops.EdgeReader's own doc gives at length: that role @@ -423,6 +429,7 @@ type Server struct { issueEdges issueops.EdgeReader issueEdgeCounter issueops.GraphCounter issueBatchGetter issueops.BatchGetter + issueReclaimer issueops.LeaseReclaimer issueRelations issueops.Relations issueCommenter issueops.Commenter issueBlocking issueops.BlockingAnnotator @@ -635,6 +642,7 @@ func Listen(cfg Config) (*Server, error) { issueEdges: cfg.EdgeReader, issueEdgeCounter: cfg.GraphCounter, issueBatchGetter: cfg.BatchGetter, + issueReclaimer: cfg.LeaseReclaimer, issueRelations: cfg.Relations, issueCommenter: cfg.Commenter, issueBlocking: cfg.BlockingAnnotator, @@ -757,12 +765,12 @@ func Listen(cfg Config) (*Server, error) { // "all or nothing" would turn an honest condition into a special case inside // three functions. It is checked once, on its own, below. func sourceRoles(cfg Config) []any { - return []any{cfg.Reader, cfg.Claimer, cfg.ReadyClaimer, cfg.Releaser, cfg.Lifecycle, cfg.BatchCloser, cfg.Settings, cfg.Stats, cfg.CycleDetector, cfg.EdgeReader, cfg.GraphCounter, cfg.BatchGetter, cfg.Relations, cfg.Commenter, cfg.BlockingAnnotator, cfg.TreeWalker, cfg.ReadyCounter, cfg.Counter, cfg.Querier, cfg.Sweeper, cfg.Deleter, cfg.BatchCreator, cfg.DependencyEditor, cfg.BatchApplier, cfg.Memories, cfg.MetadataCAS} + return []any{cfg.Reader, cfg.Claimer, cfg.ReadyClaimer, cfg.Releaser, cfg.Lifecycle, cfg.BatchCloser, cfg.Settings, cfg.Stats, cfg.CycleDetector, cfg.EdgeReader, cfg.GraphCounter, cfg.BatchGetter, cfg.LeaseReclaimer, cfg.Relations, cfg.Commenter, cfg.BlockingAnnotator, cfg.TreeWalker, cfg.ReadyCounter, cfg.Counter, cfg.Querier, cfg.Sweeper, cfg.Deleter, cfg.BatchCreator, cfg.DependencyEditor, cfg.BatchApplier, cfg.Memories, cfg.MetadataCAS} } // roleSourceNames spells sourceRoles for the refusal message, in the same // order, so a caller reading the error learns the whole set it must pass. -const roleSourceNames = "Reader, Claimer, ReadyClaimer, Releaser, Lifecycle, BatchCloser, Settings, Stats, CycleDetector, EdgeReader, GraphCounter, BatchGetter, Relations, Commenter, BlockingAnnotator, TreeWalker, ReadyCounter, Counter, Querier, Sweeper, Deleter, BatchCreator, DependencyEditor, BatchApplier, Memories and MetadataCAS" +const roleSourceNames = "Reader, Claimer, ReadyClaimer, Releaser, Lifecycle, BatchCloser, Settings, Stats, CycleDetector, EdgeReader, GraphCounter, BatchGetter, LeaseReclaimer, Relations, Commenter, BlockingAnnotator, TreeWalker, ReadyCounter, Counter, Querier, Sweeper, Deleter, BatchCreator, DependencyEditor, BatchApplier, Memories and MetadataCAS" func anyRoleSet(cfg Config) bool { return slices.ContainsFunc(sourceRoles(cfg), func(r any) bool { return r != nil }) @@ -1151,6 +1159,19 @@ func (s *Server) batchGetter(r *http.Request) (issueops.BatchGetter, error) { return src.BatchGetter() } +// leaseReclaimer returns the stale-lease sweep surface for one request, on the +// same terms as every role above and held by INTERFACE so +// uow.LeaseReclaimerSource is load-bearing rather than decorative. It goes out +// unwrapped for the sweeper's reason: ReclaimResult is a VALUE whose slice a +// nil-safe range walks. +func (s *Server) leaseReclaimer(r *http.Request) (issueops.LeaseReclaimer, error) { + if s.provider == nil { + return s.issueReclaimer, nil + } + var src uow.LeaseReclaimerSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())} + return src.LeaseReclaimer() +} + // relations returns the single-anchor neighbor surface for one request, built // the same two ways as edgeReader above and held by INTERFACE so // uow.RelationsSource is load-bearing rather than decorative. diff --git a/internal/httpapi/spec/openapi.v0.yaml b/internal/httpapi/spec/openapi.v0.yaml index 456693a41..0c087a175 100644 --- a/internal/httpapi/spec/openapi.v0.yaml +++ b/internal/httpapi/spec/openapi.v0.yaml @@ -63,7 +63,7 @@ info: outside its own parameter table with `400` / `code: invalid_argument` / `reason: "unknown_parameter"` and `param` naming the offending key. Operations that declare no QUERY parameter at all reject every query key - the same way. Today that is twenty-six of the forty-two operations + the same way. Today that is twenty-seven of the forty-three operations here — `GET /healthz`, `GET /v0/beads/context`, `GET /v0/beads/dependencies/cycles`, `POST /v0/beads/issues`, `PATCH /v0/beads/issues/{id}`, @@ -75,6 +75,7 @@ info: `POST /v0/beads/issues/{id}:casMetadata`, `POST /v0/beads/issues:sweep`, `POST /v0/beads/issues:delete`, `POST /v0/beads/issues:batchGet`, + `POST /v0/beads/issues:reclaim`, `POST /v0/beads/issues:batchCreate`, `POST /v0/beads/issues:batchApply`, `POST /v0/beads/issues:batchClose`, `GET /v0/beads/config`, @@ -3891,6 +3892,95 @@ paths: '503': $ref: '#/components/responses/Unavailable' + /v0/beads/issues:reclaim: + post: + operationId: reclaimIssues + summary: Revert stale leases back to ready + description: >- + Reverts every in-progress issue whose LEASE has gone stale back to + ready: the lease row is deleted, `status` becomes `open`, the assignee + and `started_at` are cleared, and a recovery event naming the former + holder is recorded. It is the operation behind `issueops.LeaseReclaimer` + and `bd reclaim`, and the caller it is for is a CLOCK — a supervisor + reaping the work of workers that died without releasing it — rather + than the worker that held the claim, which releases its own with + `POST /v0/beads/issues/{id}:release`. + + + A lease is stale when its expiry is more than `older_than_seconds` in + the past. A lease a worker is still heartbeating is never eligible, + whatever the request names. + + + ## Scope narrows, never widens + + + Every scope member narrows the sweep and they AND-combine: `ids`, + `assignees`, `labels` (all of them), `labels_any` (at least one) and + `exclude_labels` (none of them). A request with no scope member sweeps + every stale lease this server granted. `any_replica` is the one member + that WIDENS: it disarms the granting-replica guard, so a lease another + replica granted is reverted too. It is an operator escape hatch for a + replica that is gone for good, never a normal setting. + + + AN ID THAT IS NOT A STALE LEASE IS NOT AN ERROR. An id naming no row, or + a row whose lease is live, or a row nobody holds, is simply absent from + `reclaimed`; scoping by id narrows a sweep, it does not assert that + every named id was reclaimable. There is no `404` on this operation. + + + ## A new revision per reverted row + + + Every reverted row's `revision` is minted fresh by the revert, and each + entry of `reclaimed` carries it. A holder that wakes up after its lease + was reclaimed and writes with the revision its claim handed it gets + `412 precondition_failed`, so it cannot land over work another worker + may already have taken. + + + The whole sweep is ONE transaction. A lease a heartbeat rescues between + the snapshot that found it and its row's revert is skipped, not + reported. + + + Available when the server advertises the `issues.reclaim` capability. + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/ReclaimIssuesRequest' + security: + - bearerToken: [] + responses: + '200': + description: >- + The rows the sweep reverted, each with its former holder and its new + revision. An empty array (never null) when nothing was stale. + content: + application/json: + schema: + $ref: '#/components/schemas/ReclaimIssuesResult' + '400': + description: >- + Invalid request: an unparseable body, an unknown body member, a + missing or invalid `actor`, a negative or non-numeric + `older_than_seconds`, an `ids` carrying more than 1000 entries + (counted before deduplication), or a blank scope entry. + x-bd-codes: [invalid_argument] + content: + application/problem+json: + schema: + $ref: '#/components/schemas/Problem' + '401': + $ref: '#/components/responses/Unauthenticated' + '500': + $ref: '#/components/responses/InternalError' + '503': + $ref: '#/components/responses/Unavailable' + /v0/beads/issues:batchCreate: post: operationId: batchCreateIssues @@ -7930,7 +8020,7 @@ components: `issues.claim`, `issues.claimNext`, `issues.release`, `issues.close`, `issues.reopen`, `issues.update`, `issues.sweep`, `issues.delete`, `issues.batchGet`, - `issues.batchCreate`, + `issues.reclaim`, `issues.batchCreate`, `issues.batchApply`, `stats.get`, `config.list`, `config.get`, `config.set`, `config.unset`, `dependencies.cycles`, @@ -8518,6 +8608,108 @@ components: items: type: string + ReclaimIssuesRequest: + type: object + additionalProperties: false + required: [actor] + description: >- + One sweep of stale leases. Every member but `actor` is optional, and + every scope member NARROWS the sweep; see the operation. + + + `additionalProperties: false`, so an unknown member is a `400` naming + the member — a narrowing term this server silently ignored would widen + what is reverted. + properties: + actor: + type: string + description: >- + Who ran the sweep. REQUIRED: each reverted row's recovery event and + history entry are attributed to it. Validated by the same rules as + the claim's `actor`. + older_than_seconds: + type: number + format: double + minimum: 0 + description: >- + The grace window past a lease's own expiry, in seconds (fractions + allowed): only a lease that expired more than this long ago is + eligible. Absent means `0`, every currently expired lease. A + negative value is refused. + ids: + type: array + maxItems: 1000 + description: >- + Only these issues are eligible. An id that is not currently a stale + lease is absent from the answer rather than refused. The cap is on + the REQUEST, counted before deduplication. A blank entry is + refused. + items: + type: string + assignees: + type: array + description: Only leases held by one of these owners are eligible. + items: + type: string + labels: + type: array + description: Only issues carrying ALL of these labels are eligible. + items: + type: string + labels_any: + type: array + description: Only issues carrying AT LEAST ONE of these labels are eligible. + items: + type: string + exclude_labels: + type: array + description: An issue carrying ANY of these labels is never eligible. + items: + type: string + any_replica: + type: boolean + description: >- + Also revert leases another replica granted. The one member that + WIDENS the sweep; see the operation. Absent means `false`. + + ReclaimIssuesResult: + type: object + required: [reclaimed] + description: >- + Every row the sweep reverted, in the order the sweep found them. + properties: + reclaimed: + type: array + description: >- + One entry per reverted row. Empty array (never null) when nothing + was stale in scope. + items: + $ref: '#/components/schemas/ReclaimedLease' + + ReclaimedLease: + type: object + description: One row a lease sweep reverted. + x-go-type: types.ReclaimedLease + x-go-type-import: + name: types + path: github.com/steveyegge/beads/internal/types + required: [id, previous_owner, revision] + properties: + id: + type: string + description: The reverted issue's id. + previous_owner: + type: string + description: >- + The assignee the stale lease was taken from; empty when the row + carried none. + revision: + type: string + description: >- + The row's NEW revision, minted by the revert: the + optimistic-concurrency token a guarded write after the reclaim + needs. A decimal string, like every `revision` on this surface. + SweepRequest: type: object additionalProperties: false diff --git a/internal/httpapi/spec_parity_test.go b/internal/httpapi/spec_parity_test.go index 3d9fb664d..3f7451ac1 100644 --- a/internal/httpapi/spec_parity_test.go +++ b/internal/httpapi/spec_parity_test.go @@ -420,11 +420,11 @@ func TestSpecDefaultsMatchSharedConstants(t *testing.T) { } } - // The id bound on the two request bodies that carry an `ids` list, for the - // anchor bound's reason: each is enforced in code — maxDeleteIDs in the - // delete handler, issueops.MaxGetManyIDs in the role behind batchGet — and - // restated as the member's maxItems, which is where a generated client - // learns it. Each body is reached through its operation's own $ref, so a + // The id bound on the three request bodies that carry an `ids` list, for + // the anchor bound's reason: each is enforced in code — maxDeleteIDs in the + // delete handler, issueops.MaxGetManyIDs in the role behind batchGet and + // issueops.MaxReclaimIDs in the role behind reclaim — and restated as the + // member's maxItems, which is where a generated client learns it. Each body is reached through its operation's own $ref, so a // row cannot go on checking a schema its operation stopped using. for _, tc := range []struct { opID string @@ -432,6 +432,7 @@ func TestSpecDefaultsMatchSharedConstants(t *testing.T) { }{ {OpBatchGetIssues, issueops.MaxGetManyIDs}, {OpDeleteIssues, maxDeleteIDs}, + {OpReclaimIssues, issueops.MaxReclaimIDs}, } { so, ok := ops[tc.opID] if !ok { diff --git a/internal/httpapi/wireshape/testdata/golden.json b/internal/httpapi/wireshape/testdata/golden.json index 1bff0651a..a9c50dec3 100644 --- a/internal/httpapi/wireshape/testdata/golden.json +++ b/internal/httpapi/wireshape/testdata/golden.json @@ -3894,6 +3894,85 @@ "required": true, "item_type": "object" }, + { + "schema": "ReclaimIssuesRequest", + "member": "actor", + "type": "string", + "required": true + }, + { + "schema": "ReclaimIssuesRequest", + "member": "any_replica", + "type": "boolean", + "required": false + }, + { + "schema": "ReclaimIssuesRequest", + "member": "assignees", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "exclude_labels", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "ids", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "labels", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "labels_any", + "type": "array", + "required": false, + "item_type": "string" + }, + { + "schema": "ReclaimIssuesRequest", + "member": "older_than_seconds", + "type": "number", + "format": "double", + "required": false + }, + { + "schema": "ReclaimIssuesResult", + "member": "reclaimed", + "type": "array", + "required": true, + "item_type": "object" + }, + { + "schema": "ReclaimedLease", + "member": "id", + "type": "string", + "required": true + }, + { + "schema": "ReclaimedLease", + "member": "previous_owner", + "type": "string", + "required": true + }, + { + "schema": "ReclaimedLease", + "member": "revision", + "type": "string", + "required": true + }, { "schema": "Ref", "member": "id", @@ -6125,6 +6204,14 @@ "QueryPage": "response", "ReadyCount": "response", "ReadyPage": "response", + "ReclaimIssuesRequest": "request", + "ReclaimIssuesRequest.assignees[]": "request", + "ReclaimIssuesRequest.exclude_labels[]": "request", + "ReclaimIssuesRequest.ids[]": "request", + "ReclaimIssuesRequest.labels[]": "request", + "ReclaimIssuesRequest.labels_any[]": "request", + "ReclaimIssuesResult": "response", + "ReclaimedLease": "response", "Ref": "request", "RelatedIssues": "response", "ReleaseIssueRequest": "request", diff --git a/internal/httpclient/BUILD.bazel b/internal/httpclient/BUILD.bazel index 8e8bc8be3..1f4e4a6ad 100644 --- a/internal/httpclient/BUILD.bazel +++ b/internal/httpclient/BUILD.bazel @@ -25,6 +25,7 @@ go_library( "dispatch.go", "errors.go", "journal.go", + "leasereclaimer.go", "lifecycle.go", "list_walk.go", "memories.go", @@ -135,6 +136,7 @@ go_test( "served_hooks_test.go", "served_issue_operations_test.go", "served_journal_test.go", + "served_lease_reclaimer_test.go", "served_lifecycle_create_test.go", "served_list_test.go", "served_memories_deps_test.go", diff --git a/internal/httpclient/accessors.go b/internal/httpclient/accessors.go index ee635ebd9..8a6298155 100644 --- a/internal/httpclient/accessors.go +++ b/internal/httpclient/accessors.go @@ -384,3 +384,12 @@ func (s *Store) BatchApplier() (issueops.BatchApplier, error) { func (s *Store) BatchGetter() (issueops.BatchGetter, error) { return &httpBatchGetter{store: s}, nil } + +// LeaseReclaimer serves reclaimIssues (POST /v0/beads/issues:reclaim) — the +// stale-lease sweep behind `bd reclaim`. Like BatchGetter it dials through the +// store directly rather than through roleWire: its request is one body with no +// path id, and Store.dispatch already preflights the capability and never +// retries a POST. See leasereclaimer.go. +func (s *Store) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return &httpLeaseReclaimer{store: s}, nil +} diff --git a/internal/httpclient/encode/oracle_test.go b/internal/httpclient/encode/oracle_test.go index d127a83c4..f65bb152a 100644 --- a/internal/httpclient/encode/oracle_test.go +++ b/internal/httpclient/encode/oracle_test.go @@ -71,6 +71,7 @@ func startOracle(t *testing.T) *oracle { EdgeReader: stubEdges{}, GraphCounter: stubGraphCounter{}, BatchGetter: stubBatchGetter{}, + LeaseReclaimer: stubLeaseReclaimer{}, Relations: stubRelations{}, Commenter: stubCommenter{}, BlockingAnnotator: stubBlocking{}, @@ -270,6 +271,7 @@ type ( stubEdges struct{ issueops.EdgeReader } stubGraphCounter struct{ issueops.GraphCounter } stubBatchGetter struct{ issueops.BatchGetter } + stubLeaseReclaimer struct{ issueops.LeaseReclaimer } stubRelations struct{ issueops.Relations } stubCommenter struct{ issueops.Commenter } stubBlocking struct{ issueops.BlockingAnnotator } diff --git a/internal/httpclient/leasereclaimer.go b/internal/httpclient/leasereclaimer.go new file mode 100644 index 000000000..5630fb351 --- /dev/null +++ b/internal/httpclient/leasereclaimer.go @@ -0,0 +1,135 @@ +package httpclient + +import ( + "context" + "net/http" + "strings" + + "github.com/steveyegge/beads/internal/httpapi/apigen" + "github.com/steveyegge/beads/internal/httpclient/wire" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// httpLeaseReclaimer is issueops.LeaseReclaimer over reclaimIssues (POST +// /v0/beads/issues:reclaim). +// +// THE MAPPING IS TOTAL: every member of ReclaimRequest has a wire member, so +// nothing here refuses on shape and the ledger carries no row for this +// operation. The grace window travels as fractional seconds (a float64), which +// is exact for the whole-second and millisecond windows a reaper uses; a +// nanosecond-precise window may round. +// +// THE REQUEST RULES ARE CHECKED BEFORE THE DIAL, in the order +// internal/storage/issueops.ValidateReclaimRequest applies them, for +// checkGetManyIDs' reason: the server maps every role validation onto one +// generic invalid_value problem, so a round trip could never hand this client +// back the exact *issueops.TooManyReclaimIDsError{Requested, Cap} the role's +// contract promises. The rules are restated here rather than imported, because +// that package is the storage engine's shared body and does not belong in a +// transport client. +// +// AN EMPTY SCOPE SLICE IS NOT SENT. The server refuses a PRESENT, empty scope +// member (it would read as "no scope", the widening hazard bd reclaim refuses +// on the CLI), and on the role an empty slice and a nil one are the same +// request: no scope. Omitting it says exactly that. +type httpLeaseReclaimer struct{ store *Store } + +var _ issueops.LeaseReclaimer = (*httpLeaseReclaimer)(nil) + +// Reclaim dials POST /v0/beads/issues:reclaim. +func (l *httpLeaseReclaimer) Reclaim(ctx context.Context, req issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + if err := validateReclaimRequest(req); err != nil { + return issueops.ReclaimResult{}, err + } + + body := apigen.ReclaimIssuesRequest{Actor: req.Actor} + if req.OlderThan > 0 { + seconds := req.OlderThan.Seconds() + body.OlderThanSeconds = &seconds + } + scope := func(values []string) *[]string { + if len(values) == 0 { + return nil + } + // COPIED, never aliased: the role promises never to write through a + // caller's request. + copied := append([]string(nil), values...) + return &copied + } + body.Ids = scope(req.Filter.IDs) + body.Assignees = scope(req.Filter.Assignees) + body.Labels = scope(req.Filter.Labels) + body.LabelsAny = scope(req.Filter.LabelsAny) + body.ExcludeLabels = scope(req.Filter.ExcludeLabels) + if req.Filter.AnyReplica { + anyReplica := true + body.AnyReplica = &anyReplica + } + + var out apigen.ReclaimIssuesResult + if err := l.store.dispatch(ctx, wire.Request{ + Op: wire.OpReclaimIssues, + Method: http.MethodPost, + Path: wire.PathIssuesReclaim, + Body: body, + }, &out); err != nil { + return issueops.ReclaimResult{}, err + } + + reclaimed := make([]issueops.ReclaimedLease, 0, len(out.Reclaimed)) + for _, entry := range out.Reclaimed { + // The revision is carried as the wire's decimal string, the role's own + // spelling; it is read here only to refuse an answer whose token no + // guarded write could use. + if _, err := parseRevision("LeaseReclaimer.Reclaim", entry.Revision); err != nil { + return issueops.ReclaimResult{}, err + } + reclaimed = append(reclaimed, entry) + } + return issueops.ReclaimResult{Reclaimed: reclaimed}, nil +} + +// validateReclaimRequest restates the role's refusals, in the role's order and +// with the role's types: the actor (present, then within its column once +// trimmed), the grace window, the id cap (counted as sent), the first blank +// id, then the first blank entry of the other scopes. +func validateReclaimRequest(req issueops.ReclaimRequest) error { + field := func(name, format string, args ...any) error { + return &issueops.ReclaimFieldError{Field: name, Detail: invalid(format, args...).Error()} + } + actor := strings.TrimSpace(req.Actor) + if actor == "" { + return field(issueops.ReclaimFieldActor, "reclaim actor is required") + } + if err := types.CheckFieldLen("actor", actor); err != nil { + return field(issueops.ReclaimFieldActor, "%v", err) + } + if req.OlderThan < 0 { + return field(issueops.ReclaimFieldOlderThan, "reclaim older_than must not be negative") + } + if len(req.Filter.IDs) > issueops.MaxReclaimIDs { + return &issueops.TooManyReclaimIDsError{Requested: len(req.Filter.IDs), Cap: issueops.MaxReclaimIDs} + } + for i, id := range req.Filter.IDs { + if id == "" { + return field(issueops.ReclaimFieldIDs, "reclaim scope id at position %d is empty", i) + } + } + for _, scope := range []struct { + name string + values []string + }{ + {issueops.ReclaimFieldAssignees, req.Filter.Assignees}, + {issueops.ReclaimFieldLabels, req.Filter.Labels}, + {issueops.ReclaimFieldLabelsAny, req.Filter.LabelsAny}, + {issueops.ReclaimFieldExcludeLabels, req.Filter.ExcludeLabels}, + } { + for i, v := range scope.values { + if strings.TrimSpace(v) == "" { + return field(scope.name, "reclaim %s entry at position %d is blank", scope.name, i) + } + } + } + return nil +} diff --git a/internal/httpclient/served_harness_test.go b/internal/httpclient/served_harness_test.go index 917335a2a..8fe00ca67 100644 --- a/internal/httpclient/served_harness_test.go +++ b/internal/httpclient/served_harness_test.go @@ -297,6 +297,8 @@ func serveConfig(t *testing.T, s storage.DoltStorage) httpapi.Config { fail("Memories") cfg.BatchGetter, err = s.BatchGetter() fail("BatchGetter") + cfg.LeaseReclaimer, err = s.LeaseReclaimer() + fail("LeaseReclaimer") // THE JOURNAL IS NOT AN ACCESSOR, alone among the roles above: it is reached // by TYPE ASSERTION, because it is not on storage.DoltStorage and a backend diff --git a/internal/httpclient/served_lease_reclaimer_test.go b/internal/httpclient/served_lease_reclaimer_test.go new file mode 100644 index 000000000..8b23af9e9 --- /dev/null +++ b/internal/httpclient/served_lease_reclaimer_test.go @@ -0,0 +1,111 @@ +//go:build cgo + +package httpclient + +import ( + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" +) + +// The LeaseReclaimer contract, run through the http client against a real bd +// serve. Nothing is parked: every member of ReclaimRequest has a wire member, +// and every refusal the role lists is raised before the dial in the role's own +// order (leasereclaimer.go). +// +// The claim that writes each lease goes over the wire too, through the client's +// Claimer; only the aging of a lease and the attaching of a label reach past +// the wire, into the embedded database behind the server, because no verb can +// make a lease stale on demand. +// +// THE HOOK CASE runs the reclaimer off a HookFiringStore wrapped around the +// client, the chain cmd/bd composes for a connected workspace: bd serve runs no +// hooks, so it is the CLIENT's decorator that owes one on_update per reverted +// row, re-reading each row over the wire to hand the script. +func newServedLeaseReclaimerFixture(t *testing.T, prefix string) conformance.LeaseReclaimerFixture { + t.Helper() + env := newServedEnv(t, prefix) + reclaimer, err := env.subject.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer(): %v", err) + } + claimer, err := env.subject.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + lifecycle, err := env.subject.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + runner, fired := conformance.NewUpdateHookLog(t) + hooked := reclaimer + if runner != nil { + if hooked, err = storage.NewHookFiringStore(env.subject, runner).LeaseReclaimer(); err != nil { + t.Fatalf("hooked LeaseReclaimer(): %v", err) + } + } + return conformance.LeaseReclaimerFixture{ + IssuePrefix: env.prefix, + LeaseReclaimer: reclaimer, + CreateIssue: env.createIssue, + Claimer: claimer, + Lifecycle: lifecycle, + Exec: env.exec, + QueryScalar: env.queryScalar, + HookedLeaseReclaimer: hooked, + FiredUpdates: fired, + } +} + +func TestServedLeaseReclaimerRevertsAStaleLease(t *testing.T) { + conformance.RunLeaseReclaimerRevertsAStaleLease(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr00")) +} + +func TestServedLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t *testing.T) { + conformance.RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr01")) +} + +func TestServedLeaseReclaimerLeavesALiveLeaseAlone(t *testing.T) { + conformance.RunLeaseReclaimerLeavesALiveLeaseAlone(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr02")) +} + +func TestServedLeaseReclaimerHonorsTheGraceWindow(t *testing.T) { + conformance.RunLeaseReclaimerHonorsTheGraceWindow(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr03")) +} + +func TestServedLeaseReclaimerScopesToTheNamedIDs(t *testing.T) { + conformance.RunLeaseReclaimerScopesToTheNamedIDs(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr04")) +} + +func TestServedLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t *testing.T) { + conformance.RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr05")) +} + +func TestServedLeaseReclaimerScopesToAssigneesAndLabels(t *testing.T) { + conformance.RunLeaseReclaimerScopesToAssigneesAndLabels(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr06")) +} + +func TestServedLeaseReclaimerAttributesTheRecoveryEvent(t *testing.T) { + conformance.RunLeaseReclaimerAttributesTheRecoveryEvent(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr07")) +} + +func TestServedLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t *testing.T) { + conformance.RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr08")) +} + +func TestServedLeaseReclaimerRefusesAMalformedRequest(t *testing.T) { + conformance.RunLeaseReclaimerRefusesAMalformedRequest(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr09")) +} + +func TestServedLeaseReclaimerRefusesABlankScopeEntry(t *testing.T) { + conformance.RunLeaseReclaimerRefusesABlankScopeEntry(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr12")) +} + +func TestServedLeaseReclaimerAcceptsExactlyTheCap(t *testing.T) { + conformance.RunLeaseReclaimerAcceptsExactlyTheCap(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr10")) +} + +func TestServedLeaseReclaimerDoesNotMutateTheCallerRequest(t *testing.T) { + conformance.RunLeaseReclaimerDoesNotMutateTheCallerRequest(t, t.Context(), newServedLeaseReclaimerFixture(t, "hlr11")) +} diff --git a/internal/httpclient/served_test.go b/internal/httpclient/served_test.go index 5037cb1f7..23b16b02e 100644 --- a/internal/httpclient/served_test.go +++ b/internal/httpclient/served_test.go @@ -240,6 +240,9 @@ func serveRoles(store storage.DoltStorage) (httpapi.Config, error) { if cfg.BatchGetter, err = store.BatchGetter(); err != nil { return cfg, err } + if cfg.LeaseReclaimer, err = store.LeaseReclaimer(); err != nil { + return cfg, err + } return cfg, nil } diff --git a/internal/httpclient/unsupported_gen.go b/internal/httpclient/unsupported_gen.go index 79e696c54..2e8cde872 100644 --- a/internal/httpclient/unsupported_gen.go +++ b/internal/httpclient/unsupported_gen.go @@ -742,4 +742,4 @@ func (unsupportedDoltStorage) VersionReconciler() (_ issueops.VersionReconciler, return } -// NOTE: partial shell (145 of 199 methods generated; 54 left to this package's hand-written set). +// NOTE: partial shell (145 of 200 methods generated; 55 left to this package's hand-written set). diff --git a/internal/httpclient/wave2b_roles_test.go b/internal/httpclient/wave2b_roles_test.go index 86c053aca..c64817ac1 100644 --- a/internal/httpclient/wave2b_roles_test.go +++ b/internal/httpclient/wave2b_roles_test.go @@ -8,8 +8,10 @@ import ( "reflect" "strings" "testing" + "time" "github.com/steveyegge/beads/internal/httpapi/apigen" + storeops "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/workapi" "github.com/steveyegge/beads/issueops" @@ -542,3 +544,87 @@ func TestRelatedLeavesTheCallersTypesAlone(t *testing.T) { t.Errorf("the caller's Types changed across the call: %v, want %v", types_, snapshot) } } + +// TestReclaimValidationMatchesTheSharedValidator binds leasereclaimer.go's +// restated rules to internal/storage/issueops.ValidateReclaimRequest, for +// TestReleaseValidationMatchesTheSharedValidator's reason: a restated rule is +// how a leg drifts. It compares VERDICTS, and the cap's type, over a table that +// covers both sides of every rule. +func TestReclaimValidationMatchesTheSharedValidator(t *testing.T) { + overCap := make([]string, issueops.MaxReclaimIDs+1) + for name, req := range map[string]issueops.ReclaimRequest{ + "valid bare": {Actor: "reaper"}, + "valid scoped": {Actor: "reaper", OlderThan: time.Minute, Filter: issueops.ReclaimFilter{IDs: []string{"bd-1"}}}, + "no actor": {}, + "blank actor": {Actor: " "}, + "negative grace": {Actor: "reaper", OlderThan: -time.Second}, + "blank id": {Actor: "reaper", Filter: issueops.ReclaimFilter{IDs: []string{"bd-1", ""}}}, + "padded id is an id": {Actor: "reaper", Filter: issueops.ReclaimFilter{IDs: []string{" bd-1 "}}}, + "over the cap": {Actor: "reaper", Filter: issueops.ReclaimFilter{IDs: overCap}}, + "blank assignee": {Actor: "reaper", Filter: issueops.ReclaimFilter{Assignees: []string{" "}}}, + "blank label": {Actor: "reaper", Filter: issueops.ReclaimFilter{Labels: []string{"a", ""}}}, + "blank label-any": {Actor: "reaper", Filter: issueops.ReclaimFilter{LabelsAny: []string{"\t"}}}, + "blank exclude": {Actor: "reaper", Filter: issueops.ReclaimFilter{ExcludeLabels: []string{""}}}, + + // The actor is held to its column once trimmed, as the sweep records it. + "actor at the bound": {Actor: strings.Repeat("r", types.MaxFieldLen)}, + "padded actor at the bound": {Actor: " " + strings.Repeat("r", types.MaxFieldLen) + " "}, + "over-long actor": {Actor: strings.Repeat("r", types.MaxFieldLen+1)}, + } { + t.Run(name, func(t *testing.T) { + mine := validateReclaimRequest(req) + shared := storeops.ValidateReclaimRequest(req) + if (mine != nil) != (shared != nil) { + t.Fatalf("validateReclaimRequest = %v, storeops.ValidateReclaimRequest = %v: the client's restatement has drifted", mine, shared) + } + var mineCap, sharedCap *issueops.TooManyReclaimIDsError + if errors.As(mine, &mineCap) != errors.As(shared, &sharedCap) { + t.Fatalf("cap type differs: client %v, shared %v", mine, shared) + } + var mineField, sharedField *issueops.ReclaimFieldError + errors.As(mine, &mineField) + errors.As(shared, &sharedField) + if (mineField == nil) != (sharedField == nil) || (mineField != nil && mineField.Field != sharedField.Field) { + t.Fatalf("field differs: client %v, shared %v", mine, shared) + } + }) + } +} + +// TestReclaimSendsAnyReplicaOnlyWhenAsked pins the one filter member that +// WIDENS a sweep rather than narrowing it. ReclaimFilter.AnyReplica disarms the +// granting-replica guard, so a client that sent it unasked would revert leases +// another replica granted, and one that dropped it would keep the guard a caller +// had disarmed. Unasked it must be absent from the body; asked it must be true. +func TestReclaimSendsAnyReplicaOnlyWhenAsked(t *testing.T) { + for name, anyReplica := range map[string]bool{"unasked": false, "asked": true} { + t.Run(name, func(t *testing.T) { + store, w := recordingStore(t) + role, err := store.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer(): %v", err) + } + if _, err := role.Reclaim(context.Background(), issueops.ReclaimRequest{ + Actor: "reaper", + Filter: issueops.ReclaimFilter{IDs: []string{"bd-1"}, AnyReplica: anyReplica}, + }); err != nil { + t.Fatalf("Reclaim(): %v", err) + } + if len(w.requests) != 1 { + t.Fatalf("dialed %d times, want 1", len(w.requests)) + } + body, ok := w.requests[0].Body.(apigen.ReclaimIssuesRequest) + if !ok { + t.Fatalf("the body is %T, want the generated ReclaimIssuesRequest", w.requests[0].Body) + } + switch { + case !anyReplica && body.AnyReplica != nil: + t.Errorf("an unasked sweep sent any_replica=%v; the guard must stay armed", *body.AnyReplica) + case anyReplica && body.AnyReplica == nil: + t.Error("a sweep that disarmed the replica guard sent no any_replica; the server would keep it armed") + case anyReplica && !*body.AnyReplica: + t.Error("a sweep that disarmed the replica guard sent any_replica=false, want true") + } + }) + } +} diff --git a/internal/httpclient/wire/handshake.go b/internal/httpclient/wire/handshake.go index b7db819c2..60476683a 100644 --- a/internal/httpclient/wire/handshake.go +++ b/internal/httpclient/wire/handshake.go @@ -102,6 +102,11 @@ var opCapability = map[string]string{ // the block above states: the server publishes it ahead of the accessor // that dials it, and the set-equality gate needs its token here first. OpBatchGetIssues: "issues.batchGet", + // The stale-lease sweep, dialed by httpLeaseReclaimer. A server that + // predates it does not advertise the token, so Preflight refuses locally + // with a typed capability error instead of dialing a path the older server + // would answer as a claim of an issue called ":reclaim". + OpReclaimIssues: "issues.reclaim", } // CapProjectEnforce is the behavior capability the server advertises to announce diff --git a/internal/httpclient/wire/paths.go b/internal/httpclient/wire/paths.go index 6357c526d..94a90b83a 100644 --- a/internal/httpclient/wire/paths.go +++ b/internal/httpclient/wire/paths.go @@ -57,6 +57,7 @@ const ( OpGetMemory = "getMemory" OpForgetMemory = "forgetMemory" OpBatchGetIssues = "batchGetIssues" + OpReclaimIssues = "reclaimIssues" ) // The paths that carry no caller-supplied segment. A path with one is built by @@ -75,6 +76,7 @@ const ( PathIssuesBatchCreate = "/v0/beads/issues:batchCreate" PathIssuesBatchClose = "/v0/beads/issues:batchClose" PathIssuesBatchGet = "/v0/beads/issues:batchGet" + PathIssuesReclaim = "/v0/beads/issues:reclaim" PathIssuesClaimNext = "/v0/beads/issues:claimNext" PathIssuesBatchApply = "/v0/beads/issues:batchApply" PathSettings = "/v0/beads/config" diff --git a/internal/httpclient/wire/revision_tolerance_test.go b/internal/httpclient/wire/revision_tolerance_test.go index 9b410298f..1c9f33f16 100644 --- a/internal/httpclient/wire/revision_tolerance_test.go +++ b/internal/httpclient/wire/revision_tolerance_test.go @@ -283,6 +283,10 @@ var revisionSchemaExempt = map[string]string{ "#6053 and types.BatchGetIssue declares Revision a string, so every server that routes it " + "answers the decimal-string shape; a pre-#6053 server never advertises issues.batchGet, so " + "Preflight refuses the operation with a *CapabilityError before any body is decoded", + "ReclaimedLease": "ReclaimIssuesResult.reclaimed's element. issues:reclaim postdates #6053 and " + + "types.ReclaimedLease declares Revision a string, so every server that routes it answers the " + + "decimal-string shape; a pre-#6053 server never advertises issues.reclaim, so Preflight refuses " + + "the operation with a *CapabilityError before any body is decoded", } // TestRevisionBearingResponseCoversEveryWireShapeSchemaWithARevisionMember is diff --git a/internal/httpclient/wire/skew_test.go b/internal/httpclient/wire/skew_test.go index 04b023171..fe2fc7ace 100644 --- a/internal/httpclient/wire/skew_test.go +++ b/internal/httpclient/wire/skew_test.go @@ -110,16 +110,17 @@ func TestAgainstAFirstSliceServerEveryPostBaselineOperationRefusesWithCaseTwoDat t.Fatalf("post-baseline ops on a first-slice token = %v, want exactly [claimIssue]", compat) } // The dimension is pinned so the matrix cannot shrink silently: the first - // slice served six operations and the tip surface has forty-two, so - // thirty-six of them are things a first-slice server has never heard of. The + // slice served six operations and the tip surface has forty-three, so + // thirty-seven of them are things a first-slice server has never heard of. The // wire wave moved this from twenty-seven to thirty — claimNext, release and // count — wave 2 to thirty-two, adding the dependency-edge count and the // related read, the mini-sync to thirty-five: the comment write and the two - // config writes, and the batch read (upstream #7248) to thirty-six. Each is + // config writes, the batch read (upstream #7248) to thirty-six, and the + // stale-lease sweep to thirty-seven. Each is // an operation a first-slice server never served and this client does not // yet dial, which is exactly the shape the refusal matrix is about. - if len(refused) != 36 { - t.Fatalf("the post-baseline set has %d operations a first-slice server never served, want 36", len(refused)) + if len(refused) != 37 { + t.Fatalf("the post-baseline set has %d operations a first-slice server never served, want 37", len(refused)) } // The compat op forces the handshake yet preflights clean: issues.claim is on diff --git a/internal/httpclient/wire/vocabulary_test.go b/internal/httpclient/wire/vocabulary_test.go index 23d98908d..c650ac7f5 100644 --- a/internal/httpclient/wire/vocabulary_test.go +++ b/internal/httpclient/wire/vocabulary_test.go @@ -68,6 +68,7 @@ func TestOperationIdsMatchTheServerRouteTable(t *testing.T) { {OpListEvents, httpapi.OpListEvents}, {OpWatchEvents, httpapi.OpWatchEvents}, {OpBatchGetIssues, httpapi.OpBatchGetIssues}, + {OpReclaimIssues, httpapi.OpReclaimIssues}, } // The table is a HAND LIST, and a hand list's failure mode is omission: a new diff --git a/internal/storage/BUILD.bazel b/internal/storage/BUILD.bazel index 86a1ef4f6..e4a8b5a27 100644 --- a/internal/storage/BUILD.bazel +++ b/internal/storage/BUILD.bazel @@ -38,6 +38,7 @@ go_library( "hook_issue_claimer.go", "hook_issue_operations.go", "hook_issue_reader.go", + "hook_lease_reclaimer.go", "hook_memories.go", "hook_metadata_cas.go", "hook_querier.go", @@ -90,6 +91,7 @@ go_test( "hook_issue_claimer_test.go", "hook_issue_operations_test.go", "hook_issue_reader_test.go", + "hook_lease_reclaimer_test.go", "hook_metadata_cas_test.go", "hook_ready_claimer_test.go", "hook_relations_test.go", diff --git a/internal/storage/dolt/BUILD.bazel b/internal/storage/dolt/BUILD.bazel index 989f2d8e4..1f79aa088 100644 --- a/internal/storage/dolt/BUILD.bazel +++ b/internal/storage/dolt/BUILD.bazel @@ -47,6 +47,7 @@ go_library( "iter_stubs.go", "journal_accessor.go", "labels.go", + "lease_reclaimer.go", "memories.go", "merge_slot.go", "metadata_cas.go", @@ -268,6 +269,7 @@ go_test( "large_batch_apply_measure_test.go", "large_graph_create_bench_test.go", "lease_migration_test.go", + "lease_reclaimer_contract_test.go", "lease_test.go", "lenient_open_dirty_tables_integration_test.go", "lifecycle_close_reopen_contract_test.go", diff --git a/internal/storage/dolt/lease_reclaimer.go b/internal/storage/dolt/lease_reclaimer.go new file mode 100644 index 000000000..492a47f48 --- /dev/null +++ b/internal/storage/dolt/lease_reclaimer.go @@ -0,0 +1,50 @@ +package dolt + +import ( + "context" + "database/sql" + + "github.com/steveyegge/beads/internal/storage" + storeops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimer returns the lease-sweep surface for this store. +func (s *DoltStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + if s == nil { + return nil, &storage.ErrUnsupported{Op: "LeaseReclaimer", Backend: "nil"} + } + return &leaseReclaimer{store: s}, nil +} + +// leaseReclaimer answers a Reclaim from one retrying write transaction around +// the shared body, storeops.ExecuteReclaimInTx. +// +// THE VERSION-CONTROL ENTRY GOES THROUGH runIssueOperationTxWithMessage, the +// funnel Releaser uses, rather than the raw ReclaimExpiredLeases' in-transaction +// DOLT_COMMIT: the funnel is what honors a deferred version commit on the +// context (dolt.auto-commit batch/off), which a role a front door calls has to, +// and a sweep that reverted nothing composes no staged set and so records no +// entry describing a no-op. +type leaseReclaimer struct{ store *DoltStore } + +var _ issueops.LeaseReclaimer = (*leaseReclaimer)(nil) + +func (l *leaseReclaimer) Reclaim(ctx context.Context, request issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + if err := storeops.ValidateReclaimRequest(request); err != nil { + return issueops.ReclaimResult{}, err + } + var result issueops.ReclaimResult + if err := l.store.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) { + var err error + result, err = storeops.ExecuteReclaimInTx(ctx, tx, request) + if err != nil { + return nil, "", err + } + tables, msg := storeops.ReclaimVersionCommit(result) + return tables, msg, nil + }); err != nil { + return issueops.ReclaimResult{}, err + } + return result, nil +} diff --git a/internal/storage/dolt/lease_reclaimer_contract_test.go b/internal/storage/dolt/lease_reclaimer_contract_test.go new file mode 100644 index 000000000..cfc19f3e9 --- /dev/null +++ b/internal/storage/dolt/lease_reclaimer_contract_test.go @@ -0,0 +1,114 @@ +package dolt + +import ( + "context" + "fmt" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" +) + +// TestLeaseReclaimerContract runs the LeaseReclaimer contract against the +// server-backed store, which wraps internal/storage/issueops.ExecuteReclaimInTx +// in its own retrying write transaction. Like every role suite here the cases +// share one store, so each namespaces its seeds and scopes every sweep. +func TestLeaseReclaimerContract(t *testing.T) { + fixture, ctx, cleanup := newDoltLeaseReclaimerFixture(t, "lrc") + defer cleanup() + + t.Run("RevertsAStaleLease", func(t *testing.T) { + conformance.RunLeaseReclaimerRevertsAStaleLease(t, ctx, fixture) + }) + t.Run("MintsARevisionThatFencesTheFormerHolder", func(t *testing.T) { + conformance.RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t, ctx, fixture) + }) + t.Run("LeavesALiveLeaseAlone", func(t *testing.T) { + conformance.RunLeaseReclaimerLeavesALiveLeaseAlone(t, ctx, fixture) + }) + t.Run("HonorsTheGraceWindow", func(t *testing.T) { + conformance.RunLeaseReclaimerHonorsTheGraceWindow(t, ctx, fixture) + }) + t.Run("ScopesToTheNamedIDs", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToTheNamedIDs(t, ctx, fixture) + }) + t.Run("AnswersAnAbsentIDWithAnEmptyResult", func(t *testing.T) { + conformance.RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t, ctx, fixture) + }) + t.Run("ScopesToAssigneesAndLabels", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToAssigneesAndLabels(t, ctx, fixture) + }) + t.Run("AttributesTheRecoveryEvent", func(t *testing.T) { + conformance.RunLeaseReclaimerAttributesTheRecoveryEvent(t, ctx, fixture) + }) + t.Run("FiresOnUpdateOncePerReclaimedRow", func(t *testing.T) { + conformance.RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t, ctx, fixture) + }) + t.Run("RefusesAMalformedRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesAMalformedRequest(t, ctx, fixture) + }) + t.Run("RefusesABlankScopeEntry", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesABlankScopeEntry(t, ctx, fixture) + }) + t.Run("AcceptsExactlyTheCap", func(t *testing.T) { + conformance.RunLeaseReclaimerAcceptsExactlyTheCap(t, ctx, fixture) + }) + t.Run("DoesNotMutateTheCallerRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerDoesNotMutateTheCallerRequest(t, ctx, fixture) + }) +} + +// newDoltLeaseReclaimerFixture composes the frozen role kit with this +// backend's accessors, plus the same reclaimer taken off a HookFiringStore so +// the hook case observes the decorator a command actually holds. +func newDoltLeaseReclaimerFixture(t *testing.T, prefix string) (conformance.LeaseReclaimerFixture, context.Context, func()) { + t.Helper() + store, storeCleanup := setupTestStore(t) + ctx, cancel := testContext(t) + stop := func() { + cancel() + storeCleanup() + } + fail := func(what string, err error) { + stop() + t.Fatalf("%s: %v", what, err) + } + reclaimer, err := store.LeaseReclaimer() + if err != nil { + fail("LeaseReclaimer()", err) + } + claimer, err := store.IssueClaimer() + if err != nil { + fail("IssueClaimer()", err) + } + lifecycle, err := store.IssueLifecycle() + if err != nil { + fail("IssueLifecycle()", err) + } + runner, fired := conformance.NewUpdateHookLog(t) + hooked := reclaimer + if runner != nil { + if hooked, err = storage.NewHookFiringStore(store, runner).LeaseReclaimer(); err != nil { + fail("hooked LeaseReclaimer()", err) + } + } + kit := newDoltRoleFixtureKit(store, prefix) + return conformance.LeaseReclaimerFixture{ + IssuePrefix: kit.IssuePrefix, + LeaseReclaimer: reclaimer, + CreateIssue: kit.CreateIssue, + Claimer: claimer, + Lifecycle: lifecycle, + QueryScalar: kit.QueryScalar, + Exec: func(ctx context.Context, statements []conformance.SQLStatement) error { + for _, stmt := range statements { + if _, err := store.db.ExecContext(ctx, stmt.Query, stmt.Args...); err != nil { + return fmt.Errorf("%s: %w", stmt.Query, err) + } + } + return nil + }, + HookedLeaseReclaimer: hooked, + FiredUpdates: fired, + }, ctx, stop +} diff --git a/internal/storage/domain/db/issue.go b/internal/storage/domain/db/issue.go index a9d2e3c7a..4a1282e25 100644 --- a/internal/storage/domain/db/issue.go +++ b/internal/storage/domain/db/issue.go @@ -1309,4 +1309,11 @@ func (r *issueSQLRepositoryImpl) ReclaimExpiredLeases(ctx context.Context, older return out, nil } +// Reclaim runs the shared sweep body directly on r.runner, matching GetMany's +// bare-passthrough shape: validation and error-wrapping both happen inside +// issueops.ExecuteReclaimInTx. +func (r *issueSQLRepositoryImpl) Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + return issueops.ExecuteReclaimInTx(ctx, r.runner, request) +} + const deleteBatchSize = 200 diff --git a/internal/storage/domain/issue.go b/internal/storage/domain/issue.go index a6ce46a83..7f5ec7208 100644 --- a/internal/storage/domain/issue.go +++ b/internal/storage/domain/issue.go @@ -117,6 +117,11 @@ type IssueSQLRepository interface { UnclaimIssueIfAssignee(ctx context.Context, id, actor, expectedAssignee string) error HeartbeatIssue(ctx context.Context, id, actor string) error ReclaimExpiredLeases(ctx context.Context, olderThan time.Duration, filter types.ReclaimFilter, actor string) ([]types.ReclaimedLease, error) + // Reclaim runs the SHARED sweep body (issueops.ExecuteReclaimInTx) on this + // repository's transaction, which is how the unit-of-work provider reaches + // the same function the two store backends wrap. It is the publicops-typed + // counterpart to ReclaimExpiredLeases above, the way GetMany is to GetByIDs. + Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) WakeExpiredDefers(ctx context.Context) (issues, wisps int, err error) } @@ -344,6 +349,10 @@ type IssueUseCase interface { UnclaimIfAssignee(ctx context.Context, id, actor, expectedAssignee string) error Heartbeat(ctx context.Context, id, actor string) error ReclaimExpiredLeases(ctx context.Context, olderThan time.Duration, filter types.ReclaimFilter, actor string) ([]types.ReclaimedLease, error) + // Reclaim is the shape issueops.LeaseReclaimer publishes; see + // IssueRepository.Reclaim for why it is a separate method from + // ReclaimExpiredLeases rather than a replacement for it. + Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) WakeExpiredDefers(ctx context.Context) (issues, wisps int, err error) CreateIssue(ctx context.Context, params CreateIssueParams, actor string) (CreateIssueResult, error) @@ -1978,3 +1987,11 @@ func (u *issueUseCaseImpl) ReclaimExpiredLeases(ctx context.Context, olderThan t } return out, nil } + +// Reclaim passes straight through to the repository with no pre-check and no +// error-wrapping, matching GetMany's reasoning: the request is fully +// validated inside the shared body (issueops.ExecuteReclaimInTx), and there is +// no use-case-level concern layered on top of it. +func (u *issueUseCaseImpl) Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + return u.issueRepo.Reclaim(ctx, request) +} diff --git a/internal/storage/embeddeddolt/BUILD.bazel b/internal/storage/embeddeddolt/BUILD.bazel index 64e516d60..8241d7414 100644 --- a/internal/storage/embeddeddolt/BUILD.bazel +++ b/internal/storage/embeddeddolt/BUILD.bazel @@ -37,6 +37,7 @@ go_library( "iter_stubs.go", "journal_accessor.go", "labels.go", + "lease_reclaimer.go", "list_queries.go", "memories.go", "merge_slot.go", @@ -169,6 +170,7 @@ go_test( "label_rename_test.go", "large_batch_apply_measure_test.go", "large_graph_create_bench_test.go", + "lease_reclaimer_contract_test.go", "lease_replica_test.go", "lease_test.go", "lenient_gate_warning_test.go", diff --git a/internal/storage/embeddeddolt/lease_reclaimer.go b/internal/storage/embeddeddolt/lease_reclaimer.go new file mode 100644 index 000000000..cfbfb1c28 --- /dev/null +++ b/internal/storage/embeddeddolt/lease_reclaimer.go @@ -0,0 +1,54 @@ +//go:build cgo + +package embeddeddolt + +import ( + "context" + "database/sql" + + "github.com/steveyegge/beads/internal/storage" + storeops "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimer returns the lease-sweep surface for this store. +func (s *EmbeddedDoltStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + if s == nil { + return nil, &storage.ErrUnsupported{Op: "LeaseReclaimer", Backend: "nil"} + } + return &leaseReclaimer{store: s}, nil +} + +// leaseReclaimer answers a Reclaim from one connection's transaction around the +// shared body, storeops.ExecuteReclaimInTx, the embedded twin of the +// server-backed store's leaseReclaimer. +// +// THE VERSION-CONTROL ENTRY LANDS AFTER THE SQL COMMIT, through +// runIssueOperationTxWithMessage, Releaser's funnel: unlike the raw +// ReclaimExpiredLeases, which leaves its rows in the working set for the CLI's +// commitPendingIfEmbedded, a role is called by front doors (bd serve, a +// library caller) that have no such second step, so the role records its own +// entry, and the funnel is what lets a deferred version commit on the context +// suppress it. +type leaseReclaimer struct{ store *EmbeddedDoltStore } + +var _ issueops.LeaseReclaimer = (*leaseReclaimer)(nil) + +func (l *leaseReclaimer) Reclaim(ctx context.Context, request issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + if err := storeops.ValidateReclaimRequest(request); err != nil { + return issueops.ReclaimResult{}, err + } + var result issueops.ReclaimResult + if err := l.store.runIssueOperationTxWithMessage(ctx, func(tx *sql.Tx) (storeops.ChangedTables, string, error) { + var err error + result, err = storeops.ExecuteReclaimInTx(ctx, tx, request) + if err != nil { + return nil, "", err + } + tables, msg := storeops.ReclaimVersionCommit(result) + return tables, msg, nil + }); err != nil { + return issueops.ReclaimResult{}, err + } + return result, nil +} diff --git a/internal/storage/embeddeddolt/lease_reclaimer_contract_test.go b/internal/storage/embeddeddolt/lease_reclaimer_contract_test.go new file mode 100644 index 000000000..d834b3f92 --- /dev/null +++ b/internal/storage/embeddeddolt/lease_reclaimer_contract_test.go @@ -0,0 +1,111 @@ +//go:build cgo + +package embeddeddolt_test + +import ( + "context" + "fmt" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/embeddeddolt" +) + +// TestLeaseReclaimerContract runs the LeaseReclaimer contract against the +// embedded store, which wraps internal/storage/issueops.ExecuteReclaimInTx in +// one connection's transaction and records its version entry after the SQL +// commit. +func TestLeaseReclaimerContract(t *testing.T) { + skipUnlessEmbeddedDolt(t) + te := newTestEnv(t, "lrc") + ctx := t.Context() + fixture := newEmbeddedLeaseReclaimerFixture(t, te, "lrc") + + t.Run("RevertsAStaleLease", func(t *testing.T) { + conformance.RunLeaseReclaimerRevertsAStaleLease(t, ctx, fixture) + }) + t.Run("MintsARevisionThatFencesTheFormerHolder", func(t *testing.T) { + conformance.RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t, ctx, fixture) + }) + t.Run("LeavesALiveLeaseAlone", func(t *testing.T) { + conformance.RunLeaseReclaimerLeavesALiveLeaseAlone(t, ctx, fixture) + }) + t.Run("HonorsTheGraceWindow", func(t *testing.T) { + conformance.RunLeaseReclaimerHonorsTheGraceWindow(t, ctx, fixture) + }) + t.Run("ScopesToTheNamedIDs", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToTheNamedIDs(t, ctx, fixture) + }) + t.Run("AnswersAnAbsentIDWithAnEmptyResult", func(t *testing.T) { + conformance.RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t, ctx, fixture) + }) + t.Run("ScopesToAssigneesAndLabels", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToAssigneesAndLabels(t, ctx, fixture) + }) + t.Run("AttributesTheRecoveryEvent", func(t *testing.T) { + conformance.RunLeaseReclaimerAttributesTheRecoveryEvent(t, ctx, fixture) + }) + t.Run("FiresOnUpdateOncePerReclaimedRow", func(t *testing.T) { + conformance.RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t, ctx, fixture) + }) + t.Run("RefusesAMalformedRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesAMalformedRequest(t, ctx, fixture) + }) + t.Run("RefusesABlankScopeEntry", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesABlankScopeEntry(t, ctx, fixture) + }) + t.Run("AcceptsExactlyTheCap", func(t *testing.T) { + conformance.RunLeaseReclaimerAcceptsExactlyTheCap(t, ctx, fixture) + }) + t.Run("DoesNotMutateTheCallerRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerDoesNotMutateTheCallerRequest(t, ctx, fixture) + }) +} + +func newEmbeddedLeaseReclaimerFixture(t *testing.T, te *testEnv, prefix string) conformance.LeaseReclaimerFixture { + t.Helper() + reclaimer, err := te.store.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer(): %v", err) + } + claimer, err := te.store.IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + lifecycle, err := te.store.IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + runner, fired := conformance.NewUpdateHookLog(t) + hooked := reclaimer + if runner != nil { + if hooked, err = storage.NewHookFiringStore(te.store, runner).LeaseReclaimer(); err != nil { + t.Fatalf("hooked LeaseReclaimer(): %v", err) + } + } + kit := newEmbeddedRoleFixtureKit(te, prefix) + return conformance.LeaseReclaimerFixture{ + IssuePrefix: kit.IssuePrefix, + LeaseReclaimer: reclaimer, + CreateIssue: kit.CreateIssue, + Claimer: claimer, + Lifecycle: lifecycle, + QueryScalar: kit.QueryScalar, + Exec: func(ctx context.Context, statements []conformance.SQLStatement) error { + db, cleanup, err := embeddeddolt.OpenSQL(ctx, te.dataDir, te.database, "main") + if err != nil { + return err + } + defer func() { _ = cleanup() }() + for _, stmt := range statements { + if _, err := db.ExecContext(ctx, stmt.Query, stmt.Args...); err != nil { + return fmt.Errorf("%s: %w", stmt.Query, err) + } + } + return nil + }, + HookedLeaseReclaimer: hooked, + FiredUpdates: fired, + } +} diff --git a/internal/storage/externaldeps/uow_decorator.go b/internal/storage/externaldeps/uow_decorator.go index 0b9e9293b..f7826b43e 100644 --- a/internal/storage/externaldeps/uow_decorator.go +++ b/internal/storage/externaldeps/uow_decorator.go @@ -119,7 +119,10 @@ func (p *uowProvider) VersionReconciler() (publicops.VersionReconciler, error) { } func (p *uowProvider) MetadataCAS() (publicops.MetadataCAS, error) { return uow.NewMetadataCAS(p) } func (p *uowProvider) Releaser() (publicops.Releaser, error) { return uow.NewReleaser(p) } -func (p *uowProvider) Memories() (memoryops.Memories, error) { return uow.NewMemories(p) } +func (p *uowProvider) LeaseReclaimer() (publicops.LeaseReclaimer, error) { + return uow.NewLeaseReclaimer(p) +} +func (p *uowProvider) Memories() (memoryops.Memories, error) { return uow.NewMemories(p) } func (p *uowProvider) EventsJournalCursor() (storage.EventsJournalCursor, error) { return uow.NewEventsJournalCursor(p) } @@ -195,6 +198,7 @@ var ( _ uow.VersionReconcilerSource = (*uowProvider)(nil) _ uow.MetadataCASSource = (*uowProvider)(nil) _ uow.ReleaserSource = (*uowProvider)(nil) + _ uow.LeaseReclaimerSource = (*uowProvider)(nil) _ uow.MemoriesSource = (*uowProvider)(nil) _ uow.EventsJournalCursorSource = (*uowProvider)(nil) ) diff --git a/internal/storage/hook_batch_applier_test.go b/internal/storage/hook_batch_applier_test.go index 92844911c..a211e0908 100644 --- a/internal/storage/hook_batch_applier_test.go +++ b/internal/storage/hook_batch_applier_test.go @@ -39,6 +39,9 @@ func (r *recordingBatchApplyHooks) CompleteIssueOperationMetadata(_ context.Cont func (r *recordingBatchApplyHooks) CompleteIssueOperationRelease(_ context.Context, issueID string) { r.completions = append(r.completions, "release:"+issueID) } +func (r *recordingBatchApplyHooks) CompleteIssueOperationReclaim(_ context.Context, issueID string) { + r.completions = append(r.completions, "reclaim:"+issueID) +} func issueIDOrNil(issue *types.Issue) string { if issue == nil { @@ -230,3 +233,4 @@ func (r *recordingBatchApplyIssues) CompleteIssueOperationDependency(context.Con func (r *recordingBatchApplyIssues) CompleteIssueOperationComment(context.Context, string) {} func (r *recordingBatchApplyIssues) CompleteIssueOperationMetadata(context.Context, string) {} func (r *recordingBatchApplyIssues) CompleteIssueOperationRelease(context.Context, string) {} +func (r *recordingBatchApplyIssues) CompleteIssueOperationReclaim(context.Context, string) {} diff --git a/internal/storage/hook_decorator.go b/internal/storage/hook_decorator.go index c0342252e..c6be30885 100644 --- a/internal/storage/hook_decorator.go +++ b/internal/storage/hook_decorator.go @@ -153,6 +153,13 @@ func RoleFiresHooks(role any) bool { // hazard, on the operation that reaches it with nothing else in the way. case *hookBatchCreator: return true + // The lease reclaimer fires the update hook once per row it reverts + // (hook_lease_reclaimer.go), so a scheduled reaper sweep is N + // subprocesses per run — the same value Releaser and ReadyClaimer are + // refused for, and the one whose caller is a clock rather than a human + // or a request. + case *hookLeaseReclaimer: + return true } return false } @@ -466,6 +473,15 @@ func (h *HookFiringStore) CompleteIssueOperationRelease(ctx context.Context, iss h.fireHookByID(ctx, hooks.EventUpdate, issueID) } +// CompleteIssueOperationReclaim fires the update hook for one row a lease +// sweep reverted, which is the event the generic update path fires for the +// same write — assignee and status are exactly the fields a reclaim moves, +// the same pair a release moves. It re-reads the row rather than taking the +// caller's, so a script sees the unheld issue the sweep produced. +func (h *HookFiringStore) CompleteIssueOperationReclaim(ctx context.Context, issueID string) { + h.fireHookByID(ctx, hooks.EventUpdate, issueID) +} + func (h *HookFiringStore) fireHookByID(ctx context.Context, event, id string) { if h.runner == nil { return diff --git a/internal/storage/hook_issue_operations.go b/internal/storage/hook_issue_operations.go index ba41e5d81..7926a5694 100644 --- a/internal/storage/hook_issue_operations.go +++ b/internal/storage/hook_issue_operations.go @@ -35,6 +35,7 @@ type issueOperationHooks interface { CompleteIssueOperationComment(ctx context.Context, issueID string) CompleteIssueOperationMetadata(ctx context.Context, issueID string) CompleteIssueOperationRelease(ctx context.Context, issueID string) + CompleteIssueOperationReclaim(ctx context.Context, issueID string) } var _ issueOperationHooks = (*HookFiringStore)(nil) diff --git a/internal/storage/hook_issue_operations_test.go b/internal/storage/hook_issue_operations_test.go index 0c86b95ea..13a788060 100644 --- a/internal/storage/hook_issue_operations_test.go +++ b/internal/storage/hook_issue_operations_test.go @@ -63,6 +63,9 @@ func (r *recordingIssueOperationHooks) CompleteIssueOperationMetadata(_ context. func (r *recordingIssueOperationHooks) CompleteIssueOperationRelease(_ context.Context, issueID string) { r.completions = append(r.completions, "release:"+issueID) } +func (r *recordingIssueOperationHooks) CompleteIssueOperationReclaim(_ context.Context, issueID string) { + r.completions = append(r.completions, "reclaim:"+issueID) +} // lifecycleStore is a DoltStorage whose only real method is IssueLifecycle. type lifecycleStore struct { diff --git a/internal/storage/hook_lease_reclaimer.go b/internal/storage/hook_lease_reclaimer.go new file mode 100644 index 000000000..e94877f93 --- /dev/null +++ b/internal/storage/hook_lease_reclaimer.go @@ -0,0 +1,42 @@ +package storage + +import ( + "context" + + "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimer returns the inner store's lease-sweep surface with this +// decorator's completion hooks layered over it. +// +// It WRAPS rather than recursing unwrapped, for the reason Releaser does: a +// reclaim changes an issue's assignee and status per reverted row, which is +// an on_update in the vocabulary internal/hooks publishes, and the row it +// names is still there to hand a script. +// +// It recurses instead of delegating, so a blind delegation would hand back +// the inner store's surface and silently stop firing it. +func (h *HookFiringStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + inner, err := h.inner.LeaseReclaimer() + if err != nil { + return nil, err + } + return &hookLeaseReclaimer{inner: inner, hooks: h}, nil +} + +type hookLeaseReclaimer struct { + inner issueops.LeaseReclaimer + hooks issueOperationHooks +} + +// Reclaim fires the update hook once for every row the sweep reverted. +func (r *hookLeaseReclaimer) Reclaim(ctx context.Context, request issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + result, err := r.inner.Reclaim(ctx, request) + if err != nil { + return result, err + } + for _, reclaimed := range result.Reclaimed { + r.hooks.CompleteIssueOperationReclaim(ctx, reclaimed.ID) + } + return result, nil +} diff --git a/internal/storage/hook_lease_reclaimer_test.go b/internal/storage/hook_lease_reclaimer_test.go new file mode 100644 index 000000000..a3d8506df --- /dev/null +++ b/internal/storage/hook_lease_reclaimer_test.go @@ -0,0 +1,83 @@ +package storage + +import ( + "context" + "errors" + "reflect" + "testing" + + "github.com/steveyegge/beads/issueops" +) + +type fakeLeaseReclaimer struct { + result issueops.ReclaimResult + err error +} + +func (f *fakeLeaseReclaimer) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + return f.result, f.err +} + +// TestHookLeaseReclaimerFiresOncePerReclaimedRow is the decorator's whole +// subject: one on_update completion for every row the sweep reverted, in the +// sweep's order, and none at all for a refusal — even a refusal that somehow +// carries rows, because the role leaves its result unspecified on error. +func TestHookLeaseReclaimerFiresOncePerReclaimedRow(t *testing.T) { + for _, test := range []struct { + name string + inner *fakeLeaseReclaimer + wantFire []string + }{ + { + name: "every reverted row fires once", + inner: &fakeLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{ + {ID: "bd-1"}, {ID: "bd-2"}, + }}}, + wantFire: []string{"reclaim:bd-1", "reclaim:bd-2"}, + }, + { + name: "a sweep that reverted nothing fires nothing", + inner: &fakeLeaseReclaimer{result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{}}}, + }, + { + name: "a refusal fires nothing", + inner: &fakeLeaseReclaimer{err: errors.New("boom")}, + }, + { + name: "a refusal that carries rows still fires nothing", + inner: &fakeLeaseReclaimer{ + result: issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{{ID: "bd-1"}}}, + err: errors.New("boom"), + }, + }, + } { + t.Run(test.name, func(t *testing.T) { + recorder := &recordingIssueOperationHooks{} + reclaimer := &hookLeaseReclaimer{inner: test.inner, hooks: recorder} + + _, err := reclaimer.Reclaim(context.Background(), issueops.ReclaimRequest{Actor: "reaper"}) + if (err != nil) != (test.inner.err != nil) { + t.Fatalf("Reclaim error = %v, want %v", err, test.inner.err) + } + if !reflect.DeepEqual(recorder.completions, test.wantFire) { + t.Fatalf("hooks fired = %v, want %v", recorder.completions, test.wantFire) + } + }) + } +} + +// TestHookLeaseReclaimerPassesTheResultThrough pins that the decorator reports +// exactly what the role reported, revisions included: a hook layer that +// rewrote the result would be deciding which token a caller feeds forward. +func TestHookLeaseReclaimerPassesTheResultThrough(t *testing.T) { + want := issueops.ReclaimResult{Reclaimed: []issueops.ReclaimedLease{{ID: "bd-1", PreviousOwner: "w", Revision: "42"}}} + reclaimer := &hookLeaseReclaimer{inner: &fakeLeaseReclaimer{result: want}, hooks: &recordingIssueOperationHooks{}} + + got, err := reclaimer.Reclaim(context.Background(), issueops.ReclaimRequest{Actor: "reaper"}) + if err != nil { + t.Fatalf("Reclaim error = %v", err) + } + if !reflect.DeepEqual(got, want) { + t.Fatalf("Reclaim result = %+v, want %+v", got, want) + } +} diff --git a/internal/storage/issueops/BUILD.bazel b/internal/storage/issueops/BUILD.bazel index 0838bce48..be4703360 100644 --- a/internal/storage/issueops/BUILD.bazel +++ b/internal/storage/issueops/BUILD.bazel @@ -66,6 +66,7 @@ go_library( "journal_prune.go", "labels.go", "lease.go", + "lease_reclaimer.go", "metadata.go", "metadata_cas.go", "molecule.go", @@ -164,6 +165,7 @@ go_test( "journal_truncation_test.go", "journal_vocabulary_test.go", "label_rename_merged_count_test.go", + "lease_reclaimer_test.go", "lease_replica_test.go", "metadata_test.go", "prepare_timestamps_utc_test.go", diff --git a/internal/storage/issueops/lease.go b/internal/storage/issueops/lease.go index d2b75f7e7..2ec940280 100644 --- a/internal/storage/issueops/lease.go +++ b/internal/storage/issueops/lease.go @@ -667,6 +667,11 @@ func reclaimReplicaSQL(filter types.ReclaimFilter, localNode string) (string, [] // the lease from) so the caller can log/emit recovery events. The caller owns // Dolt versioning. // +// This is the one place a reclaimed row's revision token is minted: the row's +// freshRowLock() value is captured off the revert UPDATE and published on +// each types.ReclaimedLease as types.RevisionToken(newLock), the same +// encoding IssueDetails.Revision uses. +// // filter narrows which stale leases are eligible (see types.ReclaimFilter); the // zero filter keeps the historical global behavior. Scoping is applied to the // snapshot SELECT only — the per-row DELETE/UPDATE re-checks staleness by id, @@ -754,12 +759,13 @@ func ReclaimExpiredLeasesInTx(ctx context.Context, tx DBTX, cutoff time.Time, fi // being in_progress under us (closed) is left alone; row_lock makes a // concurrent close/update conflict at commit time rather than // cell-merge with this write. + newLock := freshRowLock() res, err = tx.ExecContext(ctx, ` UPDATE issues SET status = 'open', assignee = NULL, started_at = NULL, updated_at = ?, row_lock = ? WHERE id = ? AND status = 'in_progress' - `, time.Now().UTC(), freshRowLock(), r.ID) + `, time.Now().UTC(), newLock, r.ID) if err != nil { return nil, fmt.Errorf("reclaim %s: %w", r.ID, err) } @@ -770,6 +776,7 @@ func ReclaimExpiredLeasesInTx(ctx context.Context, tx DBTX, cutoff time.Time, fi if n == 0 { continue // no longer in_progress — its lease row was stale anyway } + r.Revision = types.RevisionToken(newLock) if err := RecordFullEventInTable(ctx, tx, "events", r.ID, types.EventLeaseReclaimed, actor, r.PreviousOwner, ""); err != nil { return nil, fmt.Errorf("record reclaim event for %s: %w", r.ID, err) diff --git a/internal/storage/issueops/lease_reclaimer.go b/internal/storage/issueops/lease_reclaimer.go new file mode 100644 index 000000000..ba0e8b4cb --- /dev/null +++ b/internal/storage/issueops/lease_reclaimer.go @@ -0,0 +1,123 @@ +package issueops + +import ( + "context" + "fmt" + "strings" + "time" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// ValidateReclaimRequest applies the request rules every LeaseReclaimer +// implementation shares. +// +// THE CAP IS CHECKED AGAINST Filter.IDs AS SENT, before deduplication, for +// ValidateGetManyRequest's reason: the obligation it bounds is reading the +// request apart, not reading the rows it resolves to. +// +// THE ACTOR IS BOUNDED HERE, trimmed as the sweep records it, for +// claimIssueInTx's reason: every reverted row's recovery event lands in a +// VARCHAR(255) column, so an over-long actor is a typed refusal on every leg +// rather than a raw backend error on the first sweep that reverts something. +// +// It is the ONE definition of the role's rules: the library legs run it, and +// the HTTP client restates it (pinned by +// TestReclaimValidationMatchesTheSharedValidator). Every refusal but the cap +// is a *publicops.ReclaimFieldError naming its field. bd serve's handler leaves +// the id cap and the blank entries to it, and adds three rules of the wire's +// own that a library caller is not held to: the claim's name rules on the +// actor (a 256-byte bound, no control characters), an `older_than_seconds` a +// time.Duration can hold, and no scope member present but empty. +func ValidateReclaimRequest(request publicops.ReclaimRequest) error { + actor := strings.TrimSpace(request.Actor) + if actor == "" { + return reclaimFieldError(publicops.ReclaimFieldActor, "reclaim actor is required") + } + if err := types.CheckFieldLen("actor", actor); err != nil { + return reclaimFieldError(publicops.ReclaimFieldActor, err.Error()) + } + if request.OlderThan < 0 { + return reclaimFieldError(publicops.ReclaimFieldOlderThan, "reclaim older_than must not be negative") + } + if len(request.Filter.IDs) > publicops.MaxReclaimIDs { + return &publicops.TooManyReclaimIDsError{Requested: len(request.Filter.IDs), Cap: publicops.MaxReclaimIDs} + } + for i, id := range request.Filter.IDs { + if id == "" { + return reclaimFieldError(publicops.ReclaimFieldIDs, fmt.Sprintf("reclaim scope id at position %d is empty", i)) + } + } + for _, scope := range []struct { + field string + values []string + }{ + {publicops.ReclaimFieldAssignees, request.Filter.Assignees}, + {publicops.ReclaimFieldLabels, request.Filter.Labels}, + {publicops.ReclaimFieldLabelsAny, request.Filter.LabelsAny}, + {publicops.ReclaimFieldExcludeLabels, request.Filter.ExcludeLabels}, + } { + for i, v := range scope.values { + if strings.TrimSpace(v) == "" { + return reclaimFieldError(scope.field, fmt.Sprintf("reclaim %s entry at position %d is blank", scope.field, i)) + } + } + } + return nil +} + +func reclaimFieldError(field, detail string) error { + return &publicops.ReclaimFieldError{Field: field, Detail: storage.ErrValidation.Error() + ": " + detail} +} + +// ExecuteReclaimInTx sweeps stale leases from tx, in ONE transaction. It is +// the body behind the LeaseReclaimer accessor on ALL THREE legs: the two +// stores wrap it in their own retrying write transaction, and the +// unit-of-work provider reaches it through the domain repository, whose +// runner publishes exactly the DBTX method set ReclaimExpiredLeasesInTx +// takes. +// +// VALIDATION HAPPENS HERE, for ExecuteGetMany's reason: this body is the only +// body, so a leg that forgot to validate would be answering a different +// contract. It runs before the cutoff is computed and before the sweep's +// query, so a request that fails validation never reaches storage. +// +// THE CUTOFF IS COMPUTED HERE, once, from time.Now().UTC().Add(-OlderThan), so +// every leg sweeps against the same clock read rather than each leg taking its +// own — the dolt and embedded-dolt wrappers and the unit-of-work repository +// all call this function with the request, never with a precomputed cutoff. +func ExecuteReclaimInTx(ctx context.Context, tx DBTX, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + if err := ValidateReclaimRequest(request); err != nil { + return publicops.ReclaimResult{}, err + } + cutoff := time.Now().UTC().Add(-request.OlderThan) + reclaimed, err := ReclaimExpiredLeasesInTx(ctx, tx, cutoff, request.Filter, strings.TrimSpace(request.Actor)) + if err != nil { + return publicops.ReclaimResult{}, fmt.Errorf("reclaim: %w", err) + } + if reclaimed == nil { + reclaimed = []publicops.ReclaimedLease{} + } + return publicops.ReclaimResult{Reclaimed: reclaimed}, nil +} + +// ReclaimVersionCommit composes the version-control entry for a finished sweep: +// the durable tables it changed and the message to record them under. A sweep +// that reverted nothing composes neither, which every leg's commit funnel reads +// as "write nothing to history", so a no-op reaper tick leaves no entry +// describing a no-op. +// +// It is ONE helper the three legs share so the message cannot drift between +// them: the same "bd: reclaim N expired lease(s)" the raw ReclaimExpiredLeases +// has always recorded. The staged set is the one the release records, issues +// and events, because a reclaim is the same revert applied by a clock. +func ReclaimVersionCommit(result publicops.ReclaimResult) (ChangedTables, string) { + if len(result.Reclaimed) == 0 { + return nil, "" + } + tables := ChangedTables{} + tables.Add("issues", "events") + return tables, fmt.Sprintf("bd: reclaim %d expired lease(s)", len(result.Reclaimed)) +} diff --git a/internal/storage/issueops/lease_reclaimer_test.go b/internal/storage/issueops/lease_reclaimer_test.go new file mode 100644 index 000000000..5b97b50af --- /dev/null +++ b/internal/storage/issueops/lease_reclaimer_test.go @@ -0,0 +1,107 @@ +package issueops + +import ( + "context" + "errors" + "strings" + "testing" + "time" + + "github.com/DATA-DOG/go-sqlmock" + + "github.com/steveyegge/beads/internal/types" + publicops "github.com/steveyegge/beads/issueops" +) + +// TestValidateReclaimRequestRefusesInTheDocumentedOrder pins the database-free +// tier of the role's refusals: each rule on both sides, and the cap checked +// before the blank scan, against the request as sent. +func TestValidateReclaimRequestRefusesInTheDocumentedOrder(t *testing.T) { + overCap := make([]string, publicops.MaxReclaimIDs+1) + atCap := make([]string, publicops.MaxReclaimIDs) + for i := range atCap { + atCap[i] = "bd-x" + } + for _, test := range []struct { + name string + request publicops.ReclaimRequest + wantErr bool + wantCap bool + }{ + {"a bare valid request", publicops.ReclaimRequest{Actor: "reaper"}, false, false}, + {"a zero grace window", publicops.ReclaimRequest{Actor: "reaper", OlderThan: 0}, false, false}, + {"exactly the cap", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{IDs: atCap}}, false, false}, + {"no actor", publicops.ReclaimRequest{}, true, false}, + {"a blank actor", publicops.ReclaimRequest{Actor: " "}, true, false}, + // The actor is bounded as the sweep records it: trimmed, then held to + // its column, so padding around a value at the bound is not over it. + {"an actor at the bound", publicops.ReclaimRequest{Actor: strings.Repeat("r", types.MaxFieldLen)}, false, false}, + {"a padded actor at the bound", publicops.ReclaimRequest{Actor: " " + strings.Repeat("r", types.MaxFieldLen) + " "}, false, false}, + {"an over-long actor", publicops.ReclaimRequest{Actor: strings.Repeat("r", types.MaxFieldLen+1)}, true, false}, + {"a negative grace window", publicops.ReclaimRequest{Actor: "reaper", OlderThan: -time.Nanosecond}, true, false}, + {"a blank id", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{IDs: []string{"bd-1", ""}}}, true, false}, + {"a blank assignee", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{Assignees: []string{"w", " "}}}, true, false}, + {"a blank label", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{Labels: []string{""}}}, true, false}, + {"a blank label-any", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{LabelsAny: []string{"\t"}}}, true, false}, + {"a blank exclude", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{ExcludeLabels: []string{"x", ""}}}, true, false}, + // Every entry blank AND one past the cap: the cap answers, so a caller + // that sent 1001 ids learns about the cap rather than about an entry. + {"over the cap of blanks", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{IDs: overCap}}, true, true}, + } { + t.Run(test.name, func(t *testing.T) { + err := ValidateReclaimRequest(test.request) + if (err != nil) != test.wantErr { + t.Fatalf("ValidateReclaimRequest = %v, want error %v", err, test.wantErr) + } + if err == nil { + return + } + if !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("ValidateReclaimRequest = %v, want ErrValidation", err) + } + var capErr *publicops.TooManyReclaimIDsError + if got := errors.As(err, &capErr); got != test.wantCap { + t.Fatalf("errors.As(*TooManyReclaimIDsError) = %v, want %v (err %v)", got, test.wantCap, err) + } + var fieldErr *publicops.ReclaimFieldError + if !test.wantCap && (!errors.As(err, &fieldErr) || fieldErr.Field == "") { + t.Fatalf("refusal %v names no field; every refusal but the cap is a *ReclaimFieldError", err) + } + }) + } +} + +// TestExecuteReclaimInTxRefusesBeforeAnyQuery pins "a request that fails +// validation never reaches storage": the mock expects nothing, so any query +// would fail the test. +func TestExecuteReclaimInTxRefusesBeforeAnyQuery(t *testing.T) { + db, mock, err := sqlmock.New() + if err != nil { + t.Fatalf("sqlmock.New: %v", err) + } + defer db.Close() + + _, err = ExecuteReclaimInTx(context.Background(), db, publicops.ReclaimRequest{}) + if !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("ExecuteReclaimInTx with no actor = %v, want ErrValidation", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatal(err) + } +} + +// TestReclaimVersionCommitComposesNothingForANoOp pins the shared commit +// composition: no entry for a sweep that reverted nothing, and one entry +// naming the count, staging issues and events, for one that did. +func TestReclaimVersionCommitComposesNothingForANoOp(t *testing.T) { + if tables, msg := ReclaimVersionCommit(publicops.ReclaimResult{Reclaimed: []publicops.ReclaimedLease{}}); len(tables) != 0 || msg != "" { + t.Fatalf("a no-op sweep composed tables=%v msg=%q, want neither", tables, msg) + } + tables, msg := ReclaimVersionCommit(publicops.ReclaimResult{Reclaimed: []publicops.ReclaimedLease{{ID: "a"}, {ID: "b"}}}) + if !tables["issues"] || !tables["events"] || len(tables) != 2 { + t.Fatalf("tables = %v, want issues and events", tables) + } + if !strings.Contains(msg, "reclaim 2 expired lease(s)") { + t.Fatalf("message = %q, want it to name the two reverted leases", msg) + } +} diff --git a/internal/storage/role_accessor_decorator_test.go b/internal/storage/role_accessor_decorator_test.go index aa8ce3b57..540518354 100644 --- a/internal/storage/role_accessor_decorator_test.go +++ b/internal/storage/role_accessor_decorator_test.go @@ -68,11 +68,11 @@ func roleAccessorNamesOf(surface reflect.Type) (names, unclassified []string) { } // TestEveryStoreRoleAccessorIsClassified fails when DoltStorage hands out an -// interface the census cannot place. Every one of the twenty-eight today is a +// interface the census cannot place. Every one of the twenty-nine today is a // facade role, so this costs nothing and closes the path where a role surface // grows a package and the census quietly stops covering it. (roleAccessorNames // is derived, so the count is prose and only this sentence goes stale — but it -// does go stale, and the last three sweeps found it saying twenty-seven.) +// does go stale, and the last three sweeps found it saying twenty-eight.) func TestEveryStoreRoleAccessorIsClassified(t *testing.T) { _, unclassified := roleAccessorNamesOf(reflect.TypeOf((*DoltStorage)(nil)).Elem()) for _, accessor := range unclassified { @@ -151,71 +151,73 @@ func assertRoleAccessorsAreDeclared(t *testing.T, decorator reflect.Type) { } // roleAccessorStore is a DoltStorage whose only real methods are the -// twenty-eight role accessors, each answering with a distinguishable sentinel +// twenty-nine role accessors, each answering with a distinguishable sentinel // so a test can tell a decorated surface from a passed-through one. type roleAccessorStore struct { DoltStorage - lifecycle issueops.Lifecycle - reader issueops.Reader - relations issueops.Relations - edges issueops.EdgeReader - blocking issueops.BlockingAnnotator - tree issueops.TreeWalker - graphCounter issueops.GraphCounter - batchGetter issueops.BatchGetter - counter issueops.Counter - settings issueops.WorkspaceConfig - memories memoryops.Memories - versions issueops.VersionReconciler - stats issueops.StatsReporter - cycles issueops.CycleDetector - commenter issueops.Commenter - claimer issueops.ReadyClaimer - closer issueops.BatchCloser - creator issueops.BatchCreator - editor issueops.DependencyEditor - applier issueops.BatchApplier - readyCounter issueops.ReadyCounter - querier issueops.Querier - sweeper issueops.Sweeper - deleter issueops.Deleter - bootstrapper issueops.Bootstrapper - verifier issueops.InitVerifier - metadataCAS issueops.MetadataCAS - releaser issueops.Releaser - err error + lifecycle issueops.Lifecycle + reader issueops.Reader + relations issueops.Relations + edges issueops.EdgeReader + blocking issueops.BlockingAnnotator + tree issueops.TreeWalker + graphCounter issueops.GraphCounter + batchGetter issueops.BatchGetter + counter issueops.Counter + settings issueops.WorkspaceConfig + memories memoryops.Memories + versions issueops.VersionReconciler + stats issueops.StatsReporter + cycles issueops.CycleDetector + commenter issueops.Commenter + claimer issueops.ReadyClaimer + closer issueops.BatchCloser + creator issueops.BatchCreator + editor issueops.DependencyEditor + applier issueops.BatchApplier + readyCounter issueops.ReadyCounter + querier issueops.Querier + sweeper issueops.Sweeper + deleter issueops.Deleter + bootstrapper issueops.Bootstrapper + verifier issueops.InitVerifier + metadataCAS issueops.MetadataCAS + releaser issueops.Releaser + leaseReclaimer issueops.LeaseReclaimer + err error } func newRoleAccessorStore() *roleAccessorStore { sentinel := &roleAccessorSentinel{} return &roleAccessorStore{ - memories: &memoryRoleSentinel{}, - lifecycle: sentinel, - reader: sentinel, - relations: sentinel, - edges: sentinel, - blocking: sentinel, - graphCounter: sentinel, - batchGetter: sentinel, - counter: sentinel, - settings: sentinel, - versions: sentinel, - stats: sentinel, - cycles: sentinel, - commenter: sentinel, - claimer: sentinel, - closer: sentinel, - creator: sentinel, - editor: sentinel, - applier: sentinel, - readyCounter: sentinel, - querier: sentinel, - sweeper: sentinel, - deleter: sentinel, - bootstrapper: sentinel, - verifier: sentinel, - metadataCAS: sentinel, - releaser: sentinel, + memories: &memoryRoleSentinel{}, + lifecycle: sentinel, + reader: sentinel, + relations: sentinel, + edges: sentinel, + blocking: sentinel, + graphCounter: sentinel, + batchGetter: sentinel, + counter: sentinel, + settings: sentinel, + versions: sentinel, + stats: sentinel, + cycles: sentinel, + commenter: sentinel, + claimer: sentinel, + closer: sentinel, + creator: sentinel, + editor: sentinel, + applier: sentinel, + readyCounter: sentinel, + querier: sentinel, + sweeper: sentinel, + deleter: sentinel, + bootstrapper: sentinel, + verifier: sentinel, + metadataCAS: sentinel, + releaser: sentinel, + leaseReclaimer: sentinel, } } @@ -286,8 +288,11 @@ func (s *roleAccessorStore) BatchApplier() (issueops.BatchApplier, error) { func (s *roleAccessorStore) Releaser() (issueops.Releaser, error) { return s.releaser, s.err } +func (s *roleAccessorStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return s.leaseReclaimer, s.err +} -// roleAccessorSentinel implements twenty-seven of the twenty-eight roles at +// roleAccessorSentinel implements twenty-eight of the twenty-nine roles at // once — every one but memoryops.Memories, whose List collides with // issueops.Reader.List and needs the second sentinel below. // Nothing calls its methods; identity is the whole point. @@ -412,6 +417,9 @@ func (*roleAccessorSentinel) CompareAndSetKey(context.Context, issueops.CompareA func (*roleAccessorSentinel) Release(context.Context, issueops.ReleaseRequest) (issueops.ReleaseResult, error) { return issueops.ReleaseResult{}, nil } +func (*roleAccessorSentinel) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + return issueops.ReclaimResult{}, nil +} // memoryRoleSentinel is the memory role's sentinel, and the one role // memoryRoleSentinel is the remaining role's sentinel, and the one role @@ -474,6 +482,7 @@ func TestHookFiringStoreWrapsTheWriteRolesAndPassesTheReadsThrough(t *testing.T) {"MetadataCAS", func() (any, error) { return store.MetadataCAS() }, inner.metadataCAS, true}, {"BatchApplier", func() (any, error) { return store.BatchApplier() }, inner.applier, true}, {"Releaser", func() (any, error) { return store.Releaser() }, inner.releaser, true}, + {"LeaseReclaimer", func() (any, error) { return store.LeaseReclaimer() }, inner.leaseReclaimer, true}, {"IssueReader", func() (any, error) { return store.IssueReader() }, inner.reader, false}, {"IssueRelations", func() (any, error) { return store.IssueRelations() }, inner.relations, false}, {"EdgeReader", func() (any, error) { return store.EdgeReader() }, inner.edges, false}, @@ -532,6 +541,7 @@ func TestHookFiringStoreRoleAccessorsPropagateInnerErrors(t *testing.T) { {"MetadataCAS", func() (any, error) { return store.MetadataCAS() }}, {"BatchApplier", func() (any, error) { return store.BatchApplier() }}, {"Releaser", func() (any, error) { return store.Releaser() }}, + {"LeaseReclaimer", func() (any, error) { return store.LeaseReclaimer() }}, {"IssueReader", func() (any, error) { return store.IssueReader() }}, {"IssueRelations", func() (any, error) { return store.IssueRelations() }}, {"EdgeReader", func() (any, error) { return store.EdgeReader() }}, diff --git a/internal/storage/storage.go b/internal/storage/storage.go index 8abe0b0fd..66553307f 100644 --- a/internal/storage/storage.go +++ b/internal/storage/storage.go @@ -378,6 +378,15 @@ type Storage interface { // assignee and status, which is on_update — the same event the journal // already records for it. See hook_releaser.go. Releaser() (issueops.Releaser, error) + // LeaseReclaimer returns the lease-sweep surface for this store: revert + // every lease that has been held past its deadline back to its + // pre-claim state, for a caller that is a clock rather than a human or a + // request. + // + // It is a WRITE role and its hook decorator WRAPS: a reclaim changes + // assignee and status per reverted row, which is on_update, the same + // event Releaser fires. See hook_lease_reclaimer.go. + LeaseReclaimer() (issueops.LeaseReclaimer, error) // Issue CRUD CreateIssue(ctx context.Context, issue *types.Issue, actor string) error diff --git a/internal/storage/uow/BUILD.bazel b/internal/storage/uow/BUILD.bazel index 8b4abc0d4..fed8ccd9a 100644 --- a/internal/storage/uow/BUILD.bazel +++ b/internal/storage/uow/BUILD.bazel @@ -31,6 +31,7 @@ go_library( "issue_operations.go", "issue_reader.go", "journal_maintenance.go", + "lease_reclaimer.go", "maintenance.go", "memories.go", "metadata_cas.go", @@ -128,6 +129,7 @@ go_test( "journal_contract_test.go", "journal_delete_actor_test.go", "journal_scope_test.go", + "lease_reclaimer_contract_test.go", "lifecycle_close_reopen_contract_test.go", "lifecycle_create_contract_test.go", "lifecycle_role_hooks_test.go", diff --git a/internal/storage/uow/lease_reclaimer.go b/internal/storage/uow/lease_reclaimer.go new file mode 100644 index 000000000..1cc522c6b --- /dev/null +++ b/internal/storage/uow/lease_reclaimer.go @@ -0,0 +1,60 @@ +package uow + +import ( + "context" + "fmt" + + storageissueops "github.com/steveyegge/beads/internal/storage/issueops" + publicops "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimerSource is the capability accessor a unit-of-work provider +// offers for the lease-sweep role, the sibling of DeleterSource and +// ReleaserSource. +type LeaseReclaimerSource interface { + LeaseReclaimer() (publicops.LeaseReclaimer, error) +} + +// leaseReclaimer sweeps stale leases through a unit of work. +type leaseReclaimer struct { + provider UnitOfWorkProvider +} + +// LeaseReclaimer returns the guarded lease-sweep surface for this provider. +func (p *doltSQLProvider) LeaseReclaimer() (publicops.LeaseReclaimer, error) { + return NewLeaseReclaimer(p) +} + +// NewLeaseReclaimer constructs a public LeaseReclaimer backed by provider. +func NewLeaseReclaimer(provider UnitOfWorkProvider) (publicops.LeaseReclaimer, error) { + if isNilUnitOfWorkProvider(provider) { + return nil, fmt.Errorf("new lease reclaimer: unit-of-work provider must not be nil") + } + return &leaseReclaimer{provider: provider}, nil +} + +var _ publicops.LeaseReclaimer = (*leaseReclaimer)(nil) + +// Reclaim reaches domain.IssueUseCase.Reclaim inside ONE committing unit of +// work, composing the same conditional commit message the dolt and +// embedded-dolt legs use: a sweep that reverts nothing leaves no history +// entry describing a no-op. +// +// VALIDATION HAPPENS BEFORE THE UNIT OF WORK OPENS, as on those two legs and +// for Releaser's reason: a refusal then changes nothing on the connection as +// well as on the rows. The shared body validates again inside the transaction, +// which is what keeps a leg that skipped this from answering a different +// contract. +func (l *leaseReclaimer) Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + if err := storageissueops.ValidateReclaimRequest(request); err != nil { + return publicops.ReclaimResult{}, err + } + return RunTxResult(ctx, l.provider, func(ctx context.Context, uw UnitOfWork) (publicops.ReclaimResult, string, error) { + result, err := uw.IssueUseCase().Reclaim(ctx, request) + if err != nil { + return result, "", err + } + _, msg := storageissueops.ReclaimVersionCommit(result) + return result, msg, nil + }) +} diff --git a/internal/storage/uow/lease_reclaimer_contract_test.go b/internal/storage/uow/lease_reclaimer_contract_test.go new file mode 100644 index 000000000..edcda454a --- /dev/null +++ b/internal/storage/uow/lease_reclaimer_contract_test.go @@ -0,0 +1,146 @@ +package uow + +import ( + "context" + "errors" + "fmt" + "testing" + + "github.com/steveyegge/beads/backend/conformance" + publicops "github.com/steveyegge/beads/issueops" +) + +// TestLeaseReclaimerContract runs the LeaseReclaimer contract against the +// unit-of-work provider, which reaches internal/storage/issueops. +// ExecuteReclaimInTx through domain.IssueUseCase.Reclaim inside one committing +// unit of work. +func TestLeaseReclaimerContract(t *testing.T) { + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + fixture := newUOWLeaseReclaimerFixture(t, ctx, "lrc") + + t.Run("RevertsAStaleLease", func(t *testing.T) { + conformance.RunLeaseReclaimerRevertsAStaleLease(t, ctx, fixture) + }) + t.Run("MintsARevisionThatFencesTheFormerHolder", func(t *testing.T) { + conformance.RunLeaseReclaimerMintsARevisionThatFencesTheFormerHolder(t, ctx, fixture) + }) + t.Run("LeavesALiveLeaseAlone", func(t *testing.T) { + conformance.RunLeaseReclaimerLeavesALiveLeaseAlone(t, ctx, fixture) + }) + t.Run("HonorsTheGraceWindow", func(t *testing.T) { + conformance.RunLeaseReclaimerHonorsTheGraceWindow(t, ctx, fixture) + }) + t.Run("ScopesToTheNamedIDs", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToTheNamedIDs(t, ctx, fixture) + }) + t.Run("AnswersAnAbsentIDWithAnEmptyResult", func(t *testing.T) { + conformance.RunLeaseReclaimerAnswersAnAbsentIDWithAnEmptyResult(t, ctx, fixture) + }) + t.Run("ScopesToAssigneesAndLabels", func(t *testing.T) { + conformance.RunLeaseReclaimerScopesToAssigneesAndLabels(t, ctx, fixture) + }) + t.Run("AttributesTheRecoveryEvent", func(t *testing.T) { + conformance.RunLeaseReclaimerAttributesTheRecoveryEvent(t, ctx, fixture) + }) + t.Run("FiresOnUpdateOncePerReclaimedRow", func(t *testing.T) { + conformance.RunLeaseReclaimerFiresOnUpdateOncePerReclaimedRow(t, ctx, fixture) + }) + t.Run("RefusesAMalformedRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesAMalformedRequest(t, ctx, fixture) + }) + t.Run("RefusesABlankScopeEntry", func(t *testing.T) { + conformance.RunLeaseReclaimerRefusesABlankScopeEntry(t, ctx, fixture) + }) + t.Run("AcceptsExactlyTheCap", func(t *testing.T) { + conformance.RunLeaseReclaimerAcceptsExactlyTheCap(t, ctx, fixture) + }) + t.Run("DoesNotMutateTheCallerRequest", func(t *testing.T) { + conformance.RunLeaseReclaimerDoesNotMutateTheCallerRequest(t, ctx, fixture) + }) +} + +// TestLeaseReclaimerRefusesBeforeOpeningAUOW: a malformed sweep is a +// deterministic validation failure and must not cost a database connection to +// discover, as on the dolt and embedded-dolt legs. The provider fails every +// open, so a refusal that came from inside a unit of work would be counted. +func TestLeaseReclaimerRefusesBeforeOpeningAUOW(t *testing.T) { + for _, tc := range []struct { + name string + request publicops.ReclaimRequest + }{ + {"no actor", publicops.ReclaimRequest{}}, + {"a negative grace window", publicops.ReclaimRequest{Actor: "reaper", OlderThan: -1}}, + {"a blank id", publicops.ReclaimRequest{Actor: "reaper", Filter: publicops.ReclaimFilter{IDs: []string{""}}}}, + } { + t.Run(tc.name, func(t *testing.T) { + provider := &mockUnitOfWorkProvider{newUOWErr: errors.New("unexpected unit-of-work open")} + reclaimer, err := NewLeaseReclaimer(provider) + if err != nil { + t.Fatalf("NewLeaseReclaimer: %v", err) + } + if _, err := reclaimer.Reclaim(context.Background(), tc.request); !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("err = %v, want ErrValidation", err) + } + if provider.newUOWCalls != 0 { + t.Errorf("an invalid request opened %d units of work", provider.newUOWCalls) + } + }) + } +} + +// newUOWLeaseReclaimerFixture takes every role through the provider's own +// capability accessors, and the hooked reclaimer off the notifying provider +// the proxied CLI route wraps around its provider. +func newUOWLeaseReclaimerFixture(t *testing.T, ctx context.Context, prefix string) conformance.LeaseReclaimerFixture { + t.Helper() + provider := newUOWRoleFixtureProvider(t, ctx, prefix) + source, ok := provider.(LeaseReclaimerSource) + if !ok { + t.Fatalf("provider %T does not offer the LeaseReclaimer accessor", provider) + } + reclaimer, err := source.LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer(): %v", err) + } + claimer, err := provider.(IssueClaimerSource).IssueClaimer() + if err != nil { + t.Fatalf("IssueClaimer(): %v", err) + } + lifecycle, err := provider.(IssueLifecycleSource).IssueLifecycle() + if err != nil { + t.Fatalf("IssueLifecycle(): %v", err) + } + runner, fired := conformance.NewUpdateHookLog(t) + hooked := reclaimer + if runner != nil { + notifying, ok := NewNotifyingProvider(provider, Sinks{Hook: runner}).(LeaseReclaimerSource) + if !ok { + t.Fatal("the notifying provider does not offer the LeaseReclaimer accessor") + } + if hooked, err = notifying.LeaseReclaimer(); err != nil { + t.Fatalf("hooked LeaseReclaimer(): %v", err) + } + } + kit := newUOWRoleFixtureKit(provider, prefix) + return conformance.LeaseReclaimerFixture{ + IssuePrefix: kit.IssuePrefix, + LeaseReclaimer: reclaimer, + CreateIssue: kit.CreateIssue, + Claimer: claimer, + Lifecycle: lifecycle, + QueryScalar: kit.QueryScalar, + Exec: func(ctx context.Context, statements []conformance.SQLStatement) error { + return RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) { + for _, stmt := range statements { + if _, err := uw.RawSQLUseCase().Exec(ctx, stmt.Query, stmt.Args...); err != nil { + return "", fmt.Errorf("%s: %w", stmt.Query, err) + } + } + return "seed lease reclaimer rows", nil + }) + }, + HookedLeaseReclaimer: hooked, + FiredUpdates: fired, + } +} diff --git a/internal/storage/uow/notifying.go b/internal/storage/uow/notifying.go index f1e9930e5..53da8a3b9 100644 --- a/internal/storage/uow/notifying.go +++ b/internal/storage/uow/notifying.go @@ -309,6 +309,10 @@ func (p *notifyingProvider) MetadataCAS() (publicops.MetadataCAS, error) { retur func (p *notifyingProvider) Releaser() (publicops.Releaser, error) { return NewReleaser(p) } +func (p *notifyingProvider) LeaseReclaimer() (publicops.LeaseReclaimer, error) { + return NewLeaseReclaimer(p) +} + func (p *notifyingProvider) Memories() (memoryops.Memories, error) { return NewMemories(p) } // EventsJournalCursor builds on THIS provider, like every role above it, so a @@ -423,6 +427,7 @@ var ( _ WorkspaceConfigSource = (*notifyingProvider)(nil) _ VersionReconcilerSource = (*notifyingProvider)(nil) _ MemoriesSource = (*notifyingProvider)(nil) + _ LeaseReclaimerSource = (*notifyingProvider)(nil) _ EventsJournalCursorSource = (*notifyingProvider)(nil) ) @@ -1008,6 +1013,25 @@ func (u *recordingIssueUC) ReleaseIssue(ctx context.Context, req publicops.Relea return result, wrote, nil } +// Reclaim records an update for every lease the sweep reverted. +// +// It is DECLARED for the same reason ReleaseIssue above is: an inherited +// accessor would compile and record nothing, silently dropping the hook the +// DoltStorage chain fires once per reverted row. +// +// A sweep reverts only issues, never wisps, so the snapshot is the issues- +// plane snap.issue rather than anyPlane. +func (u *recordingIssueUC) Reclaim(ctx context.Context, request publicops.ReclaimRequest) (publicops.ReclaimResult, error) { + result, err := u.IssueUseCase.Reclaim(ctx, request) + if err != nil { + return result, err + } + for _, r := range result.Reclaimed { + u.rec.record(opUpdate, u.snap.issue(ctx, r.ID)) + } + return result, nil +} + func (u *recordingIssueUC) UpdateWisp(ctx context.Context, id string, updates map[string]any, actor string) error { if err := u.IssueUseCase.UpdateWisp(ctx, id, updates, actor); err != nil { return err diff --git a/internal/telemetry/BUILD.bazel b/internal/telemetry/BUILD.bazel index 90792516b..ffcf6b7d5 100644 --- a/internal/telemetry/BUILD.bazel +++ b/internal/telemetry/BUILD.bazel @@ -20,6 +20,7 @@ go_library( "issue_claimer.go", "issue_operations.go", "issue_reader.go", + "lease_reclaimer.go", "memories.go", "metadata_cas.go", "querier.go", diff --git a/internal/telemetry/lease_reclaimer.go b/internal/telemetry/lease_reclaimer.go new file mode 100644 index 000000000..cf61d8356 --- /dev/null +++ b/internal/telemetry/lease_reclaimer.go @@ -0,0 +1,36 @@ +package telemetry + +import ( + "context" + + "github.com/steveyegge/beads/issueops" +) + +// LeaseReclaimer returns the inner store's lease-sweep surface wrapped in +// this layer's instrumentation. It recurses instead of delegating: a blind +// delegation would return the inner surface unspanned and untimed. +func (s *InstrumentedStorage) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + inner, err := s.Unwrap().LeaseReclaimer() + if err != nil { + return nil, err + } + return s.WrapLeaseReclaimer(inner), nil +} + +// WrapLeaseReclaimer instruments the lease sweep with this storage layer's +// existing telemetry meter and tracer. +func (s *InstrumentedStorage) WrapLeaseReclaimer(inner issueops.LeaseReclaimer) issueops.LeaseReclaimer { + return &instrumentedLeaseReclaimer{storage: s, inner: inner} +} + +type instrumentedLeaseReclaimer struct { + storage *InstrumentedStorage + inner issueops.LeaseReclaimer +} + +func (c *instrumentedLeaseReclaimer) Reclaim(ctx context.Context, request issueops.ReclaimRequest) (result issueops.ReclaimResult, err error) { + ctx, span, started := c.storage.op(ctx, "LeaseReclaimer.Reclaim") + result, err = c.inner.Reclaim(ctx, request) + c.storage.done(ctx, span, started, err) + return result, err +} diff --git a/internal/telemetry/role_accessor_decorator_test.go b/internal/telemetry/role_accessor_decorator_test.go index e7c9b7c9c..24a2d953b 100644 --- a/internal/telemetry/role_accessor_decorator_test.go +++ b/internal/telemetry/role_accessor_decorator_test.go @@ -94,7 +94,7 @@ func TestInstrumentedStorageDeclaresEveryRoleAccessor(t *testing.T) { } // roleAccessorStore is a DoltStorage whose only real methods are the -// twenty-eight role accessors, each answering with a distinguishable sentinel +// twenty-nine role accessors, each answering with a distinguishable sentinel // so a test can tell an instrumented surface from a passed-through one. // // TWO sentinels rather than one: memoryops.Memories.List and issueops.Reader.List @@ -175,8 +175,11 @@ func (s *roleAccessorStore) BatchApplier() (issueops.BatchApplier, error) { func (s *roleAccessorStore) Releaser() (issueops.Releaser, error) { return s.surface, s.err } +func (s *roleAccessorStore) LeaseReclaimer() (issueops.LeaseReclaimer, error) { + return s.surface, s.err +} -// roleAccessorSentinel implements twenty-seven of the twenty-eight roles at +// roleAccessorSentinel implements twenty-eight of the twenty-nine roles at // once — every one but memoryops.Memories, whose List collides with // issueops.Reader.List and needs the second sentinel below. // Nothing calls its methods; identity is the whole point. @@ -306,6 +309,9 @@ func (*roleAccessorSentinel) CompareAndSetKey(context.Context, issueops.CompareA func (*roleAccessorSentinel) Release(context.Context, issueops.ReleaseRequest) (issueops.ReleaseResult, error) { return issueops.ReleaseResult{}, nil } +func (*roleAccessorSentinel) Reclaim(context.Context, issueops.ReclaimRequest) (issueops.ReclaimResult, error) { + return issueops.ReclaimResult{}, nil +} // memoryRoleSentinel is the memory role's sentinel — see // memoryRoleSentinel is the remaining role's sentinel — see @@ -382,6 +388,7 @@ func TestInstrumentedStorageInstrumentsEveryRoleAccessor(t *testing.T) { {"MetadataCAS", func() (any, error) { return wrapped.MetadataCAS() }, sentinel}, {"BatchApplier", func() (any, error) { return wrapped.BatchApplier() }, sentinel}, {"Releaser", func() (any, error) { return wrapped.Releaser() }, sentinel}, + {"LeaseReclaimer", func() (any, error) { return wrapped.LeaseReclaimer() }, sentinel}, } { t.Run(test.name, func(t *testing.T) { surface, err := test.got() @@ -468,6 +475,7 @@ func TestInstrumentedStorageRoleAccessorsPropagateInnerErrors(t *testing.T) { {"MetadataCAS", func() (any, error) { return wrapped.MetadataCAS() }}, {"BatchApplier", func() (any, error) { return wrapped.BatchApplier() }}, {"Releaser", func() (any, error) { return wrapped.Releaser() }}, + {"LeaseReclaimer", func() (any, error) { return wrapped.LeaseReclaimer() }}, } { t.Run(test.name, func(t *testing.T) { surface, err := test.got() diff --git a/internal/types/types.go b/internal/types/types.go index 12f3777e0..dffc85811 100644 --- a/internal/types/types.go +++ b/internal/types/types.go @@ -2380,6 +2380,13 @@ func (s SortPolicy) IsValid() bool { type ReclaimedLease struct { ID string `json:"id"` PreviousOwner string `json:"previous_owner"` + + // Revision is the fresh RevisionToken minted for this issue's row by the + // same UPDATE that reverted its lease — the one this package's + // RevisionToken/ParseRevisionToken pair encodes, the same wire shape as + // IssueDetails.Revision. A caller that re-fetches after a reclaim sees a + // different token than one that raced it. + Revision string `json:"revision"` } // ReclaimFilter scopes which stale-lease issues bd reclaim may revert. The diff --git a/issue_roles_external_test.go b/issue_roles_external_test.go index b6e6dd38a..c9fde9947 100644 --- a/issue_roles_external_test.go +++ b/issue_roles_external_test.go @@ -238,6 +238,39 @@ func TestReleaserExposesTypedUnsupportedError(t *testing.T) { } } +// TestLeaseReclaimerLayersHooksOutsideTelemetry is the same pin for the +// lease-sweep role. It gives the same answer the release does and for the +// same reason: a reclaim moves assignee and status per reverted row, which +// is on_update — a hook the vocabulary publishes and a row a script can +// still read back — so this decorator wraps rather than recursing the way +// Sweeper and Deleter do. +func TestLeaseReclaimerLayersHooksOutsideTelemetry(t *testing.T) { + t.Setenv("BD_OTEL_STDOUT", "true") + instrumented, ok := telemetry.WrapStorage(&dolt.DoltStore{}).(*telemetry.InstrumentedStorage) + if !ok { + t.Fatal("WrapStorage() did not create InstrumentedStorage") + } + + reclaimer, err := storage.NewHookFiringStore(instrumented, nil).LeaseReclaimer() + if err != nil { + t.Fatalf("LeaseReclaimer() error = %v", err) + } + if got := reflect.TypeOf(reclaimer).String(); got != "*storage.hookLeaseReclaimer" { + t.Fatalf("outer layer = %s, want the hook wrapper", got) + } +} + +func TestLeaseReclaimerExposesTypedUnsupportedError(t *testing.T) { + reclaimer, err := (*dolt.DoltStore)(nil).LeaseReclaimer() + if reclaimer != nil { + t.Fatalf("LeaseReclaimer() reclaimer = %T, want nil", reclaimer) + } + var unsupported *beads.ErrUnsupported + if !errors.As(err, &unsupported) { + t.Fatalf("LeaseReclaimer() error = %v, want *beads.ErrUnsupported", err) + } +} + // TestIssueRelationsKeepsTelemetryOutermost is the READ role's version of the // pin, and it is deliberately the other answer: the hook decorator adds no // layer to a read, so the outermost thing a caller gets is the instrumented diff --git a/issueops/BUILD.bazel b/issueops/BUILD.bazel index dc28cd166..47fa9a549 100644 --- a/issueops/BUILD.bazel +++ b/issueops/BUILD.bazel @@ -22,6 +22,7 @@ go_library( "importer.go", "initverifier.go", "issueops.go", + "leasereclaimer.go", "metadatacas.go", "querier.go", "reader.go", diff --git a/issueops/leasereclaimer.go b/issueops/leasereclaimer.go new file mode 100644 index 000000000..4c63f5a42 --- /dev/null +++ b/issueops/leasereclaimer.go @@ -0,0 +1,183 @@ +package issueops + +import ( + "context" + "fmt" + "time" + + "github.com/steveyegge/beads/internal/types" +) + +// ReclaimFilter scopes which stale-lease issues a reclaim may revert. See +// types.ReclaimFilter: this is the same value, aliased into this package the +// way GetManyRequest's sibling roles alias their types.* shapes. +type ReclaimFilter = types.ReclaimFilter + +// ReclaimedLease names one issue whose stale lease a reclaim reverted, plus +// the owner it took the lease from and the fresh revision the revert minted. +// See types.ReclaimedLease. +type ReclaimedLease = types.ReclaimedLease + +// MaxReclaimIDs is the largest ReclaimFilter.IDs this role accepts in one +// call. It is the same number and the same reasoning as MaxGetManyIDs: a flat +// bound on the request rather than a cost model over the sweep it scopes, so +// one call cannot hold the revert transaction open over an unbounded id list. +const MaxReclaimIDs = 1000 + +// TooManyReclaimIDsError reports that a ReclaimRequest's Filter.IDs named more +// ids than this role accepts in one call. It wraps ErrValidation, matching +// TooManyIDsError's shape for the same reason: a caller that only checks for a +// usable request classifies it correctly without knowing this type exists. +type TooManyReclaimIDsError struct { + // Requested is the number of ids Filter.IDs named, counted before + // deduplication. + Requested int + // Cap is MaxReclaimIDs, carried on the error so a caller does not need the + // constant in scope to report what it hit. + Cap int +} + +func (e *TooManyReclaimIDsError) Error() string { + return fmt.Sprintf("reclaim accepts at most %d scoped ids, got %d", e.Cap, e.Requested) +} + +// Unwrap makes TooManyReclaimIDsError match ErrValidation. +func (e *TooManyReclaimIDsError) Unwrap() error { return ErrValidation } + +// The ReclaimRequest fields a ReclaimFieldError can name. +const ( + ReclaimFieldActor = "actor" + ReclaimFieldOlderThan = "older_than" + ReclaimFieldIDs = "ids" + ReclaimFieldAssignees = "assignees" + ReclaimFieldLabels = "labels" + ReclaimFieldLabelsAny = "labels_any" + ReclaimFieldExcludeLabels = "exclude_labels" +) + +// ReclaimFieldError reports a ReclaimRequest refusal and the one field it is +// about, so a front door can name that field without parsing the message. It +// wraps ErrValidation. The id cap is the one refusal that is not one of these: +// it is a *TooManyReclaimIDsError, always about ReclaimFieldIDs. +type ReclaimFieldError struct { + // Field is one of the ReclaimField* constants. + Field string + // Detail is the sentence the error reads as. + Detail string +} + +func (e *ReclaimFieldError) Error() string { return e.Detail } + +// Unwrap makes ReclaimFieldError match ErrValidation. +func (e *ReclaimFieldError) Unwrap() error { return ErrValidation } + +// ReclaimRequest describes one sweep of stale leases — the shape behind +// `bd reclaim`. +// +// Implementations never mutate caller-owned request values: Filter's slices +// are read, never written through. +type ReclaimRequest struct { + // Actor attributes every reverted row's recovery event and its journaled + // update. It is REQUIRED — an empty or all-blank Actor is ErrValidation — + // for the same reason Releaser.Release requires one: a reclaim is the + // moment work stops being owned, and the one question asked of a reverted + // row's history afterwards is who ran the sweep that freed it. It is + // recorded trimmed, and one longer than types.MaxFieldLen characters once + // trimmed is ErrValidation too: the column it is recorded in holds no more. + Actor string + // OlderThan is the grace period past a lease's own TTL: only a lease whose + // lease_expires_at is more than OlderThan in the past is eligible. It must + // not be negative — a negative grace reaches into the future and reclaims + // leases that have not yet expired by their own TTL, which is not a wider + // sweep, it is a different operation this role does not perform. + OlderThan time.Duration + // Filter narrows which stale-lease issues are eligible; the zero value + // reclaims every stale lease workspace-wide. See types.ReclaimFilter for + // the scope fields (ids, assignees, labels) and the AnyReplica escape + // hatch — none of that is restated here, and this role changes none of it. + // + // FILTER.IDS IS NOT AN ERROR WHEN IT NAMES A NON-STALE OR UNKNOWN ISSUE. + // Like GetManyRequest.IDs, an id that is not currently a stale lease + // simply does not appear in ReclaimResult.Reclaimed — scoping by id + // narrows a sweep, it does not assert that every named id was reclaimable. + // + // MORE THAN MaxReclaimIDs ENTRIES IN Filter.IDs IS A REFUSAL: a + // *TooManyReclaimIDsError, counted as sent, before deduplication. + Filter ReclaimFilter +} + +// ReclaimResult reports every issue a reclaim reverted, in the order the sweep +// found them. +type ReclaimResult struct { + // Reclaimed names every issue the sweep reverted, together with the owner + // each lease was taken from and the fresh revision the revert minted on + // each row. Never nil for a successful call, even when it is empty: a + // sweep that finds nothing stale answers with an empty, non-nil slice, the + // same promise GetManyResult.Issues makes. + Reclaimed []ReclaimedLease +} + +// LeaseReclaimer sweeps stale leases back to ready — the capability behind +// `bd reclaim` — and, like every other capability here, a role with its own +// accessor. +// +// IT IS A DIFFERENT QUESTION FROM Releaser. Releaser gives up ONE claim a +// caller names, on the claim's own say-so (Actor, or Force). This sweeps +// MANY leases that have gone stale by a clock, on nobody's say-so but the +// clock's — a reaper calls this, not the agent that held the work. A surface +// carrying both would let a caller that may only report its own abandoned +// work reach into every other holder's, which is the same split Releaser's +// own doc draws against Claimer. +// +// WRITES, AND ITS HOOK IS on_update, once per reverted row. A reclaim clears +// assignee and status on each row it reverts, which is an update and is +// already how the journal classifies it — see Releaser's identical +// reasoning. There is no on_reclaim to fire and inventing one is not this +// role's job. +// +// Reclaim only ever touches the permanent issues table: wisps are ephemeral +// and are never leased work (see internal/storage/issueops. +// ReclaimExpiredLeasesInTx). The replica guard ReclaimFilter.AnyReplica +// documents is unchanged by this role — it is enforced in the shared body +// every library leg calls through. +// +// EVERY LEG RUNS ONE BODY: dolt, embedded dolt and the unit-of-work provider +// wrap internal/storage/issueops.ExecuteReclaimInTx, and the HTTP client leg +// dials POST /v0/beads/issues:reclaim (capability issues.reclaim), behind which +// bd serve runs the same body. `bd reclaim` runs this role on both of its +// routes. Each leg records one version-control entry per sweep that reverted +// anything, and honors a deferred version commit on the context. +// +// Deterministic request-validation failures match ErrValidation. Result +// values are unspecified when error is non-nil, and every refusal below +// leaves persistent state unchanged. +type LeaseReclaimer interface { + // Reclaim sweeps every stale lease ReclaimRequest.Filter admits whose + // lease_expires_at is more than OlderThan in the past, and reports every + // issue it reverted. + // + // IT IS ONE TRANSACTION over the whole sweep: the snapshot that decides + // which leases are stale, every row's revert, and the revision each revert + // mints all see one write. A lease a concurrent heartbeat rescues between + // the snapshot and its row's revert is simply skipped — it never appears + // in Reclaimed, and is not an error. + // + // REFUSALS: + // + // - an empty or all-blank Actor, or one longer than types.MaxFieldLen + // characters once trimmed: ErrValidation, before anything is read; + // - a negative OlderThan: ErrValidation; + // - more than MaxReclaimIDs entries in Filter.IDs, or an empty-string + // entry among them: *TooManyReclaimIDsError or ErrValidation, + // respectively, before anything is read; + // - an empty or all-blank entry in Filter.Assignees, Labels, LabelsAny or + // ExcludeLabels: ErrValidation, because it would match nothing a + // caller meant to name. + // + // Every ErrValidation refusal but the cap is a *ReclaimFieldError naming + // its field. + // + // A sweep that finds no stale leases is not a refusal: it answers an empty, + // non-nil Reclaimed. + Reclaim(ctx context.Context, request ReclaimRequest) (ReclaimResult, error) +}