beads/cmd/bd/reclaim.go
Julian Knutsen 470ec832b5
feat(roles): LeaseReclaimer role, POST issues:reclaim, and bd reclaim on both routes (S9) (#7412)
* feat(roles): LeaseReclaimer role and POST issues:reclaim (S9)

Adds issueops.LeaseReclaimer.Reclaim(ReclaimRequest{Actor, OlderThan,
Filter}) -> ReclaimResult{Reclaimed []ReclaimedLease{ID, PreviousOwner,
Revision}}, the stale-lease sweep behind `bd reclaim`, on every leg:

- dolt, embedded dolt and the unit-of-work provider all run one body,
  internal/storage/issueops.ExecuteReclaimInTx over
  ReclaimExpiredLeasesInTx, and record one version commit per sweep that
  reverted anything through the issue-operation funnel (so a deferred
  version commit on the context is honored);
- each reverted row's fresh row_lock is published as its revision, so a
  former holder's guarded write fails with ErrVersionMismatch;
- an id that is not a stale lease (absent, live, unheld) is left out of
  Reclaimed, never an error; the id scope is capped at 1000
  (*TooManyReclaimIDsError, counted as sent);
- the hook decorator fires on_update once per reverted row (and the
  proxied notifying provider records one update per row); telemetry
  wraps it like every role.

bd serve publishes it as POST /v0/beads/issues:reclaim behind the new
issues.reclaim capability (httpapi.Config.LeaseReclaimer, required like
every role). Spec, apigen and the wire-shape golden are regenerated;
the change is additive and wire_revision is unchanged. The HTTP client
leg dials it (wire vocabulary, capability preflight, local validation
in the role's order).

Conformance: RunLeaseReclaimer* (12 cases: revert post-state, the
revision race, live lease, grace window, id scope with absent/live ids,
empty answer, assignee/label scopes re-homed from reclaim_scope.go,
recovery event attribution, on_update once per row, refusals, the cap
boundary, no request mutation) wired on dolt, embeddeddolt, uow and the
served http leg, and so held by the leg lock. Handler tests cover the
member mapping, the projection, every handler refusal by name and the
role's refusals reaching the wire.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(bd): bd reclaim runs the LeaseReclaimer role on both routes (S9)

`bd reclaim` takes issueops.LeaseReclaimer off the decorated store on the
direct route (so a connected HTTP workspace dials issues:reclaim instead
of refusing) and off the proxied provider's own accessor on the proxied
route, replacing the raw ReclaimExpiredLeases calls and
runReclaimProxiedServer. Visible changes: --json entries carry the
reverted row's new `revision`, the workspace's on_update hook fires once
per reverted row, and the version commit is the role's own
("bd: reclaim N expired lease(s)", honoring dolt.auto-commit through the
context the way bd prune does).

Tests: an embedded direct-route CLI test (scoped sweep with a live and an
absent id, revision, one version commit, on_update once) and the proxied
reclaim integration test now asserts the revision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(roles): S9 review LOWs: one validator, field attribution, overflow bound

- The library's ValidateReclaimRequest is the one definition of a valid
  reclaim: it now also refuses a blank assignee/label/label-any/exclude
  entry, and every refusal but the id cap is a *issueops.ReclaimFieldError
  naming its field. The httpapi handler no longer checks blank entries
  itself; failReclaimErr names the member the role's error is about
  (`ids` for the cap) instead of blaming `ids` for every validation. The
  HTTP client's restatement matches, pinned field by field.
- older_than_seconds is refused at >= MaxInt64/1e9: that float rounds to
  2^63 ns, which wrapped negative under the old > check.
- New contract case RunLeaseReclaimerRefusesABlankScopeEntry on all four
  legs; handler tests for field attribution and the overflow boundary.
- CHANGELOG names the 1000-id cap on bd reclaim --id and is rewrapped; the
  client's grace-window precision comment no longer claims exactness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(roles): S9 review: regenerate the unsupported shell, cgo-tag the embedded contract, one actor spelling

- go generate ./internal/httpclient: the new accessor moves the partial-shell
  trailer to 145 of 200.
- lease_reclaimer_contract_test.go gets //go:build cgo like its siblings, so
  the pure-Go lane compiles.
- The library trims the actor before attributing, matching the handler, so
  " alice " is attributed the same on every route.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(roles): S9 review minors: bound the reclaim actor on every leg, pin any_replica, validate before the uow

Review minors on #7412, each in code this PR adds.

- The reclaim actor lands on every reverted row's recovery event, in a
  VARCHAR(255) column, but ValidateReclaimRequest never bounded it: a
  256-character actor passed validation on the dolt, embedded-dolt and
  unit-of-work legs and failed the sweep with the backend's own
  "Error 1105: string '...' is too large for column 'actor'" on the first
  row it would have reverted (all three legs answer exactly that with the
  bound below removed). The shared validator now holds the trimmed actor to
  types.MaxFieldLen, as claimIssueInTx does, so it is a *ReclaimFieldError
  for `actor` on every leg. The HTTP client restates the rule, and the parity
  test covers both sides of it. bd serve already refused it through the
  claim's name rules. The role's Actor and REFUSALS docs name the bound, and
  the conformance case runs it on all four legs.
- ValidateReclaimRequest's doc no longer says bd serve's handler leaves every
  rule to it: it names the three wire rules the handler adds of its own.
- TestReclaimSendsAnyReplicaOnlyWhenAsked pins the one filter member that
  widens a sweep: the client sends any_replica only when the caller set it.
- The unit-of-work leg validates before opening its unit of work, as the dolt
  and embedded-dolt legs do; TestLeaseReclaimerRefusesBeforeOpeningAUOW pins
  it.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 14:56:32 +00:00

271 lines
11 KiB
Go
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

package main
import (
"errors"
"fmt"
"strings"
"github.com/spf13/cobra"
"github.com/spf13/pflag"
"github.com/steveyegge/beads/internal/metrics"
storeops "github.com/steveyegge/beads/internal/storage/issueops"
"github.com/steveyegge/beads/internal/storage/uow"
"github.com/steveyegge/beads/internal/types"
"github.com/steveyegge/beads/internal/ui"
"github.com/steveyegge/beads/issueops"
)
var reclaimCmd = &cobra.Command{
Use: "reclaim",
GroupID: "issues",
Short: "Revert stale-lease in_progress issues back to ready (dead-worker recovery)",
Long: `Revert in_progress issues whose lease has gone stale back to ready.
When a worker claims an issue it takes a lease that expires after a TTL, kept
alive by 'bd heartbeat'. A worker that dies stops heartbeating, so its lease
expires and its issue would otherwise stay in_progress forever. reclaim is the
reaper: it finds in_progress issues whose lease expired more than --older-than
ago, clears the assignee, and sets them back to open so another worker can
claim them. The previous owner's stale lease is recorded as a recovery event.
--older-than is a grace window past lease expiry: only leases that expired at
least this long ago are reclaimed, so a worker briefly paused (GC, clock skew)
is not robbed of live work. Run it from a supervisor on a timer with a window
of roughly 2× the claim TTL.
By default reclaim covers every stale lease THIS replica granted. The scope
filters below narrow it further, using the same label surface claiming is
scoped by (--label / --label-any / --exclude-label), plus --assignee and --id.
Filters AND-combine and never widen the set: a reclaimed lease must still be
stale.
Replicas and leases (federated deployments)
-------------------------------------------
A lease is only meaningful on the replica that granted it. Every other
replica's view of the holder's liveness is stale by up to one sync interval,
so a reaper elsewhere can revert a unit that is very much alive over there.
reclaim therefore records the granting replica on each lease and SKIPS a lease
another replica granted, summarizing what it declined on stderr (one line per
run; 'bd -v' expands it to the first 20 leases individually). Reap it where it
was granted; use --any-replica only when that replica is permanently gone (or
when this node was renamed and its own old leases now look foreign — an
ordinary heartbeat keeps a lease alive but does not re-home it to the node
heartbeating it). Prefer the narrow form '--any-replica --id <id>': bare
--any-replica reverts EVERY foreign stale lease, live peers included.
Two invariants the guard cannot enforce for you:
grace window > sync interval, and lease TTL > sync interval.
A TTL or grace shorter than the cadence at which replicas exchange state is
meaningless across the bridge — the remote view is a full interval old by
construction. Raise the TTL/grace above the sync interval, never the reverse.
The guard is opt-in: set BEADS_NODE_ID, or run 'bd config set node_id <name>'
(which writes the per-machine ~/.config/bd/config.yaml — never commit a node_id
to the git-tracked .beads/config.yaml, or every clone reads the same name and
the guard goes armed-but-inert). One id per STORE, not per host: machines that
are clients of the same dolt sql-server are ONE replica and must share one value
or leave it unset. There is no hostname fallback — the hostname names the client
process's machine, not the store — so an unnamed deployment keeps the old,
unguarded behavior instead of stranding its own work.
Examples:
bd reclaim # default grace window (2× the lease TTL)
bd reclaim --older-than 10m # reclaim leases expired >10m ago
bd reclaim --older-than 0s # reclaim every currently-expired lease
bd reclaim --label lane-a # only this machine's claim partition
bd reclaim --label-any lane-a,lane-b --exclude-label pinned
bd reclaim --assignee zelda --assignee epona # only these workers' leases
bd reclaim --id wy-abc --id wy-def # exactly these issues
bd reclaim --any-replica # also reap leases granted by a departed replica`,
Args: cobra.NoArgs,
SilenceUsage: true,
SilenceErrors: true,
RunE: func(cmd *cobra.Command, args []string) error {
evt := metrics.NewCommandEvent("reclaim")
defer func() {
if c := metrics.Global(); c != nil {
c.CloseEventAndAdd(evt)
}
}()
olderThan, _ := cmd.Flags().GetDuration("older-than")
if olderThan < 0 {
return HandleErrorRespectJSON("--older-than must not be negative")
}
filter, err := reclaimFilterFromFlags(cmd)
if err != nil {
return HandleErrorRespectJSON("%v", err)
}
CheckReadonly("reclaim")
reclaimer, err := openLeaseReclaimer()
if err != nil {
return HandleErrorRespectJSON("reclaim: %v", err)
}
// The role mints the sweep's version commit itself, so batch and off
// modes have to be said on the CONTEXT, the way `bd prune` says them.
// The proxied route applied the policy to rootCtx in the root pre-run
// already, and issueOpsContext leaves a context that defers alone.
ctx := rootCtx
if !usesProxiedServer() {
if ctx, err = issueOpsContext(rootCtx); err != nil {
return HandleErrorRespectJSON("%v", err)
}
}
result, err := reclaimer.Reclaim(ctx, issueops.ReclaimRequest{
Actor: currentActor(),
OlderThan: olderThan,
Filter: filter,
})
if err != nil {
return HandleErrorRespectJSON("reclaim: %v", err)
}
if len(result.Reclaimed) > 0 {
commandDidWrite.Store(true)
}
return renderReclaim(result.Reclaimed, !filter.IsEmpty())
},
}
// openLeaseReclaimer hands back the stale-lease sweep for whichever route this
// invocation is on: the store's own accessor (through every decorator the store
// was wired with, so the workspace's on_update hook fires once per reverted
// row, and over a registered HTTP backend the wire's reclaimIssues), or the
// proxied-server provider's.
func openLeaseReclaimer() (issueops.LeaseReclaimer, error) {
if usesProxiedServer() {
return proxiedLeaseReclaimer()
}
if store == nil {
if err := ensureStoreActive(); err != nil {
return nil, err
}
}
return store.LeaseReclaimer()
}
// proxiedLeaseReclaimer takes the sweep through the proxied-server provider's
// OWN capability accessor, proxiedSweeper's two-step and for its reason: the
// accessor is where each layer (the notifying provider's hooks included) is
// added.
func proxiedLeaseReclaimer() (issueops.LeaseReclaimer, error) {
if uowProvider == nil {
return nil, errors.New("proxied-server UOW provider not initialized")
}
src, ok := uowProvider.(uow.LeaseReclaimerSource)
if !ok {
return nil, fmt.Errorf("proxied-server provider %T does not offer the lease-sweep surface", uowProvider)
}
return src.LeaseReclaimer()
}
func renderReclaim(reclaimed []types.ReclaimedLease, scoped bool) error {
if jsonOutput {
return outputJSON(map[string]interface{}{
"reclaimed": reclaimed,
"count": len(reclaimed),
// Whether any scope filter was in effect, so a supervisor auditing
// its own reclaim log can tell a scoped sweep from a global one.
"scoped": scoped,
})
}
if len(reclaimed) == 0 {
if scoped {
fmt.Printf("%s No stale leases to reclaim in the filtered scope\n", ui.RenderPass("✓"))
return nil
}
fmt.Printf("%s No stale leases to reclaim\n", ui.RenderPass("✓"))
return nil
}
fmt.Printf("%s Reclaimed %d stale-lease issue(s):\n", ui.RenderPass("✓"), len(reclaimed))
for _, r := range reclaimed {
owner := r.PreviousOwner
if owner == "" {
owner = "(unassigned)"
}
fmt.Printf(" %s (was held by %s)\n", r.ID, owner)
}
return nil
}
// registerReclaimScopeFlags declares the scope filters — the same label surface
// as the claim side (bd ready --claim), so a fleet can reclaim exactly the
// partition it claims from. Split out of init() so tests can exercise the real
// flag set on a fresh command instead of a hand-copied one.
func registerReclaimScopeFlags(fs *pflag.FlagSet) {
fs.StringSlice("id", nil, "Only reclaim these issue IDs (repeatable)")
fs.StringSliceP("assignee", "a", nil, "Only reclaim leases held by these assignees (repeatable)")
fs.StringSliceP("label", "l", nil, "Only reclaim issues with ALL these labels (AND). Can combine with --label-any")
fs.StringSlice("label-any", nil, "Only reclaim issues with AT LEAST ONE of these labels (OR). Can combine with --label")
fs.StringSlice("exclude-label", nil, "Never reclaim issues carrying ANY of these labels")
fs.Bool("any-replica", false,
"Also reclaim leases granted by ANOTHER replica (unsafe unless that replica is gone; see 'Replicas and leases')")
}
// reclaimFilterFromFlags maps the scope flags onto a types.ReclaimFilter,
// rejecting a flag that was SUPPLIED but carries no usable value.
//
// The hard error is the point. `bd reclaim --label "$LANE"` with LANE unset
// parses to an empty slice, which would otherwise be indistinguishable from
// "no --label at all" — i.e. a supervisor's scoped sweep silently degrading
// into a global one that reaps every stale lease in a federated database. A
// scope flag that resolves to nothing is operator error, not a wildcard.
func reclaimFilterFromFlags(cmd *cobra.Command) (types.ReclaimFilter, error) {
get := func(name string) ([]string, error) {
values, err := cmd.Flags().GetStringSlice(name)
if err != nil {
return nil, fmt.Errorf("--%s: %w", name, err)
}
if !cmd.Flags().Changed(name) {
return nil, nil
}
var kept []string
for _, v := range values {
if strings.TrimSpace(v) != "" {
kept = append(kept, v)
}
}
if len(kept) == 0 {
return nil, fmt.Errorf("--%s was given no usable value (an empty scope flag would reclaim everything; drop the flag to sweep globally)", name)
}
return kept, nil
}
var filter types.ReclaimFilter
var err error
if filter.IDs, err = get("id"); err != nil {
return types.ReclaimFilter{}, err
}
if filter.Assignees, err = get("assignee"); err != nil {
return types.ReclaimFilter{}, err
}
if filter.Labels, err = get("label"); err != nil {
return types.ReclaimFilter{}, err
}
if filter.LabelsAny, err = get("label-any"); err != nil {
return types.ReclaimFilter{}, err
}
if filter.ExcludeLabels, err = get("exclude-label"); err != nil {
return types.ReclaimFilter{}, err
}
// --any-replica is an override, not a scope: it WIDENS the set past the
// granting-replica guard, so it deliberately skips the empty-value hard
// error above (a bool flag has no empty-value hazard) and never counts
// toward "scoped" in the reclaim report.
if filter.AnyReplica, err = cmd.Flags().GetBool("any-replica"); err != nil {
return types.ReclaimFilter{}, fmt.Errorf("--any-replica: %w", err)
}
return filter, nil
}
func init() {
reclaimCmd.Flags().Duration("older-than", 2*storeops.DefaultLeaseTTL,
"Only reclaim leases that expired at least this long ago (grace window)")
registerReclaimScopeFlags(reclaimCmd.Flags())
rootCmd.AddCommand(reclaimCmd)
}