* feat(purge): live-dependent protection, --wisps-plane selector, hour-precision --older-than An orchestrator's wisp retention step deletes every closed row in the wisps table older than N hours, except a closed wisp that an open/hooked/in_progress wisp depends on through parent-child, tracks or blocks. `bd purge` could not express that: it selected the ephemeral TIER (so closed --no-history beads, which live in the wisps table, were left behind), it deleted a closed molecule root whose step was still open, and it floored --older-than hours to whole days (36h swept rows 24h old). All three land together so `bd purge --wisps-plane --older-than 168h --force` covers that step. Live-dependent protection (issueops.SweepRequest.ProtectLiveDependents, SweepSkips.LiveDependent): a candidate is held back when a live row depends on it through a SweepLiveDependentEdgeTypes edge (parent-child, tracks, blocks), read from both dependency tables. "Live" is the same not-done set the reference scan uses, custom statuses included, so it is a superset of open/hooked/in_progress. `bd purge` always asks for it (no opt-out: removing live work's root is `bd delete`'s job); `bd prune` does not. Reported as live_dependent_skipped in --json. Wisps-plane selector (issueops.SweepWispsPlane, `--wisps-plane`): the candidates are every closed row in the wisps TABLE, whatever its ephemeral/no_history/wisp_type values, via a new single-plane search (issueops.SearchWispsPlaneInTx) that the store backends call directly and the unit-of-work provider reaches through IssueUseCase.SearchWispsPlane, so embedded, server and proxied share one body. It reaches durable-tier (no-history) rows, so it carries the durable tier's require-a-filter gate. --older-than (purge and prune): day values keep their meaning (7, 7d, 2w); anything else is a Go duration taken exactly (36h, 168h, 90m). Tests: three new Sweeper contract cases (wisps plane clears the whole table, wisps plane requires a filter, protects live dependents) run on dolt, embeddeddolt and the uow provider; workapi unit tests; a CLI retention scenario run in embedded, shared proxied and managed-local proxied modes (fails against origin/main); TestParseOlderThan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(uow): classify IssueUseCase.SearchWispsPlane as a read in the notifying parity partition TestRecordingIssueUseCaseCoversItsSurface requires every IssueUseCase method to be either wrapped by the notifying decorator or exempted with the reason it fires no hook. SearchWispsPlane reads state and changes none, like SearchIssues beside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(purge): refuse out-of-range --older-than; add --limit for bounded drains; cross-plane protection test Review follow-ups on #6883. --older-than: a day count past what a time.Duration holds (213504d, 30501w) used to wrap silently — 213504d became 25m26s, which on a destructive command selected almost every closed row. It is now refused as out of range; ParseDuration already refused oversized h/m/s values. --limit N (issueops.SweepRequest.Limit, SweepResult.Remaining): one purge deletes at most N rows, oldest closed_at first (then id), after every protection, and reports the rest. --json adds remaining and has_more when --limit is given. At ~15ms/row on the proxied path a single unbounded sweep of a large backlog can outlive a caller's deadline, roll back, and never clear; a bounded loop cannot. Negative limits are ErrValidation at the role and refused at the CLI. Tests: TestParseOlderThan out-of-range rows; contract cases LimitTakesTheOldestClosedFirst and ProtectsLiveDependentsAcrossPlanes (a live issues-table bead depending on a closed wisp via parent-child, tracks and blocks keeps it) on dolt, embeddeddolt and uow; workapi units for the ordering and the negative-limit refusal; the CLI retention scenario now drains with --limit 1 and checks the overflow refusal. CHANGELOG notes that sub-day Nh values used to round UP to a day (12h is now taken exactly) and lists the protection under Changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
392 lines
14 KiB
Go
392 lines
14 KiB
Go
package workapi
|
|
|
|
import (
|
|
"fmt"
|
|
"path/filepath"
|
|
"sort"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/steveyegge/beads/internal/types"
|
|
"github.com/steveyegge/beads/issueops"
|
|
)
|
|
|
|
// The shared, DATABASE-FREE half of issueops.Sweeper: what a sweep request
|
|
// means, which candidates survive the recheck, and what counts as a citation.
|
|
// Every implementation of the role runs these, so `bd purge` and `bd prune`
|
|
// have one definition rather than one per backend.
|
|
//
|
|
// What is NOT here is the sweep itself. Selecting rows and deleting them needs
|
|
// one transaction (issueops.Sweeper.Sweep), which no interface above a store
|
|
// publishes; the bodies live in internal/storage/issueops/sweep.go and in the
|
|
// unit-of-work provider.
|
|
|
|
// ValidateSweepRequest applies the request rules every Sweeper implementation
|
|
// shares, before anything is read.
|
|
//
|
|
// The require-a-filter refusal for the durable tier is a safety invariant, so
|
|
// it lives HERE rather than in a CLI handler: a second front door inherits it
|
|
// by calling the role. See issueops.SweepRequest.
|
|
func ValidateSweepRequest(in issueops.SweepRequest) error {
|
|
switch in.Tier {
|
|
case issueops.SweepEphemeral, issueops.SweepDurable, issueops.SweepWispsPlane:
|
|
case "":
|
|
return fmt.Errorf("%w: sweep requires a tier (%q, %q or %q), and has no default",
|
|
issueops.ErrValidation, issueops.SweepEphemeral, issueops.SweepDurable, issueops.SweepWispsPlane)
|
|
default:
|
|
return fmt.Errorf("%w: %q is not a sweep tier; use %q, %q or %q",
|
|
issueops.ErrValidation, in.Tier, issueops.SweepEphemeral, issueops.SweepDurable, issueops.SweepWispsPlane)
|
|
}
|
|
if in.IDPattern != "" {
|
|
// filepath.Match reports a malformed pattern on any subject, so one
|
|
// probe against the empty string classifies the pattern itself. The
|
|
// front doors used to discard this error, which turned `--pattern '['`
|
|
// into "nothing matched" on a command whose job is to delete matches.
|
|
if _, err := filepath.Match(in.IDPattern, ""); err != nil {
|
|
return fmt.Errorf("%w: --pattern %q is not a valid glob: %v",
|
|
issueops.ErrValidation, in.IDPattern, err)
|
|
}
|
|
}
|
|
if in.Limit < 0 {
|
|
return fmt.Errorf("%w: sweep limit %d is negative; use 0 for no limit", issueops.ErrValidation, in.Limit)
|
|
}
|
|
if sweepRequiresFilter(in.Tier) && in.ClosedBefore == nil && in.IDPattern == "" {
|
|
return fmt.Errorf("%w: a %s sweep requires a closed-before cutoff or an id pattern; "+
|
|
"pass the pattern \"*\" to sweep every closed issue deliberately",
|
|
issueops.ErrValidation, in.Tier)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// sweepRequiresFilter reports whether a tier reaches durable-tier rows and so
|
|
// carries the require-a-filter gate. The wisps plane does: it holds no-history
|
|
// beads (issueops.SweepWispsPlane).
|
|
func sweepRequiresFilter(tier issueops.SweepTier) bool {
|
|
return tier == issueops.SweepDurable || tier == issueops.SweepWispsPlane
|
|
}
|
|
|
|
// SweepSearchesWispsPlaneOnly reports whether a request's candidates come from
|
|
// the wisps table alone (issueops.SweepWispsPlane) rather than from a tier
|
|
// query merged across both planes. Every Sweeper implementation branches on
|
|
// this one answer, so the plane a tier reads is decided in one place.
|
|
func SweepSearchesWispsPlaneOnly(in issueops.SweepRequest) bool {
|
|
return in.Tier == issueops.SweepWispsPlane
|
|
}
|
|
|
|
// BuildSweepCandidateFilter turns a sweep request into the storage-level
|
|
// filter that selects its CANDIDATES: the closed rows of one tier, bounded by
|
|
// the cutoff.
|
|
//
|
|
// The pattern is deliberately NOT in the filter. Globs are matched in Go
|
|
// (MatchesSweepPattern) because a LIKE translation would silently disagree with
|
|
// filepath.Match on `[...]` and on the escape rules — a disagreement that, on
|
|
// this operation, decides which rows are deleted.
|
|
//
|
|
// Call ValidateSweepRequest first; this builder assumes a validated request
|
|
// and does not re-refuse one.
|
|
//
|
|
// For SweepWispsPlane the filter carries NO tier constraint: the plane is the
|
|
// selection, and the implementation applies it by searching the wisps table
|
|
// alone (SweepSearchesWispsPlaneOnly). Every closed row there is a candidate,
|
|
// whatever its ephemeral, no_history or wisp_type values.
|
|
func BuildSweepCandidateFilter(in issueops.SweepRequest) types.IssueFilter {
|
|
closed := types.StatusClosed
|
|
if SweepSearchesWispsPlaneOnly(in) {
|
|
filter := types.IssueFilter{Status: &closed}
|
|
if in.ClosedBefore != nil {
|
|
cutoff := *in.ClosedBefore
|
|
filter.ClosedBefore = &cutoff
|
|
}
|
|
return filter
|
|
}
|
|
ephemeral := in.Tier == issueops.SweepEphemeral
|
|
// EphemeralTier, not Ephemeral: the tier owns typed wisps whether or not
|
|
// their minting set the ephemeral flag (older creators set wisp_type but
|
|
// not ephemeral, and those rows must fall to `bd purge` — they were the
|
|
// rows it exists for). It also keeps the candidate search on the
|
|
// both-planes merge path, because legacy typed wisps live in the issues
|
|
// table, not the wisps plane an Ephemeral=true search is routed to.
|
|
filter := types.IssueFilter{
|
|
Status: &closed,
|
|
EphemeralTier: &ephemeral,
|
|
}
|
|
if in.ClosedBefore != nil {
|
|
cutoff := *in.ClosedBefore
|
|
filter.ClosedBefore = &cutoff
|
|
}
|
|
return filter
|
|
}
|
|
|
|
// MatchesSweepPattern reports whether an id is admitted by a request's
|
|
// IDPattern. An empty pattern admits everything.
|
|
//
|
|
// A malformed pattern reports false here; ValidateSweepRequest is what refuses it.
|
|
func MatchesSweepPattern(pattern, id string) bool {
|
|
if pattern == "" {
|
|
return true
|
|
}
|
|
ok, err := filepath.Match(pattern, id)
|
|
return err == nil && ok
|
|
}
|
|
|
|
// FilterSweepCandidates applies the pattern and then the two protections that
|
|
// hold a candidate back before any reference scan: the pinned flag, and the
|
|
// closed_at recheck.
|
|
//
|
|
// THE ORDER IS PART OF THE ANSWER, and issueops.Sweeper.Sweep states it: the
|
|
// pattern narrows first, so a pinned row the pattern excluded is not counted
|
|
// as protected — it was never a candidate. The three closed_at buckets are
|
|
// defenses against the tier query returning a row it was not asked for, counted
|
|
// rather than dropped so a disagreement between the query and the recheck is
|
|
// visible instead of silent.
|
|
//
|
|
// cutoff is the request's ClosedBefore; a nil cutoff performs the presence
|
|
// checks and skips the comparison.
|
|
func FilterSweepCandidates(issues []*types.Issue, pattern string, cutoff *time.Time) ([]*types.Issue, issueops.SweepSkips) {
|
|
kept := make([]*types.Issue, 0, len(issues))
|
|
var skips issueops.SweepSkips
|
|
|
|
for _, issue := range issues {
|
|
if issue == nil {
|
|
skips.Unreadable++
|
|
continue
|
|
}
|
|
if !MatchesSweepPattern(pattern, issue.ID) {
|
|
continue
|
|
}
|
|
switch {
|
|
case issue.Pinned:
|
|
skips.Pinned++
|
|
case issue.Status != types.StatusClosed:
|
|
skips.NotClosed++
|
|
case issue.ClosedAt == nil:
|
|
skips.UnknownClosedAt++
|
|
case cutoff != nil && !issue.ClosedAt.Before(*cutoff):
|
|
skips.ClosedAtOrAfterCutoff++
|
|
default:
|
|
kept = append(kept, issue)
|
|
}
|
|
}
|
|
return kept, skips
|
|
}
|
|
|
|
// SweepDefenseSkips is how many candidates the recheck rejected as
|
|
// self-inconsistent — the three closed_at buckets plus the unreadable row. It
|
|
// is separate from the two PROTECTIONS (pinned, referenced) because a non-zero
|
|
// value here is a defense firing rather than a normal outcome, and the front
|
|
// doors warn on exactly that.
|
|
func SweepDefenseSkips(s issueops.SweepSkips) int {
|
|
return s.Unreadable + s.NotClosed + s.UnknownClosedAt + s.ClosedAtOrAfterCutoff
|
|
}
|
|
|
|
// NotDoneStatusesForSweep is the status vocabulary a reference scan asks:
|
|
// every built-in status that is not "closed", plus every configured custom
|
|
// status whose category is not "done".
|
|
//
|
|
// It is an explicit LIST rather than a not-closed exclusion because that is
|
|
// what the storage seam takes reliably on both planes (a Statuses set, not an
|
|
// ExcludeStatus). Reading the workspace's custom statuses is required, not
|
|
// best-effort: a scan that missed a custom active status would under-protect
|
|
// and delete a bead a live bead still cites.
|
|
func NotDoneStatusesForSweep(custom []types.CustomStatus) []types.Status {
|
|
statuses := []types.Status{
|
|
types.StatusOpen,
|
|
types.StatusInProgress,
|
|
types.StatusBlocked,
|
|
types.StatusDeferred,
|
|
types.StatusPinned,
|
|
types.StatusHooked,
|
|
}
|
|
for _, cs := range custom {
|
|
if cs.Category != types.CategoryDone {
|
|
statuses = append(statuses, types.Status(cs.Name))
|
|
}
|
|
}
|
|
return statuses
|
|
}
|
|
|
|
// BuildSweepReferenceScanFilter selects the rows a reference scan reads: the
|
|
// not-done set, and nothing else about them.
|
|
func BuildSweepReferenceScanFilter(custom []types.CustomStatus) types.IssueFilter {
|
|
return types.IssueFilter{Statuses: NotDoneStatusesForSweep(custom)}
|
|
}
|
|
|
|
// CandidateIDMatcher finds literal, word-bounded occurrences of a candidate id
|
|
// set in arbitrary text. It is the whole of what "cited" means for
|
|
// issueops.SweepRequest.ProtectReferenced.
|
|
//
|
|
// It is bucketed by FIRST BYTE and each bucket sorted longest-first: the scan
|
|
// runs over every not-done bead's description, notes and comments, where a
|
|
// naive strings.Contains per candidate is O(candidates x text). Longest-first
|
|
// inside a bucket is what makes `be-1.1` win over `be-1` where both are
|
|
// candidates and both start at the same offset.
|
|
type CandidateIDMatcher struct {
|
|
byFirstByte map[byte][]string
|
|
}
|
|
|
|
// NewCandidateIDMatcher builds a matcher over a candidate id set. Empty ids
|
|
// are dropped: they would match at every boundary.
|
|
func NewCandidateIDMatcher(candidateIDs map[string]bool) CandidateIDMatcher {
|
|
byFirstByte := make(map[byte][]string)
|
|
for id := range candidateIDs {
|
|
if id == "" {
|
|
continue
|
|
}
|
|
byFirstByte[id[0]] = append(byFirstByte[id[0]], id)
|
|
}
|
|
for first := range byFirstByte {
|
|
ids := byFirstByte[first]
|
|
sort.Slice(ids, func(i, j int) bool {
|
|
if len(ids[i]) == len(ids[j]) {
|
|
return ids[i] < ids[j]
|
|
}
|
|
return len(ids[i]) > len(ids[j])
|
|
})
|
|
byFirstByte[first] = ids
|
|
}
|
|
return CandidateIDMatcher{byFirstByte: byFirstByte}
|
|
}
|
|
|
|
// FindAll adds every candidate id occurring in text at ASCII word boundaries
|
|
// to found.
|
|
func (m CandidateIDMatcher) FindAll(text string, found map[string]bool) {
|
|
for i := 0; i < len(text); i++ {
|
|
ids := m.byFirstByte[text[i]]
|
|
if len(ids) == 0 || !isWordBoundaryAt(text, i) {
|
|
continue
|
|
}
|
|
for _, id := range ids {
|
|
end := i + len(id)
|
|
if end <= len(text) && strings.HasPrefix(text[i:], id) && isWordBoundaryAt(text, end) {
|
|
found[id] = true
|
|
break
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
func isWordBoundaryAt(s string, idx int) bool {
|
|
var before, after byte
|
|
if idx > 0 {
|
|
before = s[idx-1]
|
|
}
|
|
if idx < len(s) {
|
|
after = s[idx]
|
|
}
|
|
return isASCIIWordByte(before) != isASCIIWordByte(after)
|
|
}
|
|
|
|
func isASCIIWordByte(b byte) bool {
|
|
return b == '_' ||
|
|
('0' <= b && b <= '9') ||
|
|
('A' <= b && b <= 'Z') ||
|
|
('a' <= b && b <= 'z')
|
|
}
|
|
|
|
// PartitionSweepReferenced splits candidates into the ones no not-done row
|
|
// cites and the ones some row does, and returns the bounded sample
|
|
// issueops.SweepResult.ReferencedIDs carries. Both the kept slice and the
|
|
// sample preserve the candidate order, so two runs over one snapshot report the
|
|
// same ids.
|
|
func PartitionSweepReferenced(candidates []*types.Issue, referenced map[string]bool) (kept []*types.Issue, referencedCount int, sample []string) {
|
|
kept = make([]*types.Issue, 0, len(candidates))
|
|
for _, issue := range candidates {
|
|
if referenced[issue.ID] {
|
|
referencedCount++
|
|
if len(sample) < issueops.SweepReferencedSampleLimit {
|
|
sample = append(sample, issue.ID)
|
|
}
|
|
continue
|
|
}
|
|
kept = append(kept, issue)
|
|
}
|
|
return kept, referencedCount, sample
|
|
}
|
|
|
|
// BuildSweepLiveDependentScanFilter selects which of a candidate set's
|
|
// dependents are LIVE for issueops.SweepRequest.ProtectLiveDependents: the
|
|
// named sources, restricted to the not-done statuses. It is the same status
|
|
// vocabulary the reference scan asks, so "live" means one thing in a sweep.
|
|
func BuildSweepLiveDependentScanFilter(dependentIDs []string, custom []types.CustomStatus) types.IssueFilter {
|
|
return types.IssueFilter{IDs: dependentIDs, Statuses: NotDoneStatusesForSweep(custom)}
|
|
}
|
|
|
|
// SweepLiveDependentSources returns the ids of the rows that depend on a
|
|
// candidate through a protecting edge (issueops.SweepLiveDependentEdgeTypes),
|
|
// given the edges pointing at the candidates keyed by target. The result is
|
|
// sorted and de-duplicated, so the follow-up status read is deterministic.
|
|
func SweepLiveDependentSources(incoming map[string][]*types.Dependency) []string {
|
|
seen := make(map[string]bool)
|
|
for _, deps := range incoming {
|
|
for _, dep := range deps {
|
|
if dep != nil && isSweepProtectingEdge(dep.Type) {
|
|
seen[dep.IssueID] = true
|
|
}
|
|
}
|
|
}
|
|
ids := make([]string, 0, len(seen))
|
|
for id := range seen {
|
|
ids = append(ids, id)
|
|
}
|
|
sort.Strings(ids)
|
|
return ids
|
|
}
|
|
|
|
// SweepLiveDependentTargets returns the candidates a live row depends on: the
|
|
// targets of protecting edges whose source is in live.
|
|
func SweepLiveDependentTargets(incoming map[string][]*types.Dependency, live map[string]bool) map[string]bool {
|
|
protected := make(map[string]bool)
|
|
for target, deps := range incoming {
|
|
for _, dep := range deps {
|
|
if dep != nil && isSweepProtectingEdge(dep.Type) && live[dep.IssueID] {
|
|
protected[target] = true
|
|
break
|
|
}
|
|
}
|
|
}
|
|
return protected
|
|
}
|
|
|
|
func isSweepProtectingEdge(t types.DependencyType) bool {
|
|
for _, protecting := range issueops.SweepLiveDependentEdgeTypes {
|
|
if t == protecting {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// PartitionSweepLiveDependents splits candidates into the ones no live row
|
|
// depends on and a count of the ones some live row does, preserving candidate
|
|
// order.
|
|
func PartitionSweepLiveDependents(candidates []*types.Issue, protected map[string]bool) (kept []*types.Issue, protectedCount int) {
|
|
kept = make([]*types.Issue, 0, len(candidates))
|
|
for _, issue := range candidates {
|
|
if protected[issue.ID] {
|
|
protectedCount++
|
|
continue
|
|
}
|
|
kept = append(kept, issue)
|
|
}
|
|
return kept, protectedCount
|
|
}
|
|
|
|
// LimitSweepCandidates applies issueops.SweepRequest.Limit to the candidates
|
|
// that survived every protection: the oldest-closed first (closed_at
|
|
// ascending, then id, so two runs over one snapshot pick the same rows), at
|
|
// most limit of them. It returns the kept rows and how many were left behind.
|
|
// A non-positive limit keeps everything. The input slice is not reordered.
|
|
func LimitSweepCandidates(candidates []*types.Issue, limit int) (kept []*types.Issue, remaining int) {
|
|
if limit <= 0 || len(candidates) <= limit {
|
|
return candidates, 0
|
|
}
|
|
ordered := append([]*types.Issue(nil), candidates...)
|
|
sort.SliceStable(ordered, func(i, j int) bool {
|
|
a, b := ordered[i], ordered[j]
|
|
if a.ClosedAt != nil && b.ClosedAt != nil && !a.ClosedAt.Equal(*b.ClosedAt) {
|
|
return a.ClosedAt.Before(*b.ClosedAt)
|
|
}
|
|
return a.ID < b.ID
|
|
})
|
|
return ordered[:limit], len(ordered) - limit
|
|
}
|