beads/internal/httpapi/problem.go
CI Bot 519fb1479b feat(issueops): put bd purge and bd prune behind the Sweeper role (bd-pn231)
`bd purge` selected closed wisps in one transaction and deleted them in
another. A bead closed, unpinned or cited in the window between the two was
judged on one state and acted on in a different one — the TOCTOU the bead
names — and the proxied twin held a second copy of the whole flow: its own
candidate filter, its own glob match, its own closed_at recheck, its own
reference scan and 190 lines of output kept character-identical to the direct
route's by hand.

PURGE AND PRUNE ARE ONE CAPABILITY, which the design left open and the code
answers plainly: both routes already ran ONE function with a `purgeScope`
struct choosing the tier. So issueops.Sweeper has ONE method, reached through
store.Sweeper() and the provider's own accessor, and SweepRequest.Tier selects
the plane — `ephemeral` is purge, `durable` is prune, and the two are disjoint
by construction. What genuinely differed became request fields rather than a
second role: the require-a-filter refusal keys on Tier, and the citation
protection is ASKED FOR (ProtectReferenced) instead of being inferred from a
command name, which is what the shared body did (`scope.cmdName == "prune"`).

Q9 ANSWERED IN THE ROLE. The require-a-filter gate is
workapi.ValidateSweepRequest, below every front door: a durable sweep with
neither a cutoff nor an id pattern is ErrValidation. It is a safety invariant,
not presentation — without it a second front door is one handler away from an
unguarded mass delete, which is exactly what this commit's HTTP operation would
otherwise have been. The CLI keeps its friendly message, and says in place that
it is there for the WORDING while the guard is the role's; the contract case is
what proves the guard survives that branch being deleted.

THE DEFECT, AND ITS PRICE. The whole sweep is now one transaction, so the set
the result describes IS the set it deleted. That gives up the server-backed
store's batched wisp deletion (200 per transaction, added for Dolt's 10s write
timeout after bd-2ehd and ff-tqm): a sweep too large now fails WHOLE. That is
the better failure — the partial one it replaces reported a deleted count of
ZERO while having deleted rows — and the other two backends already worked this
way. It is written down as a promise WITH its cost, and recorded for sign-off.

TWO MORE DEFECTS the single call site removed. `--dry-run` passed force=false
to the delete where the real run passed force=true, so a preview could FAIL
over dependents the sweep would have orphaned — and the CLI answered that
failure with zeros. And `filepath.Match`'s error was discarded on both routes,
so `--pattern '['` printed "No closed beads to prune" and exited 0; it is now
ErrValidation quoting the pattern.

THE CONTRACT IS WHAT WAS DELETED, not what was reported.
backend/conformance/sweeper_contract.go holds ten cases, each citing the leaf
doc, run at all three backends with ZERO PARKS and zero divergences. Every case
that acts asserts row counts through QueryScalar rather than trusting the
result the sweep reported about itself, which on a destructive operation is the
assertion that matters; every case scopes itself with an id pattern, because a
sweep is asked of a whole TIER and an unscoped one would clear the next case's
seeds. They are TWO INDEPENDENT BODIES plus an engine check: dolt and
embeddeddolt share internal/storage/issueops.SweepInTx and differ only in how
they reach a transaction, and the unit-of-work provider reaches the same four
questions through the domain use cases.

THE HTTP OPERATION, spec first (per bd-jtp1h's A2 scope decision):
POST /v0/beads/issues:sweep, capability `issues.sweep` — the first DESTRUCTIVE
operation on a surface that was reads plus one claim. It is a collection-level
custom method with a body rather than DELETE with a query string, because a
dropped query parameter widens what is erased where an unknown body member is
refused by name. The handler decodes six members and calls the role; it does
not implement the safety gate, the tier vocabulary or the glob, all of which
are the role's ErrValidation reaching the wire as a 400 through failSweepErr.
No component carries allOf and none is x-go-type-pinned — there is no canonical
Go struct whose JSON encoding is this body, since the CLI publishes these
numbers under `purged_count`/`pruned_count` — so the projection is tested
instead. `bd serve --help` grew a DESTRUCTIVE OPERATIONS section and the
--allow-non-loopback warning now says peers can bulk-delete closed beads.

`bd gc` keeps the closed-candidate recheck by calling the SAME pure function
the role calls (workapi.FilterSweepCandidates), so the two cannot drift; the
reference matcher and its 10K-body budget moved to internal/workapi with the
scan. cmd/bd/purge.go and purge_proxied_server.go LEFT the forbidigo exception
list, which is how a role commit proves its front doors reached the role.

USER-VISIBLE, for the wave's changelog pass: the version-control label for a
sweep is now `bd: sweep <n> <tier> bead(s)` on both routes, where proxied wrote
`bd: prune <n> bead(s)` and direct wrote `bd: delete <n> issue(s)` — an HTTP
sweep is not `bd prune`, so the label names the operation and the tier from
every door; a malformed `--pattern` now exits non-zero; `--dry-run` no longer
refuses over dependents the real sweep orphans; and `bd serve` gains a
bulk-delete endpoint. Reasoning and the alternatives are in
/data/tmp/sd-server/AMBIGUITIES.md as A-swp-1..9.

Verified: build, vet, gofmt clean; golangci-lint 0 issues; make api-check
(regenerate-and-diff plus the httpapi suite) green; TestSweeperContract 10/10 at
dolt, embeddeddolt and uow; the internal/workapi, internal/storage and
internal/telemetry suites; TestEmbeddedPurge, TestEmbeddedPurgeConcurrent,
TestEmbeddedPrune, TestPrune_protectsBeadCitedByCustomStatusBead and
TestEmbeddedGC under BEADS_TEST_EMBEDDED_DOLT=1; TestProxiedServerPrune under
BEADS_TEST_PROXIED_SERVER=1; TestProxiedLookup*; and the root layering pins.
The protocol corpus has no purge/prune case to account for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 16:40:26 +00:00

408 lines
18 KiB
Go

package httpapi
import (
"context"
"database/sql"
"database/sql/driver"
"encoding/json"
"errors"
"net"
"net/http"
"strconv"
"syscall"
"github.com/go-sql-driver/mysql"
"github.com/steveyegge/beads/internal/httpapi/apigen"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/uow"
)
// Every non-2xx byte this server emits is an RFC 9457 problem+json document
// (apigen.Problem, generated from the spec — there is one error shape here).
// This file owns the whole mapping: sentinel error in, status + machine code
// out, matched exclusively with errors.Is/errors.As. Never `err != nil ->
// status`.
//
// SENTINEL NOTE: the mapping keys on the sentinel VALUES, never on where they
// are declared. A parallel refactor relocates the canonical Err* declarations
// into a leaf issueops package and aliases them back from internal/storage
// with the same pointers, so errors.Is keeps matching either spelling with no
// edit here. Do not re-express these rows as package-path or message checks.
// Code is the machine-readable member of the problem envelope, and the only
// member a client may dispatch on.
//
// The vocabulary is a one-way door: renaming or removing a documented
// status+code pair breaks the wire. Adding one does not, which is why clients
// are told to default-branch on unknown codes within a status class. Keep it
// at what the seven v0 operations actually need.
// at what the v0 operations actually need.
type Code string
// The v0 code vocabulary. Every value here is documented in the spec, and
// TestSpecStatusCodesMatchHandlerTable fails if the two ever disagree.
const (
// CodeInvalidArgument covers every request-validation refusal: an unknown
// query parameter, a malformed value, an invalid actor, an unparseable or
// oversized body, limit=0 under --allow-non-loopback, and the Host-header
// middleware. The 400 carries `param` and `reason` so clients never have
// to tell those cases apart by reading prose.
CodeInvalidArgument Code = "invalid_argument"
// CodeInvalidCursor is a separate code because a stale or foreign cursor
// is a normal client situation with an obvious recovery (restart paging),
// not a client bug.
CodeInvalidCursor Code = "invalid_cursor"
CodeNotFound Code = "not_found"
// CodeAlreadyClaimed carries the holder in the `assignee` extension
// member, read inside the same transaction — never parsed out of the
// sentinel's message text.
CodeAlreadyClaimed Code = "already_claimed"
CodeNotClaimable Code = "not_claimable"
// CodeBusy is retryable contention: the transaction retry budget was
// exhausted, or the in-flight request limit was saturated.
CodeBusy Code = "busy"
// CodeDBUnavailable is a retryable connectivity failure reaching the
// database.
CodeDBUnavailable Code = "db_unavailable"
CodeInternal Code = "internal"
)
// codeClientClosed is a LOG-ONLY outcome, not wire vocabulary: it is
// deliberately absent from codeStatus, from operationCodes and from the
// document, and it never reaches a response body — the client it describes has
// already gone. It exists so that the request line does not book a client
// hanging up as a server fault. See failErr.
const codeClientClosed Code = "client_closed"
// codeStatus freezes one HTTP status per code. A code that could arrive with
// two different statuses would defeat the point of dispatching on it.
var codeStatus = map[Code]int{
CodeInvalidArgument: http.StatusBadRequest,
CodeInvalidCursor: http.StatusBadRequest,
CodeNotFound: http.StatusNotFound,
CodeAlreadyClaimed: http.StatusConflict,
CodeNotClaimable: http.StatusConflict,
CodeBusy: http.StatusServiceUnavailable,
CodeDBUnavailable: http.StatusServiceUnavailable,
CodeInternal: http.StatusInternalServerError,
}
// Status returns the HTTP status frozen to c, or 0 if c is not in the v0
// vocabulary.
func (c Code) Status() int { return codeStatus[c] }
// Reason distinguishes the two client postures behind a 400
// CodeInvalidArgument, so that telling them apart never requires parsing
// `detail`. The set may grow; clients default-branch on unknown values.
type Reason string
const (
// ReasonUnknownParameter means this server does not know that parameter:
// version skew. The client degrades or falls back. It is also a client's
// only per-parameter capability probe, since `capabilities` is
// operation-level.
ReasonUnknownParameter Reason = "unknown_parameter"
// ReasonInvalidValue means the server will not act on that value:
// malformed, outside the vocabulary, or legal-but-refused in this
// server's configuration (limit=0 under --allow-non-loopback). The
// recovery is always to send something different, never to retry; the
// detail says which case it was.
ReasonInvalidValue Reason = "invalid_value"
)
// staticDetail is the ONLY `detail` a 5xx may carry. The underlying error goes
// to the server log and nowhere else: driver and dial errors routinely embed
// the DSN — go-sql-driver renders connection targets as
// user@tcp(127.0.0.1:PORT)/db, net dial errors carry the same host:port — and
// query errors can carry SQL fragments. The moment the server is bound with
// --allow-non-loopback, a verbose 5xx detail is an information-disclosure
// channel to network peers. 4xx details stay specific: they reflect the
// caller's own input back, not server internals.
var staticDetail = map[Code]string{
CodeBusy: "the server is busy; retry shortly",
CodeDBUnavailable: "database temporarily unavailable; retry",
CodeInternal: "internal server error",
}
// Retry-After values, in seconds.
const (
// retryAfterContention follows an exhausted transaction retry budget. The
// budget spans many seconds of observed write contention, so a one-second
// comeback would invite a convoy of retries that each hold a slot while
// they wait — starving reads exactly when the server is busiest.
retryAfterContention = 5
// retryAfterSaturation follows an in-flight-limit wait timeout. Slot
// pressure clears quickly.
retryAfterSaturation = 1
)
// ErrBusy reports that the in-flight request limiter refused to admit the
// request within its bounded wait. The limiter itself lands with the server;
// the sentinel lives here so the mapping owns the whole 503 vocabulary.
var ErrBusy = errors.New("server busy")
// Operation ids, matching the spec's operationId values exactly.
const (
OpHealth = "health"
OpGetContext = "getContext"
OpListReadyWork = "listReadyWork"
OpGetStats = "getStats"
OpListIssues = "listIssues"
OpGetIssue = "getIssue"
OpClaimIssue = "claimIssue"
OpListSettings = "listSettings"
OpGetSetting = "getSetting"
OpListDependencyCycles = "listDependencyCycles"
// OpListDependencies reads STORED EDGE ROWS for several issues at once.
// It is a separate operation from getIssue's embedded `dependencies`
// member because it answers per named issue, reports the ids that named
// nothing, and returns edges whose target this database holds no row for.
OpListDependencies = "listDependencies"
OpCountReadyWork = "countReadyWork"
OpQueryIssues = "queryIssues"
// OpSweepIssues is the one DESTRUCTIVE operation on this surface: bulk
// clearance of closed beads from one tier, behind issueops.Sweeper.
OpSweepIssues = "sweepIssues"
)
// operationCodes is the per-operation problem vocabulary: exactly the codes
// that operation's handler can produce, and therefore exactly what the spec
// documents for it. TestSpecStatusCodesMatchHandlerTable asserts set-equality
// in both directions, so an undocumented emission and an unemittable
// documented status both fail CI.
//
// Two 400 invalid_argument paths are reachable on every route including
// /healthz — the Host-header middleware, and the unknown-query-parameter
// refusal every decoder performs — and both are deliberately absent from every
// row here. They are uniform rules, not per-operation behavior, and the spec
// documents them once at the document level rather than repeating them on all
// twelve operations; these rows carry what an operation produces beyond them.
// documents them once at the document level rather than repeating them on
// every operation; these rows carry what an operation produces beyond them.
// Keep the two documents in step: a row here and the document-level prose are
// the only two places that carve-out exists.
var operationCodes = map[string][]Code{
// Liveness answers from the process and touches nothing that can fail.
OpHealth: nil,
// v0 serves a startup snapshot without touching the database, so there is
// no 503 here. If a later slice makes this a DB-probing readiness
// endpoint, db_unavailable joins this row and the spec in the same change.
OpGetContext: {CodeInternal},
OpListReadyWork: {CodeInvalidArgument, CodeBusy, CodeDBUnavailable, CodeInternal},
OpListIssues: {CodeInvalidArgument, CodeInvalidCursor, CodeBusy, CodeDBUnavailable, CodeInternal},
OpGetIssue: {CodeNotFound, CodeBusy, CodeDBUnavailable, CodeInternal},
// No 400 of its own: the operation takes no parameters, so the only
// invalid_argument it can raise is the document-level unknown-query-key
// rule this table deliberately omits.
OpListSettings: {CodeBusy, CodeDBUnavailable, CodeInternal},
// No 404, and that is the operation's whole shape: a key nothing stored and
// a key stored as the empty string are one answer on this surface, so the
// only refusal a key can earn is the 400 that says it was not a key.
OpGetSetting: {CodeInvalidArgument, CodeBusy, CodeDBUnavailable, CodeInternal},
// The 400 here is this operation's own, not the document-level
// unknown-parameter rule: a malformed `skip_blocked`, and the EMPTY
// `assignee` the document refuses rather than answering with the rows that
// have no assignee.
OpGetStats: {CodeInvalidArgument, CodeBusy, CodeDBUnavailable, CodeInternal},
// The cycle sweep takes no parameters at all, so it has no 400 of its own:
// the two uniform ones above are the whole of its invalid-argument story.
OpListDependencyCycles: {CodeBusy, CodeDBUnavailable, CodeInternal},
// No not_found here, deliberately: an id that names nothing is reported in
// the response's `missing` member, so a batch keeps the answers for the ids
// that were found. A 404 would discard them.
OpListDependencies: {CodeInvalidArgument, CodeBusy, CodeDBUnavailable, CodeInternal},
// The same vocabulary as the listing it sizes: it takes the same filters
// and can refuse them the same way. limit=0's mode-dependent refusal has no
// analog here because there is no limit to pass — but that was a
// CodeInvalidArgument too, so this row would not change if there were.
OpCountReadyWork: {CodeInvalidArgument, CodeBusy, CodeDBUnavailable, CodeInternal},
// The listing's vocabulary minus the cursor: this operation has none, so
// invalid_cursor cannot arise. An unparseable EXPRESSION is an
// invalid_argument on `q` rather than a code of its own — a client's
// recovery is the same as for any other malformed parameter value, and a
// code it cannot act on differently is a code it should not have to know.
OpQueryIssues: {CodeInvalidArgument, CodeBusy, CodeDBUnavailable, CodeInternal},
// The 400 here is the widest on this surface, and most of it is not this
// handler's: an unknown or malformed body member is, but the unfiltered
// durable sweep, the unrecognized tier and the malformed glob are all the
// ROLE's ErrValidation reaching the wire through failSweepErr. No 404 —
// this operation names no id — and no 409: a sweep has nothing to conflict
// with, since a bead another sweep already took is simply not in the set.
OpSweepIssues: {CodeInvalidArgument, CodeBusy, CodeDBUnavailable, CodeInternal},
OpClaimIssue: {
CodeInvalidArgument, CodeNotFound, CodeAlreadyClaimed, CodeNotClaimable,
CodeBusy, CodeDBUnavailable, CodeInternal,
},
}
// Result is a problem response ready to be written: the envelope plus the
// transport-level Retry-After the code implies.
type Result struct {
Problem apigen.Problem
// RetryAfterSeconds is written as the Retry-After header when positive.
RetryAfterSeconds int
}
// WithAssignee attaches the `assignee` extension member (the current holder of
// a claimed issue). Populate it from a read in the claim's own transaction —
// never by parsing the sentinel's message fragments.
func (r Result) WithAssignee(assignee string) Result {
r.Problem.Assignee = &assignee
return r
}
// WithIssueStatus attaches the `issue_status` extension member (the issue's
// status at the moment of refusal). Same rule as WithAssignee.
func (r Result) WithIssueStatus(status string) Result {
r.Problem.IssueStatus = &status
return r
}
// WithRequestID sets the `request_id` member, the correlation id echoed in the
// request log line. It is what makes a 5xx actionable: the body carries a fixed
// static detail by design, so the id is the client's only handle on the one log
// line that has the real error. The document requires it on every problem
// response, which is why this sets it unconditionally — an id this server
// failed to mint travels as an empty string rather than as a missing required
// member.
func (r Result) WithRequestID(id string) Result {
r.Problem.RequestId = id
return r
}
func newResult(code Code, detail string) Result {
status := code.Status()
if status == 0 {
// Unreachable unless a code is added without a status; fail closed
// rather than emitting a 0 status line.
status = http.StatusInternalServerError
code = CodeInternal
detail = staticDetail[CodeInternal]
}
if static, ok := staticDetail[code]; ok {
// 5xx detail is fixed per code, whatever the caller passed.
detail = static
}
p := apigen.Problem{
Status: status,
Title: http.StatusText(status),
Code: string(code),
}
if detail != "" {
p.Detail = &detail
}
return Result{Problem: p}
}
// InvalidArgument builds the 400 for a request the server refuses to
// interpret. param names the offending query parameter, body member or header;
// pass "" only when the input has no nameable part (a body that fails to parse
// at all). detail may quote the caller's own input — it is not server state.
func InvalidArgument(param string, reason Reason, detail string) Result {
res := newResult(CodeInvalidArgument, detail)
if param != "" {
res.Problem.Param = &param
}
r := string(reason)
res.Problem.Reason = &r
return res
}
// InvalidCursor builds the 400 for a cursor this server did not issue, cannot
// decode, or issued under a different internal version.
func InvalidCursor() Result {
return newResult(CodeInvalidCursor, "cursor is not valid for this server; restart paging without it")
}
// NotFound builds the 404 for an id this server cannot resolve. It is one
// function rather than a literal per site so that a handler which decides a
// miss WITHOUT reading storage — an id no row could hold — is indistinguishable
// on the wire from one that read and missed. A client that could tell them
// apart would be probing which ids are well-formed.
func NotFound() Result {
return newResult(CodeNotFound, "no issue or wisp with that id")
}
// ClassifyError maps an error from the storage seam onto the wire. The caller
// is responsible for logging err: everything mapped to a 5xx deliberately
// drops the error text on the floor (see staticDetail).
func ClassifyError(err error) Result {
switch {
case err == nil:
return newResult(CodeInternal, "")
// Not-found normalization belongs in the shared read path, which folds the
// two miss shapes (a wrapped sql.ErrNoRows, and a nil issue with a nil
// error) into storage.ErrNotFound. The bare sql.ErrNoRows row below is
// defense in depth for a path that has not been normalized yet: a missing
// issue must never surface as a 500. The converse mistake — treating any
// error as a miss — stays closed, because only these two sentinels reach
// 404 and everything else falls through to 500.
case errors.Is(err, storage.ErrNotFound), errors.Is(err, sql.ErrNoRows):
return NotFound()
case errors.Is(err, storage.ErrAlreadyClaimed):
return newResult(CodeAlreadyClaimed, "issue is claimed by another actor")
case errors.Is(err, storage.ErrNotClaimable):
return newResult(CodeNotClaimable, "issue is not in a claimable state")
case errors.Is(err, ErrBusy):
res := newResult(CodeBusy, "")
res.RetryAfterSeconds = retryAfterSaturation
return res
// The retry budget is spent inside uow.RunTxResult; reaching here means it
// gave up, so the request is retryable at the client's cadence, not the
// server's.
case uow.IsSerializationError(err):
res := newResult(CodeBusy, "")
res.RetryAfterSeconds = retryAfterContention
return res
case isUnavailable(err):
res := newResult(CodeDBUnavailable, "")
res.RetryAfterSeconds = retryAfterContention
return res
default:
return newResult(CodeInternal, "")
}
}
// isUnavailable reports whether err is a failure to reach the database at all
// — the server or proxy being down, idle-stopped, or dropping connections —
// as opposed to a failure while executing a statement.
//
// The list is empirical and safe to extend: it only chooses between two 5xx
// codes that carry identical static detail, so a miss costs a less useful
// `code`, never a disclosure.
func isUnavailable(err error) bool {
// context.DeadlineExceeded satisfies net.Error, so it must be excluded
// before the net.Error test below: a tripped per-request deadline is a
// generic 500, not a claim that the database is unreachable.
if errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled) {
return false
}
if errors.Is(err, driver.ErrBadConn) || errors.Is(err, mysql.ErrInvalidConn) {
return true
}
if errors.Is(err, syscall.ECONNREFUSED) || errors.Is(err, syscall.ECONNRESET) || errors.Is(err, syscall.EPIPE) {
return true
}
var netErr net.Error
return errors.As(err, &netErr)
}
// Write emits res as application/problem+json. Success bodies are written by
// their handlers; every non-2xx byte goes through here.
func Write(w http.ResponseWriter, res Result) {
h := w.Header()
h.Set("Content-Type", "application/problem+json; charset=utf-8")
if res.RetryAfterSeconds > 0 {
h.Set("Retry-After", strconv.Itoa(res.RetryAfterSeconds))
}
w.WriteHeader(res.Problem.Status)
_ = json.NewEncoder(w).Encode(res.Problem)
}