beads/internal/httpclient/sweeper.go
Julian Knutsen 533f95295a
feat(httpclient): BatchApplier over HTTP — handshake cap, dep_add lineage on the wire, gc mappings (S5) (#7265)
* feat(httpclient): carry DepAddItem.HasSpawner/ThreadID over the wire

The http leg's batchApply wire shape dropped a dep_add item's HasSpawner
and ThreadID fields -- an edit the caller believed landed. Extend the
wire body (ApplyDepAddItem.has_spawner/thread_id), gate sending either
field on a new additive behavior capability
(issues.batchApply.depAddLineage / CapBatchApplyDepAddLineage) so a
client refuses locally before dialing an older server instead of
silently dropping graph lineage, and wire the storage-layer spawner-id
stamping (StampWaitsForSpawnerID) through both the dolt and uow legs
that already had the fields but never stamped them. wire_revision and
min_client are unchanged; the golden digest and openapi spec are
regenerated.

Also fixes three generic bijection/vocabulary gates this change was
missing: write_ledger_test.go's applyBatch/item/dep_add carriage table,
and the literal behaviorCapabilities lists in
internal/httpapi/server_test.go and
internal/httpclient/wire/vocabulary_test.go.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(httpclient/wire): pin the item-count gate at 356 items

Add TestApplyBatchAt356ItemsWithoutTheCapabilityRefusesBeforeDialing,
exercising internal/storage/batchfixtures.Shape356's exact item count
-- the design's primary measured large-batch-apply shape -- rather than
only the floor+1/ceiling+1 boundaries the existing cases pin. The gate
is content-agnostic (it never looks past len(body.Items)), so a
synthetic plan here is behaviorally identical to a real Shape356 plan
for this one check; realistic-content coverage follows in the next
commit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(httpclient): prove a 356-item Shape356 plan commits atomically or leaves zero rows

Add two tests against the real served stack using
batchfixtures.Shape356's realistic, heterogeneous content (102 creates,
136 blocks edges, 102 tracks-to-root edges, 16 assign updates):

- TestServedBatchApplyShape356CommitsAtomically applies the full plan
  once and verifies every create landed with a bound key, matching the
  issues table row-for-row.
- TestServedBatchApplyShape356PlusAStaleGuardLeavesZeroRows appends a
  357th item guarding the pre-existing root issue with a deliberately
  stale ExpectedVersion. Because the root is untouched by the other 356
  items, this exercises the real server-side precondition_failed round
  trip (rather than the static same-request-row ErrValidation rule),
  asserting errors.Is(err, issueops.ErrVersionMismatch) and that none of
  the 356 creates landed -- the "never chunk" ceiling: an all-or-nothing
  request that fails on its last item must leave nothing behind.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(httpclient): close S5 review HIGH findings (lineage gate, thread_id, gc mappings)

2026-10 Opus review on the S5 batch-apply slice found the wire change
correct but coverage missing:

- HIGH-1: refuseUnservedDepAddLineage (batchapplier.go) had no coverage
  at all -- deleting it passed every existing test. Add a test pinning
  that HasSpawner or ThreadID on a server without
  issues.batchApply.depAddLineage refuses *storage.ErrUnsupported
  before dialing, and that a plan touching neither field still dials.

- HIGH-2: thread_id was never asserted end to end. Add
  RunBatchApplyCarriesThreadIDOntoTheStoredEdge to the shared
  conformance package, wire it onto the dolt, embeddeddolt and uow
  legs, and add the http leg's own case
  (TestServedBatchApplyCarriesThreadIDOntoTheStoredEdge), which is the
  one case that proves the field survives both the client's encode and
  the server's decode at once.

- HIGH-3: DESIGN Sec 3.8 ("Guarded writes: no new roles") documents two
  gc mappings onto BatchApplier that had no coverage: an update carrying
  a metadata patch plus ExpectedVersion followed by a close in one
  request (CloseWithMetadataIfMatch, atomic -- a stale version leaves
  zero changes), and a keyed create/MetadataRefs/dep_add
  (parent-child included)/trailing-update plan (ApplyGraphPlan). Document
  both compositions on the BatchApplier interface and add http-leg tests
  for each.

Mutation-verified: removing the lineage gate call, dropping thread_id on
the client's encode, and dropping it on the server's decode each now
fail a test that previously passed unconditionally.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpclient): nil-guard snapshot consumers, fix spec wording, sync Bazel

2026-10 Opus review MED/LOW findings on the S5 batch-apply slice:

- LOW-6: Store.snapshot returns (nil, nil) when a Store has neither a
  wire nor a cached handshake. Store.Counter() does not call roleWire
  first (unlike Sweeper/BatchApplier), so this is reachable through the
  ordinary public accessor and a scoped count would panic on
  snap.Capabilities. Nil-guard every snapshot() consumer
  (batchapplier.go, counter.go, sweeper.go) so a nil snapshot is treated
  as "nothing advertised" and falls through to the capability refusal.
  Add TestCounterRefusesScopeGracefullyWithNoTransportAndNoSnapshot,
  mutation-verified to panic without the guard.

- LOW-7: the ApplyDepAddItem spec comment said an old server would
  "silently ignore" has_spawner/thread_id; it answers its own 400 for
  an unknown member. Fix the wording and regenerate apigen/types.gen.go.

- MED-4: internal/httpclient/BUILD.bazel was missing
  served_batch_apply_large_test.go, expected_revision_contract_test.go,
  served_dualwrite_history_test.go and the
  //internal/storage/batchfixtures dep. Sync with gazelle (confirmed
  idempotent) and go_srcs.py --check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(storage): carry thread_id on a same-type dep re-add; bound batch thread_id

A same-type re-add of an existing dependency compared and rewrote only
metadata, so on both write legs (issueops.addDependencyInTx and the uow
dependency repository) a re-add naming a new ThreadID was read as
change-free and the thread was dropped, and a metadata-plus-thread re-add
stored the metadata without the thread. Both arms now read the stored
thread_id through one shared rule, issueops.ReAddedDependencyThread: a
named thread the row does not carry is a change (UPDATE, replacement-edge
journal entry, exactly one version, #5898 leg 2), and a re-add naming no
thread keeps the stored one, so callers that never carry a thread cannot
erase it. The journal's EventDep has no thread member, so a thread-only
re-add journals the same payload as the original add; the source's version
snapshot is what records the new thread, and the comments on both arms now
say so instead of promising a complete replacement edge.

planApplyBatchDepAdd bounds DepAddItem.ThreadID with CheckFieldLen, so an
over-long thread id is ErrValidation before any database work on every
leg, as the http leg's decode already refuses it, instead of an opaque
mid-transaction "string too long" on the local legs. Only the length bound
is shared: the http leg's control-character refusal stays http-only.

The HasSpawner/ThreadID docs now name the producer that actually sets
HasSpawner (an embedder lowering a graph plan, e.g. gc's ApplyGraphPlan
mapping) instead of bd create --graph, and drop the CHANGELOG reference
that had no entry behind it.

New conformance cases, wired on dolt, embeddeddolt, uow and served:
RunBatchApplyCarriesThreadIDOntoAnExistingEdge,
RunBatchApplyRefusesAThreadIDLongerThanItsColumn and
RunDependencyEditorSameTypeReAddWithChangedThreadMintsOneVersion.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(shards): add the re-add thread test to the embedded storage shard manifest

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci: trigger the first PR CI run now that the base is main

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(httpclient): gate has_spawner to waits-for edges; refuse empty thread_id

Review iteration 1 fix-up for #7265 (synthesis F1, F2, F4, F5, F6, F7).

F2: the client's CapBatchApplyDepAddLineage gate refused HasSpawner on
every edge type, but the role reads the flag on a waits-for edge only
(StampWaitsForSpawnerID is a no-op off one). An embedder plan setting it
on, say, a blocks edge works against every existing server, yet this
client refused it during a mixed-version rollout. depAddNamesSpawner now
holds the role's own rule: the flag is gated and encoded on a waits-for
edge only, and dropped elsewhere, which loses nothing the role would
store. ThreadID stays gated on every edge type, since a thread is stored
on all of them. The "spawner flag on any other edge type" subtest is
restored to an unadvertising server and now also asserts the flag is
dropped from the body, and
TestApplyBatchRefusesUnservedDepAddLineageBeforeDialing gains a
ThreadID-on-blocks case. The W-DepAddItem.HasSpawner ledger row and the
regenerated engdocs/design/http-divergence-ledger.md say the same.

F5: the document gives thread_id minLength: 1, but the decoder read an
empty string as no thread. applyDepAddItem now refuses a present, empty
thread_id with 400 invalid_value naming items[i].dep_add.thread_id, as
it does an empty type; an absent one still names no thread. New
TestApplyBatchRefusesAnEmptyThreadID; the documented-body test now
forwards has_spawner and thread_id as well.

F1: bd create --graph does set HasSpawner (buildGraphApplyBatchRequest,
on the embedded and proxied legs, since #7209), so the DepAddItem doc
names both producers again and the conformance doc's CHANGELOG pointer
is restored; the [Unreleased] entry behind it is on the base.

F4: the CapBatchApplyDepAddLineage comment said an older server would
silently drop the members; it answers either one with
400 invalid_argument/unknown_parameter.

F6: the create-edge ledger comment said no operation publishes a thread
member; batch apply's dep_add item now does.

F7: TestServedBatchApplyGcApplyGraphPlan adds a waits-for item carrying
HasSpawner and ThreadID and reads the stored row back, pinning
metadata.spawner_id to the resolved spawner and the thread on the
served leg.

F3 (the journal's EventDep carries no thread member) is not addressed
here.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <adopt-pr@gascity.com>
2026-10-09 01:05:39 +00:00

191 lines
8.3 KiB
Go

// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/sweeper.go@49d1df2f6)
// to OSS beads under the MIT license.
package httpclient
import (
"context"
"slices"
"strings"
"github.com/steveyegge/beads/internal/httpapi/apigen"
"github.com/steveyegge/beads/internal/httpclient/wire"
"github.com/steveyegge/beads/issueops"
)
// httpSweeper serves issueops.Sweeper from the sweepIssues custom method
// (design D8 row 12) — the capability behind `bd purge` and `bd prune`.
//
// S4 closed three gaps this client had against the local Sweeper role: the
// wire now carries `tier: "wisps-plane"`, `protect_live_dependents` and
// `limit` on the
// request, and `skipped.live_dependent`/`remaining` on the response — each
// behind its own behavior-capability token (CapSweepWispsPlane,
// CapSweepLiveDependents, CapSweepLimit), since each is a parameter added to
// an operation that already shipped. refuseUnservedSweep checks all three
// before the dial, exactly as counter.go's refuseUnservedScope does for
// Count's scope fields: a caller asking for something an older server
// predates learns that LOCALLY, never after a round trip that would 400
// anyway, and never by way of a silently unprotected or unbounded sweep.
//
// WHAT IS NOT DECIDED HERE, deliberately: the require-a-filter gate, the glob's
// well-formedness and the tier's own predicate. All three are the ROLE's, they
// live below the server's handler, and the server routes this request through
// the same issueops.Sweeper a local workspace uses — so re-deciding any of them
// client-side would be a second definition of the rule that keeps a workspace's
// history from being erased by an omission. They come back as
// issueops.ErrValidation, which is what the role's contract promises.
//
// The ONE thing decided here is the tier VOCABULARY, because the tier is an
// enum on the wire and this client has to map it. An unrecognized tier refuses
// before the dial rather than as a 400, which is the rule every other write on
// this surface follows for a request its own contract calls invalid.
type httpSweeper struct {
store *Store
wire WriteWire
}
var _ issueops.Sweeper = (*httpSweeper)(nil)
// Sweep dials POST /v0/beads/issues:sweep.
//
// BOTH BOOLEANS ARE SENT EXPLICITLY, and protect_referenced is the one that
// matters: the wire DEFAULTS it ON when the member is absent (an unauthenticated
// surface is where a default must be the guarded one), while the role's zero
// value is off. A client that omitted it would turn `bd prune
// --ignore-references` into a protected sweep and report the protection as
// skips the caller never asked for — a narrower answer than the request, which
// is the same failure class refuse-not-drop exists to stop, in the other
// direction.
func (s *httpSweeper) Sweep(ctx context.Context, req issueops.SweepRequest) (result issueops.SweepResult, err error) {
// Write-side parity with reads: decorates a bare *encode.RefusedError into
// *InexpressibleError so errors.As(err, &unsupported) reaches
// *storage.ErrUnsupported, same as inexpressible does for a read role. No
// Sweep path raises one today — refuseUnservedSweep's capability errors
// pass through unchanged — so this is the write-role convention's single
// defer, kept so the next refuse() here is decorated without a new one.
defer func() { err = s.store.inexpressible("Sweeper.Sweep", err) }()
tier := apigen.SweepRequestTier(req.Tier)
if !tier.Valid() {
return issueops.SweepResult{}, invalid("sweep tier %q is not %q, %q, or %q",
string(req.Tier), apigen.Ephemeral, apigen.Durable, apigen.WispsPlane)
}
// refuseUnservedSweep is the pre-dial half of the three Sweep behavior
// tokens (routes.go, beside CapIssuesSweepWispsPlane): a request naming
// the wisps-plane tier, or setting ProtectLiveDependents or Limit, asks
// for something only a server advertising the matching token answers, and
// an older server predating it answers with a guaranteed 400
// invalid_value/unknown_parameter. Checked BEFORE the dial so a caller
// never pays for a round trip that 400s anyway, and never silently gets an
// unprotected or unbounded sweep from a server too old to honor the
// request as asked.
if err := s.refuseUnservedSweep(ctx, tier, req); err != nil {
return issueops.SweepResult{}, err
}
body := apigen.SweepRequest{
Tier: tier,
ProtectReferenced: &req.ProtectReferenced,
DryRun: &req.DryRun,
}
if strings.TrimSpace(req.Actor) != "" {
// Omitted rather than sent blank: the role accepts an empty Actor — a
// deleted row leaves nothing to attribute the deletion on — and the
// server refuses an actor that is empty AFTER TRIMMING. The trim is the
// whole test, not `!= ""`: a whitespace-only Actor is an accepted
// request locally and would come back a 400 over http.
body.Actor = &req.Actor
}
if req.IDPattern != "" {
body.Pattern = &req.IDPattern
}
if req.ClosedBefore != nil {
// Copied, never aliased: SweepRequest promises implementations never
// write through a caller's pointer, and handing this one to a marshaler
// is the kind of borrow that becomes a write when a helper is added.
cutoff := *req.ClosedBefore
body.ClosedBefore = &cutoff
}
if req.ProtectLiveDependents {
body.ProtectLiveDependents = &req.ProtectLiveDependents
}
if req.Limit != 0 {
limit := int64(req.Limit)
body.Limit = &limit
}
res, err := s.wire.SweepIssues(ctx, body)
if err != nil {
return issueops.SweepResult{}, err
}
return sweepResult(res), nil
}
// refuseUnservedSweep checks each of the three S4 additions against the
// handshake snapshot, independently — a caller may ask for any subset of
// them, and an older server may serve none, some, or (not yet, but
// structurally possible) only some of the three. A request using none of
// them never consults the snapshot at all, the same short-circuit
// refuseUnservedScope uses for Count.
func (s *httpSweeper) refuseUnservedSweep(ctx context.Context, tier apigen.SweepRequestTier, req issueops.SweepRequest) error {
if tier != apigen.WispsPlane && !req.ProtectLiveDependents && req.Limit == 0 {
return nil
}
snap, err := s.store.snapshot(ctx)
if err != nil {
return err
}
// snap is nil, nil whenever Store.snapshot has no transport AND no cached
// handshake (a Store built with a nil wire): nothing was ever advertised,
// so every check below reads a nil capability list rather than
// dereferencing a nil *apigen.ContextResponse.
var capabilities []string
if snap != nil {
capabilities = snap.Capabilities
}
if tier == apigen.WispsPlane && !slices.Contains(capabilities, wire.CapSweepWispsPlane) {
return s.store.unsupportedCapability("Sweeper.Sweep", wire.CapSweepWispsPlane)
}
if req.ProtectLiveDependents && !slices.Contains(capabilities, wire.CapSweepLiveDependents) {
return s.store.unsupportedCapability("Sweeper.Sweep", wire.CapSweepLiveDependents)
}
if req.Limit != 0 && !slices.Contains(capabilities, wire.CapSweepLimit) {
return s.store.unsupportedCapability("Sweeper.Sweep", wire.CapSweepLimit)
}
return nil
}
// sweepResult projects the wire's answer onto the role's.
//
// It is a field list rather than a cast because SweepResult is deliberately not
// x-go-type-pinned on the wire: there is no canonical Go struct whose JSON
// encoding is that body, so this is the one place the two shapes are held
// together. TestSweepResultCarriesEveryWireMember is what keeps a member the
// server grows from being dropped here in silence.
func sweepResult(res *apigen.SweepResult) issueops.SweepResult {
out := issueops.SweepResult{
DryRun: res.DryRun,
Swept: res.Swept,
Dependencies: res.Dependencies,
Labels: res.Labels,
Events: res.Events,
Skipped: issueops.SweepSkips{
Pinned: res.Skipped.Pinned,
Referenced: res.Skipped.Referenced,
NotClosed: res.Skipped.NotClosed,
UnknownClosedAt: res.Skipped.UnknownClosedAt,
ClosedAtOrAfterCutoff: res.Skipped.ClosedAtOrAfterCutoff,
Unreadable: res.Skipped.Unreadable,
},
}
if res.Skipped.LiveDependent != nil {
out.Skipped.LiveDependent = *res.Skipped.LiveDependent
}
if res.Remaining != nil {
out.Remaining = int(*res.Remaining)
}
if res.ReferencedIds != nil {
out.ReferencedIDs = append([]string(nil), *res.ReferencedIds...)
}
return out
}