beads/internal/storage/batch_apply.go
Julian Knutsen e9e2c49431
Add issueops.BatchApplier: one plan, one transaction (#5480)
* Add issueops.BatchApplier: one plan, one transaction

A heterogeneous, ORDERED list of creates, updates, closes and edges applied
as one durable act. Each of BatchCreator, BatchCloser and DependencyEditor is
one verb repeated, so a caller that needs two of them needs two transactions
with a window in between; this role's unit is a PLAN.

The role, the two bodies, both decorators and the accessors on every
implementer. The wire half and the conformance contract land beside this.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Close the empty-commit-message hole a no-op batch could open

ApplyBatchCommitMessage now reads the write as well as the result. The counts
are ItemResult.Changed, which is what a CALLER observes; a request can still
write a durable row with every count at zero — a same-type edge re-add
rewrites that edge row's metadata, a re-close can settle blocked state. An
empty message for one of those is an unlabelled Dolt commit on the store legs
and a rollback of a real write on the unit-of-work leg.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Give BatchApplier the contract its two bodies justify

Thirty-nine cases over three legs and TWO independent bodies — the store legs
share issueops.ApplyBatchInTx, the unit-of-work leg cannot reach it, and the
contract header says so rather than claiming three backends agree.

The end-gate case is the one that could not have been written before: every
edge in it is legal when it is written, and only a re-validation against the
closure the whole request produced refuses the pair.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Publish batchApply on v0, and delete batchCreate's false promise

The spec first, the types from it, the handler against the types. The item
union is a TAGGED single-shape object rather than a oneOf, because the
document's x-go-type doctrine forbids composition keywords; the create item
publishes sender, metadata, status, ephemeral and no_history, none of which
batchCreateIssues has — which is why no measured caller could use that one.

The wire result stays lean: ids, changed and a revision. The hydrated
snapshot on the Go contract is what the completion hooks consume, and hooks
never fire on this surface.

OQ2: batchCreateIssues' description claimed a dependency target could name an
earlier item of the same batch. It never could — the server mints the ids and
a wire item has no name — so the paragraph is deleted rather than qualified.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Partition the two new dependency probes on the recorder's surface

The gate halves BatchApplier re-runs are repository PROBES — an ancestry walk
and a reachability walk — so recordingDepUC declares neither and the parity
test has to say so out loud rather than let embedding answer for it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Teach the proxied topology's capability golden about batchApply

The one expectation stored as data outside the package whose gates would
otherwise have caught it — which is what its own header warns about.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Restore the CAS hook case, mint 409 already_exists, and stop over-claiming

Three fixes from review.

The rebase that brought MetadataCAS in left `case *hookMetadataCAS:` with an
EMPTY BODY: a Go type switch does not fall through, so RoleFiresHooks answered
FALSE for it and #5473's Listen refusal was silently disarmed. The compiler,
vet, lint and CI all pass on that. The case is restored and — because it had no
precondition of its own, which is why nothing caught it — serve_source_test now
pins CAS the way it pins the other four, with its peel assertion.

An occupied explicit create id becomes 409 already_exists rather than 400. The
body is well-formed and stays well-formed; what refuses it is state the client
must READ, and recovery is to look at that state. A dead 409 retires for free
where a narrowed 400 breaks clients.

Two headers claimed all three legs share the store body — every clause false,
one citing the contract header as agreeing when it says the opposite — and the
plan file gave a placement reason that is MetadataCAS's, not this role's. The
end-gate claim in ADDING_AN_ISSUEOPS_ROLE.md is narrowed to the true thing: the
two LEAF checks are shared, the orchestration around them forks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Give the scheduling-edge set one definition, in types

Four packages walked {blocks, conditional-blocks, parent-child} under four
names — isSchedulingEdge, isSchedulingDep, isSchedulingDependency, and the
isApplySchedulingEdge this branch added as the fourth. Adding a fifth
scheduling type was four edits with nothing to catch a missed one.

It goes in internal/types rather than the issueops leaf the review named,
because internal/storage/issueops imports internal/storage/domain, so domain
cannot import back and two of the four copies would have survived there. Beside
the Dep* constants themselves, every walker reaches it: nine call sites, one
definition, one edit for a fifth type.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Review Sim <ci@beads.test>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 06:59:34 -07:00

392 lines
16 KiB
Go

package storage
import (
"encoding/json"
"fmt"
"strings"
"github.com/steveyegge/beads/internal/types"
"github.com/steveyegge/beads/issueops"
)
// The DATABASE-FREE half of issueops.BatchApplier: what a batch request MEANS,
// which refs it can resolve, which guards it may carry, and how a waits-for
// edge's gate metadata is spelled. Every implementation of the role runs this,
// so those rules have ONE definition rather than one per backend — and they are
// pinned by unit tests in milliseconds, where a contract case would need a
// database to observe the same thing.
//
// IT IS HERE BECAUSE BOTH BODIES NEED IT, and they sit in different packages:
// the store-backed one in internal/storage/issueops and the unit-of-work one in
// internal/storage/uow, each of which already imports this package. This is
// NOT PlanCompareAndSetKey's reason — that plan type is here because
// internal/storage/domain has to name it in a repository signature, and nothing
// in domain names this one, because this role's unit-of-work leg has its own
// body and reaches domain through ordinary use cases. It is here beside
// PlanCompareAndSetKey, ValidateMetadataKey and the rest of this plane's
// meaning because that is where a reader looks for it.
//
// What is NOT here is anything that needs a row. Whether a key's create item
// lands on the durable or the ephemeral plane, whether an id exists, whether an
// edge closes a cycle — all of that belongs to the bodies, and each of the two
// runs it against its own transaction.
// ApplyBatchPlan is a validated apply-batch request whose waits-for edges carry
// normalized gate metadata. It is what an implementation works from, so no
// backend re-derives the ref rules and the caller's request is never written
// through.
//
// It shares the request's *issueops.Issue pointers rather than deep-copying
// them. The body clones each one before writing — every single-verb Execute…
// already snapshots its own request at entry — so the ID an implementation
// assigns still lands on the result and not on the caller's issue.
type ApplyBatchPlan struct {
// Actor, Provenance, ForceIDPrefix and SkipPerEdgeCycleCheck are the
// request's, checked.
Actor string
Provenance string
ForceIDPrefix bool
SkipPerEdgeCycleCheck bool
// Items are the request's items in declaration order, with each waits-for
// dep_add's Metadata replaced by its normalized form. The slice is fresh,
// so normalization does not write through to the caller's.
Items []issueops.ApplyItem
// KeyIndex maps each create item's Key to the index that declares it. The
// body resolves refs through it and the wire half reports an unknown key
// from it, so the two cannot disagree about what a key means.
KeyIndex map[string]int
}
// PlanApplyBatch validates an apply-batch request and normalizes its waits-for
// gate metadata. It is the whole of the role's request validation: every
// implementation calls it before touching a substrate, so a refused request
// costs no database work anywhere.
//
// THE ORDER OF THE CHECKS IS PART OF THE CONTRACT, because a request can be
// wrong in several ways at once and a caller fixing them one at a time needs
// the same answer every time. Request-level shape first, then per-item shape,
// then the ref graph, then the guards.
func PlanApplyBatch(in issueops.ApplyBatchRequest) (ApplyBatchPlan, error) {
if in.Actor == "" {
return ApplyBatchPlan{}, fmt.Errorf("%w: apply batch requires an actor", issueops.ErrValidation)
}
if len(in.Items) == 0 {
return ApplyBatchPlan{}, fmt.Errorf("%w: apply batch requires at least one item", issueops.ErrValidation)
}
if len(in.Items) > issueops.MaxApplyBatchItems {
return ApplyBatchPlan{}, fmt.Errorf("%w: apply batch accepts at most %d items, got %d",
issueops.ErrValidation, issueops.MaxApplyBatchItems, len(in.Items))
}
keyIndex, err := planApplyBatchKeys(in.Items)
if err != nil {
return ApplyBatchPlan{}, err
}
plan := ApplyBatchPlan{
Actor: in.Actor,
Provenance: in.Provenance,
ForceIDPrefix: in.ForceIDPrefix,
SkipPerEdgeCycleCheck: in.SkipPerEdgeCycleCheck,
Items: make([]issueops.ApplyItem, len(in.Items)),
KeyIndex: keyIndex,
}
copy(plan.Items, in.Items)
// touched records every row an earlier item MUTATED, addressed the way the
// request addresses it: by key for a row this request creates, by id for one
// it names. It is what makes the ExpectedVersion rule statically checkable.
touched := map[string]bool{}
for i := range plan.Items {
if err := planApplyBatchItem(&plan.Items[i], i, keyIndex, touched); err != nil {
return ApplyBatchPlan{}, err
}
}
return plan, nil
}
// planApplyBatchKeys checks every item's kind/payload agreement and collects
// the create keys. It runs as its own pass because a ref may name a key
// declared by a LATER item, which is a different diagnosis from a key nothing
// declares — and telling them apart means knowing every key before checking any
// ref.
func planApplyBatchKeys(items []issueops.ApplyItem) (map[string]int, error) {
keyIndex := make(map[string]int, len(items))
for i, item := range items {
payloads := 0
for _, present := range []bool{item.Create != nil, item.Update != nil, item.Close != nil, item.DepAdd != nil} {
if present {
payloads++
}
}
if payloads != 1 {
return nil, fmt.Errorf("%w: apply batch item %d must carry exactly one payload, got %d",
issueops.ErrValidation, i, payloads)
}
var matches bool
switch item.Kind {
case issueops.ItemCreate:
matches = item.Create != nil
case issueops.ItemUpdate:
matches = item.Update != nil
case issueops.ItemClose:
matches = item.Close != nil
case issueops.ItemDepAdd:
matches = item.DepAdd != nil
default:
return nil, fmt.Errorf("%w: apply batch item %d has unknown kind %q", issueops.ErrValidation, i, item.Kind)
}
if !matches {
return nil, fmt.Errorf("%w: apply batch item %d is kind %q but carries another kind's payload",
issueops.ErrValidation, i, item.Kind)
}
if item.Create == nil {
continue
}
if item.Create.Issue == nil {
return nil, fmt.Errorf("%w: apply batch item %d requires an issue", issueops.ErrValidation, i)
}
if len(item.Create.Issue.Comments) > 0 || len(item.Create.Issue.Dependencies) > 0 {
return nil, fmt.Errorf("%w: apply batch item %d must not carry comments or dependencies on the issue; edges are their own items",
issueops.ErrValidation, i)
}
if item.Create.Key == "" {
continue
}
if prior, dup := keyIndex[item.Create.Key]; dup {
return nil, fmt.Errorf("%w: apply batch item %d reuses key %q, already declared by item %d",
issueops.ErrValidation, i, item.Create.Key, prior)
}
keyIndex[item.Create.Key] = i
}
return keyIndex, nil
}
// planApplyBatchItem validates one item's refs, guards and edge metadata, and
// records what it touches for the items after it.
func planApplyBatchItem(item *issueops.ApplyItem, index int, keyIndex map[string]int, touched map[string]bool) error {
switch item.Kind {
case issueops.ItemCreate:
return planApplyBatchCreate(item.Create, index, keyIndex, touched)
case issueops.ItemUpdate:
return planApplyBatchUpdate(item.Update, index, keyIndex, touched)
case issueops.ItemClose:
return planApplyBatchClose(item.Close, index, keyIndex, touched)
case issueops.ItemDepAdd:
// The edge is COPIED before it is normalized. plan.Items is a fresh
// slice, but its payload members are the caller's pointers, so writing
// the normalized gate metadata in place would edit the request the
// caller still owns — the one mutation this plan can make, and the one
// the role promises never happens.
edge := *item.DepAdd
if err := planApplyBatchDepAdd(&edge, index, keyIndex); err != nil {
return err
}
item.DepAdd = &edge
return nil
}
return fmt.Errorf("%w: apply batch item %d has unknown kind %q", issueops.ErrValidation, index, item.Kind)
}
// planApplyBatchCreate checks a create's metadata refs and records the row it
// mints as touched.
//
// A METADATA REF MAY NAME ANY KEY, including this item's own and one declared
// later. Every id is minted before any splice runs, so the direction cannot
// matter — see issueops.CreateItem.MetadataRefs. What it may not do is name a
// key no item declares.
func planApplyBatchCreate(item *issueops.CreateItem, index int, keyIndex map[string]int, touched map[string]bool) error {
for metaKey, ref := range item.MetadataRefs {
if metaKey == "" {
return fmt.Errorf("%w: apply batch item %d has a metadata_ref with an empty key", issueops.ErrValidation, index)
}
if err := validateApplyRef(ref, index, "metadata_ref "+metaKey); err != nil {
return err
}
if ref.Key == "" {
continue
}
if _, ok := keyIndex[ref.Key]; !ok {
return &issueops.RefError{Index: index, Member: "metadata_ref " + metaKey, Key: ref.Key}
}
}
if item.Key != "" {
touched[applyTouchKeyRef(issueops.Ref{Key: item.Key})] = true
}
if item.Issue != nil && item.Issue.ID != "" {
touched[applyTouchKeyRef(issueops.Ref{ID: item.Issue.ID})] = true
}
return nil
}
// planApplyBatchUpdate checks an update's target and guards.
func planApplyBatchUpdate(item *issueops.UpdateItem, index int, keyIndex map[string]int, touched map[string]bool) error {
if err := validateApplyTargetRef(item.Target, index, "target", keyIndex); err != nil {
return err
}
if err := checkApplyExpectedVersion(item.ExpectedVersion, item.Target, index, touched); err != nil {
return err
}
touched[applyTouchKeyRef(item.Target)] = true
return nil
}
// planApplyBatchClose checks a close's target and guard.
func planApplyBatchClose(item *issueops.CloseItem, index int, keyIndex map[string]int, touched map[string]bool) error {
if err := validateApplyTargetRef(item.Target, index, "target", keyIndex); err != nil {
return err
}
if err := checkApplyExpectedVersion(item.ExpectedVersion, item.Target, index, touched); err != nil {
return err
}
touched[applyTouchKeyRef(item.Target)] = true
return nil
}
// planApplyBatchDepAdd checks an edge's endpoints and type and normalizes its
// gate metadata.
//
// IT RECORDS NOTHING AS TOUCHED, and that is a decision rather than an
// oversight. The ExpectedVersion rule above refuses a guard on a row this
// request has already REWRITTEN, and an edge write is a change to the graph
// rather than to either endpoint's row: the role promises nothing about whether
// it moves the source's version token. A later guard on that source is
// therefore left to the substrate, where a genuine mismatch is
// ErrVersionMismatch — an honest refusal — rather than being refused up front
// as a request a caller could not have composed.
func planApplyBatchDepAdd(item *issueops.DepAddItem, index int, keyIndex map[string]int) error {
if err := validateApplyTargetRef(item.Source, index, "source", keyIndex); err != nil {
return err
}
if err := validateApplyTargetRef(item.Target, index, "target", keyIndex); err != nil {
return err
}
if item.Source == item.Target {
return fmt.Errorf("%w: apply batch item %d: %s cannot depend on itself",
issueops.ErrSelfDependency, index, applyRefLabel(item.Source))
}
if !item.Type.IsValid() {
return fmt.Errorf("%w: apply batch item %d requires a dependency type (max %d chars)",
issueops.ErrValidation, index, types.MaxDependencyTypeLen)
}
metadata, err := normalizeApplyEdgeMetadata(item.Type, item.Metadata)
if err != nil {
return fmt.Errorf("%w: apply batch item %d: %v", issueops.ErrValidation, index, err)
}
item.Metadata = metadata
return nil
}
// checkApplyExpectedVersion refuses a version guard on a row an earlier item of
// this request already mutated.
//
// IT IS A REQUEST-SHAPE RULE, not a race. The token is server-minted and
// rewritten by the write, so mid-request there is no value a caller COULD send:
// the pre-request token is stale by construction, and a row this request just
// created never had one the caller could read. Refusing statically says so;
// letting it through would answer every such request with ErrVersionMismatch
// and leave the caller looking for a concurrent writer that does not exist.
//
// ExpectedStatus and ExpectedAssignee carry no such rule, and the difference is
// that a caller CAN know what its own earlier item set them to.
func checkApplyExpectedVersion(expected *int64, target issueops.Ref, index int, touched map[string]bool) error {
if expected == nil || !touched[applyTouchKeyRef(target)] {
return nil
}
return fmt.Errorf("%w: apply batch item %d guards on a row version, but an earlier item in this request already wrote %s; "+
"the version token is minted by the write and cannot be known mid-request",
issueops.ErrValidation, index, applyRefLabel(target))
}
// validateApplyTargetRef checks a ref used to ADDRESS a row: exactly one member
// set, and a key that reaches BACKWARD.
func validateApplyTargetRef(ref issueops.Ref, index int, member string, keyIndex map[string]int) error {
if err := validateApplyRef(ref, index, member); err != nil {
return err
}
if ref.Key == "" {
return nil
}
declaredAt, ok := keyIndex[ref.Key]
if !ok {
return &issueops.RefError{Index: index, Member: member, Key: ref.Key}
}
if declaredAt >= index {
return &issueops.RefError{Index: index, Member: member, Key: ref.Key, DeclaredLater: true}
}
return nil
}
// validateApplyRef checks the exactly-one rule every ref answers to.
func validateApplyRef(ref issueops.Ref, index int, member string) error {
switch {
case ref.Key == "" && ref.ID == "":
return fmt.Errorf("%w: apply batch item %d: %s must name a key or an id", issueops.ErrValidation, index, member)
case ref.Key != "" && ref.ID != "":
return fmt.Errorf("%w: apply batch item %d: %s names both key %q and id %q; exactly one",
issueops.ErrValidation, index, member, ref.Key, ref.ID)
}
return nil
}
// applyTouchKeyRef renders a ref as the address the touched set is keyed by.
// The two namespaces are kept apart so a key and an id that happen to read the
// same string are not confused for one row.
func applyTouchKeyRef(ref issueops.Ref) string {
if ref.Key != "" {
return "key:" + ref.Key
}
return "id:" + ref.ID
}
// applyRefLabel renders a ref for a message.
func applyRefLabel(ref issueops.Ref) string {
if ref.Key != "" {
return fmt.Sprintf("key %q", ref.Key)
}
return ref.ID
}
// normalizeApplyEdgeMetadata returns the metadata an edge is STORED with.
//
// A WAITS-FOR ROW MUST BE SELF-DESCRIBING. Readers that predate the gate's
// introduction do not default a missing one, so an absent, blank or `{}`
// metadata is written as {"gate":"all-children"} — the rule
// types.NewGraphEdgeDependency already applies on every other path that writes
// one, reached here so this role cannot drift from it. A metadata that names a
// gate keeps it, along with the spawner and also-blocks members a caller may
// carry; a gate that is neither known value is refused.
//
// Every other edge type's metadata is passed through unchanged, checked only
// for being well-formed JSON when it is present at all: the blob is
// type-specific and this role does not know the types.
func normalizeApplyEdgeMetadata(depType types.DependencyType, metadata string) (string, error) {
trimmed := strings.TrimSpace(metadata)
if depType != types.DepWaitsFor {
if trimmed == "" {
return "", nil
}
if !json.Valid([]byte(trimmed)) {
return "", fmt.Errorf("edge metadata is not well-formed JSON")
}
return metadata, nil
}
meta := types.WaitsForMeta{}
if trimmed != "" && trimmed != "{}" {
if err := json.Unmarshal([]byte(trimmed), &meta); err != nil {
return "", fmt.Errorf("waits-for metadata is not a well-formed gate object: %w", err)
}
}
if meta.Gate == "" {
meta.Gate = types.WaitsForAllChildren
}
if !types.IsValidWaitsForGate(meta.Gate) {
return "", fmt.Errorf("waits-for gate %q is neither %q nor %q",
meta.Gate, types.WaitsForAllChildren, types.WaitsForAnyChildren)
}
raw, err := json.Marshal(meta)
if err != nil {
return "", fmt.Errorf("serializing waits-for metadata: %w", err)
}
return string(raw), nil
}