beads/internal/storage/batchfixtures
Julian Knutsen ba12ffa74e
perf(issueops): indexed dep-add cycle/hierarchy walks; pin blocked-state lookup plans; split waits-for spawner checks (#7224)
* perf(issueops): index each step of the per-edge cycle and hierarchy walks

The single-edge cycle and hierarchy probes (AddDependencyInTx, i.e. bd dep
add and apply-batch dep_add items, the domain stack's HasCycle /
ValidateBlockingHierarchy, and apply-batch's end-gate hierarchy check) joined
the recursive frontier against a UNION of dependencies and wisp_dependencies.
Dolt cannot index that derived table, so every recursion step rescanned every
matching edge: on a 50k-issue / 100k-edge database one dep add took minutes
(an apply-batch of 1 create + 2 dep adds ~10 min).

Give each dependency table its own recursive member joined directly on
d.issue_id, so each step is an index lookup, and pin the lookup plan with
JOIN_ORDER/LOOKUP_JOIN hints: without them the Dolt sql-server planner picks
a type-index scan join while table statistics are not yet available (right
after a commit), back to minutes. Same reachable set (UNION distinct merges
the members per step), same edge types, same error paths, and still one
statement per probe.

Adds a "depadd" createbatchequiv scenario with a golden digest recorded from
the pre-change code: every cycle/hierarchy probe over 17 nodes x 3 table
sets (mixed tables, every target column, external and missing ids), then 20
AddDependencyInTx calls covering cycles through both tables, hierarchy
conflicts in both directions, self, missing endpoints, type conflict,
external and non-scheduling edges.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(issueops): pin lookup joins in the batched blocked-state recompute

The batched mark/unmark UPDATEs decide membership through the scoped
should-be-blocked union. On the Dolt sql-server its planner intermittently
(depending on when the background statistics refresh lands) turned the
union's issues legs around: it drove them from an index scan of every open
issue through (is_blocked, status) and probed dependencies per row. Over an
import's few thousand uncommitted issues that is ~3 s per leg and 6-10 s per
mark/unmark statement for each 200-id chunk, where the lookup plan takes
~0.1 s; an import of 5000 issues with edges took ~28 min and came close to
the 10 s read timeout. EXPLAIN on the server engine showed the two plans;
two of three identical 2000-issue imports hit the slow one (4-5 min vs 14 s).

Pin each joined leg of the scoped union to drive from the batch's own
dependency rows and look the target/parent up by primary key
(JOIN_ORDER/LOOKUP_JOIN hints, ignored by MySQL). The SQL is otherwise
unchanged, so results are identical (createbatchequiv golden digests,
including is_blocked, pass on both engines) and the statement count is the
same. The unscoped union used by full repair and the doctor count walks
whole tables and keeps the planner's choice.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(storage): pin the hinted graph-walk and blocked-state plans with EXPLAIN

The JOIN_ORDER/LOOKUP_JOIN hints on the per-edge reachability walks and the
batched blocked-state recompute were only pinned textually: the suite passed
with them deleted or naming a wrong alias. TestGraphWalkPlansHonorJoinHints
(embedded) records the statements the production entry points send
(WouldCreateSchedulingCycleInTx, CheckBlockingHierarchyInTx,
RecomputeIsBlockedInTx) through a DBTX wrapper, EXPLAINs exactly those, and
asserts each recursive member is a lookup join keyed by the frontier node and
each joined union leg a primary-key lookup, never an is_blocked/status scan.
Deleting a hint or misspelling its alias fails it.

The test found a remaining planner trap in the ancestor walk: with
"d.type = 'parent-child'" in the join condition, the embedded planner
satisfied the wisp_dependencies member through the type index keyed by the
constant, a lookup returning every parent-child row of the table per
frontier node. The type filter now sits in each member's projection (a row
of another type yields a NULL node, which joins nothing and never equals the
goal), so issue_id is the join's only predicate. The depadd golden, recorded
from the pre-change code, still matches.

Also corrects the hint comments: MySQL honors JOIN_ORDER (the order it forces
is the wanted one) and ignores the unknown LOOKUP_JOIN with a warning.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(createbatchequiv): check run scenarios against the golden only; probe a stored parent cycle

A run scenario (depadd) drives code the create fast paths do not gate, so
its fast-vs-per-row comparison compared a body with itself. Apply it once,
check it against its golden digest (the pre-change code's answers), and fail
if the run recorded nothing.

depadd now plants a parent-child 2-cycle straight into dependencies, as
legacy or merged data can hold even though every write path refuses it, and
probes across it plus three adds touching it, so termination of the walks on
stored cycles is covered. Golden re-recorded from a93498c81 (pre-change
walks); the current code matches it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* perf(issueops): index the waits-for gate's child lookups; waits-for bench fixture

The large-graph benchmark gains a formula-shaped fan-out
(batchfixtures.WaitsForInserts: one spawner, 1000 parent-child children,
20 waiters over every waits-for gate shape) and two shapes over it:
create+waits-for and closing a spawner child (which recomputes every waiter
through the waits-for gate).

It was not a statistics-dependent plan flip but a consistently bad plan:
each "spawner has an open/closed child" EXISTS matched both spawner columns
with one OR, which no index serves, so every correlated evaluation scanned
the edge table. On 50k issues / 100k edges, closing one child took ~39 s
per close on embedded, and on the server create+waits-for hit the 30-minute
read timeout. The gate now splits each EXISTS per spawner column (one per
dependency table and column, each served by its target-column index) and
pins the child lookup by primary key (JOIN_ORDER/LOOKUP_JOIN).
EXISTS(a OR b) is EXISTS(a) OR EXISTS(b), so the result is unchanged. The
template is shared with the unscoped full repair, where a per-row correlated
lookup is the right plan too.

Close spawner child per close: emb 39.5 s -> 3.3 s; create+waits-for emb
2.1 s -> 0.46 s, srv >30 min (read timeout) -> 0.29 s; srv close spawner
child 3.2 s (not reached before).

Adds a "waitsfor" createbatchequiv scenario, golden recorded from
a93498c81 (pre-change gate): issue and wisp spawners with issue and wisp
children, waiters in all five gate shapes, then closes, reopens, a scoped
recompute and the full repair, snapshotting every is_blocked. A mutation
dropping the wisp-spawner legs fails it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(createbatchequiv): compare dependency metadata canonicalized across engines

The embedded engine and the Dolt sql-server serialize a JSON metadata value
differently (key order, spacing), so the waitsfor golden recorded on embedded
failed on the server for its gate metadata alone. Re-serialize each
dependency row's metadata through encoding/json before comparing. Rows with
'{}' metadata are unchanged, so the other goldens keep their digests;
waitsfor is re-recorded from a93498c81 and now passes on both engines.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* build(bazel): auto-sync BUILD files

Applied from the bazel-sync-patch artifact of https://github.com/gastownhall/beads/actions/runs/37272437653
(`make bazel-sync`: gazelle, tools/bazel/go_srcs.py, bazel mod tidy).
Only BUILD.bazel, MODULE.bazel and MODULE.bazel.lock are ever pushed; see
scripts/bazel-autofix-push.sh.

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-04 23:43:32 -07:00
..
batchfixtures.go feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
batchfixtures_test.go feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
BUILD.bazel perf(issueops): indexed dep-add cycle/hierarchy walks; pin blocked-state lookup plans; split waits-for spawner checks (#7224) 2026-10-04 23:43:32 -07:00
largegraph.go perf(issueops): batch create/import without per-row round trips; on-demand graph checks (#7205) 2026-10-04 13:56:12 -07:00
uncapped_plan.go feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
waitsfor.go perf(issueops): indexed dep-add cycle/hierarchy walks; pin blocked-state lookup plans; split waits-for spawner checks (#7224) 2026-10-04 23:43:32 -07:00