beads/cmd/bd/export_source_wisp_subset_test.go
Bee 91aeeb0548
perf(storage/db): batch fetchIssuesByIDs, GetBlockingInfo and loadStatusByID at 200 ids; delete tolerates only a missing wisps table (#6444)
* perf(storage/db): batch the bulk IN-list readers at 200 ids; export narrows the wisp-plane reads

bd export on a 19,413-issue rig in proxied-server (UOW) mode took 264-287s.
The cmd layer is bulk, but uowExportSource.LoadExportRelations handed ALL
ids to the domain use cases, and the domain/db readers built ONE IN (...)
list per statement: 19,409 placeholders each for labels, wisp_labels,
wisp_comments rows and counts, dependencies counts and wisp_dependencies
counts, at 24-60s a statement (~240s of the run). Dolt is super-linear in
IN-list size, prepared or literal alike: on the same server the labels rows
took 31s as one statement and 0.80s as 98 batches of 200; a 200-id batch
is 3-7ms. The wisp readers were also handed every durable id (rows=0 back).

- internal/storage/domain/db/batching.go: forEachIDBatch at queryBatchSize
  (200, mirroring issueops). commentSQLRepositoryImpl.CountsByIssueIDs /
  ListByIssueIDs, labelSQLRepositoryImpl.ListByIssueIDs,
  dependencySQLRepositoryImpl.ListByIssueIDs / CountsByIssueIDs and
  issueSQLRepositoryImpl.GetByIDs loop over batches and merge; an id lands in
  exactly one batch, so per-id groups and their ORDER BY survive. Error
  strings are byte-identical; empty input still issues no statement. The
  three hand-rolled loops (getLabelsFromTable, getDependencyRecordsFromTable,
  fetchCountsByIDs) collapse onto the helper.
- cmd/bd/export_source.go: LoadExportRelations reads wisps-table membership
  once (WispPlaneIDs, now batched through GetByIDs) and sends only wisp ids
  to GetLabelsForWisps / GetCommentsForWisps / GetWispCommentCounts;
  CountsByWispIDs keeps every id because its inbound leg counts wisp->durable
  edges under the durable target. Membership that could not be read fails
  OPEN to every id (wispReaderIDs), never to "no wisps".
- WispPlaneIDs now tolerates only a missing WISPS table
  (dberrors.IsMissingTable), not any table-not-exist: its read joins leases,
  and the blanket check reported a broken database as an empty wisp plane
  (the issue_search.go missingOptionalWispTable lesson). Behavior change on
  purpose: a rig missing leases now fails bd export with "failed to classify
  wisp-plane rows" instead of silently exporting without markers.

Measured in the same clone against the same shared server, 19,433 rows:
stock 264s / 266s / 287s -> batched 16.9s / 17.8s. Stock-vs-batched byte
oracle: 19,363 of 19,433 records byte-identical; every differing record is
a live lease/heartbeat field or an updated_at inside the seven minutes
between the two runs.

Tests: internal/storage/domain/db TestBulkReadersBatch (a recording Runner
over the real Dolt suite: ceil(N/200) statements per leg, <=200 id
placeholders each, merge across batch boundaries, the Types filter's extra
placeholder, every reader routed to the wisp tables, GetByIDs, empty input),
cmd/bd TestWispPlaneSubset + TestWispReaderIDs, and the proxied export
integration fixture's wisp now carries a label and a comment that the --all
export must emit.

Follow-up (not here): fetchIssuesByIDs (SearchIssues Limit=0),
GetBlockingInfo and loadStatusByID still build one unbounded IN list, and
issue_delete.go's two GetByIDs(wisps) calls keep the blanket
IsTableNotExist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjKMk7ubzrgpEDj2JRAbu

* perf(storage/db): batch fetchIssuesByIDs, GetBlockingInfo and loadStatusByID at 200 ids; delete tolerates only a missing wisps table (wy-sm01o2)

Follow-up to #6267 (wy-237yfi), whose own body names these three readers as
the residual unbounded IN lists in internal/storage/domain/db. Dolt's cost per
IN-list placeholder is super-linear, so one statement over N ids is far slower
than ceil(N/200) statements over 200 each.

- issue_search.go fetchIssuesByIDs: one SELECT per <=200-id batch, merged into
  the same map. `ordered` is the scan order concatenated across batches and
  hydration runs ONCE over the merged slice (hydrateIssues batches its own
  relation reads since #6267), so callers get the same map and the same
  hydration for one round trip per 200 ids instead of one 19k-placeholder
  statement (export's SearchIssues with Limit=0 reaches it with ~19k ids).
  The signature and the missing-wisp-table classification are unchanged.
- dependency.go GetBlockingInfo: both legs batch through forEachIDBatch and
  their blockingRows are appended across batches; the merge that follows is
  untouched. Each leg keys its result by a column the IN list constrains
  (outbound issue_id, inbound the dependency target), and an id lands in
  exactly one batch, so every row sharing a key comes from the same batch.
  loadStatusByID batches its id list per table; the cross-plane duplicate-id
  error is unchanged, because an id is still recorded under `issues` before
  the `wisps` pass reaches it. GetBlockingInfoAcrossIssuesAndWisps needs no
  change: it only calls GetBlockingInfo twice and merges the maps.
- issue_delete.go: previewDelete's and collectConnectedIssues' two
  GetByIDs(UseWispsTable) calls read `FROM wisps LEFT JOIN leases`, so their
  blanket dberrors.IsTableNotExist reported a rig missing `leases` as a rig
  with no wisp plane (previewDelete listed live wisps under NotFound;
  hydrateNeighbors returned them unhydrated). Both now tolerate only
  dberrors.IsMissingTable(err, "wisps"), the fix #6267 applied to
  WispPlaneIDs. The other IsTableNotExist sites in that file are left alone:
  they guard reads whose FROM is the optional table itself
  (wisp_dependencies via ListByIssueIDs(UseWispsTable)).

Tests: internal/storage/domain/db TestBulkReadersBatch gains FetchIssuesByIDs
(ceil(N/200) statements, <=200 placeholders, rows on both sides of a boundary
merged, and exactly ONE label hydration over the merged slice),
GetBlockingInfoOutbound, GetBlockingInfoInbound, LoadStatusByID (both planes,
one statement per batch each) and LoadStatusByIDDuplicateAcrossTables; the
empty-input subtest now covers all three readers issuing no statement.
internal/storage/domain/issue_delete_missing_table_test.go proves a missing
`leases` (or `wisp_labels`) is no longer swallowed at either GetByIDs(wisps)
site while a missing `wisps` table still is, with an unrelated-error control.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(storage/db): consume the aborted duplicate-plane run's statements inside its own subtest (wy-sm01o2)

Salvaged WIP from the quota-killed executor (session-limit at 00:19Z): the
duplicate-plane loadStatusByID call aborts mid-run, so reset the recorder
before it and assert its statements under the cap in place, rather than
leaving them to pollute the next subtest.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rrz4umA2L5QU8s9EmCBPw8

* docs(cmd/bd): correct the LoadExportRelations plane-partition doc block

The header still documented pre-PR full-set wisp reads ("both planes are
queried with the FULL ID set and merged"), which this PR made false for the
wisp side. That block is the load-bearing record of the plane-partition
invariant, so a future editor trusting it could "restore" full-set wisp reads
and silently undo the wy-237yfi fix.

- The first bullet now describes the partitioned reads: membership is read
  from the wisps table itself (WispPlaneIDs) rather than inferred from row
  flags, the durable plane is queried with the full id set, the wisp plane
  only with that membership subset (wispReaderIDs), and unreadable membership
  fails OPEN to the full list. The "row flags are NOT a substitute" rationale
  is kept deliberately: it is *why* membership is read from the table at all.
- The IsTableNotExist sentence is extended, not replaced. It is still true of
  the three single-table wisp-relation readers, which correctly keep the
  blanket check; only the lease-joined membership probe (WispPlaneIDs) is
  pinned to a missing `wisps` table. Replacing the sentence outright would
  have introduced a new inaccuracy.

Maintainer review fixup for adopt-pr #6444. Documentation only: the tree is
AST-identical after stripping comments, so no behavior can change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <julianknutsen@users.noreply.github.com>
2026-09-12 02:19:39 -07:00

43 lines
1.5 KiB
Go

package main
import (
"reflect"
"testing"
)
// wispPlaneSubset feeds the wisp-plane relation readers in
// uowExportSource.LoadExportRelations: only ids that live in the wisps table
// go to them, in input order; a set that marks none sends nothing, so those
// readers issue no statement at all (wy-237yfi).
func TestWispPlaneSubset(t *testing.T) {
ids := []string{"a", "b", "c", "d"}
if got := wispPlaneSubset(ids, map[string]bool{}); got != nil {
t.Fatalf("empty set: want nil, got %v", got)
}
got := wispPlaneSubset(ids, map[string]bool{"d": true, "b": true, "zz": true, "a": false})
if !reflect.DeepEqual(got, []string{"b", "d"}) {
t.Fatalf("want [b d] in input order, got %v", got)
}
if got := wispPlaneSubset(nil, map[string]bool{"a": true}); got != nil {
t.Fatalf("no ids: want nil, got %v", got)
}
}
// wispReaderIDs is the policy on top: a nil set is "membership unreadable"
// and must FAIL OPEN to every id — narrowing on it would silently export
// every wisp with no labels and no comments — while a non-nil empty set is a
// real answer (no wisps) and sends nothing.
func TestWispReaderIDs(t *testing.T) {
ids := []string{"a", "b", "c"}
if got := wispReaderIDs(ids, nil); !reflect.DeepEqual(got, ids) {
t.Fatalf("nil set must fail open to all ids, got %v", got)
}
if got := wispReaderIDs(ids, map[string]bool{}); got != nil {
t.Fatalf("empty non-nil set means no wisps: want nil, got %v", got)
}
if got := wispReaderIDs(ids, map[string]bool{"c": true}); !reflect.DeepEqual(got, []string{"c"}) {
t.Fatalf("want [c], got %v", got)
}
}