* perf(storage/db): batch the bulk IN-list readers at 200 ids; export narrows the wisp-plane reads
bd export on a 19,413-issue rig in proxied-server (UOW) mode took 264-287s.
The cmd layer is bulk, but uowExportSource.LoadExportRelations handed ALL
ids to the domain use cases, and the domain/db readers built ONE IN (...)
list per statement: 19,409 placeholders each for labels, wisp_labels,
wisp_comments rows and counts, dependencies counts and wisp_dependencies
counts, at 24-60s a statement (~240s of the run). Dolt is super-linear in
IN-list size, prepared or literal alike: on the same server the labels rows
took 31s as one statement and 0.80s as 98 batches of 200; a 200-id batch
is 3-7ms. The wisp readers were also handed every durable id (rows=0 back).
- internal/storage/domain/db/batching.go: forEachIDBatch at queryBatchSize
(200, mirroring issueops). commentSQLRepositoryImpl.CountsByIssueIDs /
ListByIssueIDs, labelSQLRepositoryImpl.ListByIssueIDs,
dependencySQLRepositoryImpl.ListByIssueIDs / CountsByIssueIDs and
issueSQLRepositoryImpl.GetByIDs loop over batches and merge; an id lands in
exactly one batch, so per-id groups and their ORDER BY survive. Error
strings are byte-identical; empty input still issues no statement. The
three hand-rolled loops (getLabelsFromTable, getDependencyRecordsFromTable,
fetchCountsByIDs) collapse onto the helper.
- cmd/bd/export_source.go: LoadExportRelations reads wisps-table membership
once (WispPlaneIDs, now batched through GetByIDs) and sends only wisp ids
to GetLabelsForWisps / GetCommentsForWisps / GetWispCommentCounts;
CountsByWispIDs keeps every id because its inbound leg counts wisp->durable
edges under the durable target. Membership that could not be read fails
OPEN to every id (wispReaderIDs), never to "no wisps".
- WispPlaneIDs now tolerates only a missing WISPS table
(dberrors.IsMissingTable), not any table-not-exist: its read joins leases,
and the blanket check reported a broken database as an empty wisp plane
(the issue_search.go missingOptionalWispTable lesson). Behavior change on
purpose: a rig missing leases now fails bd export with "failed to classify
wisp-plane rows" instead of silently exporting without markers.
Measured in the same clone against the same shared server, 19,433 rows:
stock 264s / 266s / 287s -> batched 16.9s / 17.8s. Stock-vs-batched byte
oracle: 19,363 of 19,433 records byte-identical; every differing record is
a live lease/heartbeat field or an updated_at inside the seven minutes
between the two runs.
Tests: internal/storage/domain/db TestBulkReadersBatch (a recording Runner
over the real Dolt suite: ceil(N/200) statements per leg, <=200 id
placeholders each, merge across batch boundaries, the Types filter's extra
placeholder, every reader routed to the wisp tables, GetByIDs, empty input),
cmd/bd TestWispPlaneSubset + TestWispReaderIDs, and the proxied export
integration fixture's wisp now carries a label and a comment that the --all
export must emit.
Follow-up (not here): fetchIssuesByIDs (SearchIssues Limit=0),
GetBlockingInfo and loadStatusByID still build one unbounded IN list, and
issue_delete.go's two GetByIDs(wisps) calls keep the blanket
IsTableNotExist.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDjKMk7ubzrgpEDj2JRAbu
* perf(storage/db): batch fetchIssuesByIDs, GetBlockingInfo and loadStatusByID at 200 ids; delete tolerates only a missing wisps table (wy-sm01o2)
Follow-up to #6267 (wy-237yfi), whose own body names these three readers as
the residual unbounded IN lists in internal/storage/domain/db. Dolt's cost per
IN-list placeholder is super-linear, so one statement over N ids is far slower
than ceil(N/200) statements over 200 each.
- issue_search.go fetchIssuesByIDs: one SELECT per <=200-id batch, merged into
the same map. `ordered` is the scan order concatenated across batches and
hydration runs ONCE over the merged slice (hydrateIssues batches its own
relation reads since #6267), so callers get the same map and the same
hydration for one round trip per 200 ids instead of one 19k-placeholder
statement (export's SearchIssues with Limit=0 reaches it with ~19k ids).
The signature and the missing-wisp-table classification are unchanged.
- dependency.go GetBlockingInfo: both legs batch through forEachIDBatch and
their blockingRows are appended across batches; the merge that follows is
untouched. Each leg keys its result by a column the IN list constrains
(outbound issue_id, inbound the dependency target), and an id lands in
exactly one batch, so every row sharing a key comes from the same batch.
loadStatusByID batches its id list per table; the cross-plane duplicate-id
error is unchanged, because an id is still recorded under `issues` before
the `wisps` pass reaches it. GetBlockingInfoAcrossIssuesAndWisps needs no
change: it only calls GetBlockingInfo twice and merges the maps.
- issue_delete.go: previewDelete's and collectConnectedIssues' two
GetByIDs(UseWispsTable) calls read `FROM wisps LEFT JOIN leases`, so their
blanket dberrors.IsTableNotExist reported a rig missing `leases` as a rig
with no wisp plane (previewDelete listed live wisps under NotFound;
hydrateNeighbors returned them unhydrated). Both now tolerate only
dberrors.IsMissingTable(err, "wisps"), the fix#6267 applied to
WispPlaneIDs. The other IsTableNotExist sites in that file are left alone:
they guard reads whose FROM is the optional table itself
(wisp_dependencies via ListByIssueIDs(UseWispsTable)).
Tests: internal/storage/domain/db TestBulkReadersBatch gains FetchIssuesByIDs
(ceil(N/200) statements, <=200 placeholders, rows on both sides of a boundary
merged, and exactly ONE label hydration over the merged slice),
GetBlockingInfoOutbound, GetBlockingInfoInbound, LoadStatusByID (both planes,
one statement per batch each) and LoadStatusByIDDuplicateAcrossTables; the
empty-input subtest now covers all three readers issuing no statement.
internal/storage/domain/issue_delete_missing_table_test.go proves a missing
`leases` (or `wisp_labels`) is no longer swallowed at either GetByIDs(wisps)
site while a missing `wisps` table still is, with an unrelated-error control.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* test(storage/db): consume the aborted duplicate-plane run's statements inside its own subtest (wy-sm01o2)
Salvaged WIP from the quota-killed executor (session-limit at 00:19Z): the
duplicate-plane loadStatusByID call aborts mid-run, so reset the recorder
before it and assert its statements under the cap in place, rather than
leaving them to pollute the next subtest.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rrz4umA2L5QU8s9EmCBPw8
* docs(cmd/bd): correct the LoadExportRelations plane-partition doc block
The header still documented pre-PR full-set wisp reads ("both planes are
queried with the FULL ID set and merged"), which this PR made false for the
wisp side. That block is the load-bearing record of the plane-partition
invariant, so a future editor trusting it could "restore" full-set wisp reads
and silently undo the wy-237yfi fix.
- The first bullet now describes the partitioned reads: membership is read
from the wisps table itself (WispPlaneIDs) rather than inferred from row
flags, the durable plane is queried with the full id set, the wisp plane
only with that membership subset (wispReaderIDs), and unreadable membership
fails OPEN to the full list. The "row flags are NOT a substitute" rationale
is kept deliberately: it is *why* membership is read from the table at all.
- The IsTableNotExist sentence is extended, not replaced. It is still true of
the three single-table wisp-relation readers, which correctly keep the
blanket check; only the lease-joined membership probe (WispPlaneIDs) is
pinned to a missing `wisps` table. Replacing the sentence outright would
have introduced a new inaccuracy.
Maintainer review fixup for adopt-pr #6444. Documentation only: the tree is
AST-identical after stripping comments, so no behavior can change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <julianknutsen@users.noreply.github.com>