* perf(storage/db): batch the bulk IN-list readers at 200 ids; export narrows the wisp-plane reads bd export on a 19,413-issue rig in proxied-server (UOW) mode took 264-287s. The cmd layer is bulk, but uowExportSource.LoadExportRelations handed ALL ids to the domain use cases, and the domain/db readers built ONE IN (...) list per statement: 19,409 placeholders each for labels, wisp_labels, wisp_comments rows and counts, dependencies counts and wisp_dependencies counts, at 24-60s a statement (~240s of the run). Dolt is super-linear in IN-list size, prepared or literal alike: on the same server the labels rows took 31s as one statement and 0.80s as 98 batches of 200; a 200-id batch is 3-7ms. The wisp readers were also handed every durable id (rows=0 back). - internal/storage/domain/db/batching.go: forEachIDBatch at queryBatchSize (200, mirroring issueops). commentSQLRepositoryImpl.CountsByIssueIDs / ListByIssueIDs, labelSQLRepositoryImpl.ListByIssueIDs, dependencySQLRepositoryImpl.ListByIssueIDs / CountsByIssueIDs and issueSQLRepositoryImpl.GetByIDs loop over batches and merge; an id lands in exactly one batch, so per-id groups and their ORDER BY survive. Error strings are byte-identical; empty input still issues no statement. The three hand-rolled loops (getLabelsFromTable, getDependencyRecordsFromTable, fetchCountsByIDs) collapse onto the helper. - cmd/bd/export_source.go: LoadExportRelations reads wisps-table membership once (WispPlaneIDs, now batched through GetByIDs) and sends only wisp ids to GetLabelsForWisps / GetCommentsForWisps / GetWispCommentCounts; CountsByWispIDs keeps every id because its inbound leg counts wisp->durable edges under the durable target. Membership that could not be read fails OPEN to every id (wispReaderIDs), never to "no wisps". - WispPlaneIDs now tolerates only a missing WISPS table (dberrors.IsMissingTable), not any table-not-exist: its read joins leases, and the blanket check reported a broken database as an empty wisp plane (the issue_search.go missingOptionalWispTable lesson). Behavior change on purpose: a rig missing leases now fails bd export with "failed to classify wisp-plane rows" instead of silently exporting without markers. Measured in the same clone against the same shared server, 19,433 rows: stock 264s / 266s / 287s -> batched 16.9s / 17.8s. Stock-vs-batched byte oracle: 19,363 of 19,433 records byte-identical; every differing record is a live lease/heartbeat field or an updated_at inside the seven minutes between the two runs. Tests: internal/storage/domain/db TestBulkReadersBatch (a recording Runner over the real Dolt suite: ceil(N/200) statements per leg, <=200 id placeholders each, merge across batch boundaries, the Types filter's extra placeholder, every reader routed to the wisp tables, GetByIDs, empty input), cmd/bd TestWispPlaneSubset + TestWispReaderIDs, and the proxied export integration fixture's wisp now carries a label and a comment that the --all export must emit. Follow-up (not here): fetchIssuesByIDs (SearchIssues Limit=0), GetBlockingInfo and loadStatusByID still build one unbounded IN list, and issue_delete.go's two GetByIDs(wisps) calls keep the blanket IsTableNotExist. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RDjKMk7ubzrgpEDj2JRAbu * perf(storage/db): batch fetchIssuesByIDs, GetBlockingInfo and loadStatusByID at 200 ids; delete tolerates only a missing wisps table (wy-sm01o2) Follow-up to #6267 (wy-237yfi), whose own body names these three readers as the residual unbounded IN lists in internal/storage/domain/db. Dolt's cost per IN-list placeholder is super-linear, so one statement over N ids is far slower than ceil(N/200) statements over 200 each. - issue_search.go fetchIssuesByIDs: one SELECT per <=200-id batch, merged into the same map. `ordered` is the scan order concatenated across batches and hydration runs ONCE over the merged slice (hydrateIssues batches its own relation reads since #6267), so callers get the same map and the same hydration for one round trip per 200 ids instead of one 19k-placeholder statement (export's SearchIssues with Limit=0 reaches it with ~19k ids). The signature and the missing-wisp-table classification are unchanged. - dependency.go GetBlockingInfo: both legs batch through forEachIDBatch and their blockingRows are appended across batches; the merge that follows is untouched. Each leg keys its result by a column the IN list constrains (outbound issue_id, inbound the dependency target), and an id lands in exactly one batch, so every row sharing a key comes from the same batch. loadStatusByID batches its id list per table; the cross-plane duplicate-id error is unchanged, because an id is still recorded under `issues` before the `wisps` pass reaches it. GetBlockingInfoAcrossIssuesAndWisps needs no change: it only calls GetBlockingInfo twice and merges the maps. - issue_delete.go: previewDelete's and collectConnectedIssues' two GetByIDs(UseWispsTable) calls read `FROM wisps LEFT JOIN leases`, so their blanket dberrors.IsTableNotExist reported a rig missing `leases` as a rig with no wisp plane (previewDelete listed live wisps under NotFound; hydrateNeighbors returned them unhydrated). Both now tolerate only dberrors.IsMissingTable(err, "wisps"), the fix #6267 applied to WispPlaneIDs. The other IsTableNotExist sites in that file are left alone: they guard reads whose FROM is the optional table itself (wisp_dependencies via ListByIssueIDs(UseWispsTable)). Tests: internal/storage/domain/db TestBulkReadersBatch gains FetchIssuesByIDs (ceil(N/200) statements, <=200 placeholders, rows on both sides of a boundary merged, and exactly ONE label hydration over the merged slice), GetBlockingInfoOutbound, GetBlockingInfoInbound, LoadStatusByID (both planes, one statement per batch each) and LoadStatusByIDDuplicateAcrossTables; the empty-input subtest now covers all three readers issuing no statement. internal/storage/domain/issue_delete_missing_table_test.go proves a missing `leases` (or `wisp_labels`) is no longer swallowed at either GetByIDs(wisps) site while a missing `wisps` table still is, with an unrelated-error control. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(storage/db): consume the aborted duplicate-plane run's statements inside its own subtest (wy-sm01o2) Salvaged WIP from the quota-killed executor (session-limit at 00:19Z): the duplicate-plane loadStatusByID call aborts mid-run, so reset the recorder before it and assert its statements under the cap in place, rather than leaving them to pollute the next subtest. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rrz4umA2L5QU8s9EmCBPw8 * docs(cmd/bd): correct the LoadExportRelations plane-partition doc block The header still documented pre-PR full-set wisp reads ("both planes are queried with the FULL ID set and merged"), which this PR made false for the wisp side. That block is the load-bearing record of the plane-partition invariant, so a future editor trusting it could "restore" full-set wisp reads and silently undo the wy-237yfi fix. - The first bullet now describes the partitioned reads: membership is read from the wisps table itself (WispPlaneIDs) rather than inferred from row flags, the durable plane is queried with the full id set, the wisp plane only with that membership subset (wispReaderIDs), and unreadable membership fails OPEN to the full list. The "row flags are NOT a substitute" rationale is kept deliberately: it is *why* membership is read from the table at all. - The IsTableNotExist sentence is extended, not replaced. It is still true of the three single-table wisp-relation readers, which correctly keep the blanket check; only the lease-joined membership probe (WispPlaneIDs) is pinned to a missing `wisps` table. Replacing the sentence outright would have introduced a new inaccuracy. Maintainer review fixup for adopt-pr #6444. Documentation only: the tree is AST-identical after stripping comments, so no behavior can change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Eddie the Engineer <julianknutsen@users.noreply.github.com>
43 lines
1.5 KiB
Go
43 lines
1.5 KiB
Go
package main
|
|
|
|
import (
|
|
"reflect"
|
|
"testing"
|
|
)
|
|
|
|
// wispPlaneSubset feeds the wisp-plane relation readers in
|
|
// uowExportSource.LoadExportRelations: only ids that live in the wisps table
|
|
// go to them, in input order; a set that marks none sends nothing, so those
|
|
// readers issue no statement at all (wy-237yfi).
|
|
func TestWispPlaneSubset(t *testing.T) {
|
|
ids := []string{"a", "b", "c", "d"}
|
|
|
|
if got := wispPlaneSubset(ids, map[string]bool{}); got != nil {
|
|
t.Fatalf("empty set: want nil, got %v", got)
|
|
}
|
|
got := wispPlaneSubset(ids, map[string]bool{"d": true, "b": true, "zz": true, "a": false})
|
|
if !reflect.DeepEqual(got, []string{"b", "d"}) {
|
|
t.Fatalf("want [b d] in input order, got %v", got)
|
|
}
|
|
if got := wispPlaneSubset(nil, map[string]bool{"a": true}); got != nil {
|
|
t.Fatalf("no ids: want nil, got %v", got)
|
|
}
|
|
}
|
|
|
|
// wispReaderIDs is the policy on top: a nil set is "membership unreadable"
|
|
// and must FAIL OPEN to every id — narrowing on it would silently export
|
|
// every wisp with no labels and no comments — while a non-nil empty set is a
|
|
// real answer (no wisps) and sends nothing.
|
|
func TestWispReaderIDs(t *testing.T) {
|
|
ids := []string{"a", "b", "c"}
|
|
|
|
if got := wispReaderIDs(ids, nil); !reflect.DeepEqual(got, ids) {
|
|
t.Fatalf("nil set must fail open to all ids, got %v", got)
|
|
}
|
|
if got := wispReaderIDs(ids, map[string]bool{}); got != nil {
|
|
t.Fatalf("empty non-nil set means no wisps: want nil, got %v", got)
|
|
}
|
|
if got := wispReaderIDs(ids, map[string]bool{"c": true}); !reflect.DeepEqual(got, []string{"c"}) {
|
|
t.Fatalf("want [c], got %v", got)
|
|
}
|
|
}
|