STORAGE LAYER ONLY, per CONTRIBUTING_PR_GUIDELINES.md. issueops.ListRequest and issueops.ReadyRequest gain Brief; no CLI flag and no HTTP parameter sets it yet, so no front door's payload changes. The wiring stacks on top. WHAT IT DROPS is the free-form text: description, design, acceptance_criteria, notes, payload and waiters are not selected, and the row comes back carrying types.Issue.IsLitePartial. #5546 measured a 16,051-issue store where `bd ready --json` returned 454,021 bytes and notes plus description were 76% of it, on the call an agent makes to decide what to work on. Measured through issueops.Reader on 100 rows sized from those per-row ratios: 362,301 bytes full, 24,001 brief, a 93.4% reduction. THE PRIMITIVE WAS HALF-BUILT AND UNREACHABLE. #3906 landed types.IssueFilter.Lite, IssueSelectColumnsLite, HeavyDropList, ScanIssueLiteFrom and Issue.IsLitePartial, then wired filter.Lite into exactly one place: SearchIssuesInTx, the UNCOUNTED store-backed search. Every read that answers with IssueWithCounts, which is every --json route of both commands, both implementations of issueops.Reader, and both HTTP reads, runs the counts mega-query instead and ignored the field. A request that set it would have come back correct, fully hydrated, and with nothing on it to say the projection had not happened. That silent no-op is what this closes. THE APPLICATION POINT IS THE COUNTS MEGA-QUERY, and it is one point rather than four because sqlbuild.SearchCountsSQL and issueops.ScanReadyWorkRowWithCounts are already the pair every counted read passes through: the store seam's list (search_counts.go) and ready (ready_work_counts.go), and the unit-of-work seam's list (issue_search_counts.go) and ready (ready_work_union.go, via fetchCountsByIDs). sqlbuild.CountsHydration is already the value that carries hydration opt-outs across that pair on both seams, so Lite is a third field on it rather than a new channel. IT CHANGES THE PROJECTION'S SHAPE, which SkipLabels and SkipCounts do not, and that is why ScanReadyWorkRowWithCounts now takes the whole hydration instead of a bool: the value that chose the SELECT list is the value that drives the scan, so the two cannot be given different answers. Substituting empty strings in the heavy positions would have kept one scan path and was rejected, because the row would carry blank text with IsLitePartial=false, which is indistinguishable from a genuinely textless row, the ambiguity ga-clgh and CommentsOmitted already record. WHERE IS INDEPENDENT OF SELECT, so this bounds the size of a row and never which rows match: DescContains, NotesContains and EmptyDesc keep selecting exactly what they select today, and no sort key, keyset cursor or wisp-merge dedup reads a dropped column (CompareIssuesBy orders on nine fields, none of them heavy). THE CLAIM REFUSES IT, which is not the same call the page makes and is the sharper case of the rule Limit and Offset already state. A claim does not read its row through the page's query at all: ExecuteClaimNext refetches the winning row whole with GetIssueInTx and hydrates its counts itself, so the projection has nothing to apply to. Accepting the field would answer a MUTATING request with a fully hydrated row carrying IsLitePartial=false, the caller's one signal that it did not get what it asked for, reporting that it did. The check lives in issueops.ValidateClaimNextRequest, which is the shared rule both ready claimers and BatchCloser's claim already run, so all four backends refuse it on the same line. BuildReadyCountFilter CARRIES it, which is the opposite treatment and the opposite reason. A count reads no field of any row, so the projection cannot make the number wrong, and the count is not always cheap enough for that to be the end of it: the unit-of-work seam has no COUNT(*) over the ready predicate and sizes the set by running the unbounded page and taking its length (uow/ready_counter.go), so dropping the field there would hydrate every heavy column of the whole ready set to answer `bd ready --brief`, a larger read than the page it was asked for. Carrying it also keeps that builder what its doc says it is: the listing's filter with the PAGE removed and nothing else. TWO ASYMMETRIES on the page path, both deliberate and both stated in code: - Brief IS carried onto the --ready arm (ReadyFilterFromIssueFilter) where SkipLabels and SkipCounts are dropped. Those two hide a number the ready renderings print; this drops a body no listing prints, so carrying it is what makes `bd list --ready` and `bd ready` answer one request one way. - types.WorkFilter carries Lite and neither of the other two, so the ready paths get readyHydrationFor rather than hydrationFor. Before this, both uow ready call sites passed a hardcoded CountsHydration{}, a literal that could not carry a filter's answer at all, which is how that path would have been missed. CONFORMANCE at all four registered sites, one Test per case, for the two page cases and for the claim refusal. Every field the cases assert stripped is seeded heavy first and the premise is checked, so a backend that never round-trips one of the six fails loudly instead of turning that clause into one that cannot fail. Identity is asserted too (ID, Title, Status, IssueType, Priority), or a body that stripped the row to its id would satisfy every emptiness clause. IsLitePartial is asserted in both directions. THE LITE COLUMN LIST IS PINNED BY ORDER, not only by membership. Both lists are scanned positionally, so two columns of the same SQL type can be transposed with no membership change at all: the pre-existing set-based parity test stays green, the scan succeeds, and every row silently carries one column's value in another's field. TestIssueSelectColumnsLite_IsFullMinusHeavyInOrder derives the expected list by deleting HeavyDropList from IssueSelectColumns in place, which makes the lite list a derivation rather than a second hand-maintained copy. RED BEFORE GREEN, five sabotages, each verified and reverted: - deleting the lite branch from CountsHydration.IssueColumns fails both page cases with "expected 58 destination arguments in Scan, not 52"; - accepting Brief without plumbing it to either filter, the silent no-op this change exists to close, fails both page cases on all six text fields and on IsLitePartial, which is the run that proves the assertions are load-bearing; - transposing created_by and owner in IssueSelectColumnsLite fails only the new order test, on IssueSelectColumnsLite[9] = owner, want created_by, while the set-based test stays green, which is the gap it was added for; - dropping the claim's refusal fails the claimer contract with "ClaimNext with brief: error = <nil>, want ErrValidation"; and - clearing Brief in BuildReadyCountFilter fails TestBuildReadyCountFilterIsTheUnboundedReadyFilter, which already existed to catch a ReadyRequest field the count builder forgets to carry and gains Brief in its fixture here. The recorded ready-filter golden gains "Lite": false in 62 cases because types.WorkFilter gained the field and the test compares marshaled JSON byte-for-byte. False is what a request that did not ask for the projection resolves that field to, so the replay stays exact; the test comment says so. Gates: go build ./..., go vet ./..., gofmt, internal/storage/... (dolt, embeddeddolt with BEADS_TEST_EMBEDDED_DOLT=1, uow, issueops, sqlbuild, domain/db), internal/workapi/..., backend/conformance/..., issueops/..., make api-check. Refs: #5546, #3906 Co-authored-by: sjarmak <t@t.co>
287 lines
12 KiB
Go
287 lines
12 KiB
Go
package sqlbuild
|
|
|
|
import "fmt"
|
|
|
|
// ReadyWorkIssueColumns is IssueSelectColumns qualified with the "i." alias
|
|
// used by the counts mega-query. The lease overlay columns keep their own
|
|
// leases. qualifier (the mega-query FROM includes LeaseJoin("i")).
|
|
var ReadyWorkIssueColumns = QualifyColumns(IssueBaseColumns, "i.") + ", " + LeaseSelectColumns
|
|
|
|
// ReadyWorkIssueColumnsLite is the lite counterpart of ReadyWorkIssueColumns,
|
|
// selected by CountsHydration.Lite. The scan side is
|
|
// issueops.ScanIssueLiteFrom, reached through ScanReadyWorkRowWithCounts.
|
|
var ReadyWorkIssueColumnsLite = QualifyColumns(IssueBaseColumnsLite, "i.") + ", " + LeaseSelectColumns
|
|
|
|
// DepJSONObject renders one dependency row as JSON for JSON_ARRAYAGG
|
|
// aggregation in the counts mega-query. Field names must match the JSON tags
|
|
// of types.Dependency.
|
|
const DepJSONObject = `JSON_OBJECT(
|
|
'issue_id', issue_id,
|
|
'depends_on_id', COALESCE(depends_on_issue_id, depends_on_wisp_id, depends_on_external),
|
|
'type', type,
|
|
'created_at', DATE_FORMAT(created_at, '%Y-%m-%dT%H:%i:%sZ'),
|
|
'created_by', created_by,
|
|
'metadata', CAST(metadata AS CHAR),
|
|
'thread_id', thread_id
|
|
)`
|
|
|
|
// CountsHydration selects what the counts mega-query materializes per row.
|
|
// Every member is an opt-OUT: the zero value hydrates everything.
|
|
//
|
|
// None of them changes WHICH ROWS come back or in what order. The aggregates
|
|
// hang off LEFT JOINs that preserve every driver row, so dropping one drops a
|
|
// column's value and nothing else; Lite changes only the driver's own SELECT
|
|
// list, and no filter predicate is expressed in the SELECT.
|
|
//
|
|
// SkipLabels leaves labels_json NULL, which the scan turns into no labels.
|
|
// SkipCounts leaves the three cardinalities 0, which a caller must read as
|
|
// unknown rather than as none — the promise issueops.ListRequest.SkipCounts
|
|
// makes above storage. Both keep the projection's SHAPE: the scan side reads
|
|
// the six extra columns positionally, so a suppressed aggregate is projected
|
|
// as a constant in place rather than removed.
|
|
//
|
|
// LITE IS THE ONE THAT CHANGES THE SHAPE, which is why it is not merely a
|
|
// third constant substitution. It swaps ReadyWorkIssueColumns for
|
|
// ReadyWorkIssueColumnsLite, so the driver projects six fewer columns and the
|
|
// scan must read the matching dest list — ScanReadyWorkRowWithCounts takes the
|
|
// whole struct for that reason and dispatches on this field. Substituting
|
|
// empty strings in place would have kept one scan path, and was rejected: the
|
|
// row would come back with blank free-form text and IsLitePartial=false, which
|
|
// is indistinguishable from a genuinely textless row (the ambiguity ga-clgh
|
|
// and CommentsOmitted already record). ScanIssueLiteFrom sets the flag.
|
|
type CountsHydration struct {
|
|
SkipLabels bool
|
|
SkipCounts bool
|
|
Lite bool
|
|
}
|
|
|
|
// IssueColumns returns the driver's SELECT list for this hydration.
|
|
func (h CountsHydration) IssueColumns() string {
|
|
if h.Lite {
|
|
return ReadyWorkIssueColumnsLite
|
|
}
|
|
return ReadyWorkIssueColumns
|
|
}
|
|
|
|
// SearchCountsSQL renders the counts mega-query: issue rows aliased "i" plus
|
|
// labels JSON, dep/rdep/comment counts, parent ID, and dependency JSON, for one
|
|
// table family. The scan side is issueops.ScanReadyWorkRowWithCounts, which
|
|
// scans hyd.IssueColumns() positionally followed by the six extra columns in
|
|
// the order projected here — so the SAME hydration value has to reach both, and
|
|
// is the reason the scan takes the struct rather than being told the shape
|
|
// separately.
|
|
//
|
|
// There are two forms of the same projection, selected by ids:
|
|
//
|
|
// - Predicate form (ids empty): whereSQL bounds the driver in an inner
|
|
// subquery BEFORE the aggregate LEFT JOINs (see below); orderBySQL and
|
|
// limitSQL stay at the outer level, after the joins. Each count subquery
|
|
// aggregates its whole side table. The caller supplies its own args; the
|
|
// returned args slice is nil.
|
|
//
|
|
// - By-IDs form (ids non-empty): whereSQL/orderBySQL/limitSQL are ignored and
|
|
// the driver AND every count subquery are constrained to ids. This keeps
|
|
// each subquery from scanning its whole side table and, crucially, keeps
|
|
// the reverse-blocker rc self-join — whose COALESCE(...) key the pure-Go
|
|
// GMS analyzer cannot auto-index — from re-scanning its full materialization
|
|
// once per driver row. The returned args are the ids repeated once per
|
|
// injection point, in left-to-right placeholder order.
|
|
//
|
|
// The by-IDs form is result-identical to the predicate form over the same set
|
|
// of surviving rows: each per-issue count is a function of the whole dependency
|
|
// graph restricted to that issue, so filtering a subquery's input to ids cannot
|
|
// change any surviving row's count. dep/comment/parent counts are
|
|
// order-insensitive; labels are re-sorted in Go (ScanReadyWorkRowWithCounts).
|
|
// deps_json's element order is whatever JSON_ARRAYAGG emits (SQL does not
|
|
// guarantee one), but it is the same in both forms: restricting the aggregate's
|
|
// input to ids drops only rows for other issues, so the per-issue rows it
|
|
// aggregates — and their relative order — are unchanged.
|
|
//
|
|
// In the predicate form, whereSQL filters the main table in an inner subquery,
|
|
// BEFORE the aggregate LEFT JOINs. The joins all preserve every main row, so
|
|
// filtering before vs. after them yields the identical row set — but it changes
|
|
// the plan Dolt picks: instead of materializing the dep/rdep/comment/label/parent
|
|
// aggregates and joining them to every main row before the WHERE prunes the
|
|
// result, the joins now drive off the already-narrowed set. That is the win for
|
|
// the narrow ephemeral-work searches that dominate the hot path. Only the WHERE
|
|
// moves inward; orderBySQL and limitSQL remain after the joins, because ORDER BY
|
|
// and LIMIT must see the projected aggregate columns and must apply to the final
|
|
// joined result. An empty whereSQL keeps the plain "FROM <main> i" driver: there
|
|
// is no predicate to push down, so the derived table would only interpose a
|
|
// needless wrapper between the planner and the base table.
|
|
//
|
|
// That shape REQUIRES that whereSQL reference only main-table columns (or
|
|
// correlated subqueries against labels/deps/comments keyed by id) — never the
|
|
// six aggregate aliases (labels_json, dep_count, rdep_count, comment_count,
|
|
// parent_id, deps_json), which are projected by the OUTER query and are not in
|
|
// scope inside the derived subquery. The only producers of whereSQL,
|
|
// BuildIssueFilterClauses and BuildReadyWorkWhere, uphold this by construction.
|
|
// The invariant is self-enforcing: a predicate that referenced an aggregate
|
|
// would reference an out-of-scope column, so Dolt errors at query time rather
|
|
// than silently returning wrong rows — the failure is loud, not subtle. (The
|
|
// subquery scoping that makes it loud is pinned by TestSearchCountsSQLShape.)
|
|
//
|
|
// The by-IDs form keeps the plain "FROM <main> i" driver with its id predicate
|
|
// at the outer level: its subqueries are already id-constrained, so there is no
|
|
// join-then-filter blowup to avoid and no reason to interpose a derived table.
|
|
//
|
|
// The reverse-blocker rc subquery unions wisp_dependencies only when the caller
|
|
// has probed that the table exists (includeWispReverseDeps).
|
|
//
|
|
// hyd selects which aggregates are computed at all; see CountsHydration.
|
|
func SearchCountsSQL(tables FilterTables, ids []string, whereSQL, orderBySQL, limitSQL string, includeWispReverseDeps bool, hyd CountsHydration) (string, []any) {
|
|
byIDs := len(ids) > 0
|
|
inSQL, idArgs := InPlaceholders(ids)
|
|
|
|
// Per-subquery id constraints. Empty strings in the predicate form leave the
|
|
// projection unchanged; in the by-IDs form they push the page down so no
|
|
// subquery aggregates more than the page's worth of rows.
|
|
var labelWhere, depBlocksExtra, rcDepExtra, rcWispExtra, ccWhere, pcExtra, depWhere string
|
|
if byIDs {
|
|
labelWhere = fmt.Sprintf("WHERE issue_id IN (%s)", inSQL)
|
|
depBlocksExtra = fmt.Sprintf(" AND issue_id IN (%s)", inSQL)
|
|
rcDepExtra = fmt.Sprintf(" AND %s IN (%s)", DepTargetExpr, inSQL)
|
|
rcWispExtra = fmt.Sprintf(" AND %s IN (%s)", DepTargetExpr, inSQL)
|
|
ccWhere = fmt.Sprintf("WHERE issue_id IN (%s)", inSQL)
|
|
pcExtra = fmt.Sprintf(" AND issue_id IN (%s)", inSQL)
|
|
depWhere = fmt.Sprintf("WHERE issue_id IN (%s)", inSQL)
|
|
}
|
|
|
|
reverseBlockerSelect := fmt.Sprintf(`
|
|
SELECT %s AS dep_id
|
|
FROM dependencies WHERE type = 'blocks'%s
|
|
`, DepTargetExpr, rcDepExtra)
|
|
if includeWispReverseDeps {
|
|
reverseBlockerSelect += fmt.Sprintf(`
|
|
UNION ALL
|
|
SELECT %s AS dep_id
|
|
FROM wisp_dependencies WHERE type = 'blocks'%s
|
|
`, DepTargetExpr, rcWispExtra)
|
|
}
|
|
|
|
labelsSelect := "l.labels_json AS labels_json"
|
|
labelsJoin := fmt.Sprintf(`
|
|
LEFT JOIN (
|
|
SELECT issue_id, JSON_ARRAYAGG(label) AS labels_json
|
|
FROM %s
|
|
%s
|
|
GROUP BY issue_id
|
|
) l ON l.issue_id = i.id`, tables.Labels, labelWhere)
|
|
if hyd.SkipLabels {
|
|
labelsSelect = "NULL AS labels_json"
|
|
labelsJoin = ""
|
|
}
|
|
|
|
countsSelect := `COALESCE(dc.cnt, 0) AS dep_count,
|
|
COALESCE(rc.cnt, 0) AS rdep_count,
|
|
COALESCE(cc.cnt, 0) AS comment_count`
|
|
countsJoins := fmt.Sprintf(`
|
|
LEFT JOIN (
|
|
SELECT issue_id, COUNT(*) AS cnt
|
|
FROM %s
|
|
WHERE type = 'blocks'%s
|
|
GROUP BY issue_id
|
|
) dc ON dc.issue_id = i.id
|
|
LEFT JOIN (
|
|
SELECT dep_id, COUNT(*) AS cnt FROM (
|
|
%s
|
|
) all_blockers GROUP BY dep_id
|
|
) rc ON rc.dep_id = i.id
|
|
LEFT JOIN (
|
|
SELECT issue_id, COUNT(*) AS cnt
|
|
FROM %s
|
|
%s
|
|
GROUP BY issue_id
|
|
) cc ON cc.issue_id = i.id`,
|
|
tables.Dependencies, depBlocksExtra,
|
|
reverseBlockerSelect,
|
|
tables.Comments, ccWhere)
|
|
if hyd.SkipCounts {
|
|
// Constants in the same three positions: the scan reads these columns
|
|
// by index, and the rc join above is the one the embedded engine
|
|
// cannot index (see the by-IDs rationale above).
|
|
countsSelect = `0 AS dep_count,
|
|
0 AS rdep_count,
|
|
0 AS comment_count`
|
|
countsJoins = ""
|
|
}
|
|
|
|
// Predicate form with a filter: whereSQL filters the main table inside a
|
|
// derived table so the aggregate joins drive off the already-narrowed set;
|
|
// ORDER BY and LIMIT stay outer, after the joins. Everything else — empty
|
|
// whereSQL (nothing to push down) and the by-IDs form (subqueries already
|
|
// id-constrained) — keeps the plain driver.
|
|
driverSQL := fmt.Sprintf("%s i", tables.Main)
|
|
if !byIDs && whereSQL != "" {
|
|
driverSQL = fmt.Sprintf(`(
|
|
SELECT i.*
|
|
FROM %s i
|
|
%s
|
|
) i`, tables.Main, whereSQL)
|
|
}
|
|
outerClause := fmt.Sprintf("%s\n\t\t%s", orderBySQL, limitSQL)
|
|
if byIDs {
|
|
outerClause = fmt.Sprintf("WHERE i.id IN (%s)", inSQL)
|
|
}
|
|
|
|
sqlText := fmt.Sprintf(`
|
|
SELECT %s,
|
|
%s,
|
|
%s,
|
|
pc.parent_id AS parent_id,
|
|
d.deps_json AS deps_json
|
|
FROM %s
|
|
%s
|
|
%s
|
|
%s
|
|
LEFT JOIN (
|
|
SELECT issue_id,
|
|
MIN(%s) AS parent_id
|
|
FROM %s
|
|
WHERE type = 'parent-child'%s
|
|
GROUP BY issue_id
|
|
) pc ON pc.issue_id = i.id
|
|
LEFT JOIN (
|
|
SELECT issue_id, JSON_ARRAYAGG(%s) AS deps_json
|
|
FROM %s
|
|
%s
|
|
GROUP BY issue_id
|
|
) d ON d.issue_id = i.id
|
|
%s
|
|
`,
|
|
hyd.IssueColumns(),
|
|
labelsSelect,
|
|
countsSelect,
|
|
driverSQL,
|
|
LeaseJoin("i"),
|
|
labelsJoin,
|
|
countsJoins,
|
|
DepTargetExpr, tables.Dependencies, pcExtra,
|
|
DepJSONObject, tables.Dependencies, depWhere,
|
|
outerClause,
|
|
)
|
|
|
|
if !byIDs {
|
|
return sqlText, nil
|
|
}
|
|
|
|
// args follow the placeholder order in sqlText: labels join (unless
|
|
// skipped), dc, rc dependencies branch, rc wisp branch (if any), cc (all
|
|
// four unless the counts are skipped), pc, d, then the driver.
|
|
args := make([]any, 0, len(idArgs)*8)
|
|
if !hyd.SkipLabels {
|
|
args = append(args, idArgs...)
|
|
}
|
|
if !hyd.SkipCounts {
|
|
args = append(args, idArgs...) // dc
|
|
args = append(args, idArgs...) // rc dependencies
|
|
if includeWispReverseDeps {
|
|
args = append(args, idArgs...) // rc wisp_dependencies
|
|
}
|
|
args = append(args, idArgs...) // cc
|
|
}
|
|
args = append(args, idArgs...) // pc
|
|
args = append(args, idArgs...) // d
|
|
args = append(args, idArgs...) // driver
|
|
return sqlText, args
|
|
}
|