beads/internal/storage/sqlbuild/ready.go
John Zook 21822ad725
perf(search): filter the main table before the counts aggregate joins (#4480)
* perf(search): filter the main table before the counts aggregate joins (#5)

* perf(search): filter the main table before the counts aggregate joins

sqlbuild.SearchCountsSQL (the counts mega-query behind bd query/search --counts,
bd list --counts, and ready-work counts) joined six aggregate derived tables
(labels, dep/rdep/comment counts, parent, deps JSON) onto every main-table row
and only then applied the WHERE filter. For the narrow ephemeral-work searches
that dominate the hot path, Dolt materialized those aggregates across the full
tables before pruning -- the dominant cost for that access pattern.

Wrap the main table in an inner subquery that applies the WHERE filter first, so
the aggregate LEFT JOINs drive off the already-narrowed set:

    FROM ( SELECT i.* FROM <main> i <whereSQL> ) i
    LEFT JOIN ( ... ) ...

The WHERE clause references only issue columns, never the aggregates, and the
joins are all LEFT JOINs that preserve every main row, so filtering before vs.
after the joins yields the identical row set. ORDER BY and LIMIT deliberately
stay in their original position after the joins: the ready-work counts path
passes a parameterized ORDER BY, so it must render exactly once to keep the
placeholder/arg counts aligned, and callers that do not re-sort in Go (the
domain/db stack) depend on a post-join ORDER BY for the final row order.

Tests:
- TestSearchCountsFilterBeforeJoinParity (real Dolt): the new shape returns
  byte-identical rows (ids, order, dep/rdep/comment counts, parent, labels, deps
  JSON) to the prior filter-after-join shape across 11 filter/order/limit/skip
  scenarios. The old shape is kept in test code as the reference.
- TestSearchCountsSQLShape: asserts the WHERE sits inside the derived subquery
  (before the first LEFT JOIN) and that ORDER BY / LIMIT stay after the joins and
  appear exactly once.
- Full sqlbuild, issueops, dolt counts/search/ready, and domain/db suites green.

* test(search): enforce WHERE-never-references-aggregates invariant

Add TestWhereClausesNeverReferenceAggregates plus contract comments on
both WHERE producers (BuildIssueFilterClauses, BuildReadyWorkWhere) and
SearchCountsSQL, pinning the load-bearing invariant behind the
filter-before-join shape: WHERE may reference only issue columns, never
the six counts aggregate aliases (out of scope in the pre-join subquery).

Drop the stale ORDER BY/LIMIT doc paragraph (described unchanged
behavior; the arg-count invariant is already pinned by
TestSearchCountsSQLShape).

* test(search): drop the maximally-populated WHERE-aggregate test

The enumeration test only checked fields it happened to populate, so a
new filter field referencing an aggregate would pass unnoticed — false
confidence plus a standing sync cost. The invariant is already guarded
structurally: TestSearchCountsSQLShape pins the WHERE inside the pre-join
subquery, which is exactly what makes a violation an out-of-scope column
reference that Dolt errors on (fail-loud). Reword the builder/contract
comments to lean on that rather than the deleted test.

* fix(search): keep plain driver on empty WHERE; by-IDs hydration in domain/db

Review follow-ups for the filter-before-joins reshape:

- SearchCountsSQL: an empty whereSQL keeps the plain "FROM <main> i"
  driver instead of wrapping it in a no-op derived table, so the
  unfiltered list/counts path keeps its pre-existing shape and plans.
  Pinned by a new TestSearchCountsSQLShape assertion.

- domain/db fetchCountsByIDs: hydrate via the by-IDs form instead of
  hand-building "WHERE i.id IN (...)" for the predicate form - that
  clause now renders inside the derived subquery, silently coupling the
  caller to the subquery's internal alias. The by-IDs form also
  constrains every aggregate subquery to the page; IDs are chunked
  (queryBatchSize) to stay within placeholder limits, mirroring
  issueops.runReadyCountsInTx.

---------

Co-authored-by: Zook Bot <275398848+zook-bot@users.noreply.github.com>
2026-07-26 13:04:52 -07:00

215 lines
8.4 KiB
Go

package sqlbuild
import (
"fmt"
"strings"
"time"
"github.com/steveyegge/beads/internal/storage/domain"
"github.com/steveyegge/beads/internal/types"
)
// ReadyWorkExcludeTypes returns the issue types excluded from ready work by
// default, plus any caller extras (deduped, empty entries dropped). Infra types
// stay hidden from ready work, and rig identity beads are also hidden even
// though they are durable issues rather than infra wisps.
func ReadyWorkExcludeTypes(extra []types.IssueType) []types.IssueType {
out := []types.IssueType{
types.IssueType("merge-request"),
types.TypeGate,
types.TypeMolecule,
types.IssueType("rig"),
}
for _, t := range domain.DefaultInfraTypes() {
out = append(out, types.IssueType(t))
}
seen := make(map[types.IssueType]bool, len(out)+len(extra))
for _, t := range out {
seen[t] = true
}
for _, t := range extra {
if t == "" || seen[t] {
continue
}
seen[t] = true
out = append(out, t)
}
return out
}
// ReadyWorkOrder is an ORDER BY fragment plus any args its CASE expressions
// need (the hybrid policy parameterizes a recency cutoff).
type ReadyWorkOrder struct {
SQL string
Args []any
}
// BuildReadyWorkOrder renders the ready-work ORDER BY for a sort policy.
// createdCol/priorityCol name the sortable columns: real columns
// ("created_at"/"priority") for per-table queries, or the sort_* aliases
// ("sort_created"/"sort_priority") for UNION outer queries.
func BuildReadyWorkOrder(policy types.SortPolicy, createdCol, priorityCol string) ReadyWorkOrder {
switch policy {
case types.SortPolicyOldest:
return ReadyWorkOrder{SQL: fmt.Sprintf("ORDER BY %s ASC, id ASC", createdCol)}
case types.SortPolicyPriority:
return ReadyWorkOrder{SQL: fmt.Sprintf("ORDER BY %s ASC, %s ASC, id ASC", priorityCol, createdCol)}
case types.SortPolicyHybrid, "":
recentCutoff := time.Now().UTC().Add(-48 * time.Hour)
return ReadyWorkOrder{
SQL: fmt.Sprintf(`ORDER BY
CASE WHEN %s >= ? THEN 0 ELSE 1 END ASC,
CASE WHEN %s >= ? THEN %s ELSE 999 END ASC,
%s ASC, id ASC`, createdCol, createdCol, priorityCol, createdCol),
Args: []any{recentCutoff, recentCutoff},
}
default:
return ReadyWorkOrder{SQL: fmt.Sprintf("ORDER BY %s ASC, %s ASC, id ASC", priorityCol, createdCol)}
}
}
// ReadyWorkWhereInputs carries the precomputed ID sets the ready-work WHERE
// clause folds in. Computing them takes queries, which is execution-context
// work each stack does its own way.
type ReadyWorkWhereInputs struct {
// DeferredChildIDs are children of future-deferred parents; consulted
// only when !filter.IncludeDeferred.
DeferredChildIDs []string
// ParentDescendantIDs are the transitive descendants of *filter.ParentID;
// consulted only when filter.ParentID != nil.
ParentDescendantIDs []string
}
// BuildReadyWorkWhere renders the full ready-work WHERE clause for one table
// family. Both stacks must keep ready semantics identical (Seam A parity
// suite); all ready predicates live here.
//
// Invariant: every clause must reference only main-table columns or correlated
// subqueries keyed by id — never the counts mega-query's aggregate aliases
// (labels_json, dep_count, rdep_count, comment_count, parent_id, deps_json).
// SearchCountsSQL renders this WHERE inside a pre-join subquery where those
// aliases are out of scope. See the SearchCountsSQL doc comment for why a
// violation fails loud.
func BuildReadyWorkWhere(filter types.WorkFilter, tables FilterTables, in ReadyWorkWhereInputs) (string, []any, error) {
var statusClause string
var args []any
switch {
case filter.Status != "":
statusClause = "status = ?"
args = append(args, string(filter.Status))
case len(filter.Statuses) > 0:
ph, statusArgs := InPlaceholders(filter.Statuses)
statusClause = fmt.Sprintf("status IN (%s)", ph)
args = append(args, statusArgs...)
default:
statusClause = "status IN ('open', 'in_progress')"
}
whereClauses := []string{
statusClause,
"(pinned = 0 OR pinned IS NULL)",
"is_blocked = 0",
}
if !filter.IncludeEphemeral {
whereClauses = append(whereClauses, "(ephemeral = 0 OR ephemeral IS NULL)")
}
if filter.Priority != nil {
whereClauses = append(whereClauses, "priority = ?")
args = append(args, *filter.Priority)
}
if filter.Type != "" {
whereClauses = append(whereClauses, "issue_type = ?")
args = append(args, filter.Type)
} else {
ph, a := InPlaceholders(ReadyWorkExcludeTypes(filter.ExcludeTypes))
whereClauses = append(whereClauses, fmt.Sprintf("issue_type NOT IN (%s)", ph))
args = append(args, a...)
}
if filter.Unassigned {
whereClauses = append(whereClauses, "(assignee IS NULL OR assignee = '')")
} else if filter.Assignee != nil {
whereClauses = append(whereClauses, "assignee = ?")
args = append(args, *filter.Assignee)
}
if !filter.IncludeDeferred {
whereClauses = append(whereClauses, "(defer_until IS NULL OR defer_until <= UTC_TIMESTAMP())")
for start := 0; start < len(in.DeferredChildIDs); start += QueryBatchSize {
end := start + QueryBatchSize
if end > len(in.DeferredChildIDs) {
end = len(in.DeferredChildIDs)
}
placeholders, batchArgs := InPlaceholders(in.DeferredChildIDs[start:end])
args = append(args, batchArgs...)
whereClauses = append(whereClauses, fmt.Sprintf("id NOT IN (%s)", placeholders))
}
}
if len(filter.Labels) > 0 {
for _, label := range filter.Labels {
whereClauses = append(whereClauses, fmt.Sprintf("id IN (SELECT issue_id FROM %s WHERE label = ?)", tables.Labels))
args = append(args, label)
}
}
// LabelsAny is an OR-set: an issue qualifies if it carries AT LEAST ONE of
// the labels. Previously this clause was absent entirely, so --label-any was
// silently dropped on the ready/claim path (with or without --parent) — a
// worker believed it was fenced when it was not. It AND-combines with Labels
// (the flag help promises "Can combine with --label").
if len(filter.LabelsAny) > 0 {
placeholders := make([]string, len(filter.LabelsAny))
for i, label := range filter.LabelsAny {
placeholders[i] = "?"
args = append(args, label)
}
whereClauses = append(whereClauses, fmt.Sprintf("id IN (SELECT issue_id FROM %s WHERE label IN (%s))", tables.Labels, strings.Join(placeholders, ", ")))
}
if len(filter.ExcludeLabels) > 0 {
placeholders := make([]string, len(filter.ExcludeLabels))
for i, label := range filter.ExcludeLabels {
placeholders[i] = "?"
args = append(args, label)
}
whereClauses = append(whereClauses, fmt.Sprintf("id NOT IN (SELECT issue_id FROM %s WHERE label IN (%s))", tables.Labels, strings.Join(placeholders, ", ")))
}
if filter.LabelPattern != "" {
whereClauses = append(whereClauses, fmt.Sprintf("id IN (SELECT issue_id FROM %s WHERE label LIKE ? ESCAPE '|')", tables.Labels))
args = append(args, globToLikePattern(filter.LabelPattern))
}
if filter.LabelRegex != "" {
whereClauses = append(whereClauses, fmt.Sprintf("id IN (SELECT issue_id FROM %s WHERE label REGEXP ?)", tables.Labels))
args = append(args, filter.LabelRegex)
}
// Parent filtering: return all transitive descendants of parentID.
// GH#3396: a one-hop subquery silently dropped grandchildren despite the
// help text and WorkFilter.ParentID godoc both promising recursion.
if filter.ParentID != nil {
parentID := *filter.ParentID
parentClauses := []string{fmt.Sprintf("(id LIKE CONCAT(?, '.%%') AND id NOT IN (SELECT issue_id FROM %s WHERE type = 'parent-child'))", tables.Dependencies)}
args = append(args, parentID)
for start := 0; start < len(in.ParentDescendantIDs); start += QueryBatchSize {
end := start + QueryBatchSize
if end > len(in.ParentDescendantIDs) {
end = len(in.ParentDescendantIDs)
}
placeholders, batchArgs := InPlaceholders(in.ParentDescendantIDs[start:end])
parentClauses = append(parentClauses, fmt.Sprintf("id IN (%s)", placeholders))
args = append(args, batchArgs...)
}
whereClauses = append(whereClauses, "("+strings.Join(parentClauses, " OR ")+")")
}
if filter.MoleculeID != "" {
whereClauses = append(whereClauses, fmt.Sprintf("(id IN (SELECT issue_id FROM %s WHERE type = 'parent-child' AND %s = ?) OR (id LIKE CONCAT(?, '.%%') AND id NOT IN (SELECT issue_id FROM %s WHERE type = 'parent-child')))", tables.Dependencies, DepTargetExpr, tables.Dependencies))
args = append(args, filter.MoleculeID, filter.MoleculeID)
}
var err error
whereClauses, args, err = AppendMetadataClauses(whereClauses, args, filter.HasMetadataKey, filter.MetadataFields)
if err != nil {
return "", nil, err
}
return "WHERE " + strings.Join(whereClauses, " AND "), args, nil
}