beads/internal/storage/sqlbuild/sqlbuild.go
Stephanie Jarmak 7e825441b4
feat(workapi,storage): opt-in Brief projection for the list and ready pages (#5554)
STORAGE LAYER ONLY, per CONTRIBUTING_PR_GUIDELINES.md. issueops.ListRequest
and issueops.ReadyRequest gain Brief; no CLI flag and no HTTP parameter sets
it yet, so no front door's payload changes. The wiring stacks on top.

WHAT IT DROPS is the free-form text: description, design, acceptance_criteria,
notes, payload and waiters are not selected, and the row comes back carrying
types.Issue.IsLitePartial. #5546 measured a 16,051-issue store where
`bd ready --json` returned 454,021 bytes and notes plus description were 76% of
it, on the call an agent makes to decide what to work on. Measured through
issueops.Reader on 100 rows sized from those per-row ratios: 362,301 bytes
full, 24,001 brief, a 93.4% reduction.

THE PRIMITIVE WAS HALF-BUILT AND UNREACHABLE. #3906 landed
types.IssueFilter.Lite, IssueSelectColumnsLite, HeavyDropList,
ScanIssueLiteFrom and Issue.IsLitePartial, then wired filter.Lite into exactly
one place: SearchIssuesInTx, the UNCOUNTED store-backed search. Every read that
answers with IssueWithCounts, which is every --json route of both commands,
both implementations of issueops.Reader, and both HTTP reads, runs the counts
mega-query instead and ignored the field. A request that set it would have come
back correct, fully hydrated, and with nothing on it to say the projection had
not happened. That silent no-op is what this closes.

THE APPLICATION POINT IS THE COUNTS MEGA-QUERY, and it is one point rather than
four because sqlbuild.SearchCountsSQL and issueops.ScanReadyWorkRowWithCounts
are already the pair every counted read passes through: the store seam's list
(search_counts.go) and ready (ready_work_counts.go), and the unit-of-work
seam's list (issue_search_counts.go) and ready (ready_work_union.go, via
fetchCountsByIDs). sqlbuild.CountsHydration is already the value that carries
hydration opt-outs across that pair on both seams, so Lite is a third field on
it rather than a new channel.

IT CHANGES THE PROJECTION'S SHAPE, which SkipLabels and SkipCounts do not, and
that is why ScanReadyWorkRowWithCounts now takes the whole hydration instead of
a bool: the value that chose the SELECT list is the value that drives the scan,
so the two cannot be given different answers. Substituting empty strings in the
heavy positions would have kept one scan path and was rejected, because the row
would carry blank text with IsLitePartial=false, which is indistinguishable
from a genuinely textless row, the ambiguity ga-clgh and CommentsOmitted
already record.

WHERE IS INDEPENDENT OF SELECT, so this bounds the size of a row and never
which rows match: DescContains, NotesContains and EmptyDesc keep selecting
exactly what they select today, and no sort key, keyset cursor or wisp-merge
dedup reads a dropped column (CompareIssuesBy orders on nine fields, none of
them heavy).

THE CLAIM REFUSES IT, which is not the same call the page makes and is the
sharper case of the rule Limit and Offset already state. A claim does not read
its row through the page's query at all: ExecuteClaimNext refetches the winning
row whole with GetIssueInTx and hydrates its counts itself, so the projection
has nothing to apply to. Accepting the field would answer a MUTATING request
with a fully hydrated row carrying IsLitePartial=false, the caller's one signal
that it did not get what it asked for, reporting that it did. The check lives
in issueops.ValidateClaimNextRequest, which is the shared rule both ready
claimers and BatchCloser's claim already run, so all four backends refuse it on
the same line.

BuildReadyCountFilter CARRIES it, which is the opposite treatment and the
opposite reason. A count reads no field of any row, so the projection cannot
make the number wrong, and the count is not always cheap enough for that to be
the end of it: the unit-of-work seam has no COUNT(*) over the ready predicate
and sizes the set by running the unbounded page and taking its length
(uow/ready_counter.go), so dropping the field there would hydrate every heavy
column of the whole ready set to answer `bd ready --brief`, a larger read than
the page it was asked for. Carrying it also keeps that builder what its doc
says it is: the listing's filter with the PAGE removed and nothing else.

TWO ASYMMETRIES on the page path, both deliberate and both stated in code:

  - Brief IS carried onto the --ready arm (ReadyFilterFromIssueFilter) where
    SkipLabels and SkipCounts are dropped. Those two hide a number the ready
    renderings print; this drops a body no listing prints, so carrying it is
    what makes `bd list --ready` and `bd ready` answer one request one way.
  - types.WorkFilter carries Lite and neither of the other two, so the ready
    paths get readyHydrationFor rather than hydrationFor. Before this, both
    uow ready call sites passed a hardcoded CountsHydration{}, a literal that
    could not carry a filter's answer at all, which is how that path would have
    been missed.

CONFORMANCE at all four registered sites, one Test per case, for the two page
cases and for the claim refusal. Every field the cases assert stripped is
seeded heavy first and the premise is checked, so a backend that never
round-trips one of the six fails loudly instead of turning that clause into one
that cannot fail. Identity is asserted too (ID, Title, Status, IssueType,
Priority), or a body that stripped the row to its id would satisfy every
emptiness clause. IsLitePartial is asserted in both directions.

THE LITE COLUMN LIST IS PINNED BY ORDER, not only by membership. Both lists are
scanned positionally, so two columns of the same SQL type can be transposed
with no membership change at all: the pre-existing set-based parity test stays
green, the scan succeeds, and every row silently carries one column's value in
another's field. TestIssueSelectColumnsLite_IsFullMinusHeavyInOrder derives the
expected list by deleting HeavyDropList from IssueSelectColumns in place, which
makes the lite list a derivation rather than a second hand-maintained copy.

RED BEFORE GREEN, five sabotages, each verified and reverted:

  - deleting the lite branch from CountsHydration.IssueColumns fails both page
    cases with "expected 58 destination arguments in Scan, not 52";
  - accepting Brief without plumbing it to either filter, the silent no-op this
    change exists to close, fails both page cases on all six text fields and on
    IsLitePartial, which is the run that proves the assertions are
    load-bearing;
  - transposing created_by and owner in IssueSelectColumnsLite fails only the
    new order test, on IssueSelectColumnsLite[9] = owner, want created_by,
    while the set-based test stays green, which is the gap it was added for;
  - dropping the claim's refusal fails the claimer contract with "ClaimNext
    with brief: error = <nil>, want ErrValidation"; and
  - clearing Brief in BuildReadyCountFilter fails
    TestBuildReadyCountFilterIsTheUnboundedReadyFilter, which already existed to
    catch a ReadyRequest field the count builder forgets to carry and gains
    Brief in its fixture here.

The recorded ready-filter golden gains "Lite": false in 62 cases because
types.WorkFilter gained the field and the test compares marshaled JSON
byte-for-byte. False is what a request that did not ask for the projection
resolves that field to, so the replay stays exact; the test comment says so.

Gates: go build ./..., go vet ./..., gofmt, internal/storage/... (dolt,
embeddeddolt with BEADS_TEST_EMBEDDED_DOLT=1, uow, issueops, sqlbuild,
domain/db), internal/workapi/..., backend/conformance/..., issueops/...,
make api-check.

Refs: #5546, #3906

Co-authored-by: sjarmak <t@t.co>
2026-08-10 10:45:23 -07:00

134 lines
6.2 KiB
Go

// Package sqlbuild holds the pure SQL-text builders shared by the classic
// issueops stack (production, *sql.Tx) and the domain/db repository stack
// (proxied-server, Runner). Functions here map (filter, tables) -> (sql, args)
// and are execution-context-free: no database handles, no transactions.
//
// Both stacks must produce identical row sets for identical filters (pinned
// by the Seam A parity suite in internal/storage/domain/db). Keeping the
// query text in one place is what prevents the two implementations from
// drifting; see bd-6dnrw.46 for the unify-vs-duplicate decision. Execution
// orchestration (issues+wisps merge strategy, hydration, probes) is
// intentionally NOT unified and stays in each stack.
package sqlbuild
import "strings"
// FilterTables configures table names for the filter builders, allowing the
// same filter logic to target both the issues and wisps table families.
type FilterTables struct {
Main string // "issues" or "wisps"
Labels string // "labels" or "wisp_labels"
Dependencies string // "dependencies" or "wisp_dependencies"
Comments string // "comments" or "wisp_comments"
}
var (
IssuesFilterTables = FilterTables{Main: "issues", Labels: "labels", Dependencies: "dependencies", Comments: "comments"}
WispsFilterTables = FilterTables{Main: "wisps", Labels: "wisp_labels", Dependencies: "wisp_dependencies", Comments: "wisp_comments"}
)
// DepTargetExpr resolves a dependency row's target across the three
// mutually-exclusive target columns.
const DepTargetExpr = "COALESCE(depends_on_issue_id, depends_on_wisp_id, depends_on_external)"
// IssueBaseColumns is the column list for the issues/wisps row itself,
// without the lease overlay. Use IssueSelectColumns for full hydration;
// this split exists so callers that alias the main table (QualifyColumns)
// can qualify the row columns without mangling the leases.* references.
const IssueBaseColumns = `id, content_hash, title, description, design, acceptance_criteria, notes,
status, priority, issue_type, assignee, estimated_minutes,
created_at, created_by, owner, updated_at, started_at, closed_at, external_ref, spec_id,
compaction_level, compacted_at, compacted_at_commit, original_size, source_repo, close_reason, closed_by_session,
sender, ephemeral, no_history, wisp_type, pinned, is_template,
await_type, await_id, timeout_ns, waiters,
mol_type,
event_kind, actor, target, payload,
due_at, defer_until,
work_type, source_system, metadata, row_lock, storage_class`
// IssueBaseColumnsLite is IssueBaseColumns minus the heavy TEXT columns
// (issueops.HeavyDropList). It lives here rather than beside its scan function
// for the same reason IssueBaseColumns does: the counts mega-query aliases the
// main table and has to qualify the row columns without mangling the leases.*
// references, and that query is rendered in this package. issueops re-exports
// the full list as IssueSelectColumnsLite and pins it against IssueBaseColumns
// with the schema-parity test.
const IssueBaseColumnsLite = `id, content_hash, title,
status, priority, issue_type, assignee, estimated_minutes,
created_at, created_by, owner, updated_at, started_at, closed_at, external_ref, spec_id,
compaction_level, compacted_at, compacted_at_commit, original_size, source_repo, close_reason, closed_by_session,
sender, ephemeral, no_history, wisp_type, pinned, is_template,
await_type, await_id, timeout_ns,
mol_type,
event_kind, actor, target,
due_at, defer_until,
work_type, source_system, metadata, row_lock, storage_class`
// LeaseSelectColumns is the lease overlay for full issue hydration. Leases
// live in the ephemeral leases table (bd-lrgn1), not on the issues row, so
// every query selecting these must also add LeaseJoin to its FROM clause —
// a query that forgets the join fails loudly on the leases.* reference.
const LeaseSelectColumns = `leases.lease_expires_at, leases.heartbeat_at, leases.granted_node`
// IssueSelectColumns is the canonical column list for full issue hydration.
// Every query that reads a complete types.Issue should use this constant
// and include LeaseJoin(table) in its FROM clause; the scan side is
// issueops.ScanIssueFrom, which scans positionally and must stay in
// column-for-column agreement with this list.
const IssueSelectColumns = IssueBaseColumns + `,
` + LeaseSelectColumns
// IssueSelectColumnsLite is the lite counterpart of IssueSelectColumns. The
// scan side is issueops.ScanIssueLiteFrom and the same LeaseJoin requirement
// applies.
const IssueSelectColumnsLite = IssueBaseColumnsLite + `,
` + LeaseSelectColumns
// LeaseJoin returns the FROM-clause fragment that overlays the ephemeral
// leases table onto the given issues/wisps table reference (a table name or
// alias). LEFT JOIN: rows without a live claim have no lease row and hydrate
// nil lease fields.
func LeaseJoin(tableRef string) string {
return "LEFT JOIN leases ON leases.issue_id = " + tableRef + ".id"
}
// QueryBatchSize bounds IN-clause sizes when long ID lists are folded into
// WHERE fragments.
const QueryBatchSize = 200
// QualifyColumns prefixes every column in a comma-separated list (e.g.
// IssueSelectColumns) with the given table qualifier such as "i.".
func QualifyColumns(columns, prefix string) string {
raw := strings.ReplaceAll(columns, "\n", " ")
raw = strings.ReplaceAll(raw, "\t", " ")
parts := strings.Split(raw, ",")
for i, p := range parts {
parts[i] = prefix + strings.TrimSpace(p)
}
return strings.Join(parts, ", ")
}
// InPlaceholders renders "?,?,..." and the matching arg slice for an IN clause.
func InPlaceholders[T ~string](values []T) (string, []any) {
ph := make([]string, len(values))
args := make([]any, len(values))
for i, v := range values {
ph[i] = "?"
args[i] = string(v)
}
return strings.Join(ph, ","), args
}
// CompactNonEmptyStrings drops empty entries, returning nil when none remain.
func CompactNonEmptyStrings(values []string) []string {
if len(values) == 0 {
return nil
}
out := make([]string, 0, len(values))
for _, value := range values {
if value != "" {
out = append(out, value)
}
}
return out
}