beads/engdocs/EXTENDING.md
Jim Wordelman 132601bfa0
feat(storage): opt-in lite SELECT shape (Issue body columns deferred) (#3906)
* feat(storage): be-uwvs.1 lite SELECT shape foundation

Add IssueFilter.Lite + IssueSelectColumnsLite + HeavyDropList +
ScanIssueLiteFrom so listing paths can opt out of materializing heavy
TEXT columns (description, design, acceptance_criteria, notes, payload,
waiters). Default Lite=false preserves today's behavior everywhere;
Lite=true (once wired to a call site) would return issues with
IsLitePartial=true and heavy fields blank.

A schema-parity test (cols(Lite) ∪ HeavyDropList == cols(Full)) guards
against future IssueSelectColumns drift, and happy-path tests exercise
both scan helpers.

Out of scope (per bead): CLI wiring, WorkFilter.Lite,
per-backend correctness tests, benchmark — those are be-uwvs.{2,3,4,5}.

Refs: be-uwvs (designer §1/§4/§5), be-jp5s (architecture D2/R-07).

Note (rework, PR #3906): this commit is quad341's original patch
cherry-picked onto current main, minus its internal/storage/issueops/
search.go hunk. That hunk targeted searchTableInTx, which #4150
("perf(id_parser): narrow SearchIssues to id-only projection", merged
2026-07-04) replaced with a generic searchProjection[T] dispatch; the
old hunk has no target to apply to and is dropped here rather than
force-merged. search.go is therefore unchanged by this commit — no
caller can reach the Lite branch yet. The follow-up commit re-homes the
filter.Lite switch inside the new projection system.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: release gate PASS for be-uwvs.1 (lite SELECT foundation)

* rework(storage): re-home lite SELECT switch into projection dispatch (PR #3906)

Re-homes quad341's filter.Lite SELECT-shape switch (PR #3906) into the
generic searchProjection[T] dispatch that #4150 ("perf(id_parser): narrow
SearchIssues to id-only projection", merged 2026-07-04) introduced in
internal/storage/issueops/search.go, replacing the searchTableInTx
function the PR originally patched.

- internal/storage/issueops/search.go: add issueLiteProjection, a
  searchProjection[*types.Issue] literal mirroring issueProjection but
  with columns()->IssueSelectColumnsLite and scan()->ScanIssueLiteFrom.
  SearchIssuesInTx now selects issueProjection or issueLiteProjection
  based on filter.Lite before calling the shared searchInTx wrapper, so
  the Lite branch gets wisp-merge, id-shrink Pattern B, and label/dep
  hydration for free, same as the full projection. No new top-level
  function and no patched SQL-string branch inside searchTableInTxT,
  matching the PR's own "adding a narrow-projection variant means adding
  a projection literal" design note already on issueProjection/idProjection.

- internal/storage/issueops/scan.go: classify the two column groups added
  to IssueSelectColumns since this PR's original patch predates them:
  `row_lock` (optimistic-concurrency token) and the `leases.lease_expires_at`
  / `leases.heartbeat_at` overlay (migration 0054, joined via
  sqlbuild.LeaseJoin). Both are small, non-TEXT, routing-relevant columns,
  so both go into IssueSelectColumnsLite/ScanIssueLiteFrom rather than
  HeavyDropList, matching the maintainer review's reasoning for the lease
  columns and extending it to row_lock (not discussed in the review because
  it wasn't yet on main when the review was written). issueLiteProjection
  sets joinLeases: true so the FROM clause gets the LEFT JOIN these columns
  require.

Verification (2026-07-23, current upstream/main = 1bbca0c1f):
- go build -tags gms_pure_go ./...                            -> clean
- go vet -tags gms_pure_go ./...                               -> clean
- gofmt -l on changed files                                    -> clean
- go test -tags gms_pure_go -run 'Lite|IssueSelectColumns' \
    ./internal/storage/...                                     -> PASS
    (TestIssueSelectColumns_LitePlusHeavyEqualsFull,
     TestScanIssueLiteFrom_LeavesHeavyFieldsBlank,
     TestScanIssueFrom_PopulatesHeavyFields,
     TestIssueSelectColumns_MatchClassic)
- go test -tags gms_pure_go ./internal/storage/issueops/...    -> PASS
- go test -tags gms_pure_go ./internal/storage/embeddeddolt/... \
    ./internal/storage/schema/... \
    ./internal/storage/versioncontrolops/... \
    ./internal/types/...                                       -> PASS

Scope preserved from the PR: foundation only. filter.Lite still has no
caller on main (grepped for `.Lite = true` / `Lite: true` outside this
diff -- none found); CLI wiring (bd list / bd dep tree / bd dep graph
--json / WorkFilter.Lite) remains deferred to be-uwvs.2+ per the PR body.

Rework of gastownhall/beads#3906 (feat/be-uwvs.1-lite-select-foundation by
quad341), per maphew's 2026-07-05 review comment on that PR, which
identified #4150 as the timing conflict and named the lease columns as
needing classification.

Agent-Signature: claude-sonnet-medium on behalf of maphew

* chore(storage): drop stale release-gate artifact from PR #3906

release-gates/be-uwvs.1-gate.md is a contributor-process artifact from
the original PR (contains stale fork references) and isn't something
this repo's process wants carried on the branch; docs/EXTENDING.md stays
as the legitimate caller-contract doc introduced by this PR.

Per Opus review (APPROVE-WITH-NITS) on the rework in
rework(storage): re-home lite SELECT switch into projection dispatch
(6882a4513).

Agent-Signature: claude-sonnet-medium on behalf of maphew

* docs(storage): note domain/db path doesn't honor filter.Lite yet

Per Codex review: the proxied-server (domain/db) path —
issueSQLRepositoryImpl.searchTable / fetchIssuesByIDs in
internal/storage/domain/db/issue_search.go — always issues the full
issueSelectColumns SELECT and scanIssue scanner, ignoring filter.Lite
entirely. Confirmed: neither function references filter.Lite.

Adjudicated as document-and-defer, no code change: no domain/db caller
sets Lite=true yet, so this is graceful degradation (correct results,
no lite optimization) rather than a bug, and wiring Lite through
domain/db belongs to the be-uwvs.2+ CLI-wiring follow-up alongside the
rest of the caller work.

- docs/EXTENDING.md: new "Backend coverage" subsection under the Lite
  SELECT shape contract; also fixes the "SELECT dispatch" bullet, which
  still named the pre-#4150 searchTableInTx function, to name the
  current issueProjection/issueLiteProjection dispatch.
- internal/types/types.go: same one-liner added to the IssueFilter.Lite
  field doc comment.

Agent-Signature: claude-sonnet-medium on behalf of maphew

* fix(storage): be-uwvs.1 wire less comparator into issueLiteProjection

issueLiteProjection was missing the less: sqlbuild.Less field that
issueProjection carries, per maphew's 2026-07-24 review comment on
PR #3906 flagging this as the remaining follow-up once #4160 landed.
Without it, searchInTx's merge branch (issues+wisps) has no comparator
to re-sort the concatenation of two independently-ordered legs before
trimming to filter.Limit when a Lite search takes that path — the trim
would keep whichever leg happens to come first regardless of sort rank,
unlike the full (non-Lite) projection which already re-sorts correctly.

Also relocates the PR's new docs/EXTENDING.md to engdocs/EXTENDING.md:
the file's own text says "It is not user-facing; it is for code that
embeds bd or talks to the storage layer directly," and
test/docsync.TestEveryDocsPageIsPublished requires every docs/ page to
carry a docs.json navigation entry (Mintlify-published) or live under
engdocs/ instead. engdocs/ is the existing home for this class of
internal contract doc, so move it there rather than adding a new
docs.json entry or exemption. Updates the two code comments that
pointed at the old path.

Verification (2026-07-25, rebased onto current origin/main):
- go build ./...                                     -> clean
- go build -tags gms_pure_go ./...                    -> clean
- go vet ./...                                        -> clean
- go test ./internal/storage/issueops/... ./internal/types/... ./test/docsync/...
                                                        -> PASS
- go test ./... (full suite)                          -> only pre-existing,
  environment-induced failures unrelated to this diff (verified identical
  on a clean origin/main baseline checkout: internal/beads, internal/config,
  internal/configfile, cmd/bd/doctor TMPDIR/shared-dolt-server/config-leak
  artifacts of this sandbox, plus TestCommitBeadsConfigSkipsGitHooks). Docker
  unavailable in this environment, so the cgo/Dolt-backed gate tests in
  cmd/bd (lite_filter_spy-based) self-skip, same as on baseline.

Rebased gastownhall/beads#3906 (feat/be-uwvs.1-lite-select-foundation)
onto current origin/main; the only conflict was the additive
IssueFilter struct-field collision with #3908/#4160's MaxRows/
MaxRowsSource (both kept).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(issueops): be-yrtwi regression test for issueLiteProjection merge-sort

issueLiteProjection was missing the less: sqlbuild.Less comparator that
issueProjection carries (fixed in 64778027e), so sortMergedResults
silently no-op'd on the issues+wisps merge branch and trimToSearchLimit
kept issues-leg concatenation order instead of the true globally-sorted
top-N. No test exercised IssueFilter{Lite: true} through SearchIssuesInTx
end-to-end, so this bug had no regression guard.

Adds a sqlmock-based end-to-end test (matching this package's existing
convention — no test in internal/storage/issueops uses a live DB) that
seeds an issues leg and a wisps leg where the true created_at-DESC top-2
lives entirely in the wisps leg, and asserts SearchIssuesInTx returns
that true order rather than the stale issues-leg rows. Verified this
fails against the pre-64778027e state (temporarily stripped less: and
confirmed red) and passes with it restored.

* fix(storage): classify leases.granted_node into IssueSelectColumnsLite (PR #3906)

Merging upstream/main into the PR #3906 rework branch (feat/be-uwvs.1-lite-
select-foundation) brought in wy-jpd3.7's replica-aware leases, which added
leases.granted_node to LeaseSelectColumns and therefore to IssueSelectColumns
(sqlbuild package). TestIssueSelectColumns_LitePlusHeavyEqualsFull caught the
drift: the new column was in neither IssueSelectColumnsLite nor HeavyDropList,
so the schema-parity invariant (lite ∪ drop == full) broke.

granted_node is a small, non-TEXT lease-overlay column — the same class as
lease_expires_at/heartbeat_at, which IssueSelectColumnsLite already retains
(see #4150) — so it belongs in the lite list, not HeavyDropList. Adds it to
IssueSelectColumnsLite, scans it in ScanIssueLiteFrom into
issue.LeaseGrantedNode (mirroring ScanIssueFrom), and updates the sqlmock
column/row fixtures in search_lite_merge_test.go (be-yrtwi's issueLiteProjection
merge-sort regression test) to the new column count.

This is downstream of #4160's `less: sqlbuild.Less` comparator wiring on
issueLiteProjection, already present on this branch and preserved by the
merge (search.go issueLiteProjection literal) — this commit only follows up
the schema-parity fallout from bringing that history in. Bead mybd-hu2k.
Agent-Signature: claude-fable-5-medium on behalf of matt wilkie

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: matt wilkie <maphew@gmail.com>
2026-07-26 15:29:08 -07:00

2.9 KiB

Extending bd

This file documents contracts that callers of the storage API must honor. It is not user-facing; it is for code that embeds bd or talks to the storage layer directly.

Lite SELECT shape — IssueFilter.Lite

store.SearchIssues(ctx, query, filter) accepts an IssueFilter value. When filter.Lite == true, the storage layer issues a narrower SELECT that omits these heavy TEXT columns:

  • description
  • design
  • acceptance_criteria
  • notes
  • waiters
  • payload

Contract for callers

Code that calls store.SearchIssues with IssueFilter.Lite == true:

  • MUST NOT read Description, Design, AcceptanceCriteria, Notes, Payload, or Waiters from any returned *types.Issue. These fields are zero-valued after a lite scan; they did not come from the row. Reading them yields no signal.
  • MAY read every other field — identity, status, priority, timestamps, labels, dependencies, metadata, etc. Lite preserves them.
  • MUST detect lite-fetched records via issue.IsLitePartial if branching behavior on hydration depth is required. The field is internal-only (json:"-") — it never crosses the wire.

To recover the full body for a specific issue after a lite listing, call store.GetIssue(ctx, id) — GetIssue always returns the full row.

Default behavior

IssueFilter.Lite defaults to false. Every existing call site that does not opt in retains today's behavior: heavy columns are fully hydrated, and Issue.IsLitePartial is false.

Where the contract is enforced

  • Column lists: internal/storage/issueops/scan.go (IssueSelectColumns, IssueSelectColumnsLite, HeavyDropList).
  • Scan helpers: ScanIssueFrom (full) and ScanIssueLiteFrom (lite, sets IsLitePartial).
  • SELECT dispatch: internal/storage/issueops/search.go — SearchIssuesInTx selects issueProjection or issueLiteProjection on filter.Lite; both are searchProjection[*types.Issue] literals sharing the wisp-merge and hydration machinery in searchTableInTxT.
  • Schema-parity guard: internal/storage/issueops/scan_test.go::TestIssueSelectColumns_LitePlusHeavyEqualsFull fails CI if a future column is added to IssueSelectColumns without being classified into IssueSelectColumnsLite or HeavyDropList.

Backend coverage

filter.Lite is currently honored only by the issueops-backed stores (Dolt, embedded Dolt) via the dispatch above. The proxied-server (internal/storage/domain/db) path — issueSQLRepositoryImpl.searchTable / fetchIssuesByIDs — does not check filter.Lite yet: it always issues the full issueSelectColumns SELECT and returns fully-hydrated issues with IsLitePartial == false. This is correct-but-unoptimized (no lite callers exist yet, so the difference is invisible today); wiring filter.Lite through the domain/db stack is deferred to the CLI-wiring follow-up (be-uwvs.2+), not part of this foundation.