Commit graph

14 commits

Author SHA1 Message Date
Jim Wordelman
2db739e8b0
perf(storage): SearchIssueSummaries narrow-projection foundation (be-nu4.3) (#3458)
* feat(search): re-cut SearchIssueSummaries atop Lite as searchProjection[T]

Re-cuts release/be-nu4.3-stack onto current main, resolved against the
merged Lite foundation (#3906) instead of the stale pre-Lite conflict set.

SearchIssueSummaries moves from a standalone search_summary.go into a
summaryProjection literal alongside issueProjection/issueLiteProjection/
idProjection in issueops/search.go, sharing the generic searchInTx core
instead of a parallel top-level function — retiring the anti-pattern
coffeegoddd's CHANGES_REQUESTED flagged. LessSummary shares lessFields
with Less via a summarySortFields adapter so the two comparators can't
drift on NULL-handling or tie-break rules.

Adds parity coverage (SearchIssues vs SearchIssueSummaries agree on ID
set/order and on every IssueSummary field) and the three-way wide/lite/
summary A/B benchmark, extending dolt_benchmark_test.go — which still
only compiles together with the projection method, the fact that
retired the earlier "land the bench file first" plan.

* test(feat): red — fix parity test to compare order directly + add comparator/wisp-limit coverage (refs be-be3pj)

- TestSearchIssuesAndSearchIssueSummaries_Parity now compares
  SearchIssues/SearchIssueSummaries IDs order-sensitively (slices.Equal)
  instead of via the sort-then-compare equalStringSlices helper, which
  would silently pass an ordering/top-N regression.
- Add TestLessSummaryMatchesLessAcrossSortKeys: Less vs LessSummary
  produce identical order across every SortDefs key + "id", both
  directions.
- Add TestSearchIssueSummariesInTx_MergesAcrossIssuesAndWisps: be-yrtwi
  regression pattern for summaryProjection specifically, covering
  wisp-merge top-N truncation under a limit.

* fix(types): give IssueSummary the same JSON tags as Issue

IssueSummary carried no struct tags, so encoding/json emitted Go's default
field names ({"ID":…,"Title":…}). The type exists to back list-shaped
rendering and `bd list --json` is one of that command's primary modes, so
the wiring follow-up would have silently changed the wire shape for every
consumer parsing bd output. Zero production callers today makes this a
zero-risk moment to fix it.

Tags mirror the same-named fields on types.Issue exactly, including
priority carrying no omitempty (0 is a valid P0). Three tests pin it: a
reflection test comparing every tag against Issue's so the two cannot
drift, an exact-key-set marshal test, and an omitempty test.

Reviewer item 1 (maphew, 2026-08-03).

* docs(storage): pin the --sort id contract on SearchIssueSummaries

The interface said SortBy/SortDesc are honored identically to SearchIssues,
which is true but left the id case looking like a gap in the summary path.
It is not a gap, and the doc now says why: sqlbuild.OrderBy emits no ORDER BY
for id (sqlbuild.IsGoSideSort), so *neither* path gets a SQL id order, and
both carry a comparator used only where the issues+wisps merge re-sorts
(sqlbuild.Less / sqlbuild.LessSummary).

The user-facing id order is cmd/bd's own, applied after the call via
utils.NaturalCompareIDs (cmd/bd/list.go sortIssues -> compareIssuesBy) —
numeric-aware, so bd-2 sorts before bd-10. Adding a lexicographic
storage-side summary sorter would therefore diverge from SearchIssues and
still not match what bd list renders.

Reviewer question (maphew, 2026-08-03): the contract no longer depends on
an unfiled follow-up to be readable.

* fix(backend): alias IssueSummary into the public backend contract

#5321 landed the public backend surface and its reflection drift guard on
2026-08-03, about two hours after this branch's last push, so the 08-11
maintenance rebase brought the branch under a gate that did not exist when
the engine interface was widened. TestPublicSurfaceComplete fails on both
platforms at the rebased head:

    internal type github.com/steveyegge/beads/internal/types.IssueSummary
    is reachable from the public backend contract but has no alias in
    package backend

The gate is right and the rebase broke nothing. SearchIssueSummaries
returns []*types.IssueSummary on storage.Storage, which backend.DoltStorage
aliases whole, so the row type is now part of the contract an out-of-tree
backend writes against — and without the alias that backend cannot name it.

Verified this is ours rather than base breakage: the same test passes on
origin/main and fails only on this branch.

* ci: retrigger after infra-flake red on dc92c3c3

Run 34381971686 went red on two failures, neither of them in this PR's
15-file diff:

- Test (storage domain + uow): internal/tracker lost its Dolt
  testcontainer mid-suite after a Ryuk reaper handshake reset
  (54 failed, 1994 passed). Known parallel-binary reaper flake; the
  fix is PR #5770, open and unmerged upstream since 2026-08-15. Third
  recorded recurrence, same package and same mapped port as the prior
  two.

- PR Core (wrapper timing): TestDoltServer_LogFile_CapturesOutput in
  internal/storage/dbproxy/server, a package this PR does not touch.
  Not reproducible locally in 55 attempts against clean main, 30 of
  them under 2x CPU load with the CI-pinned dolt 2.2.0.

We are pull-only on the upstream repo and cannot rerun jobs, so an
empty commit is the only retrigger lever available.

* test(issueops): tie the summary column list to its scanner, delete a third copy

Review point (minor 2): IssueSummaryColumns, ScanIssueSummaryFrom's dest list
and search_summary_merge_test.go's summaryColumnNames were three hand-synced
copies of the same list, with only the real-database field-parity test to
catch a drift between them.

Two changes rather than one guard over three lists:

  - summaryColumnNames now derives from IssueSummaryColumns instead of
    restating it, so the fixture cannot drift at all. summaryMergeRow places
    its values by column NAME against that list rather than by literal
    position, so adding a column widens the row automatically instead of
    silently shifting every value one slot left.

  - TestIssueSummaryColumnsMatchScanner pins the remaining pair by count,
    using a counting scanner that needs no database. This is the summary
    counterpart of TestIssueSelectColumns_LitePlusHeavyEqualsFull.

TestIssueSummaryColumnsAreRealIssueColumns covers what a count cannot: a
misspelled column keeps the widths equal and fails only at query time against
a real database, as an opaque unknown-column error from whichever caller runs
first. Requiring membership in IssueSelectColumns catches it in a unit test.

Membership, not order: unlike IssueSelectColumnsLite, IssueSummaryColumns is
deliberately not a subsequence of the full list, so there is no analogue of
TestIssueSelectColumnsLite_IsFullMinusHeavyInOrder.

Both guards verified red-before-green by mutation: adding a column without a
dest fails the first, and misspelling one fails the second.

* fix(types): carry the wisp-plane markers through the summary projection

Review point (MAJOR): IssueSummary's doc comment promises a summary-backed
list serializes to the same wire shape a full-Issue-backed one does, and
issueops.searchInTx merges wisp rows into every result whose filter does not
set SkipWisps. But IssueSummary had no Ephemeral, NoHistory, WispType or
StorageClass, so a wisp rendered through it dropped ephemeral, no_history,
wisp_type and storage_class from `bd list --json` while every other key stayed
identical -- a wisp and a durable bead became indistinguishable on the wire.
maphew raised this on 2026-08-03 and it was still open.

Adding the fields, not excluding wisps. Excluding them would break the
same-wisp-merge-semantics claim that is this PR's central design argument, and
would make the wiring follow-up silently drop wisps from `bd list` entirely --
a strictly worse outcome than the missing keys.

Correcting our own 2026-08-03 reasoning, which argued the other way. That
comment observed that no list *renderer* reads Ephemeral or WispType back off
the row, which is true and beside the point: `--json` does not read fields,
it marshals the struct. It also treated the cost as the one D3 exists to
remove. These four are narrow scalars -- two TINYINT(1), two short VARCHARs,
already present on both the issues and wisps tables and already scanned by
ScanIssueFrom -- not the multi-KB TEXT bodies and JSON parse the projection is
narrow to avoid. Zero callers today makes this free; after the wiring lands it
is a wire-format change.

Verified red-before-green at both levels:

  - TestIssueSummaryCarriesWispMarkers failed on all four fields before the
    change (reflection against Issue, asserting tag AND type -- WispType and
    StorageClass are named string types that a plain string would impersonate
    under a tag-only check).

  - TestIssueSummaryWispKeysMatchIssue marshals the same marker values through
    both types and compares the four keys byte for byte, including a durable
    row that must emit none of them.

  - TestSearchIssueSummaries_FieldsMatchSearchIssues now seeds an ephemeral
    wisp and a no-history wisp (mutually exclusive, so one row each) and
    compares all four markers against SearchIssues on a real Dolt database. It
    refuses to pass vacuously: fewer than two wisp rows in the result is a
    fatal, since zero-vs-zero marker comparisons would otherwise succeed if
    the merge stopped returning wisps. Mutating ScanIssueSummaryFrom to stop
    mapping wisp_type fails it on both seeded rows.

ScanIssueSummaryFrom maps the four exactly as ScanIssueFrom does, including a
NULL storage_class staying empty for EffectiveStorageClass to resolve per
plane rather than being pinned to a literal.

* perf(sqlbuild): compare sortFields through pointers, not two copies per call

Review point (minor 3): the shared-comparator refactor made Less build two
sortFields values per comparison and pass them by value, a small regression on
the pre-existing issues+wisps merge sort inside a perf PR.

lessFields and sortKeyCompareFields now take *sortFields. sortFields is a
~150-byte struct, so passing by value copied it twice into lessFields and
twice more into sortKeyCompareFields on every comparison; that is now zero
copies after the adapters run. The adapters still build one value per row --
that normalization is exactly what buys Less and LessSummary a single shared
body, and giving it up would reintroduce the drift this PR exists to remove.

No allocation either way, before or after: escape analysis reports nothing in
Less/LessSummary/lessFields escaping to heap (the only escapes in sort.go are
the pre-existing SortDefs map and OrderBy's string formatting), and both arms
measure 0 B/op, 0 allocs/op. A throwaway A/B in one binary, -benchtime=3s
-count=3 on the priority key, put the by-value median at 26.33 ns/op and the
pointer median at 25.35 ns/op, with non-overlapping ranges -- a consistent but
small ~4% win.

Stating that plainly rather than claiming more: the value here is removing four
struct copies per comparison from the hot path, not the wall-clock number,
which is within a couple of nanoseconds. Behavior is unchanged; the sqlbuild
suite, including the parity and LessID-honors-sortDesc tests, passes.

* test(dolt): vary SortBy across the summary parity table

Review point (minor 4): none of the nine parity cases set SortBy, so
LessSummary-versus-SQL ORDER BY agreement was proven by unit test alone
(TestLessSummaryMatchesLessAcrossSortKeys) and never against a real database.
Every case ran at the default order, priority ASC.

Nine rows added, covering created, title, status, updated and assignee -- the
first two in both directions -- plus id in both directions. Each carries a
Limit, so the merge's sort-before-trim actually decides which rows survive
rather than the ordering being cosmetic: an ordering bug that only shows up in
which rows a top-N keeps is the failure mode this table exists to catch.

id is included deliberately even though sqlbuild.OrderBy emits no ORDER BY for
it. It is the one Go-side key, so these two rows pin that both paths still
agree on the order the merge comparator produces where SQL contributes none.

All nine pass against a real Dolt database.

* docs(storage): document the summary projection's caller contract

Two review points, both about the contract being legible to a caller who has
only the interface and the extension guide.

Minor 1 -- IncludeDependencies is a silent no-op on the summary path, because
types.IssueSummary has no Dependencies field to populate. hydrateSummaryLabels
says why in its own comment; the interface a caller actually reads did not.
SearchIssueSummaries' doc now states it, alongside the neighbouring facts that
SkipLabels IS honored identically and that wisps are not excluded.

Minor 5 -- engdocs/EXTENDING.md documented the Lite projection's contract and
never mentioned the third one. It now carries a parallel section: what a caller
may read, why there is no IsLitePartial equivalent (a narrower type makes a
missing field a compile error rather than a silent zero value, which is the
substantive difference from Lite), the dependency and wisp rules above, the
ordering guarantee, and where each is enforced -- including the note that
IssueSummaryColumns is deliberately not an ordered subsequence of the full
list, so it has no analogue of the lite path's in-order guard.

The Lite section is left intact, its own Backend coverage subsection included;
the new section follows it rather than interleaving.

* chore(bazel): enumerate the two new test files in their go_test srcs

The PR adds internal/types/issue_summary_test.go and
internal/storage/issueops/search_summary_merge_test.go. Both are untagged,
and both packages carry a base-added BUILD.bazel whose go_test enumerates
srcs explicitly, so bazel silently would not have compiled or run them:
measured on //internal/types:types_test, the unrepaired binary lists 0 of
the 5 new tests (68 total) and the repaired one lists 5 of 5 (73), with
both builds exiting 0 -- a fake green, since bazel reports "1 test passes"
either way.

Base parity at 43a526b828 is clean (102 packages / 1192 untagged test files
enumerated, 0 srcs gaps), so the omission is PR-introduced rather than an
unenforced list. make bazel-sync-check and tools/bazel/go_srcs.py --check
both exit 0 on the broken head; generated with make bazel-sync (gazelle),
which is idempotent here.

Maintainer fixup applied during adopt-pr rebase (PR #3458).

* docs: announce the required SearchIssueSummaries method and inventory its span

Review follow-through for #3458 (maintainer fixup, iteration 1).

CHANGELOG.md: backend/backend.go promises that a required method added to
the engine interface is called out in CHANGELOG.md. SearchIssueSummaries is
one and had no entry. Adds an [Unreleased] "Changed" bullet in the #4911
form, next to the #6731 GetReadyWorkWithCountsAndTotal announcement: the
method and its signature, that callers are unaffected while implementers
must add it to compile, and its contract (SortBy/SortDesc and SkipLabels as
in SearchIssues, IncludeDependencies a silent no-op, wisps merged with their
markers set).

engdocs/design/otel/otel-data-model.md: the direct-operations block calls
itself a complete inventory and was exact at the PR base. The PR's new
storage.SearchIssueSummaries span was the one missing name. Inserted in
alphabetical order and bumped the count to 74. A set-diff against the
s.op(ctx, "...") names in internal/telemetry/storage.go now finds 74 == 74
with no drift in either direction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 01:19:24 -07:00
Vishnu J
66aa73bb99
fix(storage): include custom active-category statuses in bd ready (#5831) (#5918)
bd ready pins WorkFilter.Status to StatusOpen. BuildReadyWorkWhere treated
that pin as an exact match, so issues in a custom status with category
active never appeared — even though bd statuses --help, the ready_issues
view (migration 0025), and ClaimableSourceStatusesInTx all treat that
category as ready-eligible.

Widen the singular StatusOpen clause to the same predicate the view uses:

  status = ? OR status IN (SELECT name FROM custom_statuses WHERE category = 'active')

Non-open singular statuses stay exact. Built-in open still matches. Custom
wip/done/frozen/unspecified statuses stay out, matching claim's GH-3570
anti-steal set.

Reported by Justin Clemens (@justin-candor).

Addresses #5831
2026-09-02 18:11:45 -07:00
Denis Redozubov
5f56ab2a53
fix(deps): enforce external capability blockers across ready/claim/close (#4769) (#4753)
External (cross-project) dependency blockers are now enforced everywhere a bead's readiness or closability is decided: `bd ready` / `--claim`, `bd blocked`, `bd close` / done-status `bd update` (direct and proxied), `bd dep tree`, and served HTTP ready/claim. The policy is a storage-boundary decorator (`internal/storage/externaldeps`) applied on both the DoltStorage chain and the UOW provider, with full capability-forwarding parity, `Unwrap` at every level, and the backend's advisory defer-wake preserved on the claim path.

Review trail: maphew rounds 1–4 (Jul–Aug), bee rounds 1–4 (Sep 2): D1 recording-parity exemption, D2 ready-filter goldens, D3 UOW-level Unwrap for the proxied importer, D4 claim-path defer wake. Verified at head 3468d62f on a clean merge with current main; CI fully green at that head.

Fixes #4769.
2026-09-02 16:56:23 -07:00
Julian Knutsen
7e7dce8177
fix(storage): stop reporting a broken wisp plane as an empty one in search (re-land of #5805) (#5932)
* fix(storage): stop reporting a broken wisp plane as an empty one in search

A wisp search folded any table-not-exist error into "there are no wisps". The
tolerance is there for pre-migration databases, which have no wisp plane at
all, but the wisp query reads more tables than the wisp plane owns: its FROM
clause carries sqlbuild.LeaseJoin and its hydration reads wisp_labels. So a
database that has wisps but has lost wisp_labels is reported as having no wisps
-- with a nil error, from an ordinary list.

Reproduced end-to-end against embedded Dolt, with the row present in the same
transaction that is told it is not:

  RENAME TABLE wisp_labels TO wisp_labels_renamed_away
  SearchIssues(ephemeral)  before: 1 row   after: 0 rows, nil error
  SearchIssues(no filter)  before: 2 rows  after: 1 row,  nil error

GetIssueInTx already surfaces a missing wisp_labels as an error (pinned by the
existing embedded-Dolt wisp_label_table_error_propagates case), so the two read
paths disagreed today about whether that database is broken or empty.

Narrow the tolerance to the tables the wisp plane may legitimately not have.
sqlbuild.OptionalWispTable becomes the one place that set is written down --
issueops.optionalBlockedTable now reads it, so the classic and domain/db stacks
cannot drift on what a missing table means, which is the same reason the query
text itself lives in sqlbuild. The check is gated on the existing
IsTableNotExist classification via dberrors.MissingTableName, so nothing that
was previously an error becomes tolerated.

The leases half of the same over-broad check is included for the same reason
but is not, on its own, a reachable wrong answer today: the ephemeral branch
falls through to an issues-plane query over the same join, which re-raises the
missing table, so the caller does get an error. That was verified rather than
assumed, and the tests pin it at the guard rather than claiming a user-visible
repro.

Tests are red-first: sqlmock cases on both stacks and an end-to-end
RENAME TABLE against real embedded Dolt. Both directions have controls -- a
pre-migration database missing wisps must still search to an empty result, and
a missing wisp_dependencies must stay invisible -- because over-tightening to
"never tolerate" passes every must-error assertion and only those controls
catch it.

* test(storage): cover the union hydrate-wisps guard, which nothing reached

The review council mutated all four guarded call sites and found one that
no test in the repository reaches: the domain/db searchUnion hydrate-wisps
check. Reverting it to the old blanket table-not-exist test survived the
full domain tree and the nine-minute uow contract suite with embedded Dolt
enabled.

It is the site that matters most. The ephemeral branch needs an explicit
filter; the union is what a plain `bd list` runs. And the union is exactly
where the two stacks diverge: the leg's FROM clause is plan.FromSQL, with
no lease join and no label read, so a database missing leases or
wisp_labels answers the union query successfully and returns wisp ids --
the breakage surfaces only in fetchIssuesByIDs, which adds
sqlbuild.LeaseJoin and hydrates labels. Tolerating it there sends every
wisp through reassembleBySrc as a miss and drops it behind a nil error.

Also adds the fail-open control the domain/db file was missing. The
council's other surviving mutant made the guard return true for any error
it could not parse a table name out of; issueops caught it, domain/db was
green across the whole package.

Red before green (real edits, restored by trap):

  union guard -> blanket IsTableNotExist   TestSearchUnionBrokenWispPlane
                                           IsAnError, both subtests; every
                                           other test stays green
  fail open on unparseable errors          both UnrelatedErrorPropagates
                                           tests, ephemeral and union

The e2e wisp_dependencies control keeps its name but loses its overclaim:
a search that never asks for dependencies cannot see that table missing,
so it stays green whatever the tolerance set is. The sqlmock control is
what pins that entry; this one proves the query still answers end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-21 16:58:43 -07:00
Steve Yegge
96388ea019
fix(search): order Go-side-sort pages before every trim; honor sortDesc for id (#5639)
* fix(search): order Go-side-sort pages before every trim; honor sortDesc for id

Two live defects, both named-but-deferred in #5587's commit message and
both reachable from bd query '<expr>' --sort id (DefaultQueryLimit means a
bound is always set):

1. sqlbuild.Less ignored sortDesc for "id" — the one Go-side sort key —
   so every merge sort feeding a reversed id page kept the byte-FIRST
   rows. idSrcPage.sortGoSide's doc named this sibling bug explicitly.

2. #5587's fix (a page bound is only ever pushed under an order the query
   can express) covered the union seam only. The per-table seams
   (searchTable/scanFilterIDs, and the counts twin's
   finishSearchCountsPage) received the full matching set — searchWindowFor
   already withholds the page bound — but trimmed it with no sort anywhere:
   measured, 10 rows sort id desc limit 3 answered [000 001 002] where
   [009 008 007] was owed. The store-backed seam (issueops.searchTableInTxT)
   was worse off: it still pushed LIMIT into the leg SQL with no ORDER BY,
   so each leg contributed an arbitrary engine-ordered subset.

Fixes, one per seam, all converging on the same rule:

- sqlbuild.Less: sortDesc now flips the id comparison (byte order, matching
  the SQL the key would render if it could).
- domain/db: new sortRowsGoSide helper (the per-table copy of
  idSrcPage.sortGoSide) runs before finishWindow at searchTable's wide
  path, scanFilterIDs, and finishSearchCountsPage.
- issueops.searchTableInTxT: a Go-side sort leg no longer carries a SQL
  LIMIT; the leg scans the complete matching set, orders it in Go, and
  applies the same EffectiveSearchLimit bound after the order exists
  (goSideSortAndTrim), so cap accounting is unchanged and Pattern B
  hydrates only surviving rows.
- idProjection gains a comparator for the one key its payload can order
  ("id"), removing the merged-id-search landmine the old nil documented.

Tests: red-before-green verified on all three seams. sqlbuild unit test
pins Less; sqlmock tests pin the store-backed leg shape (no LIMIT under a
Go-side sort, bound still pushed under a SQL sort) and the ordered+trimmed
page; the domain/db suite adds per-table membership cases (asc, the
measured desc repro, and a four-route parity check: wisps-empty probe,
SkipWisps, wide path, counts twin must answer one page) plus a reversed
union page fence.

(bd-jao3t, bd-69c1a)

* fix(search): extend the Go-side-sort bound rule to the store-backed counts seam; one LessID comparator

Review round (adversarial pass on PR #5639):

F1 — runFilterSearchQueryInTx (issueops/search_counts.go) had the identical
defect the first commit fixed in searchTableInTxT: LIMIT eff pushed
unconditionally while sqlbuild.OrderBy renders no ORDER BY for "id". This
is the seam bd query '<expr>' --sort id actually reaches on the
store-shaped backends (storequerier → SearchIssuesWithCounts →
SearchIssuesWithCountsInTx), where BuildQueryPlan always pushes a bound —
so the first commit's repro claim held only for the uow/domain-db bodies.
Now the counts leg scans the complete matching set under a Go-side sort
and applies the same eff bound after ordering (goSideSortAndTrim),
leaving merge/trim/cap accounting exactly what the SQL LIMIT handed them.
bd list was already safe here (workapi.SQLLimit zeroes the pushed limit
for --sort id); only the querier path pushed a bound.

F2 — the byte-order id comparison existed in five hand-rolled copies
(Less, idProjection.less, goSideSortAndTrim, sortRowsGoSide,
idSrcPage.sortGoSide) — the exact duplication that let Less drift from
sortGoSide in the first place, and that hid F1's sixth site. All five now
call the one exported comparator, sqlbuild.LessID.

F3 — the ascending per-table suite case is annotated as the weak witness
it is (engine order for a PK-clustered prefix scan tends to be
byte-ascending already); the DESC cases are the red-before-green fences.

New shape test pins the counts legs: under --sort id neither the issues
nor the wisps counts query may carry a LIMIT (red on the previous commit:
the leg query carried LIMIT 3).

(bd-jao3t, bd-69c1a)
2026-08-11 00:24:28 -07:00
CI Bot
aa10a725e5 feat(list): put bd list's row cap and counts hydration on ListRequest (bd-b34o7)
STAGE ONE OF TWO. Fields, not methods, and no route flips — those are stage 2
(bd-5k21g). `issueops.ListRequest` gains MaxRows, MaxRowsSource and SkipCounts,
each with a conformance contract running at all three backends and zero parks.

MAXROWS IS THE EXACT INVERSE OF OFFSET, which is why it needed no new shape.
Offset is honored by the unit-of-work body — it renders OFFSET — and refused by
the store-backed one with a typed *ErrUnsupported. The cap points the other way:
the store search path enforces it after the scan (EnforceMaxRowsCap), and the
unit-of-work query path (internal/storage/domain/db) reads no cap at all, so a
request carrying one would come back as the FULL uncapped answer with nothing on
it to say so — a caller that set a circuit breaker handed exactly the runaway
query it was guarding against. It refuses instead, naming the operation and the
backend. `bd list --max-rows --proxied-server` has always refused one layer up
in the CLI for that reason; this is the same refusal for the callers that are
not the CLI. RunReaderListMaxRowsIsHonoredOrRefused mirrors
RunReaderOffsetIsHonoredOrRefused, including the never-silently-ignored clause,
and asserts the complement too: a cap the result set FITS inside must still come
back as an ordinary page, or a body that refused every cap out of hand would
pass.

MAXROWSSOURCE is the third field and the bead named two. It is attribution
only — it decides no answer — but `handleMaxRowsError` falls back to
"BEADS_MAX_ROWS" when it is empty, so a request that carried the number without
the source would make stage 2's route flip start telling operators to raise the
wrong knob. The contract asserts it comes back on the cap error.

SKIPCOUNTS is the SkipLabels precedent one column over: the three cardinality
joins are dropped and DependencyCount/DependentCount/CommentCount come back
zero, to be read as unknown rather than as none. It has to land WITH MaxRows —
stage 2 moves `bd list`'s TEXT path onto the role, and that path calls
SearchIssues today and projects no counts at all, so arriving on a counted page
without a way to opt out is a straight perf regression. The expensive member is
the reverse-blocker aggregate, which joins on a COALESCE key the pure-Go
analyzer cannot auto-index — the known embedded sore point, and the reason the
knob is worth having at all.

The contract's second half is the load-bearing one: NOTHING ELSE MOVES. Same
rows, same order, same has-more, same Parent — asserted by running the identical
request both ways rather than against a written-out expectation. Parent is the
tripwire: it rides the same mega-query and is not a cardinality, so an
implementation that made an aggregate join inner, or suppressed the parent join
too, would answer a strict subset and still look like it had "skipped the
counts".

WHAT THE SQL DOES. sqlbuild.SearchCountsSQL's two trailing hydration bools
become one CountsHydration value, so a path cannot pick up one opt-out and drop
the other, and both stores plus the unit-of-work repository read the pair
through one helper. A suppressed aggregate is projected as a constant IN PLACE
rather than removed: the scan reads the six extra columns positionally, so the
shape is invariant. Ready work passes the zero value at every call site —
types.WorkFilter carries neither knob, which is what "SkipLabels and SkipCounts
are not carried onto the ReadyFlag arm" means, now stated where a caller reads
it. MaxRows IS carried there; reader_ready_scope.go already claimed so, and this
is the commit that makes the claim true, since until now the field existed only
on the filter.

ONE HAND-ROLL DELETED. `bd list` built the filter through the shared builder and
then reached in and stamped filter.MaxRows/MaxRowsSource onto it — the
"build it, then change it" half-step this programme exists to make unreachable,
and the reason the cap was invisible to every implementation of Reader.List. The
cap is resolved in gatherListInput onto the request, BuildListFilter is now the
only writer of those two filter fields, and the proxied refusal reads the
already-resolved value (rejectResolvedMaxRowsUnderProxiedServer) so a malformed
BEADS_MAX_ROWS still warns exactly once.

PROVABLY INERT AT THE CLI, measured rather than argued: 22 `bd list` / `bd
children` invocations — every output mode, both max-rows knobs, the malformed
and negative cases — run against a binary built from integration/facade-wave-2
and one built from this commit, over ONE shared workspace so both read identical
data, differ on exactly one line of 617. `bd list --max-rows -1 --status bogus`
now reports the malformed flag instead of the invalid status; both exit 1, it
needs two bad arguments to reach, and it is recorded as A-lst1-3.

NO OPENAPI OPERATION AND NO NEW QUERY PARAMETER, deliberately, recorded as
A-lst1-1. `bd list`'s operation (GET /v0/beads/issues) already exists and is
already on the role, and neither field can be published on it yet: `bd serve`
always builds a provider-backed server, which REFUSES max_rows, so the parameter
would be one the reference server can never honor; and skip_counts would set
three unconditional integers on a PINNED response component to zero, leaving a
client unable to tell "no dependencies" from "not computed" — the failure
comments_omitted exists to prevent on the detail view. `make api-check`
regenerates nothing.

Verified: build, vet, gofmt clean; golangci-lint 0 issues; the two new cases at
dolt, embeddeddolt (BEADS_TEST_EMBEDDED_DOLT=1) and uow, plus the whole Reader
contract at all three; the builder golden (35 recorded cases plus two new ones
for the new fields) and the counts-SQL shape test; internal/httpapi and
make api-check; internal/storage/{issueops,domain/...}, internal/workapi,
internal/types, internal/telemetry, the root package; the embedded
`bd list`/`bd children`/max-rows CLI suites, TestProxiedServerList under
BEADS_TEST_PROXIED_SERVER=1 (including reject_max_rows_flag,
reject_max_rows_env and allow_max_rows_zero), TestProxiedLookup*, and the
protocol corpus package.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 16:36:31 +00:00
John Zook
21822ad725
perf(search): filter the main table before the counts aggregate joins (#4480)
* perf(search): filter the main table before the counts aggregate joins (#5)

* perf(search): filter the main table before the counts aggregate joins

sqlbuild.SearchCountsSQL (the counts mega-query behind bd query/search --counts,
bd list --counts, and ready-work counts) joined six aggregate derived tables
(labels, dep/rdep/comment counts, parent, deps JSON) onto every main-table row
and only then applied the WHERE filter. For the narrow ephemeral-work searches
that dominate the hot path, Dolt materialized those aggregates across the full
tables before pruning -- the dominant cost for that access pattern.

Wrap the main table in an inner subquery that applies the WHERE filter first, so
the aggregate LEFT JOINs drive off the already-narrowed set:

    FROM ( SELECT i.* FROM <main> i <whereSQL> ) i
    LEFT JOIN ( ... ) ...

The WHERE clause references only issue columns, never the aggregates, and the
joins are all LEFT JOINs that preserve every main row, so filtering before vs.
after the joins yields the identical row set. ORDER BY and LIMIT deliberately
stay in their original position after the joins: the ready-work counts path
passes a parameterized ORDER BY, so it must render exactly once to keep the
placeholder/arg counts aligned, and callers that do not re-sort in Go (the
domain/db stack) depend on a post-join ORDER BY for the final row order.

Tests:
- TestSearchCountsFilterBeforeJoinParity (real Dolt): the new shape returns
  byte-identical rows (ids, order, dep/rdep/comment counts, parent, labels, deps
  JSON) to the prior filter-after-join shape across 11 filter/order/limit/skip
  scenarios. The old shape is kept in test code as the reference.
- TestSearchCountsSQLShape: asserts the WHERE sits inside the derived subquery
  (before the first LEFT JOIN) and that ORDER BY / LIMIT stay after the joins and
  appear exactly once.
- Full sqlbuild, issueops, dolt counts/search/ready, and domain/db suites green.

* test(search): enforce WHERE-never-references-aggregates invariant

Add TestWhereClausesNeverReferenceAggregates plus contract comments on
both WHERE producers (BuildIssueFilterClauses, BuildReadyWorkWhere) and
SearchCountsSQL, pinning the load-bearing invariant behind the
filter-before-join shape: WHERE may reference only issue columns, never
the six counts aggregate aliases (out of scope in the pre-join subquery).

Drop the stale ORDER BY/LIMIT doc paragraph (described unchanged
behavior; the arg-count invariant is already pinned by
TestSearchCountsSQLShape).

* test(search): drop the maximally-populated WHERE-aggregate test

The enumeration test only checked fields it happened to populate, so a
new filter field referencing an aggregate would pass unnoticed — false
confidence plus a standing sync cost. The invariant is already guarded
structurally: TestSearchCountsSQLShape pins the WHERE inside the pre-join
subquery, which is exactly what makes a violation an out-of-scope column
reference that Dolt errors on (fail-loud). Reword the builder/contract
comments to lean on that rather than the deleted test.

* fix(search): keep plain driver on empty WHERE; by-IDs hydration in domain/db

Review follow-ups for the filter-before-joins reshape:

- SearchCountsSQL: an empty whereSQL keeps the plain "FROM <main> i"
  driver instead of wrapping it in a no-op derived table, so the
  unfiltered list/counts path keeps its pre-existing shape and plans.
  Pinned by a new TestSearchCountsSQLShape assertion.

- domain/db fetchCountsByIDs: hydrate via the by-IDs form instead of
  hand-building "WHERE i.id IN (...)" for the predicate form - that
  clause now renders inside the derived subquery, silently coupling the
  caller to the subquery's internal alias. The by-IDs form also
  constrains every aggregate subquery to the page; IDs are chunked
  (queryBatchSize) to stay within placeholder limits, mirroring
  issueops.runReadyCountsInTx.

---------

Co-authored-by: Zook Bot <275398848+zook-bot@users.noreply.github.com>
2026-07-26 13:04:52 -07:00
Remus Cazacu
9208279712
feat(ready): WorkFilter.Statuses — multi-status ready work in one query (#4752)
Callers that need ready work across several statuses (e.g. Gas City's
native store Ready(), which spans open/blocked/deferred/pinned/hooked/
review/testing) currently issue one GetReadyWork per status. Each call
re-pays the deferred-parents pre-query, the wisps arm, and the
per-transaction round trips — measured at 30-70ms per call on a live
server-mode store, ~7x per Ready().

WorkFilter.Statuses (OR semantics) renders a single status IN (...)
clause through BuildReadyWorkWhere and propagates to the wisp-side
IssueFilter.Statuses, so multi-status callers pay the fixed per-call
work once. Singular Status keeps precedence; both empty keeps the
legacy ('open', 'in_progress') default.

Review round 2: a conformance audit case (ReadyMultiStatusFilter) pins
the contract at the result level against both backends — real issues
and wisps across several statuses, OR semantics in one call, singular-
Status precedence, the empty default, a custom status through the same
IN clause, and the ephemeral exclusion of wisps without
IncludeEphemeral. The library guide documents Statuses, and
BuildReadyWorkWhere computes the status clause and args in one pass.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-26 12:53:23 -07:00
Athos Bernardes
83a5858044
fix(storage): wire --label-pattern into the WHERE builder (#4882)
* fix(storage): wire --label-pattern/--label-regex into the WHERE builder (ga-hqchm)

IssueFilter.LabelPattern/LabelRegex and WorkFilter.LabelPattern/LabelRegex
were parsed from `bd list --label-pattern`/`--label-regex` (and threaded
through to `bd list --ready`) but never read by BuildIssueFilterClauses or
BuildReadyWorkWhere — the SQL query builder silently ignored both fields, so
a non-matching pattern returned every issue instead of an empty/filtered set.

Wire both fields into a label-subquery clause, mirroring the existing
Labels/LabelsAny/ExcludeLabels idiom: LabelPattern converts glob syntax (*, ?)
to a SQL LIKE pattern (escaping literal %, _, and \ in the input first);
LabelRegex passes through to SQL REGEXP directly.

Confirmed via direct binary execution, not just unit tests: built the fixed
bd, ran `bd list --label-pattern 'warrant*'` against a scratch workspace with
no matching label — now returns [] instead of 5 arbitrary unrelated issues,
matching the exact repro reported on ga-hqchm.

Also strengthens the existing `label_pattern` CLI test
(cmd/bd/list_embedded_test.go), which only asserted the matching issue was
present, never that a non-matching issue was absent — which is why it kept
passing despite the dead filter — and adds a sibling `label_regex` case.

Note: the standalone `bd ready` subcommand doesn't expose --label-pattern/
--label-regex as flags at all (separate gap from this bug; `bd list --ready`,
which shares WorkFilter/BuildReadyWorkWhere, works correctly with this fix).
Filed separately, not fixed here.

* fix(storage): drop LabelRegex/SQL REGEXP wiring, narrow to --label-pattern

engdocs/ICU-POLICY.md is explicit: "bd does not use SQL REGEXP functions,
so [excluding ICU] has zero practical impact" — that sentence is the stated
justification for shipping without libicu at all. The previous commit's
LabelRegex wiring used `WHERE label REGEXP ?`, which is real SQL REGEXP
usage and directly contradicts that invariant, even though it happens to
still work under gms_pure_go (RE2 via Go's regexp instead of ICU).

--label-regex needs a different implementation (Go-side regexp.MatchString
against fetched candidates, not a SQL REGEXP push-down) to stay consistent
with the policy, and ga-hqchm's reported bug only covers --label-pattern
(glob/LIKE) in the first place. Per CONTRIBUTING.md's "one issue per PR"
rule, splitting it out rather than doing the redesign inline here.

Rebuilt and re-verified end to end with the project's own canonical build
(`go build -tags gms_pure_go`, confirmed ICU-free via `otool -L`): both
`bd list --label-pattern` and `bd list --ready --label-pattern` now filter
correctly, matching-pattern issues included, non-matching issues excluded.

* fix(ready): wire --label-pattern and --label-regex into BuildReadyWorkWhere and bd ready

Upstream #3971 landed LabelPattern/LabelRegex for the list path; the
ready path still silently ignored both. Add the LIKE ESCAPE and REGEXP
subquery clauses to BuildReadyWorkWhere mirroring filter.go, and register
both flags on the standalone bd ready subcommand.

The bd ready flag registration absorbs the surviving scope of #4884 —
credit to athosmartins for both the pattern (this PR) and regex (#4884)
ready-path intent.

Addresses the 2026-07-23 review on gastownhall/beads#4882.

Agent-Signature: claude-sonnet-5-medium on behalf of maphew

* fix(ready): honor --label-pattern/--label-regex under the proxied path

Reviewer finding: flag reads happened after the proxied early-return and
gatherReadyInput never copied them into WorkFilter, so both filters were
silently ignored under --proxied-server. Hoist the reads, wire them
through readyInput, and pin with a gatherReadyInput unit test plus a
proxied integration subtest covering both filters.

Agent-Signature: codex-gpt-5.6-terra-medium on behalf of maphew

---------

Co-authored-by: matt wilkie <maphew@gmail.com>
2026-07-25 19:05:27 -07:00
matt wilkie
fbdf7badbc Merge remote-tracking branch 'upstream/main' into pr-4095-fix 2026-07-13 20:39:27 -07:00
Steve Yegge
67ccdbfd44 fix(ready): honor --label-any on the ready/claim path (wy-jpd3.2)
BuildReadyWorkWhere emitted clauses for --label (AND-set) and
--exclude-label but none at all for --label-any, so the OR-set filter was
silently dropped by `bd ready` and `bd ready --claim` on every backend
(with or without --parent), while `bd list`/`bd search` honored it.

On an atomic claim the silent drop is dangerous rather than merely wrong:
a worker fencing itself to its own lane
(`bd ready --claim --label-any lane-a --parent epic-1`) would claim
another lane's issue and believe it was fenced. Reproduced on current
main: the new conformance case claims a lane-b issue when lane-a was
requested.

Emit an OR-set membership clause (same shape as the list path in
sqlbuild/filter.go) that AND-combines with Labels, ExcludeLabels and the
parent filter. All claim paths (Dolt, embedded-Dolt, sqlkit SQL family)
funnel through ClaimReadyIssueInTx -> GetReadyWorkInTx ->
BuildReadyWorkWhere, so the single seam fixes every backend.

Tests: a SQL-shape unit test in sqlbuild, plus a behavioral conformance
case (ClaimReadyIssueLabelFilters) asserting the claim is fenced by the
OR-set alone, AND-combined with --label and --parent, and — the safety
property — that an exhausted lane claims NOTHING rather than falling back
to unfenced ready work.
2026-07-12 07:58:22 -07:00
Julian Knutsen
fb527d034a
perf(ready): page-push ready-work counts and size the total with cheap COUNT(*)s (#4679)
`bd ready --json` had two O(N) hotspots on main. The counts mega-query's
reverse-blocker `rc` subquery keys on COALESCE(depends_on_issue_id, …), an
expression the pure-Go GMS analyzer cannot auto-index, so it re-scanned its
whole materialization once per candidate — O(candidates × blockers). And the
"Showing X of N" total re-ran that entire mega-query UNBOUNDED just to size N.

Changes:
- Generalize one counts builder: sqlbuild.SearchCountsSQL now takes an optional
  `ids []string`. Empty ids = the predicate form (unchanged callers); non-empty
  ids = the page-pushed form that constrains the driver and every count
  subquery to the page, bounding rc by the page instead of the full graph.
  Removes the drifted SearchCountsSQL/SearchCountsByIDsSQL duplication; the
  projection order is defined once and DepTargetExpr is used throughout.
- issueops.runReadyCountsInTx resolves the ready-ID page with the cheap indexed
  ID query, then hydrates counts constrained to those IDs, CHUNKED by
  sqlbuild.QueryBatchSize so large pages stay within backend placeholder limits
  (no quadratic fallback, no wasted ID pass). The page order is restored so the
  result is byte-identical to the unbounded ORDER BY … LIMIT.
- issueops.CountReadyWorkInTx sizes N with indexed COUNT(*)s over the ready
  predicate for the issues AND wisps families, minus the wisp-wins overlap —
  identical to len(GetReadyWorkWithCounts(Limit=0)) even when wisps exist, with
  no mega-query fallback.
- Add ReadyWorkCounter to internal/storage beside the other optional
  capabilities; DoltStore and EmbeddedDoltStore implement CountReadyWork.
  cmd/bd/ready.go zeroes the limit at the call site and unwraps to the
  capability via storage.UnwrapStore, falling back to the unbounded query only
  when a store predates it.
- Extract the shared scan/dedupe loop (scanCountsRowsInTx).
- Conformance tests: page counts byte-identical to unbounded, and
  CountReadyWork == len(unbounded), with fixtures for wisps and a page that
  spans the chunk boundary.

Co-authored-by: Test User <test@test.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 18:07:41 -07:00
matt wilkie
8ad064a961 Merge upstream/main into PR 4095
Resolve ready-work ordering against current main after upstream moved the shared ordering into sqlbuild. Priority ready work now breaks same-priority ties with created_at ASC, and regression coverage checks both the shared helper and ClaimReadyIssue.

Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie
2026-07-08 12:59:23 -07:00
Steve Yegge
1f8f1fff31 refactor(storage): extract shared SQL builders into internal/storage/sqlbuild (bd-6dnrw.46)
Move all pure SQL-text builders shared by issueops (classic) and
domain/db (proxied) into one package; both stacks now delegate, so
ready/filter/counts query semantics can no longer drift (Seam A).
Execution orchestration stays per-stack per the extract-don't-unify
decision. Canonical SQL text is classic's; the only proxied-side text
deltas are IN-list whitespace.

Adversarial re-review (bd-6dnrw.47) verified all 10 attack items with
no production divergence, and adds an import-boundary test pinning the
layering (domain must never import issueops/sqlbuild) plus a single-use
constraint note on LabelSearchPlan.MergeInto. Follow-ups filed:
bd-emrs2 (+1-overfetch convention), bd-kzeme (IN-helper consolidation).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 15:56:23 -04:00