Commit graph

10 commits

Author SHA1 Message Date
Steve Yegge
96388ea019
fix(search): order Go-side-sort pages before every trim; honor sortDesc for id (#5639)
* fix(search): order Go-side-sort pages before every trim; honor sortDesc for id

Two live defects, both named-but-deferred in #5587's commit message and
both reachable from bd query '<expr>' --sort id (DefaultQueryLimit means a
bound is always set):

1. sqlbuild.Less ignored sortDesc for "id" — the one Go-side sort key —
   so every merge sort feeding a reversed id page kept the byte-FIRST
   rows. idSrcPage.sortGoSide's doc named this sibling bug explicitly.

2. #5587's fix (a page bound is only ever pushed under an order the query
   can express) covered the union seam only. The per-table seams
   (searchTable/scanFilterIDs, and the counts twin's
   finishSearchCountsPage) received the full matching set — searchWindowFor
   already withholds the page bound — but trimmed it with no sort anywhere:
   measured, 10 rows sort id desc limit 3 answered [000 001 002] where
   [009 008 007] was owed. The store-backed seam (issueops.searchTableInTxT)
   was worse off: it still pushed LIMIT into the leg SQL with no ORDER BY,
   so each leg contributed an arbitrary engine-ordered subset.

Fixes, one per seam, all converging on the same rule:

- sqlbuild.Less: sortDesc now flips the id comparison (byte order, matching
  the SQL the key would render if it could).
- domain/db: new sortRowsGoSide helper (the per-table copy of
  idSrcPage.sortGoSide) runs before finishWindow at searchTable's wide
  path, scanFilterIDs, and finishSearchCountsPage.
- issueops.searchTableInTxT: a Go-side sort leg no longer carries a SQL
  LIMIT; the leg scans the complete matching set, orders it in Go, and
  applies the same EffectiveSearchLimit bound after the order exists
  (goSideSortAndTrim), so cap accounting is unchanged and Pattern B
  hydrates only surviving rows.
- idProjection gains a comparator for the one key its payload can order
  ("id"), removing the merged-id-search landmine the old nil documented.

Tests: red-before-green verified on all three seams. sqlbuild unit test
pins Less; sqlmock tests pin the store-backed leg shape (no LIMIT under a
Go-side sort, bound still pushed under a SQL sort) and the ordered+trimmed
page; the domain/db suite adds per-table membership cases (asc, the
measured desc repro, and a four-route parity check: wisps-empty probe,
SkipWisps, wide path, counts twin must answer one page) plus a reversed
union page fence.

(bd-jao3t, bd-69c1a)

* fix(search): extend the Go-side-sort bound rule to the store-backed counts seam; one LessID comparator

Review round (adversarial pass on PR #5639):

F1 — runFilterSearchQueryInTx (issueops/search_counts.go) had the identical
defect the first commit fixed in searchTableInTxT: LIMIT eff pushed
unconditionally while sqlbuild.OrderBy renders no ORDER BY for "id". This
is the seam bd query '<expr>' --sort id actually reaches on the
store-shaped backends (storequerier → SearchIssuesWithCounts →
SearchIssuesWithCountsInTx), where BuildQueryPlan always pushes a bound —
so the first commit's repro claim held only for the uow/domain-db bodies.
Now the counts leg scans the complete matching set under a Go-side sort
and applies the same eff bound after ordering (goSideSortAndTrim),
leaving merge/trim/cap accounting exactly what the SQL LIMIT handed them.
bd list was already safe here (workapi.SQLLimit zeroes the pushed limit
for --sort id); only the querier path pushed a bound.

F2 — the byte-order id comparison existed in five hand-rolled copies
(Less, idProjection.less, goSideSortAndTrim, sortRowsGoSide,
idSrcPage.sortGoSide) — the exact duplication that let Less drift from
sortGoSide in the first place, and that hid F1's sixth site. All five now
call the one exported comparator, sqlbuild.LessID.

F3 — the ascending per-table suite case is annotated as the weak witness
it is (engine order for a PK-clustered prefix scan tends to be
byte-ascending already); the DESC cases are the red-before-green fences.

New shape test pins the counts legs: under --sort id neither the issues
nor the wisps counts query may carry a LIMIT (red on the previous commit:
the leg query carried LIMIT 3).

(bd-jao3t, bd-69c1a)
2026-08-11 00:24:28 -07:00
CI Bot
aa10a725e5 feat(list): put bd list's row cap and counts hydration on ListRequest (bd-b34o7)
STAGE ONE OF TWO. Fields, not methods, and no route flips — those are stage 2
(bd-5k21g). `issueops.ListRequest` gains MaxRows, MaxRowsSource and SkipCounts,
each with a conformance contract running at all three backends and zero parks.

MAXROWS IS THE EXACT INVERSE OF OFFSET, which is why it needed no new shape.
Offset is honored by the unit-of-work body — it renders OFFSET — and refused by
the store-backed one with a typed *ErrUnsupported. The cap points the other way:
the store search path enforces it after the scan (EnforceMaxRowsCap), and the
unit-of-work query path (internal/storage/domain/db) reads no cap at all, so a
request carrying one would come back as the FULL uncapped answer with nothing on
it to say so — a caller that set a circuit breaker handed exactly the runaway
query it was guarding against. It refuses instead, naming the operation and the
backend. `bd list --max-rows --proxied-server` has always refused one layer up
in the CLI for that reason; this is the same refusal for the callers that are
not the CLI. RunReaderListMaxRowsIsHonoredOrRefused mirrors
RunReaderOffsetIsHonoredOrRefused, including the never-silently-ignored clause,
and asserts the complement too: a cap the result set FITS inside must still come
back as an ordinary page, or a body that refused every cap out of hand would
pass.

MAXROWSSOURCE is the third field and the bead named two. It is attribution
only — it decides no answer — but `handleMaxRowsError` falls back to
"BEADS_MAX_ROWS" when it is empty, so a request that carried the number without
the source would make stage 2's route flip start telling operators to raise the
wrong knob. The contract asserts it comes back on the cap error.

SKIPCOUNTS is the SkipLabels precedent one column over: the three cardinality
joins are dropped and DependencyCount/DependentCount/CommentCount come back
zero, to be read as unknown rather than as none. It has to land WITH MaxRows —
stage 2 moves `bd list`'s TEXT path onto the role, and that path calls
SearchIssues today and projects no counts at all, so arriving on a counted page
without a way to opt out is a straight perf regression. The expensive member is
the reverse-blocker aggregate, which joins on a COALESCE key the pure-Go
analyzer cannot auto-index — the known embedded sore point, and the reason the
knob is worth having at all.

The contract's second half is the load-bearing one: NOTHING ELSE MOVES. Same
rows, same order, same has-more, same Parent — asserted by running the identical
request both ways rather than against a written-out expectation. Parent is the
tripwire: it rides the same mega-query and is not a cardinality, so an
implementation that made an aggregate join inner, or suppressed the parent join
too, would answer a strict subset and still look like it had "skipped the
counts".

WHAT THE SQL DOES. sqlbuild.SearchCountsSQL's two trailing hydration bools
become one CountsHydration value, so a path cannot pick up one opt-out and drop
the other, and both stores plus the unit-of-work repository read the pair
through one helper. A suppressed aggregate is projected as a constant IN PLACE
rather than removed: the scan reads the six extra columns positionally, so the
shape is invariant. Ready work passes the zero value at every call site —
types.WorkFilter carries neither knob, which is what "SkipLabels and SkipCounts
are not carried onto the ReadyFlag arm" means, now stated where a caller reads
it. MaxRows IS carried there; reader_ready_scope.go already claimed so, and this
is the commit that makes the claim true, since until now the field existed only
on the filter.

ONE HAND-ROLL DELETED. `bd list` built the filter through the shared builder and
then reached in and stamped filter.MaxRows/MaxRowsSource onto it — the
"build it, then change it" half-step this programme exists to make unreachable,
and the reason the cap was invisible to every implementation of Reader.List. The
cap is resolved in gatherListInput onto the request, BuildListFilter is now the
only writer of those two filter fields, and the proxied refusal reads the
already-resolved value (rejectResolvedMaxRowsUnderProxiedServer) so a malformed
BEADS_MAX_ROWS still warns exactly once.

PROVABLY INERT AT THE CLI, measured rather than argued: 22 `bd list` / `bd
children` invocations — every output mode, both max-rows knobs, the malformed
and negative cases — run against a binary built from integration/facade-wave-2
and one built from this commit, over ONE shared workspace so both read identical
data, differ on exactly one line of 617. `bd list --max-rows -1 --status bogus`
now reports the malformed flag instead of the invalid status; both exit 1, it
needs two bad arguments to reach, and it is recorded as A-lst1-3.

NO OPENAPI OPERATION AND NO NEW QUERY PARAMETER, deliberately, recorded as
A-lst1-1. `bd list`'s operation (GET /v0/beads/issues) already exists and is
already on the role, and neither field can be published on it yet: `bd serve`
always builds a provider-backed server, which REFUSES max_rows, so the parameter
would be one the reference server can never honor; and skip_counts would set
three unconditional integers on a PINNED response component to zero, leaving a
client unable to tell "no dependencies" from "not computed" — the failure
comments_omitted exists to prevent on the detail view. `make api-check`
regenerates nothing.

Verified: build, vet, gofmt clean; golangci-lint 0 issues; the two new cases at
dolt, embeddeddolt (BEADS_TEST_EMBEDDED_DOLT=1) and uow, plus the whole Reader
contract at all three; the builder golden (35 recorded cases plus two new ones
for the new fields) and the counts-SQL shape test; internal/httpapi and
make api-check; internal/storage/{issueops,domain/...}, internal/workapi,
internal/types, internal/telemetry, the root package; the embedded
`bd list`/`bd children`/max-rows CLI suites, TestProxiedServerList under
BEADS_TEST_PROXIED_SERVER=1 (including reject_max_rows_flag,
reject_max_rows_env and allow_max_rows_zero), TestProxiedLookup*, and the
protocol corpus package.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 16:36:31 +00:00
John Zook
21822ad725
perf(search): filter the main table before the counts aggregate joins (#4480)
* perf(search): filter the main table before the counts aggregate joins (#5)

* perf(search): filter the main table before the counts aggregate joins

sqlbuild.SearchCountsSQL (the counts mega-query behind bd query/search --counts,
bd list --counts, and ready-work counts) joined six aggregate derived tables
(labels, dep/rdep/comment counts, parent, deps JSON) onto every main-table row
and only then applied the WHERE filter. For the narrow ephemeral-work searches
that dominate the hot path, Dolt materialized those aggregates across the full
tables before pruning -- the dominant cost for that access pattern.

Wrap the main table in an inner subquery that applies the WHERE filter first, so
the aggregate LEFT JOINs drive off the already-narrowed set:

    FROM ( SELECT i.* FROM <main> i <whereSQL> ) i
    LEFT JOIN ( ... ) ...

The WHERE clause references only issue columns, never the aggregates, and the
joins are all LEFT JOINs that preserve every main row, so filtering before vs.
after the joins yields the identical row set. ORDER BY and LIMIT deliberately
stay in their original position after the joins: the ready-work counts path
passes a parameterized ORDER BY, so it must render exactly once to keep the
placeholder/arg counts aligned, and callers that do not re-sort in Go (the
domain/db stack) depend on a post-join ORDER BY for the final row order.

Tests:
- TestSearchCountsFilterBeforeJoinParity (real Dolt): the new shape returns
  byte-identical rows (ids, order, dep/rdep/comment counts, parent, labels, deps
  JSON) to the prior filter-after-join shape across 11 filter/order/limit/skip
  scenarios. The old shape is kept in test code as the reference.
- TestSearchCountsSQLShape: asserts the WHERE sits inside the derived subquery
  (before the first LEFT JOIN) and that ORDER BY / LIMIT stay after the joins and
  appear exactly once.
- Full sqlbuild, issueops, dolt counts/search/ready, and domain/db suites green.

* test(search): enforce WHERE-never-references-aggregates invariant

Add TestWhereClausesNeverReferenceAggregates plus contract comments on
both WHERE producers (BuildIssueFilterClauses, BuildReadyWorkWhere) and
SearchCountsSQL, pinning the load-bearing invariant behind the
filter-before-join shape: WHERE may reference only issue columns, never
the six counts aggregate aliases (out of scope in the pre-join subquery).

Drop the stale ORDER BY/LIMIT doc paragraph (described unchanged
behavior; the arg-count invariant is already pinned by
TestSearchCountsSQLShape).

* test(search): drop the maximally-populated WHERE-aggregate test

The enumeration test only checked fields it happened to populate, so a
new filter field referencing an aggregate would pass unnoticed — false
confidence plus a standing sync cost. The invariant is already guarded
structurally: TestSearchCountsSQLShape pins the WHERE inside the pre-join
subquery, which is exactly what makes a violation an out-of-scope column
reference that Dolt errors on (fail-loud). Reword the builder/contract
comments to lean on that rather than the deleted test.

* fix(search): keep plain driver on empty WHERE; by-IDs hydration in domain/db

Review follow-ups for the filter-before-joins reshape:

- SearchCountsSQL: an empty whereSQL keeps the plain "FROM <main> i"
  driver instead of wrapping it in a no-op derived table, so the
  unfiltered list/counts path keeps its pre-existing shape and plans.
  Pinned by a new TestSearchCountsSQLShape assertion.

- domain/db fetchCountsByIDs: hydrate via the by-IDs form instead of
  hand-building "WHERE i.id IN (...)" for the predicate form - that
  clause now renders inside the derived subquery, silently coupling the
  caller to the subquery's internal alias. The by-IDs form also
  constrains every aggregate subquery to the page; IDs are chunked
  (queryBatchSize) to stay within placeholder limits, mirroring
  issueops.runReadyCountsInTx.

---------

Co-authored-by: Zook Bot <275398848+zook-bot@users.noreply.github.com>
2026-07-26 13:04:52 -07:00
Remus Cazacu
9208279712
feat(ready): WorkFilter.Statuses — multi-status ready work in one query (#4752)
Callers that need ready work across several statuses (e.g. Gas City's
native store Ready(), which spans open/blocked/deferred/pinned/hooked/
review/testing) currently issue one GetReadyWork per status. Each call
re-pays the deferred-parents pre-query, the wisps arm, and the
per-transaction round trips — measured at 30-70ms per call on a live
server-mode store, ~7x per Ready().

WorkFilter.Statuses (OR semantics) renders a single status IN (...)
clause through BuildReadyWorkWhere and propagates to the wisp-side
IssueFilter.Statuses, so multi-status callers pay the fixed per-call
work once. Singular Status keeps precedence; both empty keeps the
legacy ('open', 'in_progress') default.

Review round 2: a conformance audit case (ReadyMultiStatusFilter) pins
the contract at the result level against both backends — real issues
and wisps across several statuses, OR semantics in one call, singular-
Status precedence, the empty default, a custom status through the same
IN clause, and the ephemeral exclusion of wisps without
IncludeEphemeral. The library guide documents Statuses, and
BuildReadyWorkWhere computes the status clause and args in one pass.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-26 12:53:23 -07:00
Athos Bernardes
83a5858044
fix(storage): wire --label-pattern into the WHERE builder (#4882)
* fix(storage): wire --label-pattern/--label-regex into the WHERE builder (ga-hqchm)

IssueFilter.LabelPattern/LabelRegex and WorkFilter.LabelPattern/LabelRegex
were parsed from `bd list --label-pattern`/`--label-regex` (and threaded
through to `bd list --ready`) but never read by BuildIssueFilterClauses or
BuildReadyWorkWhere — the SQL query builder silently ignored both fields, so
a non-matching pattern returned every issue instead of an empty/filtered set.

Wire both fields into a label-subquery clause, mirroring the existing
Labels/LabelsAny/ExcludeLabels idiom: LabelPattern converts glob syntax (*, ?)
to a SQL LIKE pattern (escaping literal %, _, and \ in the input first);
LabelRegex passes through to SQL REGEXP directly.

Confirmed via direct binary execution, not just unit tests: built the fixed
bd, ran `bd list --label-pattern 'warrant*'` against a scratch workspace with
no matching label — now returns [] instead of 5 arbitrary unrelated issues,
matching the exact repro reported on ga-hqchm.

Also strengthens the existing `label_pattern` CLI test
(cmd/bd/list_embedded_test.go), which only asserted the matching issue was
present, never that a non-matching issue was absent — which is why it kept
passing despite the dead filter — and adds a sibling `label_regex` case.

Note: the standalone `bd ready` subcommand doesn't expose --label-pattern/
--label-regex as flags at all (separate gap from this bug; `bd list --ready`,
which shares WorkFilter/BuildReadyWorkWhere, works correctly with this fix).
Filed separately, not fixed here.

* fix(storage): drop LabelRegex/SQL REGEXP wiring, narrow to --label-pattern

engdocs/ICU-POLICY.md is explicit: "bd does not use SQL REGEXP functions,
so [excluding ICU] has zero practical impact" — that sentence is the stated
justification for shipping without libicu at all. The previous commit's
LabelRegex wiring used `WHERE label REGEXP ?`, which is real SQL REGEXP
usage and directly contradicts that invariant, even though it happens to
still work under gms_pure_go (RE2 via Go's regexp instead of ICU).

--label-regex needs a different implementation (Go-side regexp.MatchString
against fetched candidates, not a SQL REGEXP push-down) to stay consistent
with the policy, and ga-hqchm's reported bug only covers --label-pattern
(glob/LIKE) in the first place. Per CONTRIBUTING.md's "one issue per PR"
rule, splitting it out rather than doing the redesign inline here.

Rebuilt and re-verified end to end with the project's own canonical build
(`go build -tags gms_pure_go`, confirmed ICU-free via `otool -L`): both
`bd list --label-pattern` and `bd list --ready --label-pattern` now filter
correctly, matching-pattern issues included, non-matching issues excluded.

* fix(ready): wire --label-pattern and --label-regex into BuildReadyWorkWhere and bd ready

Upstream #3971 landed LabelPattern/LabelRegex for the list path; the
ready path still silently ignored both. Add the LIKE ESCAPE and REGEXP
subquery clauses to BuildReadyWorkWhere mirroring filter.go, and register
both flags on the standalone bd ready subcommand.

The bd ready flag registration absorbs the surviving scope of #4884 —
credit to athosmartins for both the pattern (this PR) and regex (#4884)
ready-path intent.

Addresses the 2026-07-23 review on gastownhall/beads#4882.

Agent-Signature: claude-sonnet-5-medium on behalf of maphew

* fix(ready): honor --label-pattern/--label-regex under the proxied path

Reviewer finding: flag reads happened after the proxied early-return and
gatherReadyInput never copied them into WorkFilter, so both filters were
silently ignored under --proxied-server. Hoist the reads, wire them
through readyInput, and pin with a gatherReadyInput unit test plus a
proxied integration subtest covering both filters.

Agent-Signature: codex-gpt-5.6-terra-medium on behalf of maphew

---------

Co-authored-by: matt wilkie <maphew@gmail.com>
2026-07-25 19:05:27 -07:00
matt wilkie
fbdf7badbc Merge remote-tracking branch 'upstream/main' into pr-4095-fix 2026-07-13 20:39:27 -07:00
Steve Yegge
67ccdbfd44 fix(ready): honor --label-any on the ready/claim path (wy-jpd3.2)
BuildReadyWorkWhere emitted clauses for --label (AND-set) and
--exclude-label but none at all for --label-any, so the OR-set filter was
silently dropped by `bd ready` and `bd ready --claim` on every backend
(with or without --parent), while `bd list`/`bd search` honored it.

On an atomic claim the silent drop is dangerous rather than merely wrong:
a worker fencing itself to its own lane
(`bd ready --claim --label-any lane-a --parent epic-1`) would claim
another lane's issue and believe it was fenced. Reproduced on current
main: the new conformance case claims a lane-b issue when lane-a was
requested.

Emit an OR-set membership clause (same shape as the list path in
sqlbuild/filter.go) that AND-combines with Labels, ExcludeLabels and the
parent filter. All claim paths (Dolt, embedded-Dolt, sqlkit SQL family)
funnel through ClaimReadyIssueInTx -> GetReadyWorkInTx ->
BuildReadyWorkWhere, so the single seam fixes every backend.

Tests: a SQL-shape unit test in sqlbuild, plus a behavioral conformance
case (ClaimReadyIssueLabelFilters) asserting the claim is fenced by the
OR-set alone, AND-combined with --label and --parent, and — the safety
property — that an exhausted lane claims NOTHING rather than falling back
to unfenced ready work.
2026-07-12 07:58:22 -07:00
Julian Knutsen
fb527d034a
perf(ready): page-push ready-work counts and size the total with cheap COUNT(*)s (#4679)
`bd ready --json` had two O(N) hotspots on main. The counts mega-query's
reverse-blocker `rc` subquery keys on COALESCE(depends_on_issue_id, …), an
expression the pure-Go GMS analyzer cannot auto-index, so it re-scanned its
whole materialization once per candidate — O(candidates × blockers). And the
"Showing X of N" total re-ran that entire mega-query UNBOUNDED just to size N.

Changes:
- Generalize one counts builder: sqlbuild.SearchCountsSQL now takes an optional
  `ids []string`. Empty ids = the predicate form (unchanged callers); non-empty
  ids = the page-pushed form that constrains the driver and every count
  subquery to the page, bounding rc by the page instead of the full graph.
  Removes the drifted SearchCountsSQL/SearchCountsByIDsSQL duplication; the
  projection order is defined once and DepTargetExpr is used throughout.
- issueops.runReadyCountsInTx resolves the ready-ID page with the cheap indexed
  ID query, then hydrates counts constrained to those IDs, CHUNKED by
  sqlbuild.QueryBatchSize so large pages stay within backend placeholder limits
  (no quadratic fallback, no wasted ID pass). The page order is restored so the
  result is byte-identical to the unbounded ORDER BY … LIMIT.
- issueops.CountReadyWorkInTx sizes N with indexed COUNT(*)s over the ready
  predicate for the issues AND wisps families, minus the wisp-wins overlap —
  identical to len(GetReadyWorkWithCounts(Limit=0)) even when wisps exist, with
  no mega-query fallback.
- Add ReadyWorkCounter to internal/storage beside the other optional
  capabilities; DoltStore and EmbeddedDoltStore implement CountReadyWork.
  cmd/bd/ready.go zeroes the limit at the call site and unwraps to the
  capability via storage.UnwrapStore, falling back to the unbounded query only
  when a store predates it.
- Extract the shared scan/dedupe loop (scanCountsRowsInTx).
- Conformance tests: page counts byte-identical to unbounded, and
  CountReadyWork == len(unbounded), with fixtures for wisps and a page that
  spans the chunk boundary.

Co-authored-by: Test User <test@test.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 18:07:41 -07:00
matt wilkie
8ad064a961 Merge upstream/main into PR 4095
Resolve ready-work ordering against current main after upstream moved the shared ordering into sqlbuild. Priority ready work now breaks same-priority ties with created_at ASC, and regression coverage checks both the shared helper and ClaimReadyIssue.

Agent-Signature: codex-gpt-5.5-high on behalf of matt wilkie
2026-07-08 12:59:23 -07:00
Steve Yegge
1f8f1fff31 refactor(storage): extract shared SQL builders into internal/storage/sqlbuild (bd-6dnrw.46)
Move all pure SQL-text builders shared by issueops (classic) and
domain/db (proxied) into one package; both stacks now delegate, so
ready/filter/counts query semantics can no longer drift (Seam A).
Execution orchestration stays per-stack per the extract-don't-unify
decision. Canonical SQL text is classic's; the only proxied-side text
deltas are IN-list whitespace.

Adversarial re-review (bd-6dnrw.47) verified all 10 attack items with
no production divergence, and adds an import-boundary test pinning the
layering (domain must never import issueops/sqlbuild) plus a single-use
constraint note on LabelSearchPlan.MergeInto. Follow-ups filed:
bd-emrs2 (+1-overfetch convention), bd-kzeme (IN-helper consolidation).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-12 15:56:23 -04:00