`bd query "type=bug OR label=urgent"` has been answering with a prefix of its
answer and calling it complete. The expression language has OR, NOT and
parentheses, which no storage filter expresses, so both routes fetched a WINDOW
of candidate rows — max(3*limit, 100) — and applied the expression to those in
memory. A match past row 100 was absent from the page and unreported by
has-more. The same six lines of arithmetic were written twice, once per route,
and both copies were wrong in the same way.
THE ROLE. issueops.Querier, reached through store.Querier() and the provider's
own accessor. One method over a request whose predicate is a SENTENCE rather
than a set of fields, which is what makes it a role instead of a mode of
Reader: a ListRequest is a conjunction — every field narrows — and no
arrangement of its twelve callers' fields expresses a disjunction. It is
equally not a raw query passthrough; the language names no table, no join and
no column outside a closed vocabulary, which is why a contract can promise
anything about it at all.
THE FIX, which Q3 approved as user-visible. A predicate query's filter now
carries NO row limit: the predicate sees every candidate row, the page is the
first Limit MATCHES of the complete matching set, and has-more is exact. The
decision has one home, internal/workapi.BuildQueryPlan, which both bodies
execute. The cost is stated rather than hidden — a broad OR over a large
workspace now reads every row its base filters admit — and CHANGELOG.md says so
in the same words.
WHAT PINS THE FIX, at three levels, because the contract alone cannot see it.
TestBuildQueryPlanLeavesAPredicateQueryUNBOUNDED asserts the arithmetic in
microseconds. TestAPredicateQueryFindsAMatchBeyondTheOldWindow seeds 101 rows at
ONE backend and reads back the match at row 101 — verified to discriminate:
with the old window temporarily reinstated it returns zero rows. The portable
contract asserts the PROMISE the window broke (a page is a prefix, has-more is
exact) and, confirmed by the same experiment, does NOT discriminate the old
window at its small scale; the contract's header says so rather than implying
coverage it does not have.
THE CONTRACT. backend/conformance/querier_contract.go holds nine cases, each
citing the leaf doc by line, run at all three backends with ZERO parks. TWO
independent bodies plus an engine check: dolt and embeddeddolt share
internal/workapi/storequerier, and the unit-of-work provider is genuinely
separate — it is the only leg on which the Offset case takes its HONORED branch,
since that seam renders OFFSET where the store body refuses uniformly.
TWO SMALLER USER-VISIBLE CHANGES ride along, both filed in AMBIGUITIES.md
(Q22, Q23) for the owner's batch. `--offset` with an OR/NOT expression is no
longer refused: the offset skips MATCHES, which is only meaningful now that
every match is seen, so an error became an answer. `--limit -1` is now
ErrValidation instead of an undocumented second spelling of unlimited. The
`--offset` + `--sort` refusal survives and moved INWARD — the role makes it, for
every caller, because the order is applied to the rows the query bounded and
each page of an offset walk would be sorted for itself — so its wording changed
with its address, as did the wording of a malformed expression on both routes.
A DEFECT FOUND AND NOT FIXED, recorded as A-qry-1 rather than papered over: an
UNLIMITED request carrying an Offset renders SQL the Dolt engine answers with a
recovered panic (makeslice: cap out of range, out of topRowsIter). It predates
this role — `bd query --offset N --limit 0 --proxied-server` built the same
filter — and it is a storage-seam bug, so the contract case asks its question at
a bounded limit and says why at the case site.
THE HTTP OPERATION, spec first (per bd-jtp1h's A2 scope decision):
GET /v0/beads/issues:query, capability `issues.query`, answering a new QueryPage
{items, has_more}. It publishes no cursor and no `offset`: a predicate query's
matching set is assembled outside the database, so there is no keyset position
to encode, and an offset is honored by one database source and refused by the
other, which is not a difference to put on a wire. QueryPage carries no allOf
and its Items are pinned in pinning.go beside ReadyPage's and IssuesPage's — the
edit wave 2 missed once (bd-8ri3m). Adding a second sort enum made oapi-codegen
disambiguate the FIRST one, renaming the generated ListReadyWorkParamsSort
constants (Hybrid -> ListReadyWorkParamsSortHybrid); nothing outside the
generated file named them. httpapi.Config's roles source grows a required eighth
member, and a Config complete under the previous release is now refused at
Listen rather than nil-dereferencing on the first issues:query request.
WHAT ELSE CONVERGED. Both output modes read ONE query — the counted one --json
always used — so a predicate can no longer be applied to two different row sets
depending on the flag (A-qry-1's neighbour, Q24). The direct route prints the
truncation hint for the first time, not by new choice but because it finally has
an honest has-more to print.
Incidental: internal/storage/role_accessor_decorator_test.go's four
near-identical "the four read roles" paragraphs — one appended by each wave-2
slice — are collapsed into the one accurate paragraph the table needs. Same
defect as bd-8ri3m, comments only.
Verified: build, vet, gofmt clean; golangci-lint 0 issues; make api-check green;
TestQuerierContract 9/9 at dolt, embeddeddolt and uow;
TestAPredicateQueryFindsAMatchBeyondTheOldWindow; the internal/httpapi package
suite; internal/workapi, internal/storage, internal/telemetry, internal/query
and the root package; TestEmbeddedQuery under BEADS_TEST_EMBEDDED_DOLT=1 and
TestProxiedServerQuery under BEADS_TEST_PROXIED_SERVER=1; TestProxiedLookup*.
The protocol corpus drives no `bd query` invocation and is untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
181 lines
7.1 KiB
Go
181 lines
7.1 KiB
Go
package workapi
|
|
|
|
import (
|
|
"fmt"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/steveyegge/beads/internal/query"
|
|
"github.com/steveyegge/beads/internal/types"
|
|
"github.com/steveyegge/beads/issueops"
|
|
)
|
|
|
|
// DefaultQueryLimit is the default number of rows `bd query` returns when the
|
|
// caller does not ask for a specific limit. Both surfaces register their limit
|
|
// knob from this constant so they cannot drift apart; 0 still means unlimited.
|
|
const DefaultQueryLimit = 50
|
|
|
|
// QueryPlan is an evaluated query request: what to ask the database, what to
|
|
// decide in Go afterwards, and how to cut the page.
|
|
//
|
|
// It exists because the two executions of one query are NOT the same shape.
|
|
// A filter-expressible expression is a filter the database answers whole; an
|
|
// expression with an OR or a general NOT is a base filter plus a predicate,
|
|
// and the predicate is a Go function that no seam can push down. Both
|
|
// implementations of issueops.Querier build one of these, so the decision
|
|
// about which shape an expression takes — and, crucially, what row limit each
|
|
// shape may carry — is made once here rather than once per backend.
|
|
type QueryPlan struct {
|
|
// Filter is what the database is asked. For a predicate query it carries
|
|
// the expression's base filters and NO row limit; see BuildQueryPlan.
|
|
Filter types.IssueFilter
|
|
// Predicate is the in-memory half of the expression, or nil when Filter
|
|
// alone answers it exactly.
|
|
Predicate func(*types.Issue) bool
|
|
// Limit is the number of rows the CALLER receives, which is not always
|
|
// what Filter carries. 0 means unlimited.
|
|
Limit int
|
|
// Offset is the number of matching rows to skip. It is reported here
|
|
// rather than written onto Filter because only one of the two seams can
|
|
// render OFFSET, and only the filter-expressible shape may push it down at
|
|
// all — skipping candidate rows before a predicate runs would drop rows
|
|
// the predicate never accepted (issueops/querier.go:70-86).
|
|
Offset int
|
|
// SortBy and Reverse are the display order, applied by the page epilogue.
|
|
SortBy string
|
|
Reverse bool
|
|
}
|
|
|
|
// RequiresPredicate reports whether this plan is answered in two steps.
|
|
func (p QueryPlan) RequiresPredicate() bool { return p.Predicate != nil }
|
|
|
|
// BuildQueryPlan turns a query request into its executable plan. It is the
|
|
// single definition of what `bd query` MEANS: the parse, the evaluation, the
|
|
// default closed exclusion, the limit defaulting, and the row bound each shape
|
|
// of query may carry.
|
|
//
|
|
// THE ROW BOUND IS THE WHOLE REASON THIS IS ONE FUNCTION. A predicate query's
|
|
// Filter gets NO limit — the predicate must see every candidate row, or the
|
|
// page it produces is an arbitrary prefix of the answer reported as the whole
|
|
// of it (issueops/querier.go:118-133). Both front doors used to bound that
|
|
// query at max(3*Limit, 100) and filter what came back, in two places, with
|
|
// the arithmetic written out twice. The bound is gone and the decision has one
|
|
// home.
|
|
//
|
|
// It reads no configuration and touches no store: the query vocabulary is
|
|
// closed and the evaluator resolves it entirely, which is why both
|
|
// implementations can share this without either supplying a config source.
|
|
func BuildQueryPlan(in issueops.QueryRequest) (QueryPlan, error) {
|
|
expression := strings.TrimSpace(in.Expression)
|
|
if expression == "" {
|
|
return QueryPlan{}, invalidQueryExpression("an expression is required")
|
|
}
|
|
limit := LimitOr(in.Limit, DefaultQueryLimit)
|
|
if limit < 0 {
|
|
return QueryPlan{}, fmt.Errorf("invalid limit %d: a query limit must be zero or greater; 0 means unlimited%.0w",
|
|
limit, issueops.ErrValidation)
|
|
}
|
|
if in.Offset < 0 {
|
|
return QueryPlan{}, fmt.Errorf("invalid offset %d: a query offset must be zero or greater%.0w",
|
|
in.Offset, issueops.ErrValidation)
|
|
}
|
|
if in.Offset > 0 && in.SortBy != "" {
|
|
return QueryPlan{}, fmt.Errorf(
|
|
"invalid offset: an offset cannot be combined with a display order, because the order is applied to the rows the query bounded and each page would be sorted for itself%.0w",
|
|
issueops.ErrValidation)
|
|
}
|
|
|
|
node, err := query.Parse(expression)
|
|
if err != nil {
|
|
return QueryPlan{}, invalidQueryExpression(err.Error())
|
|
}
|
|
result, err := query.NewEvaluator(time.Now()).Evaluate(node)
|
|
if err != nil {
|
|
return QueryPlan{}, invalidQueryExpression(err.Error())
|
|
}
|
|
|
|
plan := QueryPlan{
|
|
Filter: result.Filter,
|
|
Limit: limit,
|
|
Offset: in.Offset,
|
|
SortBy: in.SortBy,
|
|
Reverse: in.Reverse,
|
|
}
|
|
if result.RequiresPredicate {
|
|
plan.Predicate = result.Predicate
|
|
} else {
|
|
// The database answers this one exactly, so it may carry the page.
|
|
plan.Filter.Limit = limit
|
|
}
|
|
|
|
// The default closed exclusion, applied only to an expression that has no
|
|
// opinion of its own about status (issueops/querier.go:29-41).
|
|
if !in.IncludeClosed && plan.Filter.Status == nil && !mentionsStatus(node) {
|
|
plan.Filter.ExcludeStatus = append(plan.Filter.ExcludeStatus, types.StatusClosed)
|
|
}
|
|
return plan, nil
|
|
}
|
|
|
|
// invalidQueryExpression is the one refusal shape a bad expression takes. The
|
|
// prefix is load-bearing: internal/httpapi maps it back to the `q` parameter
|
|
// so an unparseable expression is the documented 400 rather than a 500.
|
|
func invalidQueryExpression(detail string) error {
|
|
return fmt.Errorf("invalid query expression: %s%.0w", detail, issueops.ErrValidation)
|
|
}
|
|
|
|
// mentionsStatus reports whether the expression compares `status` anywhere.
|
|
// An expression that does keeps its own answer about closed rows; one that
|
|
// does not gets the default exclusion.
|
|
func mentionsStatus(node query.Node) bool {
|
|
switch n := node.(type) {
|
|
case *query.ComparisonNode:
|
|
return n.Field == "status"
|
|
case *query.AndNode:
|
|
return mentionsStatus(n.Left) || mentionsStatus(n.Right)
|
|
case *query.OrNode:
|
|
return mentionsStatus(n.Left) || mentionsStatus(n.Right)
|
|
case *query.NotNode:
|
|
return mentionsStatus(n.Operand)
|
|
default:
|
|
return false
|
|
}
|
|
}
|
|
|
|
// ApplyQueryPredicate keeps the rows the predicate accepts, in the order they
|
|
// arrived. Both implementations of issueops.Querier call it, so "what the
|
|
// predicate is applied to" cannot come to mean two things: it is applied to
|
|
// every row the database returned, and the database was not asked for a
|
|
// bounded set.
|
|
//
|
|
// A nil row, or a row carrying no issue, is dropped rather than offered to the
|
|
// predicate — a predicate reads issue fields and would be handed nothing to
|
|
// read.
|
|
func ApplyQueryPredicate(rows []*types.IssueWithCounts, predicate func(*types.Issue) bool) []*types.IssueWithCounts {
|
|
if predicate == nil {
|
|
return rows
|
|
}
|
|
out := make([]*types.IssueWithCounts, 0, len(rows))
|
|
for _, row := range rows {
|
|
if row == nil || row.Issue == nil {
|
|
continue
|
|
}
|
|
if predicate(row.Issue) {
|
|
out = append(out, row)
|
|
}
|
|
}
|
|
return out
|
|
}
|
|
|
|
// SkipRows drops the first n rows, and answers with an empty slice rather than
|
|
// nil when n consumes them all. It is how an Offset is honored for a predicate
|
|
// query, where there is no SQL OFFSET to push it into: the skip runs AFTER the
|
|
// predicate, so it skips matches rather than candidates.
|
|
func SkipRows(rows []*types.IssueWithCounts, n int) []*types.IssueWithCounts {
|
|
if n <= 0 {
|
|
return rows
|
|
}
|
|
if n >= len(rows) {
|
|
return []*types.IssueWithCounts{}
|
|
}
|
|
return rows[n:]
|
|
}
|