* fix(httpclient): unlimited ready/query/list reads over http, client-capped
An unlimited read never puts limit=0 on the wire, which a non-loopback
bd serve refuses ("unlimited reads are loopback-only"):
- bd ready --limit 0 (the WorkFilter bridge) sent no limit at all, so the
server's 100-row ready default silently capped every routed-pool queue.
- Reader.Ready and Querier.Query with Limit 0 sent limit=0 and got a 400
from a non-loopback server.
listReadyWork and queryIssues publish no cursor, so an unlimited read is
one request bounded at the client cap plus one (one server read, so one
snapshot). listIssues keeps its cursor walk, now bounded by the same cap
(per-page reads; the snapshot semantics are documented in
unlimited_read.go). Past BEADS_HTTP_UNLIMITED_READ_CAP (default 10000) the
read is refused with *UnlimitedReadCapError, never truncated; a server that
reports has_more below the requested bound is refused the same way. A
dispatch tripwire refuses limit=0 on the three listing operations, so a new
path that forgets fails loudly instead of only off loopback.
Served tests run against a server in --allow-non-loopback mode with 150
rows: ready (role, bridge, bridge+total), query and list at limit 0 return
every row with a bounded limit on every request, and the cap refuses with
the typed error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(backend/http/corpus): drop the three marks S15 retires
With unlimited reads served (one request bounded at the cap plus one for
ready and query), gc-ready-unlimited, gc-ready-pool-demand-unlimited and
gc-query-wisps-unlimited return every row on the reference profile instead
of the server's default page or the loopback-only refusal. mc-list-id stays
pending on S15b (--id).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(changelog): unlimited reads over http and BEADS_HTTP_UNLIMITED_READ_CAP
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(httpclient): uncap the list walk and bound the ready-count fallback
Review fixes for #7497 (review loop iterations 1 and 2).
F1 (major): the client cap no longer applies to an unlimited list.
listIssues publishes a keyset cursor, so no response is larger than a
page; the walk is back to base, bounded only by --max-rows /
BEADS_MAX_ROWS as on every local backend. The cap stays on the
cursorless reads (ready, query), whose whole answer is one response the
server buffers. The CHANGELOG, unlimited_read.go and the env doc say so.
F2 (major): externaldeps.CountReadyWork's no-counter fallback, which an
http store reaches whenever its server does not apply the
external-dependency policy (every released server), asked for the whole
ready set to report one number: over http, up to the cap's full rows,
and a refusal past it that text `bd ready` swallows into a missing
footer. It now asks for a one-row Lite page; the bridge pairs it with
the server's countReadyWork, and refetchedReadyTotal still sizes the
externally blocked case.
F3: TestCheckUnlimitedPage pins a full cap+1 page that also reports
has_more as the cap's refusal, not a truncation, and the env test
accepts the inclusive 1000000 bound. Served: the cap test adds the
unlimitedRows-2 case (the server's 149 rows with has_more), the list is
pinned as uncapped with MaxRows its only bound, and the bridge's
MaxRows-vs-cap precedence is pinned with the limit on the wire.
F5: readyBridgePage sends MaxRows+1 rather than cap+1 when MaxRows is
the tighter bound (tightenUnlimitedPage). The MaxRows check still runs
first, so an answer past it keeps its *ErrTooManyRows attribution.
F6: the decorator and test comments describe the cap+1 bound; the
refusal reads "narrow the query or raise BEADS_HTTP_UNLIMITED_READ_CAP";
the Ready and Query Limit contracts name the http exception; mc-list-id's
pending reason drops the S15 --limit 0 clause and keeps the --id refusal.
F7: unlimited_read.go states that the tripwire guards Store.dispatch
only, so a direct wire.Client caller (readyClaimPage) must send a
positive limit itself.
Not taken: F4 (an unlimited walk that meets has_more with no cursor ends
as a success rather than a ServerTruncated refusal). After F1 the walk
is base code again, the shape needs a skewed or older peer, and the
change would land in fetchIssuePages (cyclomatic 21, cognitive 34),
turning an untouched base function into a changed one in the major
complexity band. Left as a follow-up.
Iteration 2, N1: F5 reached only the raw bridge. `bd ready` reads
through the external-dependency decorator, and against a server that
leaves that policy to the client, one externally blocked issue made
readyExclusion take MaxRows off the unlimited inner query. The bridge
then asked for cap+1 rows: past the cap it refused with the read cap's
error (exit 1, "raise BEADS_HTTP_UNLIMITED_READ_CAP") rather than
--max-rows' (exit 2), and under it fetched the whole ready set to refuse
on the rows kept. An unlimited inner query now keeps MaxRows plus one
row per blocked issue: past that many rows more than MaxRows are kept
whatever the drop takes, so the bridge sizes the request by it under the
cap, and innerCapError hands the refusal back as the caller's own cap.
The suggested inner Limit of MaxRows+blocked+1 was not taken: a bounded
request skips the cap and its server-truncation check, so with any
external blocker a --max-rows above the cap would fetch past it.
TestServedReadyBridgeMaxRowsSizesTheRequest pins the request and the
refusal through externaldeps.New with one blocker, the cap still in
force past MaxRows plus the blocked rows;
TestServedReadyMaxRowsCountsTheRowsKept adds an unlimited case whose
refusal names the caller's cap on all three decorator reads.
N4: QueryRequest.Limit's "NOTHING BACKSTOPS IT" names the http
exception in its lead, as ReadyRequest.Limit does.
Not taken in iteration 2, as non-gating follow-ups: N2
(refetchedReadyTotal's Limit = total+blocked read sits outside the cap;
pre-existing), N3 (the list walk's has_more-without-cursor end, the
deferred F4), N5 (prefixing the malformed-cap error with its operation;
it already names the variable and its value).
Validation: the served tier (BEADS_TEST_EMBEDDED_DOLT=1
BEADS_HTTP_TEST_REQUIRED=1) for internal/httpclient and the command
corpus; go test for externaldeps, issueops and workapi. Each new test
was mutation-checked red: the zero-limit count, the tightening removed,
the MaxRows check removed, the PR's list cap restored, and a cap+1
has_more page read as a truncation. Iteration 2: the served
internal/httpclient ready, unlimited-read and external-policy tests (34
PASS, 0 SKIP) and go test for externaldeps, issueops and workapi; the
new pins go red with MaxRows taken off again (the pre-fix code), with no
re-attribution, with no growth by the blocked rows, with the suggested
inner Limit, and with Found left unadjusted.
Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
164 lines
8.8 KiB
Go
164 lines
8.8 KiB
Go
package issueops
|
|
|
|
import (
|
|
"context"
|
|
)
|
|
|
|
// QueryRequest describes one boolean-expression query: the question `bd query`
|
|
// asks, plus the page it wants back.
|
|
//
|
|
// THE EXPRESSION IS THE PREDICATE, and that is what makes this request unlike
|
|
// every other one here. CountRequest, ListRequest and ReadyRequest spell their
|
|
// predicate as FIELDS — one per filter, ANDed together — and a caller composes
|
|
// them by setting more of them. This request carries a SENTENCE, in the query
|
|
// mini-language `bd query` publishes, and that language has OR, NOT and
|
|
// parentheses, which no conjunction of fields can express. Parsing, evaluation
|
|
// and the choice of how to execute the result all happen inside; a caller says
|
|
// what it wants matched and never how the match is shaped.
|
|
type QueryRequest struct {
|
|
// Expression is the query, in the language `bd query --help` documents:
|
|
// field comparisons (`status=open`, `priority>1`, `created>7d`) combined
|
|
// with AND, OR, NOT and parentheses.
|
|
//
|
|
// It is parsed INSIDE. An expression that is blank, unparseable, or that
|
|
// names a field or operator the language does not have is ErrValidation
|
|
// naming the fault, never an empty page: a caller that misspelled a field
|
|
// and got zero rows back has no way to tell that from a store with nothing
|
|
// in it.
|
|
Expression string
|
|
|
|
// IncludeClosed admits closed issues, which the query otherwise hides.
|
|
//
|
|
// THE HIDING IS CONDITIONAL, and the condition is part of this contract
|
|
// because both front doors have always applied it: closed rows are
|
|
// excluded UNLESS this is set, or unless the expression itself compares
|
|
// `status`. So `status=closed` answers with closed rows on its own, and
|
|
// `NOT status=open` does too — an expression that has an opinion about
|
|
// status keeps it, and only an expression with none gets the default. The
|
|
// flag `bd query --all` sets is this one.
|
|
IncludeClosed bool
|
|
|
|
// SortBy names the display order and Reverse inverts it, over the same
|
|
// vocabulary ListRequest.SortBy takes. Empty leaves the rows in the order
|
|
// the query returned them.
|
|
//
|
|
// WHAT THE ORDER IS APPLIED TO depends on what bounded the query, and the
|
|
// difference is worth stating because it is observable. The order is
|
|
// applied to the rows the QUERY bounded, then the page is cut from them —
|
|
// the epilogue `bd list` runs (workapi.FinishPage). For a
|
|
// filter-expressible query the database applied Limit, so the order sorts
|
|
// the page. For a PREDICATE query nothing bounded the query — see
|
|
// Querier.Query — so the order sorts the whole matching set and the page
|
|
// is its head. `bd list` has the same split for the same reason: a sort
|
|
// SQL cannot express fetches everything and trims afterwards.
|
|
SortBy string
|
|
Reverse bool
|
|
|
|
// Limit bounds the page the caller RECEIVES. Nil means the shared query
|
|
// default; 0 means unlimited. It is a pointer so that "unset" and
|
|
// "explicitly unlimited" stay distinguishable, which is what lets one
|
|
// constant serve both surfaces. A negative Limit is ErrValidation rather
|
|
// than a synonym for unlimited: two spellings of "no bound" is one more
|
|
// than a caller can check for.
|
|
//
|
|
// NOTHING BACKSTOPS IT on any backend but http (below). ListRequest pairs
|
|
// its Limit with MaxRows, a circuit breaker that refuses an oversized
|
|
// answer outright; this request has no twin to it by design, and neither
|
|
// knob that sets that one (`--max-rows`, BEADS_MAX_ROWS) is read anywhere
|
|
// on the query path. So an unlimited query is unbounded in the plain sense:
|
|
// it reads every row the expression matches, however many that is, and a
|
|
// caller who asked for one gets it rather than an ErrTooManyRows. The
|
|
// ceiling here is Limit itself. This is reachable without asking: a piped
|
|
// `bd query` resolves its unflagged limit to 0 before the request, the same
|
|
// policy `bd list` has applied to piped output since GH#4094.
|
|
//
|
|
// The http backend is the one exception, because its whole answer is one
|
|
// response: it bounds an unlimited query at BEADS_HTTP_UNLIMITED_READ_CAP
|
|
// rows and refuses a larger answer with httpclient.ErrUnlimitedReadCap
|
|
// rather than truncating it.
|
|
Limit *int
|
|
|
|
// Offset skips the first N MATCHING rows, on EVERY implementation and for
|
|
// EVERY shape of expression, exactly as ListRequest.Offset does. A
|
|
// negative Offset is ErrValidation everywhere.
|
|
//
|
|
// IT IS UNIFORM, not per-expression, and that is a decision rather than an
|
|
// accident of the seams. One body renders SQL OFFSET for a
|
|
// filter-expressible query and the other reaches past the skipped rows and
|
|
// drops them in Go; both skip in Go for a predicate query, which is
|
|
// unbounded and has nothing to push down. Which shape an expression takes
|
|
// is decided by the evaluator, not by the caller, so an Offset that
|
|
// behaved differently for `type=bug OR type=task` than for `type=bug`
|
|
// would be unpredictable from the outside. One answer everywhere is the
|
|
// only checkable promise.
|
|
//
|
|
// IT SKIPS MATCHES — never candidate rows. A skip
|
|
// applied before the predicate would discard rows the predicate would have
|
|
// rejected anyway and hand back a short page, which is the failure that
|
|
// made `bd query --offset` refuse OR queries outright.
|
|
//
|
|
// AN OFFSET WITH A SortBy IS ErrValidation, at every backend. The order is
|
|
// applied to the rows the query BOUNDED (see SortBy), so under a page
|
|
// bound each page is sorted for itself: a walk with an offset would neither
|
|
// visit every row nor visit any row once. Refusing the combination is the
|
|
// only answer that is true on both shapes of query; accepting it would mean
|
|
// publishing a walk that silently is not one.
|
|
Offset int
|
|
}
|
|
|
|
// Querier answers boolean-expression queries: the operation `bd query`
|
|
// performs, and — like Counter, Reader and ReadyCounter — a role with its own
|
|
// accessor. A new capability gets a new role interface and its own accessor;
|
|
// never append a method here.
|
|
//
|
|
// IT IS ITS OWN ROLE RATHER THAN A SHAPE OF Reader.List because the two take
|
|
// predicates that are not the same KIND of thing. A ListRequest is a
|
|
// conjunction: every field it carries narrows the answer, and there is no
|
|
// arrangement of them that expresses "type=bug OR label=urgent", let alone
|
|
// "NOT (priority<2 AND assignee=none)". Handing this expression to List would
|
|
// mean either adding a free-text member to ListRequest — a field the other
|
|
// eleven callers must ignore, on the request type this library's largest
|
|
// surface already shares — or teaching List a second execution mode. The
|
|
// question is different, so the role is.
|
|
//
|
|
// EQUALLY, IT IS NOT A RAW QUERY PASSTHROUGH. The language has no SQL in it:
|
|
// no table names, no joins, no ORDER BY, no way to name a column the
|
|
// vocabulary does not publish. It is a closed set of fields over one row
|
|
// shape, which is why a role can promise anything about it at all.
|
|
type Querier interface {
|
|
// Query returns the page of issues the expression matches.
|
|
//
|
|
// THE PREDICATE IS EVALUATED AGAINST THE WHOLE CANDIDATE SET. That
|
|
// sentence is the point of this method and the fix it shipped with.
|
|
//
|
|
// An expression the storage filter vocabulary can express — a comparison,
|
|
// a chain of ANDs, a NOT over status or type, an OR over labels — is
|
|
// executed by the database, which applies Limit itself. Every other
|
|
// expression is a PREDICATE query: the database is asked for the base
|
|
// filters the expression implies and the predicate is applied to the rows
|
|
// in Go. For those, the query is issued UNBOUNDED and the predicate sees
|
|
// every candidate row, so the page is the first Limit MATCHES of the
|
|
// complete matching set.
|
|
//
|
|
// It was not always. Both front doors used to bound that query at
|
|
// max(3*Limit, 100) rows and filter what came back, so a match beyond that
|
|
// window was absent from the page AND unreported by has-more: an OR query
|
|
// over a workspace of any size returned an arbitrary prefix of its answer
|
|
// and called it complete. The window is gone. What it cost is stated
|
|
// rather than hidden: a predicate query reads every row its base filters
|
|
// admit, so a broad expression over a large workspace is a large read.
|
|
//
|
|
// HasMore is true exactly when the page is shorter than the matching set,
|
|
// which for a predicate query it can now compute rather than guess.
|
|
//
|
|
// An expression that matches nothing is an empty page and a nil error —
|
|
// Items is never nil on a successful call. There is no ErrNotFound here:
|
|
// a question about a set has an answer even when the set is empty.
|
|
//
|
|
// Querying is a READ. Nothing here records a history entry, fires a
|
|
// completion hook or changes a row, and a refusal changes nothing either.
|
|
// Deterministic request-validation failures match ErrValidation, and
|
|
// result values are unspecified when error is non-nil. Implementations
|
|
// never mutate caller-owned request values.
|
|
Query(ctx context.Context, req QueryRequest) (IssuePage, error)
|
|
}
|