beads/internal/httpclient/bridge.go
Julian Knutsen b630f96f5b
fix(httpclient): unlimited ready/query/list reads over http, client-capped (S15) (#7497)
* fix(httpclient): unlimited ready/query/list reads over http, client-capped

An unlimited read never puts limit=0 on the wire, which a non-loopback
bd serve refuses ("unlimited reads are loopback-only"):

- bd ready --limit 0 (the WorkFilter bridge) sent no limit at all, so the
  server's 100-row ready default silently capped every routed-pool queue.
- Reader.Ready and Querier.Query with Limit 0 sent limit=0 and got a 400
  from a non-loopback server.

listReadyWork and queryIssues publish no cursor, so an unlimited read is
one request bounded at the client cap plus one (one server read, so one
snapshot). listIssues keeps its cursor walk, now bounded by the same cap
(per-page reads; the snapshot semantics are documented in
unlimited_read.go). Past BEADS_HTTP_UNLIMITED_READ_CAP (default 10000) the
read is refused with *UnlimitedReadCapError, never truncated; a server that
reports has_more below the requested bound is refused the same way. A
dispatch tripwire refuses limit=0 on the three listing operations, so a new
path that forgets fails loudly instead of only off loopback.

Served tests run against a server in --allow-non-loopback mode with 150
rows: ready (role, bridge, bridge+total), query and list at limit 0 return
every row with a bounded limit on every request, and the cap refuses with
the typed error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(backend/http/corpus): drop the three marks S15 retires

With unlimited reads served (one request bounded at the cap plus one for
ready and query), gc-ready-unlimited, gc-ready-pool-demand-unlimited and
gc-query-wisps-unlimited return every row on the reference profile instead
of the server's default page or the loopback-only refusal. mc-list-id stays
pending on S15b (--id).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): unlimited reads over http and BEADS_HTTP_UNLIMITED_READ_CAP

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(httpclient): uncap the list walk and bound the ready-count fallback

Review fixes for #7497 (review loop iterations 1 and 2).

F1 (major): the client cap no longer applies to an unlimited list.
listIssues publishes a keyset cursor, so no response is larger than a
page; the walk is back to base, bounded only by --max-rows /
BEADS_MAX_ROWS as on every local backend. The cap stays on the
cursorless reads (ready, query), whose whole answer is one response the
server buffers. The CHANGELOG, unlimited_read.go and the env doc say so.

F2 (major): externaldeps.CountReadyWork's no-counter fallback, which an
http store reaches whenever its server does not apply the
external-dependency policy (every released server), asked for the whole
ready set to report one number: over http, up to the cap's full rows,
and a refusal past it that text `bd ready` swallows into a missing
footer. It now asks for a one-row Lite page; the bridge pairs it with
the server's countReadyWork, and refetchedReadyTotal still sizes the
externally blocked case.

F3: TestCheckUnlimitedPage pins a full cap+1 page that also reports
has_more as the cap's refusal, not a truncation, and the env test
accepts the inclusive 1000000 bound. Served: the cap test adds the
unlimitedRows-2 case (the server's 149 rows with has_more), the list is
pinned as uncapped with MaxRows its only bound, and the bridge's
MaxRows-vs-cap precedence is pinned with the limit on the wire.

F5: readyBridgePage sends MaxRows+1 rather than cap+1 when MaxRows is
the tighter bound (tightenUnlimitedPage). The MaxRows check still runs
first, so an answer past it keeps its *ErrTooManyRows attribution.

F6: the decorator and test comments describe the cap+1 bound; the
refusal reads "narrow the query or raise BEADS_HTTP_UNLIMITED_READ_CAP";
the Ready and Query Limit contracts name the http exception; mc-list-id's
pending reason drops the S15 --limit 0 clause and keeps the --id refusal.

F7: unlimited_read.go states that the tripwire guards Store.dispatch
only, so a direct wire.Client caller (readyClaimPage) must send a
positive limit itself.

Not taken: F4 (an unlimited walk that meets has_more with no cursor ends
as a success rather than a ServerTruncated refusal). After F1 the walk
is base code again, the shape needs a skewed or older peer, and the
change would land in fetchIssuePages (cyclomatic 21, cognitive 34),
turning an untouched base function into a changed one in the major
complexity band. Left as a follow-up.

Iteration 2, N1: F5 reached only the raw bridge. `bd ready` reads
through the external-dependency decorator, and against a server that
leaves that policy to the client, one externally blocked issue made
readyExclusion take MaxRows off the unlimited inner query. The bridge
then asked for cap+1 rows: past the cap it refused with the read cap's
error (exit 1, "raise BEADS_HTTP_UNLIMITED_READ_CAP") rather than
--max-rows' (exit 2), and under it fetched the whole ready set to refuse
on the rows kept. An unlimited inner query now keeps MaxRows plus one
row per blocked issue: past that many rows more than MaxRows are kept
whatever the drop takes, so the bridge sizes the request by it under the
cap, and innerCapError hands the refusal back as the caller's own cap.
The suggested inner Limit of MaxRows+blocked+1 was not taken: a bounded
request skips the cap and its server-truncation check, so with any
external blocker a --max-rows above the cap would fetch past it.
TestServedReadyBridgeMaxRowsSizesTheRequest pins the request and the
refusal through externaldeps.New with one blocker, the cap still in
force past MaxRows plus the blocked rows;
TestServedReadyMaxRowsCountsTheRowsKept adds an unlimited case whose
refusal names the caller's cap on all three decorator reads.

N4: QueryRequest.Limit's "NOTHING BACKSTOPS IT" names the http
exception in its lead, as ReadyRequest.Limit does.

Not taken in iteration 2, as non-gating follow-ups: N2
(refetchedReadyTotal's Limit = total+blocked read sits outside the cap;
pre-existing), N3 (the list walk's has_more-without-cursor end, the
deferred F4), N5 (prefixing the malformed-cap error with its operation;
it already names the variable and its value).

Validation: the served tier (BEADS_TEST_EMBEDDED_DOLT=1
BEADS_HTTP_TEST_REQUIRED=1) for internal/httpclient and the command
corpus; go test for externaldeps, issueops and workapi. Each new test
was mutation-checked red: the zero-limit count, the tightening removed,
the MaxRows check removed, the PR's list cap restored, and a cap+1
has_more page read as a truncation. Iteration 2: the served
internal/httpclient ready, unlimited-read and external-policy tests (34
PASS, 0 SKIP) and go test for externaldeps, issueops and workapi; the
new pins go red with MaxRows taken off again (the pre-fix code), with no
re-attribution, with no growth by the blocked rows, with the suggested
inner Limit, and with Found left unadjusted.

Agent-Signature: claude-code-claude-opus-5-5-unknown-reasoning on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-12 09:02:22 +00:00

217 lines
8.8 KiB
Go

// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/bridge.go@49d1df2f6)
// to OSS beads under the MIT license.
package httpclient
import (
"context"
"errors"
"fmt"
"net/http"
"net/url"
"github.com/steveyegge/beads/internal/httpapi/apigen"
"github.com/steveyegge/beads/internal/httpclient/encode"
"github.com/steveyegge/beads/internal/httpclient/wire"
storageops "github.com/steveyegge/beads/internal/storage/issueops"
"github.com/steveyegge/beads/internal/types"
"github.com/steveyegge/beads/issueops"
)
// The off-role ready bridge, and the raw issue read the `--parent` walk probes
// with.
//
// The bridge exists because `bd ready`'s listing is still raw at tip
// (cmd/bd/ready.go:172 and :211): the front door has not moved onto
// issueops.Reader, so a reverse types.WorkFilter mapper is required until it
// does. D8 calls it deliberately throwaway and it is — what is not throwaway is
// the rule it holds to: every field it cannot express REFUSES, because a dropped
// filter widens a result set invisibly and the server can only reject the
// parameters it receives (L12).
//
// The SEARCH bridge is not here. Its two shapes are id resolution's (resolve.go)
// and the descendant walk's, and both live beside the surfaces that own them.
// GetReadyWork serves `bd ready`'s text listing.
//
// It is the projection and GetReadyWorkWithCounts is the primitive, not the
// other way round: the wire hydrates the three cardinalities either way, because
// the page element IS types.IssueWithCounts.
func (s *Store) GetReadyWork(ctx context.Context, filter types.WorkFilter) ([]*types.Issue, error) {
rows, err := s.GetReadyWorkWithCounts(ctx, filter)
if err != nil {
return nil, err
}
out := make([]*types.Issue, 0, len(rows))
for _, row := range rows {
out = append(out, row.Issue)
}
return out, nil
}
// GetReadyWorkWithCounts is the same listing with the counts `bd ready --json`
// prints.
//
// ONE REQUEST, NO PAGING. listReadyWork publishes no cursor — its sort policies
// admit no keyset predicate — so `limit` and the server's own has_more ARE the
// page contract, exactly as they are for the Reader.Ready role this bridge dies
// in favor of. The request is built by the same encoder and dialed through the
// same door, so the two spellings of "list ready work" cannot drift apart.
func (s *Store) GetReadyWorkWithCounts(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, error) {
rows, _, err := s.readyBridgePage(ctx, "GetReadyWorkWithCounts", filter)
return rows, err
}
// GetReadyWorkWithCountsAndTotal is GetReadyWorkWithCounts plus the size of the
// whole ready set, which `bd ready --json` prints as its pagination total.
//
// AT MOST TWO REQUESTS. listReadyWork's ReadyPage carries items and has_more
// but no total, so the total cannot ride the page the way it does in a SQL
// backend's single transaction. When the server says the page is the whole set
// (has_more false — always the case for an unlimited listing) the total IS the
// page length and no second request is made. Only a truncated page pays one
// countReadyWork round trip, encoded from the same filter by
// encode.ReadyBridgeCountParams, which is the round trip the text path's
// ReadyCounter role already makes.
//
// The two requests are two server transactions, not one snapshot: a write
// landing between them can make the total disagree with the page by that
// write. The total is clamped to at least the page length so the envelope
// never reports fewer items than it carries.
//
// Every refusal the listing makes is made before either request (the count
// encoder runs the listing's encoder), so a filter the wire cannot express —
// ExcludeIDs included — fails here exactly as it fails GetReadyWorkWithCounts.
func (s *Store) GetReadyWorkWithCountsAndTotal(ctx context.Context, filter types.WorkFilter) ([]*types.IssueWithCounts, int, error) {
const op = "GetReadyWorkWithCountsAndTotal"
countParams, err := encode.ReadyBridgeCountParams(filter)
if err != nil {
return nil, 0, s.inexpressible(op, err)
}
rows, hasMore, err := s.readyBridgePage(ctx, op, filter)
if err != nil {
return nil, 0, err
}
if !hasMore {
return rows, len(rows), nil
}
var body apigen.ReadyCount
if err := s.dispatch(ctx, wire.Request{
Op: wire.OpCountReadyWork,
Method: http.MethodGet,
Path: wire.PathReadyCount,
Query: countParams,
}, &body); err != nil {
return nil, 0, err
}
total := int(body.Total)
if total < len(rows) {
total = len(rows)
}
return rows, total, nil
}
// readyBridgePage is the one listReadyWork request both bridge listings make,
// returning the rows and the server's has_more.
func (s *Store) readyBridgePage(ctx context.Context, op string, filter types.WorkFilter) ([]*types.IssueWithCounts, bool, error) {
params, err := encode.ReadyBridgeParams(filter)
if err != nil {
return nil, false, s.inexpressible(op, err)
}
// A legacy filter spells unlimited as Limit <= 0, and the encoder therefore
// emits no `limit` for it — which the server reads as ITS default of 100,
// not as unlimited. That was a silent cap on every `bd ready --limit 0`
// (and every gc routed-pool queue read through it). The pager decides the
// bound instead: the client cap plus one, refused past the cap rather than
// truncated, and never the wire's own `limit=0` (unlimited_read.go).
readCap, err := boundUnlimitedPage(params, filter.Limit <= 0)
if err != nil {
return nil, false, err
}
readCap = tightenUnlimitedPage(params, readCap, filter.MaxRows)
var body apigen.ReadyPage
if err := s.dispatch(ctx, wire.Request{
Op: wire.OpListReadyWork,
Method: http.MethodGet,
Path: wire.PathReady,
Query: params,
}, &body); err != nil {
return nil, false, err
}
// filter.Lite IS ReadyRequest.Brief — internal/workapi assigns one to the
// other — so the two doors have to stamp the same marker or the same request
// answers differently depending on which one it came in. That agreement is
// the whole of E-WorkFilter.Lite's retirement note.
rows := wireRows(body.Items, filter.Lite)
// The cap bounds the rows this client accepted off the wire, which for a
// cursorless operation is one page. It still fires: a caller whose page is
// wider than their cap gets exit 2 rather than a silently oversized answer.
if filter.MaxRows > 0 && len(rows) > filter.MaxRows {
return nil, false, &storageops.ErrTooManyRows{
Found: len(rows),
Cap: filter.MaxRows,
Source: filter.MaxRowsSource,
}
}
if err := checkUnlimitedPage(wire.OpListReadyWork, readCap, len(rows), body.HasMore); err != nil {
return nil, false, err
}
return rows, body.HasMore, nil
}
// GetIssue serves getIssue. It is the `bd list --parent` walk's
// parent-existence probe and the molecule loader's read half.
//
// A MISS IS storage.ErrNotFound (issueops.ErrNotFound), wrapped exactly as the local stores wrap it
// (issueops.GetIssueInTx: "not found: issue <id>"), so a caller written against
// the backend contract — `errors.Is(err, storage.ErrNotFound)` — reads an http
// miss the same as a local one. The empty id is a miss too, as it is locally.
func (s *Store) GetIssue(ctx context.Context, id string) (*types.Issue, error) {
if id == "" {
return nil, issueNotFound(id)
}
path, err := wire.IssuePath(id)
if err != nil {
return nil, err
}
// Neither row list: this is the existence probe and the loader's read, and
// both discard everything but the row. Asking for dependents and comments
// would buy two joins nobody reads.
var details types.IssueDetails
err = s.dispatch(ctx, wire.Request{
Op: wire.OpGetIssue,
Method: http.MethodGet,
Path: path,
Query: url.Values{},
IssueID: id,
}, &details)
if errors.Is(err, issueops.ErrNotFound) {
return nil, issueNotFound(id)
}
if err != nil {
return nil, err
}
// The detail view carries labels in a field of its OWN rather than on the
// embedded issue, and the local GetIssue hydrates them onto the issue.
// Lifting them across is what makes the two answers the same value — the
// same correction getIssuesByExactID makes for the resolver's fast path.
issue := details.Issue
if len(issue.Labels) == 0 && len(details.Labels) > 0 {
issue.Labels = append([]string(nil), details.Labels...)
}
// Issue.RowVersion is json:"-", so the decode above left it at 0 and
// Revision — getIssue's only wire spelling of the token — carries the real
// value. A molecule loader (or any other caller of this read) that then
// guards a write off issue.RowVersion must see the row's actual token, not
// every row's as though none had ever been written.
version, err := parseRevision("getIssue", details.Revision)
if err != nil {
return nil, err
}
issue.RowVersion = version
return &issue, nil
}
// issueNotFound is the raw GetIssue miss, spelled as the local stores spell it.
func issueNotFound(id string) error {
return fmt.Errorf("%w: issue %s", issueops.ErrNotFound, id)
}