beads/backend/http/corpus
Julian Knutsen 4450fe99bb
feat(http): bd blocked over bd serve (blocked.list), plus bd blocked --flags [--raw] (#7649)
* feat(http): bd blocked over bd serve (blocked.list) through issueops.BlockedLister

`bd blocked` refused against an HTTP workspace: it called the legacy
GetBlockedIssues, which the HTTP backend stubs as unsupported. It now runs
one role on every route, and bd serve publishes that role.

- Role: issueops.BlockedLister.ListBlocked(BlockedRequest{ParentID, Labels,
  LabelsAny, ExcludeLabels}) -> BlockedResult{Items}. One body,
  internal/storage/issueops.ExecuteListBlockedInTx over the unchanged
  GetBlockedIssuesInTx; labels normalized in the role (BlockedFilter);
  Items never nil and in a total order (priority, created_at desc, id).
- Legs: DoltStore (read tx), EmbeddedDoltStore (read conn), uow (read unit
  of work through the use case's GetBlockedIssues, so the
  external-dependency uow decorator applies). Census: storage.DoltStorage
  gains BlockedLister(); HookFiringStore passes it through (a read),
  telemetry wraps it, uow.notifyingProvider, externaldeps.uowProvider and
  httpapi.timedProvider build it.
- externaldeps.Store.BlockedLister is built over its own GetBlockedIssues,
  so local `bd blocked` output is unchanged. Over a remote store it takes
  the server's listing when the server advertises
  policy.external_dependencies, else decorates the server's listing here
  through EdgeReader and BatchGetter (decorateBlocked, factored out of
  GetBlockedIssues).
- Wire: GET /v0/beads/blocked (listBlockedIssues; parent, label,
  label_any, exclude_label) answering BlockedPage {items, has_more} with
  the new x-go-type-pinned BlockedIssue schema, behind the new
  blocked.list token. apigen and the wire-shape golden regenerated; the
  golden diff is additive only, so wire_revision stays 2.
  Config.BlockedLister is a required role, wired in bd serve.
- Client: httpclient.Store.BlockedLister dials through Store.dispatch, so
  an older server is refused BEFORE the dial with *wire.CapabilityError
  naming blocked.list. The legacy Store.GetBlockedIssues rides the role
  (page and row cap applied to its answer) and leaves the unsupported
  shell. Divergence ledger row F-blocked retired.
- CLI: `bd blocked` takes BlockedLister() from the store on the direct
  routes and from the provider's BlockedListerSource on the proxied route;
  runBlockedProxiedServer is deleted. --json output is the same rows in
  the same order, [] when empty.

Tests: backend/conformance RunBlockedLister (10 cases: direct blocker,
child attributed to its blocked parent, closed blocker, wisp plane, pinned
status, parent filter, label filters with normalization, order, empty
slice, request not mutated) on dolt, embedded, uow and the served HTTP
client; httpapi handler tests (mapping, unknown/repeated params,
projection, capability); httpclient stub tests (pre-dial refusal naming
the token, encoding, projection, legacy door paging); skew and vocabulary
counts; TestE2E_BlockedOverHTTP (served `bd blocked --json` equals the
local command's after mapping ids and timestamps, member order included,
external blocker named, older-server refusal with no dial); corpus
gc-blocked re-marked and gc-blocked-parent added, both pending until the
reference server advertises blocked.list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(blocked): pin server-side external decoration, refuse has_more, serve the legacy door over http

Review fixes for the blocked.list slice.

- Server-side external decoration is now pinned on the compositions bd
  serve answers from. TestWrapUOWProviderBlockedListerNamesExternalBlockers
  lists through the wrapped unit-of-work provider's own BlockedLister
  accessor (the bd serve provider path) and requires an external-only row
  and a mixed row to carry their external: refs; building that lister over
  the inner provider fails it. TestServedBlockedListerCarriesExternalRefs-
  WhenTheServerEnforces serves the store-shaped composition
  (externaldeps.Store.BlockedLister, policy advertised) and requires the
  operation's own answer, through the undecorated client role, to carry the
  refs, and the client's decorated chain to return it unchanged; serving
  the inner store's lister fails it.
- The client refuses an answer reporting has_more with the new typed
  httpclient.TruncatedListingError instead of returning a short listing:
  the operation has no cursor, and a caller reads a missing row as
  unblocked. The legacy door inherits it.
- externaldeps.Store.GetBlockedIssues over an http store reads the extra
  sources through the served roles (EdgeReader, BatchGetter) instead of the
  raw GetIssuesByIDs / GetDependencyRecordsForIssues the http backend
  refuses, and the decoration skips the source read when there is nothing
  to load. The legacy door on the CLI chain (bd mol stale, the wisp
  commands) now answers over http in both policy modes;
  TestServedLegacyGetBlockedIssuesThroughTheDecorator pins it, paging
  included. bd ready --explain still refuses on its other raw reads; the
  CHANGELOG says so.
- Docs: the spec and the retired F-blocked ledger row state that the
  listing is unpaged, bounded by the client's 32 MiB response limit
  (refused, never read in part), and that has_more is refused.
- Contract: new cases for waits-for and conditional-blocks blockers, no
  type exclusions (gate, epic), and a stale flag with no nameable blocker
  (on the legs that can write the column: dolt, embedded; skipped
  elsewhere with the reason). Order seeds explicit created_at values with
  five rows tied on priority and created_at, two of them wisps, so only the
  id tiebreak orders them; dropping the tiebreak now fails every run.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(blocked): bd blocked --flags, the blocked flag per id from stdin

`bd blocked --flags [--json]` reads ids from stdin, one per line (blank
lines and repeats ignored), and prints the blocked flag bd holds for each:
with --json, [{"id","is_blocked"}] in the order the ids were first given,
omitting an id that names no row on either plane, [] for no input. It is
the readiness column a consumer otherwise had to read with `bd sql`, so no
such consumer needs a local-only command.

It is the FLAG, not the listing: `bd blocked` narrows the stored flag to
rows it can name a blocker for, and --flags does not, so a row whose flag
is set with no nameable blocker reads true here and is absent from the
listing. Where the external-dependency policy applies, an unsatisfied
external: blocker sets the flag, as it keeps the row out of `bd ready`.

One library entry on every route: issueops.BlockedFlags over the route's
BlockingAnnotator (the store's accessor on dolt, embedded and HTTP, the
provider's on the proxied route). Over HTTP that is listBlockingAnnotations
with is_blocked behind dependencies.blocking.isBlocked; a server without
that token is refused before the dial, naming it. No wire change. The
listing's filter flags are refused with --flags rather than ignored.

Tests: TestE2E_BlockedFlagsOverHTTP (served JSON equals the local command's
after mapping ids to titles, unknown id omitted, repeat collapsed, [] on
empty stdin, filters refused; a stale-flagged row with no blocker reads
true and is absent from `bd blocked --json`, which fails if --flags is
derived from the listing; an older server's handshake refused naming the
token with no dial); TestProxiedServerBlocked/flags_reads_ids_from_stdin on
the proxied route; corpus gc-blocked-flags, pending until the reference
server advertises the token. CHANGELOG entry.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat(blocked): bd blocked --flags --raw, the stored is_blocked column

`bd blocked --flags --raw [--json]` answers the stored is_blocked column
for each id read from stdin, exactly what reading is_blocked from the issues
and wisps tables returns, without the external-dependency policy: a row
blocked only by an unsatisfied external: ref reads false (the policy-aware
--flags reads true), and a stale flag reads true either way. Same output
shape as --flags. This lets a consumer move off a `bd sql` read of the
column with no change in what it reads. The default --flags is unchanged.

The role is taken from beneath every decorator: storage.UnwrapStore on the
direct routes, uow.UnwrapProvider on the proxied route. Over HTTP the
innermost store is the http client, whose annotation read is the server's
answer: against a server that does not advertise
policy.external_dependencies that answer is the column, and --raw is
served. A server that advertises it decorates the flag it answers, the v0
wire has no member asking it for the column, and the decorated flag cannot
be turned back into the column client-side, so --raw against such a server
is refused before the annotation read, naming the token, rather than
answered with the decorated flag. --raw without --flags is refused.

Tests: TestE2E_BlockedFlagsOverHTTP/raw_is_the_stored_column (served --raw
equals local --raw; the externally-blocked-only row reads true with --flags
and false with --raw on both; locally blocked rows agree; a stale-flagged
row reads true in both modes; --raw without --flags exits 1; a server
advertising the policy refuses --raw with no annotation request);
TestProxiedServerBlocked/flags_raw_is_the_stored_column; corpus
gc-blocked-flags-raw (pending until the reference server advertises
dependencies.blocking.isBlocked). Routing --raw through the decorated
annotator fails the e2e. CHANGELOG entry.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* docs(changelog): bd blocked --flags --raw is for the local and proxied routes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(blocked): give the e2e gate-kid its own priority; document --flags answers closed ids

TestE2E_BlockedOverHTTP seeded gate-parent and gate-kid at one priority, so
when both landed in the same second their order fell to the generated ids,
which differ between the local and the served workspace: the comparison
failed intermittently. gate-kid now has its own priority. Ten runs on the
embedded lane pass.

bd blocked --flags answers a closed id too, with its stored flag; a SQL read
of the column filtered on status <> 'closed' leaves it out. Said in the
command's doc and the CHANGELOG.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(blocked): apply label filters to external-only blocked rows

The external-dependency policy appends a row blocked only by an
unsatisfied external: ref after the label query has run, so --label,
--label-any and --exclude-label never judged it: bd blocked and
GET /v0/beads/blocked listed it whatever its labels, on every route.
Both decorators now filter those rows by the request's labels. The uow
decorator reads them through GetMany, chunked at MaxGetManyIDs, so their
labels are hydrated; GetIssuesByIDs left them empty, which also printed
them unlabelled on the proxied route.

Review fixes (adopt-pr apply-fixes, iteration 1):
- major #1: label filters on external-only rows in the store and uow
  decorators, with absence-asserting arms in the unit, served httpclient
  (server- and client-enforced policy), e2e and proxied tiers; the spec
  and the role doc say such rows are narrowed like every other row.
- minor #2: the policy.external_dependencies token prose names the
  blocked listing (spec, routes.go, regenerated types.gen.go).
- minor #6: SERVE_RUNBOOK and the allowUnlimited comment record the
  blocked listing's unbounded whole-set answer and why.
- minor #10: BlockedLister telemetry-outermost and typed-unsupported pins
  beside the ReadyCounter pair, plus the externaldeps wrap pin.
- nit #9: F-blocked names --flags --raw's refusal against a server that
  applies the policy (ledger.go and the regenerated ledger doc).
- nit #11: the remote BlockedLister no longer builds a local lister it
  then discards.

Review fixes (adopt-pr apply-fixes, iteration 2; the review approved at
940 with one minor and two nits, none gating):
- minor (the --flags refusal list was unpinned): the new
  TestBlockedFlagsRefusesEveryListingFilter pins blockedFlagFilterFlags
  to the flags blockedCmd registers, in both directions. Every
  registered flag except --flags and --raw must be refused, and every
  entry must name a registered flag. The test walks Flags() rather than
  LocalFlags(), so it does not merge the root's persistent flags into
  blockedCmd, and it skips inherited root flags and cobra's --help.
  Mutants:
  - four fail it: a new filter flag left off the list, a dropped
    entry, a misspelled entry, and --raw added to the list;
  - a simulated earlier in-process run (merged root flags plus --help)
    stays green, and fails once either skip is removed.
- nit (the remote branch's span): the remote BlockedLister comment now
  says why that lister is not wrapped again. The inner lister already
  carries telemetry's span, so wrapping would nest a second span of the
  same name. The comment also says what runs outside that span.
- nit (duplicated order comparator): left as is. finishBlockedIssues
  predates this PR, so merging the two sorts would refactor existing
  code with no review pass left after this one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <338826946+gascity-eddie[bot]@users.noreply.github.com>
2026-10-17 12:29:13 +00:00
..
testdata feat(http): bd blocked over bd serve (blocked.list), plus bd blocked --flags [--raw] (#7649) 2026-10-17 12:29:13 +00:00
BUILD.bazel fix(http): gate check records a bead gate's sighting through Lifecycle.Update (#7604) 2026-10-16 15:14:53 +00:00
corpus_parts_test.go test(corpus): seed served/embedded sides concurrently; size served shards from CI timings (#7538) 2026-10-13 12:30:52 +00:00
corpus_test.go perf(http): answer a command's repeated reads from one response; bd show --json makes 1 request (port of bd-enterprise e626072b8) (#7588) 2026-10-16 04:26:37 +00:00
gate_check_served_test.go fix(http): bd gate create, add-waiter and discover over a connected HTTP workspace (#7621) 2026-10-17 02:15:07 +00:00
harness_test.go feat(http): bd blocked over bd serve (blocked.list), plus bd blocked --flags [--raw] (#7649) 2026-10-17 12:29:13 +00:00
README.md perf(http): answer a command's repeated reads from one response; bd show --json makes 1 request (port of bd-enterprise e626072b8) (#7588) 2026-10-16 04:26:37 +00:00
reference_proxy_test.go test(backend/http/corpus): served MC command corpus against the reference server profile (S23) (#7483) 2026-10-11 05:34:51 +00:00
required_corpus_tier_test.go test(backend/http/corpus): served MC command corpus against the reference server profile (S23) (#7483) 2026-10-11 05:34:51 +00:00
served_pinned_shards.txt test(bazel): shard http_served_test and corpus_served_test (embedded lane critical path) (#7537) 2026-10-13 11:35:34 +00:00

Served command corpus

This suite is the working definition of "bd fully supports a remote HTTP backend". It runs every distinct bd command shape that an orchestrator (gc) and its packs and formulas issue against a work store. Each shape runs against a served workspace and is compared with the same command on an embedded workspace.

What runs

The shapes are dealt round robin, in corpus order, across corpusParts parts (corpus_test.go). Each part is a top-level test, TestServedCorpusPart1 .. TestServedCorpusPartN, and each does the following for its share of the shapes:

  1. Builds bd from this tree. BEADS_TEST_BD_BINARY=<path> reuses a prebuilt binary instead.
  2. Starts an in-process bd serve (httpapi.Listen over embedded Dolt) the way a deployed server is bound: AllowNonLoopback plus a bearer token file. The socket is still 127.0.0.1, but the bind mode decides every behaviour the server keys on, including its refusal of unlimited reads.
  3. Puts a reference proxy in front of the server. The proxy makes it answer like the deployed reference server described in testdata/reference_server.json:
    • the context handshake publishes exactly that server's capability set and its ContextResponse members (so no wire_revision);
    • an operation the reference server does not publish gets the server's own 404;
    • a query parameter or top-level request member it does not declare gets the server's own 400 unknown_parameter.
  4. Runs bd connect on one workspace and bd init on another. Both get the seed from testdata/corpus.yaml: one graph of named rows, plus bulk graphs that put more than one server page of ready rows and wisps in the store. Both sides use BEADS_ACTOR=worker and set CLAUDE_SESSION_ID. The embedded side is initialized and seeded on its own goroutine while steps 2 and 3 run and the served side is seeded (prepareSide): the two sides share no directory, env or process, and each runs the seed in order, so the seeded data does not depend on the overlap.
  5. Runs each of its shapes as a subtest, on both sides. The embedded run must exit with the shape's expect_exit and match its expect_stderr. If it does not, the shape is wrong or bd changed, and the subtest fails as a corpus bug that no pending mark or allowlist entry hides. The served run must:
    • exit like the embedded run;
    • carry no refusal (ErrUnsupported, a v0-wire refusal, the loopback-only 400, or a reference-profile 404/400);
    • match the embedded output under the shape's compare mode, after timestamps and revision tokens are dropped (but not defer_until or due_at: a shape sets those itself, as absolute dates, and they must match exactly);
    • for writes, give the same JSON from every verify read on both sides afterwards.

A part takes its own server, workspaces and seed, so no shape can see another part's writes. That is safe because no shape depends on another shape's effects:

  • every check compares the served run with the embedded run after the same commands, so it holds whatever ran before it on both sides;
  • each part keeps corpus order, so its reads still run on the bare seed, before any of its writes;
  • each write owns its target ids (e2e-t-*, or ids it creates).

TestCorpusPartsDealEveryShapeOnce checks that every shape is in exactly one part, in corpus order. A new shape must keep that independence: it may rely on the seed and its own setup, never on another shape having run.

Every part seeds again, so running all the parts in one process takes longer than one run over every shape did; each part logs its setup time (part N setup ...). Run one part with -run 'TestServedCorpusPart2$'.

There are four parts. In the embedded lane each part's setup (server start plus both seeds, run concurrently) takes ~62-76s and the shapes ~242s in total. Four parts ran 114-146s each and three ran 170-180s. Three would save one setup (~75 test-seconds a run), but the longest part is the lane's served long pole and sits on the CI gate's critical path, so four parts win. To change the count, update corpusParts, servedCorpusParts, served_pinned_shards.txt and the target's shard_count together.

CGO_ENABLED=1 BEADS_TEST_EMBEDDED_DOLT=1 BEADS_HTTP_TEST_REQUIRED=1 \
  go test -tags gms_pure_go -run 'TestServedCorpus|TestCorpus' ./backend/http/corpus/

In CI, //backend/http/corpus:corpus_served_test runs the suite with both variables set, in the Embedded Dolt lane (--config=embedded; see engdocs/TESTING.md). Its served_pinned_shards.txt pins part k to Bazel shard k, so the parts run concurrently. corpus_test on its own sets neither variable, so it skips the parts.

Two environment variables change the run:

  • BEADS_CORPUS_REPORT=<file> writes a JSON summary with per-shape status and request counts.
  • BEADS_CORPUS_SKIP_PENDING=1 skips pending shapes without running them.

The data

File Holds
testdata/corpus.yaml assets, the seed, and the shapes
testdata/allowlist.yaml permanent refusals, each with its reason and the refusal text the served run must match
testdata/reference_server.json the reference server's wire surface

Each shape in corpus.yaml has these fields:

Field Meaning
id gc-* for orchestrator subprocess shapes, mc-* for pack/formula/prompt shapes
source where the shape comes from
args the argv; {file:NAME} expands to an asset path and {rev:ID} to that side's current revision of ID
stdin text piped to bd on both sides
setup commands run on both sides before the shape; one that fails fails the shape
compare json (default), ids, count, text or exit
ordered under json and ids, array order is significant; by default arrays are sorted before comparing
prefix_of for a limited ids listing: its unlimited form. Each side's limited rows must be a prefix, in order, of that side's own unlimited listing, both sides return as many rows, and the two unlimited listings hold the same ids. A limit that cuts through a tie set (same priority, created_at in the same second) lands on different rows on two workspaces seeded seconds apart, so the limited rows are not compared across sides directly
verify reads run after the shape
ignore_fields JSON keys dropped, at any depth, before a json comparison and before verify output is compared; for example comment ids, which each workspace mints itself
expect_exit, expect_stderr expected outcome on both sides (exit 0 by default). An allowlisted shape must state expect_exit; only its embedded run is held to these
pending a known failure
requests exactly how many requests the served run of args makes through the reference proxy, not counting setup, verify or {rev:ID} reads. Each request is a sequential round trip, so the count is the shape's latency floor over a gateway. A change that makes the shape cheaper updates the pin

pending: {slice, reason} marks a known failure and names the slice or slices that fix it, for example S6c+S15. A pending shape still runs:

  • while it fails, the subtest is skipped and the skip records what was observed;
  • once it passes, the subtest fails with "now passes; remove the pending mark".

So removing a mark is how a slice proves itself, and any other failure is unexpected and fails the suite.

Updating the reference profile

reference_server.json was derived from two things:

  • the deployed server's live GET /v0/beads/context capabilities;
  • its published OpenAPI document (bd 1.3.0, no wire_revision). For each operation the profile keeps the method, the path template, the query parameter names, and the top-level request members.

When the deployed server is upgraded (SRV-1), regenerate the file the same way. Then remove the SRV-1 pending marks that now pass.

Known fidelity limits

  • Responses pass through unfiltered. A response member the reference server would not send still reaches the client.
  • Only top-level request members are checked; nested members (for example batchApply items) are not.
  • The served backend underneath is embedded Dolt, not the reference deployment's storage backend.
  • Out of scope, because they are local by design or never touch a remote store: init, bootstrap, migrate, doctor, flatten, gc, and the dolt subcommands. context, version, backup status and prime are in the corpus, but only checked for exit codes.