beads/internal/httpapi
Julian Knutsen 6cd1184687
feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051)
* test(storage): measure ApplyBatchInTx SQL statement counts (B0)

Adds a driver.Connector-level counting/latency-injecting wrapper
(internal/storage/sqlcount) plus shared plan-request fixtures
(internal/storage/batchfixtures: mol-normalized 356/712-item shapes and a
40-item classic shape) and backend-specific pinning tests + benchmarks
(internal/storage/embeddeddolt, internal/storage/dolt) that measure the
ACTUAL number of SQL statements issueops.ApplyBatchInTx issues for each
shape, on both the embedded and Dolt (TCP) backends.

Measured totals (Prepare+Exec+Query+StmtExec+StmtQuery inside one
transaction): 356 items ~7008-7009, 712 items 14014-14015, 40 items 846,
identical on both backends (expected: statement issuance is driven by Go
application logic, not backend-specific SQL planning). Pinned with a small
(+/-2) tolerance to absorb an observed, host-contention-driven +/-1 Query
drift (Exec is deterministic on every run). This is the regression baseline
slice B2 (a later, lighter fast path) must demonstrably lower.

batchfixtures.UncappedPlan bypasses issueops.MaxApplyBatchItems so this
measurement does not depend on slice B1's cap raise.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* feat(httpapi,issueops): raise batchApply envelope to 1000 items with bounded large-apply handling

POST /v0/beads/issues:batchApply now accepts up to 1000 items (up from 100);
issueops.MaxApplyBatchItems moves accordingly and the request body cap rises
to 16 MiB. The whole plan still applies in exactly ONE transaction. A request
over 100 items runs under a separate, flat, operator-configured whole-run
budget (`bd serve --large-apply-ceiling`, 5 minutes by default) instead of
the ordinary 60s request deadline; a request at or under 100 items is
unaffected and keeps exactly the 60s deadline and 15s commit-retry ceiling it
always had.

Concurrency and shutdown are bounded, not just documented:
- Oversized requests serialize through a dedicated one-wide semaphore, and at
  most one caller may wait for that semaphore at a time (a CAS-guarded single
  waiter) — a burst of oversized requests is shed with 503 rather than
  queuing and starving ordinary traffic behind the general semaphore.
- The run deadline is stamped only once the slot is actually acquired, not
  at the time the request arrived.
- Graceful shutdown closes new admissions before computing its drain budget,
  then waits out an in-flight large apply up to its own remaining deadline
  (plus a few seconds' grace for the response to write); any large apply
  still queued for the semaphore when shutdown begins is refused rather than
  extending the drain further. A queued caller that disconnects gets a clean
  503, never a misclassified context error. `bd serve`'s orchestrator stop
  grace must be at least `--large-apply-ceiling` — see
  engdocs/SERVE_RUNBOOK.md.
- ApplyBatch's commit-retry budget only extends past the ordinary 15s ceiling
  on this explicit large-apply path (via a context marker), with headroom
  reserved so the final retry attempt can't be killed mid-commit by the
  context's own expiry.

A new `issues.batchApplyLarge` capability token on `GET /v0/beads/context`
lets a client learn whether a given server accepts the raised envelope.

Wall-clock measurement (embedded Dolt, whole-transaction build+commit) backs
the 5-minute ceiling default: 356 items ~12.6s (23.8x headroom), 1000 items
~57.5s (5.2x headroom).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(httpapi,serve): wire the large-apply retry budget; repair stale pins

Review fixes for the large batchApply envelope, on top of the test
repairs the PR needed against main.

The extended commit-retry budget was never switched on.
internal/storage/uow's retryTxBudget lets a large apply keep retrying a
commit-time conflict past the ordinary 15s ceiling only when its context
carries issueops.WithExtendedRetryBudget's marker, and the docs on both
sides say admitLargeApply sets it, but nothing did: the marker's only
callers were the uow tests, which mark their contexts by hand. A
1000-item apply that hit a conflict still gave up after 15s with most of
its 5-minute ceiling unused. admitLargeApply now marks the run context
it builds, and TestApplyBatchMarksOnlyALargeApplyForTheExtendedRetryBudget
pins the seam over HTTP: marked at 101 and 1000 items, unmarked at 100.
Removing the mark fails the 101 and 1000 cases; marking every request in
handleApplyBatch fails the 100 case.

Smaller fixes from the same review:

- `bd serve --large-apply-ceiling` repeated the 5m default as a literal
  of its own, and since bd serve always passes a positive ceiling, that
  literal, not httpapi's, was the default every deployment ran. Export
  httpapi.DefaultLargeApplyCeiling, default the flag to it, and pin the
  flag's 5m0s default in TestServeFlags.
- admitLargeApply's release cancels the run context before clearing
  s.largeApplyDeadline, the order acquireLargeApply's doc states; it
  cleared the deadline first.
- Drop the batchfixtures test's dead t.Skipf, which waited for the cap
  to reach 1000, and correct a graph-counter comment that still cited
  the 100-item cap and the TestServeFlags doc line that called the
  no-argument server unchanged.

The PR raises issueops.MaxApplyBatchItems from 100 to 1000 and adds
`bd serve --large-apply-ceiling`, but two base-owned tests still pinned
the old surface and went red on every lane that runs them (Bazel / test,
Test (macos-latest)):

- TestServeFlags pins the exact `bd serve` flag list; add
  large-apply-ceiling.
- TestPlanApplyBatchRefusesAnUnusableRequest/too_many_items built 101
  items and expected "at most 100 items", a request the 1000 cap now
  plans successfully. Size the over-cap request and its message from
  issueops.MaxApplyBatchItems, as TestPlanApplyBatchAcceptsTheBoundary
  already does for the other side of the cap; the cap's value stays
  pinned by TestCapBatchApplyLargeTiesAllThreeLimits.

The PR's own TestLargeApplyBurstNeverStarvesSmallRequestsForSemTimeout
also went red in PR Core (-race, -p 4 on a 4-vCPU runner): all eight
small requests took 160-165ms against a 100ms bound. The stall it
guards against lasts semTimeout (5s), while a small request can still
queue briefly behind the refused large requests, each holding a general
sem slot while its body decodes. Bound it at a second, the "want well
under a second" its sibling TestLargeApplyQueueDoesNotStarveOrdinaryRequests
already uses. Pinned to four cores beside 24 busy loops, small requests
measured 183ms mean and 338ms max, and the 1s bound passed 80/80 runs;
with the one-waiter gate in acquireLargeApply disabled the test still
fails 3/3, on its 2s never-completed check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(uow): stop a doomed large-apply retry before ctx cuts it off

retryTxBudget sizes maxElapsed once at ApplyBatch entry, but says nothing
about how long an individual attempt takes. A slow first attempt (e.g.
157s of a large apply's commit) left little of the budget for a retry,
which then got killed mid-transaction by ctx's own deadline instead of
finishing and reporting its own outcome -- surfacing as
context.DeadlineExceeded (isUnavailable excludes it -> a generic 500)
instead of the exhausted-write-conflict result this loop already reports
correctly (-> 503 + Retry-After).

RunTxResultWithin now tracks the previous attempt's duration and, before
starting a new one, stops early with the last real serialization error
when time left on ctx is less than that duration plus a fixed headroom --
but only when publicops.HasExtendedRetryBudget(ctx) is set (the same
marker retryTxBudget itself checks). Applying the look-ahead
unconditionally regressed TestApplyBatchHonorsAnExtendedRetryBudgetFromContext's
unmarked case: ordinary requests have cheap attempts and are meant to run
until their own ctx deadline ends the loop, and the large-apply-sized 5s
headroom was tripping that for any short-ctx caller with a near-instant
failing attempt.

Adds TestRunTxResultWithin_SlowAttemptStopsBeforeADoomedRetry (a 1.2s
commit against a 2s marked ctx must stop after one attempt with its own
serialization error, never DeadlineExceeded, and never touch a second
unit of work) and its control, TestRunTxResultWithin_ShortAttemptStillRetries
(a 50ms attempt against the same shape must still retry and succeed).
Both mutation-checked against the look-ahead condition and its gate.

Also updates the stale "items accepts at most 100 entries" prose on the
applyBatch endpoint description (openapi.v0.yaml) to the large-envelope
wording already used elsewhere in the spec: 1000-item cap behind the
issues.batchApplyLarge capability, large-apply-ceiling deadline extension,
<=100 unaffected. dependencies:count's own "at most 100" lines are
unrelated and untouched.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(ci,serve): fit the 1000-item cap's tests in the storage shard budget

Review fixes for the relaunched #7051, iteration 1.

Raising issueops.MaxApplyBatchItems from 100 to 1000 made the base-owned
TestBatchApplyContract take 604s under -race, against 97s at the old cap
(main run 35862663577). It hashed onto the legacy test-embedded-storage
shard 2/5, beside ~5m of other tests, and that shard hit its 15m budget
twice (PR Risk run 36807191489). PR Risk no longer runs the legacy tier
on same-repo PRs (#7071), but Main's push run and fork PRs still do, so
the merge would have carried the timeout to Main.

Raising the jobs' timeout-minutes alone would not fix it: the shard
script passes the binary its own -test.timeout=15m. So:

- embedded-storage-test-shard.sh passes -test.timeout=20m (and its go
  test fallback -timeout 20m), as embedded-test-shard.sh does for
  cmd/bd.
- test-embedded-storage in main.yml and pr-risk.yml: timeout-minutes 15
  -> 25, the script's 20m plus setup.
- embeddeddolt_embedded_test appends -test.timeout=19m, as
  bd_embedded_test does, so Go's timeout, which dumps every goroutine,
  fires before rbe-west's 1200s action kill, which does not.
  TestBazelEmbeddedJobMirrorsEmbeddedTier now checks the override on
  both sharded targets instead of only cmd/bd's, and the PR Risk
  coverage test pins the storage target's new args.
- embedded-storage-test-shards.txt places TestBatchApplyContract on
  shard 4, which has ~3m of other tests, and the PR's two large-batch
  tests, TestLargeBatchApplyStatementCounts_Embedded (247s) and
  TestLargeBatchApplyWallClock_Embedded (261s), on shards 2 and 5. From
  the PR run's per-test times, the heaviest shard projects to ~13m, down
  from shard 2's 15m-plus.

Also correct the stop-grace guidance. For an in-flight large apply,
Serve's drain waits out the apply's remaining deadline plus drainGrace
(5s), so the orchestrator's stop grace must be at least
--large-apply-ceiling plus 5s, as Serve's own doc says. The runbook, the
flag help, the LargeApplyCeiling field doc and the CHANGELOG entry asked
for the ceiling alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-10-01 08:32:56 -07:00
..
apigen feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
spec feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
auth.go feat(serve): optional bearer auth and a Host allowlist for bd serve (#5516) 2026-08-09 18:05:56 -07:00
auth_test.go test(httpapi): match missing token file with errors.Is, not Unix text (#6628) 2026-09-26 09:45:11 -07:00
batch_apply.go feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
batch_apply_test.go fix(update): fence unforced --notes overwrites behind --force across all surfaces (#5946) 2026-10-01 05:02:40 -07:00
batch_close.go feat(serve): name the blockers on a live-blocker not_closable (#6893) 2026-10-01 00:51:06 -07:00
batch_close_test.go feat(serve): name the blockers on a live-blocker not_closable (#6893) 2026-10-01 00:51:06 -07:00
batch_create.go feat(httpapi): serve closeIssue over v0 2026-08-07 22:25:38 +00:00
batch_create_test.go feat(httpapi): serve closeIssue over v0 2026-08-07 22:25:38 +00:00
blocking.go Trim this series' comments toward the repo's own density 2026-08-06 08:06:59 +00:00
blocking_test.go Trim this series' comments toward the repo's own density 2026-08-06 08:06:59 +00:00
brief_projection_test.go feat(list,ready,httpapi): reach the Brief projection from both front doors (#5586) 2026-08-10 13:05:00 -07:00
BUILD.bazel feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
claim.go feat(httpapi): PATCH /v0/beads/issues/{id} accepts claim, the wire spelling of bd update --claim (#6890) 2026-09-30 15:42:37 -07:00
claim_next.go claimNext + batchClose + incremental labels: the three role-backed ops with no wire (ga-g3gg8) (#5510) 2026-08-09 16:02:52 -07:00
claim_next_test.go claimNext + batchClose + incremental labels: the three role-backed ops with no wire (ga-g3gg8) (#5510) 2026-08-09 16:02:52 -07:00
claim_test.go Publish POST /v0/beads/issues/{id}:release: give a claim back (ga-lzcuv) (#5507) 2026-08-09 14:41:28 -07:00
close.go feat(serve): name the blockers on a live-blocker not_closable (#6893) 2026-10-01 00:51:06 -07:00
close_test.go feat(serve): name the blockers on a live-blocker not_closable (#6893) 2026-10-01 00:51:06 -07:00
comments.go Publish POST /v0/beads/issues/{id}/comments, the wire half of issueops.Commenter (ga-rslox) (#5594) 2026-08-10 18:59:57 -07:00
comments_test.go Publish POST /v0/beads/issues/{id}/comments, the wire half of issueops.Commenter (ga-rslox) (#5594) 2026-08-10 18:59:57 -07:00
context_test.go feat(serve): bd serve skeleton — lifecycle, mode gate, /healthz, /v0/beads/context (bd-60z) 2026-07-31 10:07:14 +00:00
count_test.go test(count): pin --include-ephemeral as the plane bit alone, and fix the plane prose (post-merge #6240) (#6998) 2026-10-01 01:14:25 -07:00
create.go Publish POST /v0/beads/issues: the single create, with the whole vocabulary (#5483) 2026-08-09 09:21:56 -07:00
create_test.go fix(contracts): JSON contract fixes before 1.3.0 freezes them — string revision tokens, --set-metadata scalar typing, create --json nano timestamps (#6053) 2026-08-29 05:20:27 +00:00
cursor.go feat(httpapi): serve bd list's order with sort on listIssues (#5666) 2026-08-27 09:21:12 -07:00
cycles.go Trim this series' comments toward the repo's own density 2026-08-06 08:06:59 +00:00
cycles_test.go Trim this series' comments toward the repo's own density 2026-08-06 08:06:59 +00:00
delete.go Publish the row-version guard on close, reopen and delete (#5506) 2026-08-09 17:02:53 -07:00
delete_test.go fix(contracts): JSON contract fixes before 1.3.0 freezes them — string revision tokens, --set-metadata scalar typing, create --json nano timestamps (#6053) 2026-08-29 05:20:27 +00:00
dependency_edit.go docs: the ext-api-store documentation tranche — post-#5516 auth sweep, wave doc coherence, role checklist refresh (#5545) 2026-08-10 08:05:07 -07:00
dependency_edit_test.go docs: the ext-api-store documentation tranche — post-#5516 auth sweep, wave doc coherence, role checklist refresh (#5545) 2026-08-10 08:05:07 -07:00
doc.go docs(httpapi): stop the orientation prose enumerating a surface it no longer has 2026-08-08 01:34:33 +00:00
edge_counts_test.go Publish GET /v0/beads/dependencies:count, the wire half of issueops.GraphCounter (#5536) 2026-08-10 02:06:57 -07:00
edges.go Publish GET /v0/beads/dependencies:count, the wire half of issueops.GraphCounter (#5536) 2026-08-10 02:06:57 -07:00
edges_test.go Trim this series' comments toward the repo's own density 2026-08-06 08:06:59 +00:00
events.go feat(httpapi): stream the events journal over SSE (bd-opisf) 2026-08-09 07:55:23 +00:00
events_test.go feat(httpapi): publish the events journal on the v0 surface (bd-opisf) 2026-08-09 06:01:23 +00:00
events_watch.go feat(httpapi): stream the events journal over SSE (bd-opisf) 2026-08-09 07:55:23 +00:00
events_watch_test.go feat(httpapi): stream the events journal over SSE (bd-opisf) 2026-08-09 07:55:23 +00:00
get_issue_test.go fix(contracts): JSON contract fixes before 1.3.0 freezes them — string revision tokens, --set-metadata scalar typing, create --json nano timestamps (#6053) 2026-08-29 05:20:27 +00:00
handlers.go fix(httpapi): make beads_dir and repo_root optional in ContextResponse (#5757) 2026-08-19 22:58:03 -07:00
large_batch_apply_test.go feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
list_comments_omitted_test.go fix(list): expose comment bodies in bd list/show --json (be-73x) (#6473) 2026-09-27 02:12:13 -07:00
list_sort_test.go feat(httpapi): serve bd list's order with sort on listIssues (#5666) 2026-08-27 09:21:12 -07:00
memories.go Put the memory plane behind memoryops, conformance-tested at three backends (#5387) 2026-08-07 03:57:04 -07:00
memories_test.go feat(httpapi): serve closeIssue over v0 2026-08-07 22:25:38 +00:00
metadata_cas.go Add issueops.MetadataCAS: compare-and-set over one metadata key (#5473) 2026-08-09 04:15:34 -07:00
metadata_cas_test.go Add issueops.MetadataCAS: compare-and-set over one metadata key (#5473) 2026-08-09 04:15:34 -07:00
pinning.go feat(httpapi): publish the events journal on the v0 surface (bd-opisf) 2026-08-09 06:01:23 +00:00
problem.go fix(update): fence unforced --notes overwrites behind --force across all surfaces (#5946) 2026-10-01 05:02:40 -07:00
problem_test.go feat(httpapi): OpenAPI v0 spec, types-only codegen, and the drift gates (bd-ilb) 2026-07-31 08:37:34 +00:00
project_stamp_test.go docs: the ext-api-store documentation tranche — post-#5516 auth sweep, wave doc coherence, role checklist refresh (#5545) 2026-08-10 08:05:07 -07:00
query.go feat(httpapi): publish the events journal on the v0 surface (bd-opisf) 2026-08-09 06:01:23 +00:00
reads.go test(count): pin --include-ephemeral as the plane bit alone, and fix the plane prose (post-merge #6240) (#6998) 2026-10-01 01:14:25 -07:00
reads_test.go feat(count): accept --has-metadata-key (#6282) 2026-09-29 13:16:37 -07:00
ready_count_test.go Trim this series' comments toward the repo's own density 2026-08-06 08:06:59 +00:00
related.go Publish GET /v0/beads/issues/{id}/related, the wire half of issueops.Relations (ga-81p36) (#5540) 2026-08-10 05:23:30 -07:00
related_test.go Publish GET /v0/beads/issues/{id}/related, the wire half of issueops.Relations (ga-81p36) (#5540) 2026-08-10 05:23:30 -07:00
release.go fix(contracts): JSON contract fixes before 1.3.0 freezes them — string revision tokens, --set-metadata scalar typing, create --json nano timestamps (#6053) 2026-08-29 05:20:27 +00:00
release_test.go fix(contracts): JSON contract fixes before 1.3.0 freezes them — string revision tokens, --set-metadata scalar typing, create --json nano timestamps (#6053) 2026-08-29 05:20:27 +00:00
reopen.go fix(contracts): JSON contract fixes before 1.3.0 freezes them — string revision tokens, --set-metadata scalar typing, create --json nano timestamps (#6053) 2026-08-29 05:20:27 +00:00
reopen_test.go fix(contracts): JSON contract fixes before 1.3.0 freezes them — string revision tokens, --set-metadata scalar typing, create --json nano timestamps (#6053) 2026-08-29 05:20:27 +00:00
roles.go Publish POST /v0/beads/issues/{id}/comments, the wire half of issueops.Commenter (ga-rslox) (#5594) 2026-08-10 18:59:57 -07:00
roles_test.go Publish PUT and DELETE /v0/beads/config/{key}, the write half of issueops.WorkspaceConfig (ga-098zm) (#5596) 2026-08-10 19:47:21 -07:00
routes.go feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
server.go feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
server_test.go feat(httpapi): opt-in large issues:batchApply envelope (≤1000 items, one transaction) (#7051) 2026-10-01 08:32:56 -07:00
settings.go Publish PUT and DELETE /v0/beads/config/{key}, the write half of issueops.WorkspaceConfig (ga-098zm) (#5596) 2026-08-10 19:47:21 -07:00
settings_write_test.go Publish PUT and DELETE /v0/beads/config/{key}, the write half of issueops.WorkspaceConfig (ga-098zm) (#5596) 2026-08-10 19:47:21 -07:00
spec_parity_test.go fix(httpapi): make beads_dir and repo_root optional in ContextResponse (#5757) 2026-08-19 22:58:03 -07:00
stats.go Merge branch 'facade/wave-3' into facade/wave-4 2026-08-06 21:39:05 +00:00
stats_test.go Merge branch 'facade/wave-3' into facade/wave-4 2026-08-06 21:39:05 +00:00
sweep.go docs: the ext-api-store documentation tranche — post-#5516 auth sweep, wave doc coherence, role checklist refresh (#5545) 2026-08-10 08:05:07 -07:00
sweep_test.go docs: the ext-api-store documentation tranche — post-#5516 auth sweep, wave doc coherence, role checklist refresh (#5545) 2026-08-10 08:05:07 -07:00
timed_provider_binding_test.go feat(httpapi): publish removeDependency on the v0 surface 2026-08-08 00:19:48 +00:00
tree.go Trim this series' comments toward the repo's own density 2026-08-06 08:06:59 +00:00
tree_test.go Trim this series' comments toward the repo's own density 2026-08-06 08:06:59 +00:00
update.go fix(update): fence unforced --notes overwrites behind --force across all surfaces (#5946) 2026-10-01 05:02:40 -07:00
update_test.go fix(update): fence unforced --notes overwrites behind --force across all surfaces (#5946) 2026-10-01 05:02:40 -07:00
wire_bijection_test.go feat(httpapi): publish the events journal on the v0 surface (bd-opisf) 2026-08-09 06:01:23 +00:00