beads/docs/reference
CI Bot 19e4af709a feat(httpapi): stream the events journal over SSE (bd-opisf)
GET /v0/beads/events:watch holds a text/event-stream response open and emits
each committed mutation as it lands, so a consumer learns about a write when it
happens instead of on its next interval. It is a SIBLING of the paged read, not
a mode of it: `follow=true` would have given one operationId two media types,
two lifetimes, two limit rules and two capacity stories, and a client generated
from the document could not have described either.

THE CURSOR IS STILL THE CONTRACT, which is what makes this cheap to publish.
The server keeps no per-consumer state and remembers no subscription; a stream
is the `since` reads a poller would have performed, performed on its behalf
against the same ReadEventsJournalPage through the same read-only cursor. Every
event carries `id: <seq>`, so the checkpoint a client holds while streaming is
the number it would have held while polling, and `Last-Event-ID` — the header a
browser's EventSource resends by itself — takes exactly the value `since` takes.
That header WINS over the parameter, because the client that sends it did not
choose to: an EventSource reconnects to the URL it was built with, so honoring
`since` there would replay every record since the consumer started, on every
reconnect, forever. A nonempty unparseable one is a 400 naming the header rather
than a silent fallback, which would start a stream somewhere the client did not
ask for and never tell it; an EMPTY one means "no id yet" and defers to `since`,
because a client that always sets the header sends it empty on its first
connect and refusing that would break the request the header exists to serve.
Delivery is at-least-once against the client's own checkpoint: exactly once and
in order within a stream, duplicates only across a reconnect. No engine or
storage change — this consumes the poll read path.

EVERY REFUSAL IS DECIDED BEFORE THE FIRST BYTE — validation, the disabled 409,
the flush check, the stream cap, and then the first read, which is what turns a
pruned-past cursor into the paged read's own 410 rather than a 200 that
immediately reports its own failure. Exactly one failure can arrive after the
status is spent: a prune racing an open stream, emitted as a NAMED `truncated`
event carrying the 410's own body in the same encoding on one line (asserted
against Write's output, not merely agreeing with it), behind a raised `retry`
and followed by a close. Naming only the exception is what lets a bare
`onmessage` consumer treat an unrecognized event as "stop, this is not a
record"; one that ignores it does not lose records silently, it stalls loudly on
connect-time 410s, slowly. The other unreportable failure — a stored payload
that will not encode — ends the stream too, and names its seq in the log,
because it will end every stream that reaches that row and nothing else would
say which one it is.

THREE LIMITS HAD TO MOVE, and the middle one is the trap.

  - requestDeadline (60s) does not apply to a streaming row. For every other
    operation it is the backstop against a request holding resources forever;
    here it would do nothing but sever the stream. Each READ carries a 15s
    deadline instead — a liveness budget rather than a database one, since it
    also bounds how long a stream can go without a heartbeat and how long it can
    ignore a shutdown.
  - http.Server.ReadTimeout (30s) had to be cleared per request, BEFORE the
    connect read. There is no WriteTimeout on this server — statusWriter rolls a
    deadline forward before every write, which bounds one stalled write rather
    than a stream, and needs nothing here — but ReadTimeout is a deadline on the
    WHOLE request, and net/http keeps a background read running while a handler
    writes: the read that notices a disconnect. Its expiry cancels the request
    context, so an idle stream would end itself at thirty seconds with no client
    involved, and a database wedged during the connect read would be booked as
    `client_closed`. Clearing it costs no disconnect detection — a client that
    leaves still produces a read error — and a peer that vanishes without a FIN
    is reaped by TCP keepalive, which Go's listener enables by default
    (measured: 15s idle, 15s interval, 9 probes; ~150s).
  - The database slot moves from the request to the read. Sixteen slots and
    forty-eight stream places means one slot per open stream would let a handful
    of idle consumers starve every other operation for hours with /healthz
    green; a stream now takes a slot for the moment of each poll and gives it
    back, and slot pressure delays a pass instead of ending a stream.

Streams are capped at 48, SIXTEEN BELOW the 64-connection limit, and the gap is
load-bearing arithmetic rather than caution: LimitListener returns a connection
slot only when the connection closes, which is after the handler returned and
after this counter came down, so a cap at maxConns could never be observed — the
connect that should earn the 503 would never be accepted, and neither would the
poll it is told to fall back to, nor a mutation, nor a fresh-connection
/healthz. The headroom makes the refusal deliverable; `events_watch_admitted`
carries the live gauge on every connect so accumulation is visible before the
cliff. Each stream jitters its own poll phase by ±10%, because consumers
reconnect together after a restart on one shared `retry` and a fixed interval
would march them into a single read burst per second forever. And a graceful
shutdown signals its streams (RegisterOnShutdown) rather than waiting them out;
the loop consults that signal and the client on EVERY pass, including the
draining ones, so a week-old backlog stays cancellable within one batch.

Tests: handler-level over live delivery (mutations land AFTER the connect),
id=seq, heartbeats, Last-Event-ID precedence including a stale header against a
live `since` and the empty-header polarity, the 400/409/410 connect legs, a
prune under an open stream, the cap and its 503, the cap's headroom against
maxConns, backlog drains cancelled by both a hangup and a shutdown, slot release
on every exit, a stream sharing a single-slot semaphore with a paged read, the
flusher refusal, the read-deadline clear and its ordering against the connect
read, jitter spread, and a clean drain with a stream open — all under -race.
Framing is pinned against a record whose payload carries literal newlines, a CR
and a `data:` line of its own. Two e2e legs through a real `bd serve`: mutate on
one connection while streaming on another under a 10s arrival bound, then
reconnect with Last-Event-ID and require the resume, plus the disabled refusal.
Mutated and each caught: the mid-stream truncated event, Last-Event-ID
precedence, the empty-header polarity, the cap decrement, the cap's headroom,
the drain-cancel fast path, the read-deadline clear and its ordering, the poll
jitter, the shutdown exit, and the deadline exemption.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agent-Signature: claude-code-claude-opus-5[1m]-unknown-reasoning on behalf of julianknutsen
2026-08-09 07:55:23 +00:00
..
advanced.md docs(events): document the events journal and bd events (bd-opisf) 2026-08-09 01:58:17 +00:00
antivirus.md docs: apply PR #4722 review feedback to user docs 2026-07-12 11:06:57 -07:00
configuration.md Revert "Simplify the supported backends: roll back the direct PostgreSQL and MySQL adapters" (#4857) 2026-07-16 16:35:51 -07:00
events-journal.md feat(httpapi): stream the events journal over SSE (bd-opisf) 2026-08-09 07:55:23 +00:00
faq.md docs: align user docs and CLI reference with the v1.1.0 release (#4874) 2026-07-17 14:17:28 -07:00
git-integration.md docs: apply PR #4722 review feedback to user docs 2026-07-12 11:06:57 -07:00
index.md docs(events): document the events journal and bd events (bd-opisf) 2026-08-09 01:58:17 +00:00
json-schema.md docs: re-review and bump the three 90-day-stale freshness markers (#5394) 2026-08-07 02:35:11 -07:00
observability.md refactor(telemetry): adopt standard OTel SDK env vars with BD_OTEL_* back-compat (#3859) 2026-08-03 00:05:06 -07:00
protected-branches.md docs: apply PR #4722 review feedback to user docs 2026-07-12 11:06:57 -07:00
troubleshooting.md docs: apply PR #4722 review feedback to user docs 2026-07-12 11:06:57 -07:00
worktrees.md docs: apply PR #4722 review feedback to user docs 2026-07-12 11:06:57 -07:00