* feat(http): serve list/show counts/dep tree/ping, close+reopen under a session, one-lookup not-found (S6b/S6c/S6d) S6c. The external-dependency decorator rebuilt its role views from the legacy DoltStorage seam (storereader over itself, IsBlocked, GetDependencyTree, ClaimReadyIssue), which a remote store only partly serves: `bd list`/`bd children` refused on the SearchIssuesWithCounts stub, `bd show --json` printed zero counts, the default `bd dep tree` refused, `bd ready --claim` and `bd close` of an externally blocked issue reached unsupported paths. For a storage.RemoteBackendStore each view now passes the inner role through and lays the policy over it in role terms (externaldeps/remote_roles.go): ready reads and the ready claim exclude the workspace-wide blocked set over a widened window; per-issue checks (claim, close, batch close, list annotations, tree leaves) read only the named issues' edges through EdgeReader instead of every edge in the workspace. A server advertising policy.external_dependencies (MC's does) gets a plain passthrough. `bd ping` uses a new storage.Pinger, which the http store answers with its authenticated context read. S6d. A store reporting storage.CallerAttributionLimitedStore (http) cannot carry a Provenance label or a patch's closed_by_session. The library still refuses both; the CLI omits them where nothing is lost: `bd reopen`'s fixed "bd: reopen <id>" label (the server labels the same entry), and the closed_by_session of `bd update -s closed` (attribution only, usually the ambient CLAUDE_SESSION_ID; an explicit --session prints a notice; `bd close` still records it). See cmd/bd/remote_backend.go. S6b. utils.ExactIDLookupStore: the resolver stops at exact lookups for a store with no substring search, so a missing prefixed id is a not-found (beadserrors.ErrNotFound, "no issue found matching") after one getIssue. The general resolver no longer repeats an identical normalized probe. The http SearchIssueIDs answers D11's partial-id refusal, which now unwraps to ErrNotFound. Contributor auto-routing for a remote workspace reads local config only, so the not-found fallback pays no settings round trip. `bd update <absent> --claim` (gc's hook claim) answers a gc-classifiable not-found in at most two requests. Served e2e (backend/http/e2e_remote_roles_test.go): every gc list shape and `bd children` against embedded, `show --json` count parity, default dep tree parity, external blocking still enforced, close/reopen under CLAUDE_SESSION_ID, missing-id request counts, absent-id claim against the current server and the MC handshake mask, policy passthrough under the MC mask, ping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(externaldeps): role-level remote reader and claimer cases bd ready and bd update --claim reach the policy through the legacy seam and the lifecycle guard, so the remote Reader.Ready exclusion and the remote Claimer guard (what bd serve and library callers reach) are pinned at the role level with a fake remote store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(http): bd serve advertises policy.external_dependencies; clients pass through bd serve composes every ready, claim, claim-next, close, blocking and tree role through the external-dependency policy, but never said so. Every client therefore ran the policy a second time, with the CLIENT machine's external-project config, and claim-next became a non-atomic read-then-claim loop on the client. The server now advertises the token exactly when the roles it serves were composed through the policy: httpapi.Config.ExternalDependencyPolicy, which bd serve derives from its actual chain (externaldeps.Applied for the store arm, externaldeps.AppliedToProvider for the provider arm). A server handed raw roles still withholds it, so the token stays truthful. Against a server that advertises it, every remote role view hands the request to the server's own role untouched: no client validation, no edge reads, claim-next is the server's one atomic call, and the ready count is the server's countReadyWork (new remoteReadyCounter). The client-side policy remains only for servers that do not advertise the token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(create): default created_by to the request's actor in the create role created_by was stamped in two front doors and never in the library: the CLI (creationActorStamp, with a remote-workspace special case) and the three httpapi create handlers. A library caller of Lifecycle.Create or BatchCreator with an empty CreatedBy stored "" locally and the actor over http. PreparePublicCreateRequest, which every create shape prepares through (Lifecycle.Create, BatchCreator items, batch-apply create items), now defaults an empty CreatedBy to request.Actor; an explicit value is kept. The handlers no longer stamp, `bd create` and the state event create pass only the actor, and creationActorStamp/isRemoteBackendWorkspace are gone. The CLI sites that still create through the raw CreateIssue seam (swarm, todo, gate, the create form) keep stamping currentActor(): that seam is shared with import, where defaulting would re-attribute imported rows, and the http store refuses it anyway, so the remote special case was never reachable there. Conformance: the lifecycle-create, batch-creator and batch-apply contracts now pin created_by == actor for an item that names none, on every leg. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(httpclient): GetIssue answers a miss with storage.ErrNotFound, like the local stores The http store's raw GetIssue answered a missing id with (nil, nil), where DoltStore and the embedded store return a wrapped storage.ErrNotFound. A caller written against the backend contract read the http miss as a hit: `bd create --graph` refused every explicit-id plan over http (patched in cmd/bd on feat/http-mc-corpus; that workaround is reverted there). GetIssue now returns the miss (and the empty id) as storage.ErrNotFound, spelled as issueops.GetIssueInTx spells it. GetLabels, which reads through GetIssue, keeps the local read's "no labels" answer for a missing id; the `bd list --parent` walk reads the typed miss as "parent issue not found". The served bridge test now holds the local reference store and the http store to the same miss: ErrNotFound with the same text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(createbatchequiv): re-record the apply golden for the created_by default c2866ea7fe made the create role default an empty CreatedBy to the request's actor, so every create item of the apply scenario's batch (actor "apply-writer") now stores created_by "apply-writer" instead of "". TestCreateBatchFastPathsMatchPerRow_*/apply compared the stored rows to a digest recorded before that change and failed on exactly the three tables the column reaches: issues and wisps (the row) and events (the update event's old-value snapshot). Fast and per-row bodies still agree; the batch-apply conformance contract pins the same value. Bisected with bazel: passes at be92efe40f's integ twin 696f7acc8e, fails from the created_by commit on. Only apply.json's three digests move. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style: gofmt ledger.go and the externaldeps remote-roles test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmd/bd): bd create reads routing settings local-only on a remote backend too determineAutoRoutedRepoPath skipped a remote backend's settings document (routing targets are local paths that one shared document cannot name for every client), but getRoutingConfigValue, which `bd create` and migrate-personal read routing.* and contributor.* through, still asked it. The same workspace could route a create one way and a lookup another, and every create paid a settings round trip per key. Both now go through routingSettingsStore: nil for a remote backend, so routing comes from config.yaml / the environment only there, and a local store is still consulted. A unit test holds both reads to the rule with a store that counts settings reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(cmd/bd): say so when update -s closed drops an ambient CLAUDE_SESSION_ID Over http `bd update -s closed` cannot carry closed_by_session, so the CLI drops it rather than refusing the close. An explicit --session printed a notice; the ambient CLAUDE_SESSION_ID every agent under Claude Code exports was dropped without a word, a silent data difference from a local backend (the served corpus's mc-update-meta-status-closed). The drop is now named on stderr in both cases, once per command, pointing at `bd close`, whose wire operation records the session. The served e2e asserts the notice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(backend/http): hold bd list --limit 2 to each side's own unlimited order TestE2E_ListWithGCFiltersMatchesEmbedded compared the --limit 2 rows of the served and embedded workspaces as sets. The seed has four P2 rows, so the limit cuts a priority tie whose created_at order differs between two workspaces seeded seconds apart: a test-only flake, the order itself being total. Each side's two rows must now be, in order, the first two of that same side's --limit 0 listing; the unlimited listings are still compared across sides by the shape loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: unit-tier pins for the created_by default, the policy token and the GetIssue miss Three review mutations were caught only by a heavier tier (the uow conformance legs, the served httpclient tier) or by one package's test alone. Each now has a unit test where its rule lives: - issueops: PreparePublicCreateRequest defaults an empty CreatedBy to the actor and keeps an explicit one, without mutating the caller's request. - externaldeps: the policy layer AppliedToProvider finds becomes the handshake's policy.external_dependencies through httpapi.Listen, the composition bd serve performs, and a raw provider's handshake lacks it. - httpclient (no embedded engine): GetIssue over the real wire client against a stub server answers a 404 not_found, and the empty id without dialing, as storage.ErrNotFound spelled like the local stores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changelog): S6b/c/d entries match the code The S6 entry still said `bd create` suppresses created_by on a remote workspace and that the external-dependency policy runs client-side. Since S6b/c/d, created_by defaults to the actor in the library on every backend, and every bd serve composed through the policy advertises policy.external_dependencies, so clients pass those operations through; the client-side policy (now in role terms) runs only against a server that does not advertise it. New entries: the advertised policy token, bd ping over http, the created_by library default, closed_by_session dropped by an http `bd update -s closed` (with a notice, typed or ambient), reopen without a Provenance label over http, local-only routing.* / contributor.* on a remote backend, GetIssue's miss as storage.ErrNotFound, and list/children/show counts/dep tree over http. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(externaldeps): keep S6's list --ready, --claim-next refusal and lost-race error on the S6c composition The re-adopted S6 head (#7288) carries the review fixup's remote role composition; S6c replaces it with its own (rolesAreRemote, the widened ready window, the policy passthrough). Two behaviours S6 shipped and its reviews pinned would otherwise go: - bd close --claim-next against a server that does not apply the policy is refused whole, before anything is read or closed, with "the served batch close cannot carry a next claim, so nothing was closed; close without --claim-next, then run `bd ready --claim`". S6c passed the batch to the http closer, whose W-CloseBatchRequest.ClaimNext refusal drops the remedy, so TestE2E_BDOverHTTP's two refuseClaimNext steps went red. A request invalid on every backend is still ErrValidation first, and a server that applies the policy still gets the request untouched. - remoteReadyClaimer.ClaimNext reports a page on which every candidate was lost to another actor as "claim ready: lost N races ...", not as the empty answer that means there is no work. - `bd list --ready` (List with ReadyFlag) is answered by the policy store's own reader over the served ready reads, as S6 does. S6c passed it to the served listing, which cannot carry ReadyFlag (httpclient ledger row E-ListRequest.ReadyFlag), so it exited 1 over http with or without external refs; TestE2E_BDOverHTTP and TestServedRolesKeepTheExternalDependencyPolicy were red on it. TestRemoteReaderAnswersAReadyListingWithoutTheWireFlag pins it, with the fake's served List refusing ReadyFlag as the wire does. S6's composition tests now run against S6c as remote_roles_composition_test.go. Their fake serves Ready the way a server does (the S6 composition read ready work through the legacy policy reader instead), and two call logs gain that Ready. TestRemoteBatchCloserRefusesANextClaim, TestRemoteBatchCloserValidatesANextClaimFirst and TestRemoteReadyClaimerReportsLostRaces were red on S6c before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(externaldeps): unassigned claim-next and fail-closed close reads Review fixes for the S6c composition against a server that does not advertise policy.external_dependencies: the two majors of review attempt 1, plus the minors on the same code. - ReadyClaimer: the composed claim-next reads only open, unassigned ready rows (Unassigned=true, Assignee cleared), the candidates the local ClaimReadyIssueInTx selects. A row another actor holds, or one already assigned to the caller, is no longer taken or counted as a lost race. Each pass reads 25 candidates past the blocked rows, as httpclient's composed claim does, and a full window lost end to end is read once more: claimed rows leave the unassigned set. Only a claim that tried candidates and won none is an error. - BatchCloser: an externally blocked item travels to the server only when its status read finds it closed or not found (liveRefusal). Any other read error is that item's outcome, so a transient or auth failure no longer forwards a refused close to a server that will not refuse it. - TreeWalker: a failed capability probe renders the external leaves over the walked tree instead of failing a walk the server already answered. appendTreeExternalReferences skips an id the tree holds, so an enforcing server's own leaves are not doubled. - remoteEdges names each anchor once and skips an empty one, which the http EdgeReader refuses with ErrValidation. The composition fakes now honor Unassigned/Assignee as workapi.BuildReadyFilter does, model a rival claim landing first and a failing GetIssue, and refuse an empty edge anchor as httpclient does. New: TestRemoteReadyClaimerTakesOnlyUnassignedWork, TestRemoteReadyClaimerWalksAContendedFront, TestRemoteBatchCloserAnswersAFailedReadWithIt, TestRemoteEdgesNamesEachAnchorOnce, and the probe-failure cases of TestRemoteTreeWalkerAddsExternalLeaves. Each is red with its fix reverted. From review attempt 2, three nearby cleanups: - TestServedRolesKeepTheExternalDependencyPolicy seeds open rows held by another actor and by the claimant ahead of the free work, then claims with and without an assignee filter. The unassigned-only claim is now pinned over a real served server, not only through the fakes: red with the pre-fixup remote_roles.go, with filter.Unassigned dropped, and (the filtered claim) with an explicit assignee honored. - The capabilities description in openapi.v0.yaml (and the regenerated types.gen.go comment) no longer calls the whole list build-level: policy.external_dependencies depends on how the server composed its roles, as the paragraph above it says. - createbatchequiv's golden note names issueops.PreparePublicCreateRequest instead of a commit id that is not on this branch. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(backend/http): count a not_found read as reaching the remote in #7469's credential doors main's #7469 added TestExplicitCredentialIgnoresTheEnvPlaintextGrant. Its OpenWith and Open doors read bd-1 from plaintextRemote, which answers every issue read not_found, and the granted leg needs that read to return a nil error. That held while the http client answered a miss with (nil, nil). This branch's "fix(httpclient): GetIssue answers a miss with storage.ErrNotFound, like the local stores" makes the same read return storage.ErrNotFound. So on main plus this branch both doors fail at per_open_credential_test.go:392 ("with Target.AllowInsecureCredential: not found: issue bd-1") under plain go test. The merge is textually clean, so only a build of the merge shows it. The two store doors now count storage.ErrNotFound as success, since that answer came off the wire. The assertions are unchanged. The refused leg still needs the "refusing to send a credential" error and no request on the transport. The granted leg still needs exactly one request carrying "Bearer explicit-token": a granted read that never reaches the wire still fails at :408. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2410 lines
107 KiB
Go
2410 lines
107 KiB
Go
package httpapi
|
|
|
|
import (
|
|
"context"
|
|
"crypto/rand"
|
|
"encoding/hex"
|
|
"encoding/json"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
"log"
|
|
"maps"
|
|
"net"
|
|
"net/http"
|
|
"os"
|
|
"runtime/debug"
|
|
"slices"
|
|
"strconv"
|
|
"strings"
|
|
"sync"
|
|
"sync/atomic"
|
|
"time"
|
|
"unicode/utf8"
|
|
|
|
"golang.org/x/net/netutil"
|
|
|
|
"github.com/steveyegge/beads/internal/httpapi/apigen"
|
|
"github.com/steveyegge/beads/internal/storage"
|
|
"github.com/steveyegge/beads/internal/storage/domain"
|
|
"github.com/steveyegge/beads/internal/storage/uow"
|
|
"github.com/steveyegge/beads/issueops"
|
|
"github.com/steveyegge/beads/memoryops"
|
|
)
|
|
|
|
// APIVersion is the path major this package serves, reported as
|
|
// ContextResponse.api_version. It changes only when /v1 is cut.
|
|
const APIVersion = "v0"
|
|
|
|
// The operating envelope. Every one of these is a bound on how much of the
|
|
// process a client can occupy, and the two that matter most are the ones with
|
|
// no natural limit: without semAcquireTimeout a queue behind a wedged database
|
|
// grows without end, and without requestDeadline a request that got a slot
|
|
// never gives it back. Both are deliberately generous — this is a loopback
|
|
// service for automation clients, not a public endpoint — and all of them can
|
|
// become operator flags later without touching the wire.
|
|
const (
|
|
// maxInflight bounds handlers that touch the database. Every unit of work
|
|
// pins one SQL connection, so this is also the steady-state connection
|
|
// count.
|
|
//
|
|
// It also bounds a request body's decode memory, worst case: every
|
|
// inflight slot is free to decode a full maxApplyBatchBodyBytes body
|
|
// concurrently before either the large-apply semaphore or anything else
|
|
// narrows it further, so the raw-bytes ceiling for POST issues:batchApply
|
|
// alone is maxInflight * maxApplyBatchBodyBytes (16 * 16 MiB = 256 MiB at
|
|
// today's defaults), and the decoded Go structures (strings, maps, slices
|
|
// of *CreateItem etc.) cost a further multiple of that on top — operators
|
|
// sizing this process's memory limit should budget for it. This is not
|
|
// actively capped narrower than maxInflight itself: only one of those
|
|
// concurrent decodes can go on to hold the one-wide large-apply slot
|
|
// (acquireLargeApply), so a decode that loses the race is refused with
|
|
// ErrBusy and its now-unreferenced ApplyBatchRequest is left for the next
|
|
// GC cycle — nothing in this file extends its lifetime past that refusal.
|
|
maxInflight = 16
|
|
// maxConns bounds ACCEPTED connections. The semaphore does not: Go spawns
|
|
// a goroutine per connection, and one parked on a full semaphore still
|
|
// holds its goroutine, fd and buffers. Excess connections wait in the
|
|
// kernel accept backlog instead of in Go memory.
|
|
maxConns = 64
|
|
// semAcquireTimeout bounds the queue in TIME as well as width. A timed-out
|
|
// acquisition is the already-documented 503 busy, so shedding load
|
|
// introduces no new status vocabulary.
|
|
semAcquireTimeout = 10 * time.Second
|
|
// requestDeadline is the whole-request backstop, needed because
|
|
// WriteTimeout is 0 (below). It covers semaphore wait + unit of work +
|
|
// query, and deliberately not the response write.
|
|
requestDeadline = 60 * time.Second
|
|
// saturationWarn is how long a semaphore wait has to last before it is
|
|
// worth a log line of its own. This is the wedge-detection signal: /healthz
|
|
// stays green while the database is hung, so saturation events are what
|
|
// distinguish "wedged" from "no traffic".
|
|
saturationWarn = time.Second
|
|
// drainTimeout covers a claim inside its serialization-retry budget plus
|
|
// the commit, so a graceful shutdown does not kill a connection whose write
|
|
// may already have landed.
|
|
drainTimeout = 20 * time.Second
|
|
// drainGrace is added on top of an in-flight large apply's own remaining
|
|
// deadline when extending the drain budget, so the response has time to
|
|
// be written back to the client after that apply's transaction commits
|
|
// right at its deadline, instead of the drain ending the instant the
|
|
// deadline is reached.
|
|
drainGrace = 5 * time.Second
|
|
// uowCloseTimeout bounds the DETACHED close described on WithUOW.
|
|
uowCloseTimeout = 5 * time.Second
|
|
// DefaultLargeApplyCeiling is Config.LargeApplyCeiling's default: the
|
|
// whole-run budget a POST issues:batchApply request over
|
|
// largeApplyItemThreshold items gets once it holds the one-wide
|
|
// large-apply slot (acquireLargeApply). It is an operator-configurable
|
|
// CEILING rather than a per-request formula — every request that clears
|
|
// the threshold gets this whole budget from the moment it acquires the
|
|
// slot, whether it carries 101 items or 1000, because scaling it down for
|
|
// a request well under the envelope's top end would narrow a caller's
|
|
// budget for a reason unrelated to what it actually sent.
|
|
//
|
|
// It is exported so `bd serve --large-apply-ceiling` defaults to this
|
|
// same value instead of repeating it: bd serve always passes a positive
|
|
// ceiling, so a second literal there, not this constant, would be the
|
|
// default every bd serve deployment actually runs with.
|
|
DefaultLargeApplyCeiling = 5 * time.Minute
|
|
)
|
|
|
|
// Pool limits for the provider's *sql.DB. The semaphore bounds handlers, not
|
|
// connections: a poisoned connection replaced after a failed ROLLBACK, each
|
|
// retry attempt of a committing transaction (a fresh unit of work is a fresh
|
|
// pinned connection), and any semaphore-exempt handler that later touches the
|
|
// database all escape it.
|
|
var servePoolLimits = uow.PoolLimits{
|
|
MaxOpenConns: maxInflight + 4,
|
|
MaxIdleConns: maxInflight,
|
|
// Keep this below Dolt's supported 30s wait_timeout configuration. A
|
|
// longer server-specific override defeats the storage layer's stale-pool
|
|
// protection and lets the next request receive a server-reaped connection.
|
|
ConnMaxIdleTime: 20 * time.Second,
|
|
ConnMaxLifetime: time.Hour,
|
|
}
|
|
|
|
// HTTP-level timeouts. WriteTimeout is deliberately absent: `limit=0` means
|
|
// unlimited on both list operations, and a whole-response deadline would
|
|
// truncate a large body mid-write.
|
|
//
|
|
// writeStallTimeout is what replaces it — a deadline rolled forward before
|
|
// every write (statusWriter.extendWriteDeadline), which bounds a STALLED write
|
|
// without bounding total transfer. That bound is load-bearing, not hygiene:
|
|
// route() releases the database slot when the handler returns, and the handler
|
|
// returns only after writing the body, so without it a client that opens
|
|
// maxInflight requests and then stops reading pins every slot and its pinned
|
|
// connection until the process is restarted — while /healthz stays green. A
|
|
// context deadline cannot substitute: nothing cancels a blocked socket write.
|
|
//
|
|
// The read, header and idle timeouts bound request READING and keep-alive idle.
|
|
// They say nothing about a response write, and must not be cited as if they did.
|
|
const (
|
|
readHeaderTimeout = 10 * time.Second
|
|
readTimeout = 30 * time.Second
|
|
idleTimeout = 120 * time.Second
|
|
writeStallTimeout = 30 * time.Second
|
|
maxHeaderBytes = 64 << 10
|
|
)
|
|
|
|
// Config is everything the server needs to answer. It is assembled by the
|
|
// caller — the package resolves no workspace state of its own.
|
|
type Config struct {
|
|
// Addr is the host:port to bind. The host must be a numeric IP literal;
|
|
// see ValidateBindAddr.
|
|
Addr string
|
|
// AllowNonLoopback permits a bind beyond loopback. There is no TLS, so
|
|
// this is an operator decision that is never taken by default, and it
|
|
// requires either Auth or InsecureNoAuth — see ValidateAuthPosture.
|
|
AllowNonLoopback bool
|
|
// Auth verifies bearer credentials. NIL MEANS NO AUTHENTICATION, which is
|
|
// the pre-existing behavior and stays the default on loopback: a zero
|
|
// Config serves exactly what it served before this field existed.
|
|
Auth *TokenFileAuth
|
|
// InsecureNoAuth is the operator's explicit waiver for serving a
|
|
// non-loopback bind with no credential. It only ever permits; it never
|
|
// disables a configured Auth (that combination is refused).
|
|
InsecureNoAuth bool
|
|
// AllowedHosts are extra Host header values to answer to, beyond the
|
|
// loopback spellings and the bind address. In a cluster the client dials a
|
|
// service DNS name, which the rebinding defense would otherwise refuse;
|
|
// see newHostPolicy. Empty leaves today's policy exactly as it was.
|
|
AllowedHosts []string
|
|
// LargeApplyCeiling bounds how long POST /v0/beads/issues:batchApply may
|
|
// hold a write transaction for a request over largeApplyItemThreshold
|
|
// items (batch_apply.go). Zero takes DefaultLargeApplyCeiling (5
|
|
// minutes). It is wired to `bd serve --large-apply-ceiling`.
|
|
//
|
|
// It is a CEILING, not a target — see DefaultLargeApplyCeiling for why
|
|
// every request over the threshold gets the whole budget rather than a
|
|
// value scaled to its item count. Raising or lowering it is an
|
|
// operational decision, not a wire change: it does not move
|
|
// largeApplyItemThreshold, maxApplyBatchBodyBytes or
|
|
// issueops.MaxApplyBatchItems, which is why TestCapBatchApplyLargeTiesAllThreeLimits
|
|
// exists — a build that lowers this without also removing the
|
|
// issues.batchApplyLarge capability token would otherwise advertise a
|
|
// budget it no longer honors.
|
|
//
|
|
// The orchestrator's stop grace must be at least this long plus
|
|
// drainGrace (5s): Serve's drain extends its budget to an in-flight large
|
|
// apply's remaining deadline plus drainGrace, but a SIGKILL from outside
|
|
// this process does not consult it. See engdocs/SERVE_RUNBOOK.md.
|
|
LargeApplyCeiling time.Duration
|
|
// Provider is where every database-touching handler opens its one unit of
|
|
// work per request.
|
|
Provider uow.UnitOfWorkProvider
|
|
// The fields below are the roles this server answers from, for a backend
|
|
// whose facade is a STORE rather than a unit-of-work provider. sourceRoles
|
|
// is the authoritative list.
|
|
//
|
|
// Set them ALL, and only when Provider is nil: together they are the other
|
|
// complete database source, not an override of one. Listen refuses every
|
|
// other combination, including a partial set — a server missing one role
|
|
// would bind, answer every other route, and fail that one with a nil
|
|
// dereference inside a handler on a live server, which is worse than not
|
|
// starting.
|
|
//
|
|
// A caller with a store takes them off the store's own accessors, and WHICH
|
|
// store value it takes them off is the whole question. Every decorator a
|
|
// store wears is on the value its accessor returns — that is what the
|
|
// accessors are for — and bd's chain is
|
|
// HookFiringStore -> InstrumentedStorage -> raw, so `store.IssueClaimer()`
|
|
// there returns a claimer that runs the workspace's on_update hook script
|
|
// for every claim it lands. That is precisely what this server documents it
|
|
// does not do (cmd/bd/serve.go). Take the roles from BENEATH the hook layer:
|
|
//
|
|
// src := store
|
|
// if hooked, ok := src.(*storage.HookFiringStore); ok {
|
|
// src = hooked.Unwrap() // keeps the telemetry layer, drops the hooks
|
|
// }
|
|
// rd, err := src.IssueReader()
|
|
// cl, err := src.IssueClaimer()
|
|
// ... one per field below, all off the same src ...
|
|
// httpapi.Listen(httpapi.Config{Reader: rd, Claimer: cl, /* and the rest */})
|
|
//
|
|
// cmd/bd's serveIssueRoles is that loop, written out.
|
|
//
|
|
// Listen refuses a hook-firing role rather than trusting the paragraph
|
|
// above — see checkDatabaseSource.
|
|
//
|
|
// WHAT LISTEN CANNOT CHECK, and the caller therefore owns: each call must
|
|
// commit ON ITS OWN, atomically and durably. That is what this server's
|
|
// contract states per request, and nothing here can observe the commit
|
|
// protocol of the backend behind an interface — every check available would
|
|
// be a self-declaration by the same caller-supplied code being checked.
|
|
// Embedded Dolt is the backend that does not qualify (its commit runs
|
|
// outside the SQL transaction on a separate connection) and it is refused
|
|
// where the workspace is actually known: serveDatabaseSource in
|
|
// cmd/bd/serve.go.
|
|
//
|
|
// Unlike the provider path these are built ONCE, before Listen, rather than
|
|
// per request. The provider path rebuilds its roles per request for exactly
|
|
// one reason — so the units of work they open land in that request's uow_ms
|
|
// (see Server.reader) — and a role reached this way opens none through this
|
|
// server, so a rebuild would buy nothing.
|
|
Reader issueops.Reader
|
|
Claimer issueops.Claimer
|
|
// BatchCloser closes many issues as one transaction, behind
|
|
// POST /v0/beads/issues:batchClose. It is its own field rather than a mode
|
|
// of Lifecycle for the role's reason: the request is the transaction
|
|
// boundary, and a loop over Lifecycle.Close is N transactions.
|
|
BatchCloser issueops.BatchCloser
|
|
// ReadyClaimer is the atomic take of ready work, behind
|
|
// POST /v0/beads/issues:claimNext. It is its own field rather than a second
|
|
// verb on Claimer for the reason the role is its own interface: the caller
|
|
// names a QUESTION and the implementation picks the answer, so selection is
|
|
// part of the operation and not a patch.
|
|
ReadyClaimer issueops.ReadyClaimer
|
|
// Releaser is the claim's inverse, behind
|
|
// POST /v0/beads/issues/{id}:release. It is its own field rather than a
|
|
// method on Claimer for the reason the role is its own interface: a caller
|
|
// entitled to give its own work back is very often not entitled to take
|
|
// new work, so a surface carrying both hands out a capability it should not
|
|
// be able to reach.
|
|
Releaser issueops.Releaser
|
|
// Lifecycle is the guarded-mutation role behind the issue lifecycle
|
|
// operations. Required on the same terms as every field here, and the
|
|
// hook-firing refusal below bites hardest on it: a store's own
|
|
// IssueLifecycle() returns a role that fires on_create, on_update and the
|
|
// close hooks for every mutation it lands.
|
|
Lifecycle issueops.Lifecycle
|
|
Settings issueops.WorkspaceConfig
|
|
Stats issueops.StatsReporter
|
|
CycleDetector issueops.CycleDetector
|
|
EdgeReader issueops.EdgeReader
|
|
// GraphCounter is the edge-count role behind
|
|
// GET /v0/beads/dependencies:count. It is a SEPARATE field from EdgeReader
|
|
// for the reason Counter is separate from ReadyCounter: that role answers
|
|
// with the edge ROWS in one direction, this one with a number in either,
|
|
// and neither can answer the other's question. Required on the same terms
|
|
// as every field here.
|
|
GraphCounter issueops.GraphCounter
|
|
// BatchGetter is the one-snapshot, many-ids read behind
|
|
// POST /v0/beads/issues:batchGet. It is a SEPARATE field from Reader for
|
|
// the reason EdgeReader and GraphCounter are separate from each other:
|
|
// Reader.Get answers ONE id with *IssueDetails or ErrNotFound, and this
|
|
// answers MANY ids with a result whose Missing field reports an absent row
|
|
// rather than refusing the call. Required on the same terms as every field
|
|
// here.
|
|
BatchGetter issueops.BatchGetter
|
|
// LeaseReclaimer is the stale-lease sweep behind
|
|
// POST /v0/beads/issues:reclaim. It is a SEPARATE field from Releaser for
|
|
// the reason issueops.LeaseReclaimer's own doc gives: Releaser gives up ONE
|
|
// claim on its holder's say-so, and this reverts MANY on a clock's.
|
|
// Required on the same terms as every field here.
|
|
LeaseReclaimer issueops.LeaseReclaimer
|
|
// Relations is the single-anchor neighbor read behind
|
|
// GET /v0/beads/issues/{id}/related. It is a SEPARATE field from EdgeReader
|
|
// for the reason issueops.EdgeReader's own doc gives at length: that role
|
|
// answers with the edge ROWS for many anchors and reports a miss per anchor,
|
|
// this one answers with the hydrated ISSUES on the far end for ONE anchor and
|
|
// answers ErrNotFound. Different answer shape, different miss policy,
|
|
// different arity. Required on the same terms as every field here.
|
|
Relations issueops.Relations
|
|
// Commenter is the append-one-comment role behind
|
|
// POST /v0/beads/issues/{id}/comments. It is its own field rather than a
|
|
// verb on Lifecycle for the role's own reason: a comment is not a patch to
|
|
// an issue — it appends a row to a thread the issue owns and leaves every
|
|
// field of the issue untouched, so an IssuePatch has nothing to carry and an
|
|
// UpdateResult has nowhere to put the comment. Required on the same terms as
|
|
// every field here.
|
|
Commenter issueops.Commenter
|
|
BlockingAnnotator issueops.BlockingAnnotator
|
|
TreeWalker issueops.TreeWalker
|
|
ReadyCounter issueops.ReadyCounter
|
|
// Counter is the issue-count role behind GET /v0/beads/issues:count. It is
|
|
// a SEPARATE field from ReadyCounter because it is a separate role: that
|
|
// one sizes the ready predicate, this one sizes a filter, and neither can
|
|
// answer the other's question. Required on the same terms as every field
|
|
// here.
|
|
Counter issueops.Counter
|
|
Querier issueops.Querier
|
|
// Sweeper is the DESTRUCTIVE one, required on the same terms as every other
|
|
// role rather than opt-in: whether this build erases beads is a decision
|
|
// for the operator who chose to run bd serve, not a consequence of whether
|
|
// a caller remembered a field.
|
|
Sweeper issueops.Sweeper
|
|
// Deleter is the OTHER destructive one, required for the same reason.
|
|
Deleter issueops.Deleter
|
|
BatchCreator issueops.BatchCreator
|
|
// DependencyEditor is the graph's write side. Required on the same terms as
|
|
// the two destructive roles above: whether this build can rewire the
|
|
// dependency graph is a decision for the operator who chose to run bd serve,
|
|
// not a consequence of whether a caller remembered a field.
|
|
DependencyEditor issueops.DependencyEditor
|
|
// MetadataCAS is the conditional single-key metadata write behind
|
|
// POST /v0/beads/issues/{id}:casMetadata. Required on the same terms as
|
|
// every other role: a Config missing it would bind and nil-dereference on
|
|
// the first request that reached that handler.
|
|
MetadataCAS issueops.MetadataCAS
|
|
// BatchApplier is the ordered, heterogeneous plan behind
|
|
// POST /v0/beads/issues:batchApply. Required on the same terms as every
|
|
// other role: a Config missing it would bind and nil-dereference on the
|
|
// first request that reached that handler.
|
|
//
|
|
// It is the field where the hook-firing refusal below bites HARDEST. A
|
|
// store's own accessor returns an applier that fires on_create, on_update
|
|
// AND the close hooks — once per landed item, plus once per distinct edge
|
|
// source — so a single hundred-item request served unpeeled would run a
|
|
// hundred of the workspace's subprocesses inside one HTTP call.
|
|
BatchApplier issueops.BatchApplier
|
|
// Memories is the workspace's persistent memory plane, and the one field
|
|
// here that is not an issueops role: memories are user data riding in the
|
|
// config table under their own merge class, not settings, so they have
|
|
// their own leaf package. Required on the same terms as every field above —
|
|
// a partial set is refused, so the field and the operations that reach it
|
|
// land together.
|
|
Memories memoryops.Memories
|
|
// EventsJournal is the durable mutation journal's READ side, and the ONE
|
|
// role here that is required CONDITIONALLY — which is why it is absent from
|
|
// sourceRoles and checked on its own. Like Memories it is not an issueops
|
|
// role: the journal is a replay feed over engine state on a dolt_ignored
|
|
// table, not a bead query, so it has its own leaf package (journalops,
|
|
// whose doc states why). storage.EventsJournalCursor is an ALIAS of
|
|
// journalops.Journal, so this field names the role whichever spelling
|
|
// reaches it.
|
|
//
|
|
// Required exactly when EventsJournalEnabled. A workspace that records
|
|
// nothing needs no reader, and a storage backend that cannot read the
|
|
// journal at all is a perfectly ordinary backend as long as nobody asked it
|
|
// to journal — the same deal eventsjournal.Apply takes when it binds
|
|
// activation to a store. Demanding it unconditionally would make a
|
|
// capability nothing uses a precondition for running the server.
|
|
//
|
|
// It is the READ role and deliberately NOT the wider
|
|
// storage.EventsJournalAccessor the CLI takes. That interface also prunes,
|
|
// and a server that is documented to publish the journal and never retain
|
|
// it should not be holding a delete it merely promises not to call.
|
|
EventsJournal storage.EventsJournalCursor
|
|
// EventsJournalEnabled is whether the served workspace actually records
|
|
// mutations, resolved by the caller through eventsjournal.EnabledFor.
|
|
//
|
|
// It is a resolved BOOLEAN rather than something this package works out,
|
|
// for the reason Config gives at the top: activation reads the target
|
|
// workspace's own config.yaml and the BD_EVENTS_JOURNAL environment
|
|
// override, which is workspace state, and this package resolves none.
|
|
//
|
|
// It cannot be inferred from the data either, which is the whole reason the
|
|
// field exists. A disabled journal presents as zero rows and a head of
|
|
// zero — byte-identical to an enabled journal nothing has written yet — so
|
|
// a server without this flag would answer "you are caught up" to a consumer
|
|
// polling a workspace that will never emit a record.
|
|
EventsJournalEnabled bool
|
|
// ExternalDependencyPolicy reports that every ready, claim, claim-next,
|
|
// close, blocking-annotation and tree role this server answers from — the
|
|
// Provider's, or the individual role fields' — was composed through bd's
|
|
// external-dependency policy (internal/storage/externaldeps). It makes the
|
|
// handshake advertise CapExternalDependencies, which tells a client to
|
|
// pass those operations through without applying the policy itself.
|
|
//
|
|
// Set it only from the composition actually handed in (cmd/bd's serve
|
|
// derives it with externaldeps.Applied / externaldeps.AppliedToProvider):
|
|
// a server that advertises the token without enforcing the policy makes
|
|
// every client skip it.
|
|
ExternalDependencyPolicy bool
|
|
// Workspace is the startup snapshot GET /v0/beads/context answers from.
|
|
// Only the allowlisted fields are ever serialized — see contextResponse,
|
|
// which names the whole set and the reasons for the exclusions.
|
|
Workspace domain.ContextInfo
|
|
// SchemaVersion is the CLI's stdout JSON envelope version, reported for
|
|
// diagnostics. Clients are told not to branch on it.
|
|
SchemaVersion int
|
|
// Mode names the resolved storage topology ("proxied", "external") for the
|
|
// startup log line. Cosmetic: nothing dispatches on it.
|
|
Mode string
|
|
// Stdout receives exactly one line, the bound address, so a caller that
|
|
// asked for an ephemeral port can discover it. Stderr receives the
|
|
// operational log. Both default to the process streams.
|
|
Stdout io.Writer
|
|
Stderr io.Writer
|
|
}
|
|
|
|
// Server is one bound listener and the routes behind it. Build it with Listen,
|
|
// which binds before returning so the caller can read Addr, then run Serve.
|
|
type Server struct {
|
|
cfg Config
|
|
provider uow.UnitOfWorkProvider
|
|
|
|
// The configured roles, set exactly when provider is nil. They are what
|
|
// reader(), claimer(), cycleDetector() and the rest of those accessors hand
|
|
// back on the store-shaped source; the field names differ from the method
|
|
// names because a struct cannot carry both.
|
|
issueReader issueops.Reader
|
|
issueClaimer issueops.Claimer
|
|
issueBatchCloser issueops.BatchCloser
|
|
issueReadyClaimer issueops.ReadyClaimer
|
|
issueReleaser issueops.Releaser
|
|
issueLifecycle issueops.Lifecycle
|
|
settings issueops.WorkspaceConfig
|
|
issueStats issueops.StatsReporter
|
|
issueCycles issueops.CycleDetector
|
|
issueEdges issueops.EdgeReader
|
|
issueEdgeCounter issueops.GraphCounter
|
|
issueBatchGetter issueops.BatchGetter
|
|
issueReclaimer issueops.LeaseReclaimer
|
|
issueRelations issueops.Relations
|
|
issueCommenter issueops.Commenter
|
|
issueBlocking issueops.BlockingAnnotator
|
|
issueTree issueops.TreeWalker
|
|
issueReadyCounter issueops.ReadyCounter
|
|
issueCounter issueops.Counter
|
|
issueQuerier issueops.Querier
|
|
issueSweeper issueops.Sweeper
|
|
issueDeleter issueops.Deleter
|
|
issueBatchCreator issueops.BatchCreator
|
|
issueDependencies issueops.DependencyEditor
|
|
issueMetadataCAS issueops.MetadataCAS
|
|
issueBatchApplier issueops.BatchApplier
|
|
workspaceMemories memoryops.Memories
|
|
eventsJournal storage.EventsJournalCursor
|
|
|
|
listener net.Listener
|
|
http *http.Server
|
|
|
|
// sem bounds handlers that touch the database. Buffered channel rather
|
|
// than sync.Semaphore so the acquisition can select on a timer.
|
|
sem chan struct{}
|
|
// largeApplySem is the ONE-WIDE "large write" semaphore: a POST
|
|
// issues:batchApply request carrying more than largeApplyItemThreshold
|
|
// items holds this slot for the whole call, so at most one oversized
|
|
// batch-apply transaction runs at a time no matter how many ordinary
|
|
// requests sem admits concurrently. It is separate from sem (which
|
|
// bounds ALL database-touching handlers) because the two answer
|
|
// different questions: sem is "how many handlers may run", this is "how
|
|
// many may hold a write transaction open for minutes". A small apply
|
|
// never touches it.
|
|
//
|
|
// The wait to acquire this slot is itself BOUNDED — see
|
|
// acquireLargeApply — deliberately: a goroutine queued here still holds
|
|
// its sem slot (route() acquires sem before the handler ever runs), so
|
|
// an unbounded wait here would starve every other request behind a full
|
|
// sem the instant enough large applies queued up.
|
|
largeApplySem chan struct{}
|
|
// largeApplyCeiling is the resolved Config.LargeApplyCeiling (defaulted
|
|
// via orDefault to DefaultLargeApplyCeiling). acquireLargeApply builds
|
|
// each run's deadline from this, fresh, at the moment the slot is
|
|
// acquired — never before, so time spent queued is never charged against
|
|
// the run itself.
|
|
largeApplyCeiling time.Duration
|
|
// largeApplyDeadline is the wall-clock deadline of the large apply
|
|
// currently holding largeApplySem, or nil when none is in flight.
|
|
// Serve's graceful-drain shutdown reads it to decide how long to wait
|
|
// for that request rather than cutting it off at the ordinary
|
|
// drainTimeout: a graceful drain waits for an in-flight large apply up to
|
|
// its own deadline, and — because acquireLargeApply refuses a NEW
|
|
// acquisition once s.closing fires — that in-flight request is the only
|
|
// one the budget ever has to cover.
|
|
largeApplyDeadline atomic.Pointer[time.Time]
|
|
// largeApplyWaiting gates the BOUNDED wait in acquireLargeApply to at
|
|
// most one goroutine at a time. Without this, every large request beyond
|
|
// the one holding largeApplySem would still queue for up to semTimeout
|
|
// waiting on the slot — and each queued goroutine holds its own sem slot
|
|
// the whole time it waits (route() acquires sem before the handler runs),
|
|
// so a burst of large requests could occupy up to semTimeout worth of
|
|
// general concurrency slots simply queueing, starving ordinary small
|
|
// requests for that whole window even though only one of them could ever
|
|
// win the large-apply slot. A second large request arriving while one is
|
|
// already queued is refused immediately (ErrBusy, no wait at all) instead
|
|
// of queueing behind it.
|
|
largeApplyWaiting atomic.Bool
|
|
// auth is nil on an unauthenticated server, which is the loopback default.
|
|
auth *TokenFileAuth
|
|
// semTimeout, semWarn, writeStall, watchPoll, watchBeat and drainTimeout
|
|
// default to the constants above. They are fields rather than constants
|
|
// at the point of use so the queueing, stalled-write, streaming and
|
|
// drain behavior can be exercised in milliseconds instead of tens of
|
|
// seconds.
|
|
semTimeout time.Duration
|
|
semWarn time.Duration
|
|
writeStall time.Duration
|
|
watchPoll time.Duration
|
|
watchBeat time.Duration
|
|
// minClientWireRevision is the field checkWireRevision enforces against.
|
|
// It starts at the package constant MinClientWireRevision and exists as
|
|
// a field, rather than a direct read of that constant, only so a test can
|
|
// raise the floor in milliseconds the way it shrinks semTimeout above —
|
|
// production never overrides it. Unlike semTimeout et al it is an int
|
|
// with no zero-value ambiguity to paper over with orDefault: the
|
|
// constant's own real-world default is 0, so a freshly zeroed Server
|
|
// behaves identically whether or not Listen ran yet.
|
|
minClientWireRevision int
|
|
// drainTimeout overrides the package-level drainTimeout constant used as
|
|
// drainBudget's floor. A test shrinks this to prove Serve's drain
|
|
// actually EXTENDS past it for an in-flight large apply, instead of
|
|
// merely asserting the pure drainBudget helper's return value — see
|
|
// TestServeExtendsDrainPastAFixedFloorForAnInFlightLargeApply.
|
|
drainTimeout time.Duration
|
|
|
|
// closing is closed when a graceful shutdown begins, and it is the ONLY
|
|
// notice a streaming handler gets: http.Server.Shutdown waits for active
|
|
// requests without canceling their contexts, so a stream that watched only
|
|
// for a client disconnect would hold every shutdown open for the whole drain
|
|
// timeout and then be killed anyway. Registered on the http.Server (which
|
|
// calls it exactly once) and guarded so a second call cannot close twice.
|
|
closing chan struct{}
|
|
closingOnce sync.Once
|
|
|
|
// watchStreams is the live count of open events:watch streams and
|
|
// maxWatchStreams mirrors the constant, so a test can saturate the cap
|
|
// without opening sixty-four of them.
|
|
watchStreams atomic.Int64
|
|
maxWatchStreams int
|
|
|
|
log *log.Logger
|
|
stdout io.Writer
|
|
ctxBody apigen.ContextResponse
|
|
|
|
// hosts is the Host-header allowlist, the DNS-rebinding defense. It is
|
|
// derived from the bind address and there is no configuration that turns it
|
|
// off; see newHostPolicy.
|
|
hosts hostPolicy
|
|
|
|
idPrefix string
|
|
idSeq atomic.Uint64
|
|
|
|
// maxConns mirrors the constant so a test can exercise the cap without
|
|
// opening 64 sockets. liveConns is the accepted-connection gauge, reported
|
|
// on every request line; connCapWarned makes the saturation event
|
|
// edge-triggered rather than once per connection.
|
|
maxConns int
|
|
liveConns atomic.Int64
|
|
connCapWarned atomic.Bool
|
|
}
|
|
|
|
// ValidateBindAddr enforces the bind posture, following the policy the managed
|
|
// Dolt child already lives under (validateManagedServerConfigPolicy in
|
|
// cmd/bd/proxied_server.go): the host must be a NUMERIC IP literal.
|
|
//
|
|
// Hostnames are refused, "localhost" included. A name is not a listener
|
|
// specification — it resolves to whatever the host's resolver says today, so
|
|
// the operator cannot tell from the flag which interfaces they just opened.
|
|
// Unix sockets are not supported at all; they fail here because they do not
|
|
// parse as host:port.
|
|
func ValidateBindAddr(addr string, allowNonLoopback bool) (net.IP, error) {
|
|
host, port, err := net.SplitHostPort(addr)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("--addr %q must be HOST:PORT with a numeric IP literal host (unix sockets are not supported): %w", addr, err)
|
|
}
|
|
if _, err := strconv.ParseUint(port, 10, 16); err != nil {
|
|
return nil, fmt.Errorf("--addr %q: port must be a number from 0 to 65535 (0 picks an ephemeral port)", addr)
|
|
}
|
|
ip := net.ParseIP(host)
|
|
if ip == nil {
|
|
return nil, fmt.Errorf("--addr %q: host must be a numeric IP literal, not a name — use 127.0.0.1 rather than localhost", addr)
|
|
}
|
|
if !ip.IsLoopback() && !allowNonLoopback {
|
|
return nil, fmt.Errorf("--addr %q binds beyond loopback, which requires --allow-non-loopback (and, with it, --auth-token-file)", addr)
|
|
}
|
|
return ip, nil
|
|
}
|
|
|
|
// Listen validates the configuration, binds the listener, and reports the
|
|
// bound address on stdout and the startup state on stderr. It does not accept
|
|
// anything until Serve runs.
|
|
//
|
|
// There is no lock file, pid file or discovery file: bd serve is
|
|
// operator-invoked and the TCP bind IS the mutual exclusion, so a second
|
|
// instance on the same fixed port fails here with the operating system's own
|
|
// address-in-use error. (Under the ephemeral default that exclusion does not
|
|
// exist — N instances simply run on N ports — which is why fixed ports are the
|
|
// deployment recommendation.)
|
|
func Listen(cfg Config) (*Server, error) {
|
|
if err := checkDatabaseSource(cfg); err != nil {
|
|
return nil, err
|
|
}
|
|
ip, err := ValidateBindAddr(cfg.Addr, cfg.AllowNonLoopback)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
// The same posture and allowlist rules the CLI applies, applied again here
|
|
// so a second caller of this package cannot assemble a Config that serves
|
|
// the whole surface to a network with no credential.
|
|
if err := ValidateAuthPosture(cfg.AllowNonLoopback, cfg.Auth != nil, cfg.InsecureNoAuth); err != nil {
|
|
return nil, err
|
|
}
|
|
for _, host := range cfg.AllowedHosts {
|
|
if err := ValidateAllowedHost(host); err != nil {
|
|
return nil, err
|
|
}
|
|
}
|
|
if cfg.Stdout == nil {
|
|
cfg.Stdout = os.Stdout
|
|
}
|
|
if cfg.Stderr == nil {
|
|
cfg.Stderr = os.Stderr
|
|
}
|
|
|
|
prefix, err := newIDPrefix()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
s := &Server{
|
|
cfg: cfg,
|
|
provider: cfg.Provider,
|
|
issueReader: cfg.Reader,
|
|
issueClaimer: cfg.Claimer,
|
|
issueBatchCloser: cfg.BatchCloser,
|
|
issueReadyClaimer: cfg.ReadyClaimer,
|
|
issueReleaser: cfg.Releaser,
|
|
issueLifecycle: cfg.Lifecycle,
|
|
settings: cfg.Settings,
|
|
issueStats: cfg.Stats,
|
|
issueCycles: cfg.CycleDetector,
|
|
issueEdges: cfg.EdgeReader,
|
|
issueEdgeCounter: cfg.GraphCounter,
|
|
issueBatchGetter: cfg.BatchGetter,
|
|
issueReclaimer: cfg.LeaseReclaimer,
|
|
issueRelations: cfg.Relations,
|
|
issueCommenter: cfg.Commenter,
|
|
issueBlocking: cfg.BlockingAnnotator,
|
|
issueTree: cfg.TreeWalker,
|
|
issueReadyCounter: cfg.ReadyCounter,
|
|
issueCounter: cfg.Counter,
|
|
issueQuerier: cfg.Querier,
|
|
issueSweeper: cfg.Sweeper,
|
|
issueDeleter: cfg.Deleter,
|
|
issueBatchCreator: cfg.BatchCreator,
|
|
issueDependencies: cfg.DependencyEditor,
|
|
issueMetadataCAS: cfg.MetadataCAS,
|
|
issueBatchApplier: cfg.BatchApplier,
|
|
workspaceMemories: cfg.Memories,
|
|
eventsJournal: cfg.EventsJournal,
|
|
|
|
sem: make(chan struct{}, maxInflight),
|
|
semTimeout: semAcquireTimeout,
|
|
semWarn: saturationWarn,
|
|
minClientWireRevision: MinClientWireRevision,
|
|
largeApplySem: make(chan struct{}, 1),
|
|
largeApplyCeiling: orDefault(cfg.LargeApplyCeiling, DefaultLargeApplyCeiling),
|
|
|
|
closing: make(chan struct{}),
|
|
maxWatchStreams: maxWatchStreams,
|
|
|
|
log: log.New(cfg.Stderr, "bd serve: ", log.LstdFlags|log.LUTC),
|
|
stdout: cfg.Stdout,
|
|
ctxBody: contextResponse(cfg.Workspace, cfg.SchemaVersion, advertisedCapabilities(cfg)),
|
|
hosts: newHostPolicy(ip, cfg.AllowedHosts),
|
|
auth: cfg.Auth,
|
|
idPrefix: prefix,
|
|
maxConns: maxConns,
|
|
}
|
|
|
|
ln, err := net.Listen("tcp", cfg.Addr)
|
|
if err != nil {
|
|
return nil, fmt.Errorf("bind %s: %w", cfg.Addr, err)
|
|
}
|
|
s.listener = netutil.LimitListener(ln, s.maxConns)
|
|
|
|
s.http = &http.Server{
|
|
Handler: s.handler(),
|
|
ReadHeaderTimeout: readHeaderTimeout,
|
|
ReadTimeout: readTimeout,
|
|
IdleTimeout: idleTimeout,
|
|
MaxHeaderBytes: maxHeaderBytes,
|
|
ErrorLog: log.New(cfg.Stderr, "bd serve: http: ", log.LstdFlags|log.LUTC),
|
|
ConnState: s.connState,
|
|
}
|
|
// Tell the streams to wind up as soon as a drain starts. Without it a
|
|
// graceful shutdown waits out the whole drain timeout on any open stream and
|
|
// then reports itself forced, which is the one shutdown signal an operator
|
|
// is meant to be able to trust.
|
|
s.http.RegisterOnShutdown(s.closeStreams)
|
|
|
|
// Bound what a burst of requests can open on the database. The knob is
|
|
// optional on the interface, so say so out loud when a provider does not
|
|
// carry it rather than silently running unbounded.
|
|
//
|
|
// Nothing to bound on the roles source: the pool belongs to whatever the
|
|
// backend is, and this server neither owns it nor can reach it. Saying the
|
|
// knob is "unavailable" there would report a missing capability for a
|
|
// provider that was never asked for.
|
|
if cfg.Provider != nil {
|
|
if tuner, ok := cfg.Provider.(uow.PoolTuner); ok {
|
|
tuner.SetPoolLimits(servePoolLimits)
|
|
} else {
|
|
s.event("pool_limits_unavailable", "provider", fmt.Sprintf("%T", cfg.Provider))
|
|
}
|
|
}
|
|
|
|
fmt.Fprintf(s.stdout, "bd serve: listening on http://%s\n", s.Addr())
|
|
s.logStartup()
|
|
return s, nil
|
|
}
|
|
|
|
// checkDatabaseSource enforces exactly one complete database source.
|
|
//
|
|
// There are two, and a Config carries one or the other: a unit-of-work
|
|
// provider, or the roles this surface answers from (sourceRoles). A PARTIAL
|
|
// set is refused with the same message as none at all, because it is the same
|
|
// mistake and the failure it would otherwise produce is the worst shape
|
|
// available — a Config missing one role binds, answers every other route, and
|
|
// fails that one with a nil dereference in a handler on a live server.
|
|
//
|
|
// The set GROWS as this surface grows: every operation added here is an
|
|
// operation a roles-backed deployment must be able to answer, so a role added
|
|
// to the set turns "this build serves an operation your Config cannot answer"
|
|
// into a startup error instead of a 500 on the first client that finds it.
|
|
//
|
|
// Both together is refused rather than resolved by precedence: a caller that
|
|
// set both holds two different opinions about where this server reads from, and
|
|
// silently honoring one of them leaves the other as configuration that looks
|
|
// live and is not.
|
|
//
|
|
// The last refusal is the one a caller does not see coming. A store wears
|
|
// decorators and its accessors hand them out — that is what the accessors are
|
|
// FOR — so the obvious `store.IssueClaimer()` on bd's own storage chain returns
|
|
// a claimer that fires the workspace's on_update hook for every claim it lands.
|
|
// This server's contract says hooks do not fire (cmd/bd/serve.go), and a
|
|
// contract broken by the caller's most natural line is not a contract. Refusing
|
|
// at Listen is the difference between a startup error naming the store to take
|
|
// roles from and a server that has been quietly running a user's subprocess per
|
|
// claim since it booted.
|
|
|
|
// sourceRoles is the store-shaped source's roles in ONE place, so the three
|
|
// questions checkDatabaseSource asks — is any set, is any missing, does any
|
|
// fire hooks — cannot drift apart as the set grows. An operation that reaches a
|
|
// role this source does not yet carry adds a line here and a line to
|
|
// roleSourceNames, and nothing else in this file.
|
|
//
|
|
// A role is compared against nil as an INTERFACE, which is what the caller
|
|
// actually sets; a typed nil stored in one of these fields is a value as far as
|
|
// this check is concerned.
|
|
//
|
|
// It carries only the roles every deployment must have. EventsJournal is
|
|
// deliberately NOT here: it is required only when the workspace's journal is
|
|
// enabled, and folding a conditional field into a set whose whole value is
|
|
// "all or nothing" would turn an honest condition into a special case inside
|
|
// three functions. It is checked once, on its own, below.
|
|
func sourceRoles(cfg Config) []any {
|
|
return []any{cfg.Reader, cfg.Claimer, cfg.ReadyClaimer, cfg.Releaser, cfg.Lifecycle, cfg.BatchCloser, cfg.Settings, cfg.Stats, cfg.CycleDetector, cfg.EdgeReader, cfg.GraphCounter, cfg.BatchGetter, cfg.LeaseReclaimer, cfg.Relations, cfg.Commenter, cfg.BlockingAnnotator, cfg.TreeWalker, cfg.ReadyCounter, cfg.Counter, cfg.Querier, cfg.Sweeper, cfg.Deleter, cfg.BatchCreator, cfg.DependencyEditor, cfg.BatchApplier, cfg.Memories, cfg.MetadataCAS}
|
|
}
|
|
|
|
// roleSourceNames spells sourceRoles for the refusal message, in the same
|
|
// order, so a caller reading the error learns the whole set it must pass.
|
|
const roleSourceNames = "Reader, Claimer, ReadyClaimer, Releaser, Lifecycle, BatchCloser, Settings, Stats, CycleDetector, EdgeReader, GraphCounter, BatchGetter, LeaseReclaimer, Relations, Commenter, BlockingAnnotator, TreeWalker, ReadyCounter, Counter, Querier, Sweeper, Deleter, BatchCreator, DependencyEditor, BatchApplier, Memories and MetadataCAS"
|
|
|
|
func anyRoleSet(cfg Config) bool {
|
|
return slices.ContainsFunc(sourceRoles(cfg), func(r any) bool { return r != nil })
|
|
}
|
|
|
|
func everyRoleSet(cfg Config) bool {
|
|
return !slices.Contains(sourceRoles(cfg), nil)
|
|
}
|
|
|
|
func anyRoleFiresHooks(cfg Config) bool {
|
|
return slices.ContainsFunc(sourceRoles(cfg), storage.RoleFiresHooks)
|
|
}
|
|
|
|
func checkDatabaseSource(cfg Config) error {
|
|
switch {
|
|
case cfg.Provider != nil && (anyRoleSet(cfg) || cfg.EventsJournal != nil):
|
|
return errors.New("httpapi: both a unit-of-work provider and issue roles were set; pass exactly one database source")
|
|
case cfg.Provider == nil && !everyRoleSet(cfg):
|
|
return errors.New("httpapi: no database source: set Provider, or " + roleSourceNames + " together")
|
|
// The conditional role, checked where every other configuration mistake is.
|
|
// A workspace that HAS a journal and a server that cannot read it is the
|
|
// one combination that would bind, answer every other route, and fail this
|
|
// one — with a nil dereference, which is the shape checkDatabaseSource
|
|
// exists to prevent. A workspace with the journal off needs no reader and
|
|
// this says nothing about it.
|
|
case cfg.Provider == nil && cfg.EventsJournalEnabled && cfg.EventsJournal == nil:
|
|
return errors.New("httpapi: this workspace's events journal is enabled but no EventsJournal reader was configured; " +
|
|
"take one off the store (storage.EventsJournalCursor), or serve a workspace with the journal off")
|
|
case anyRoleFiresHooks(cfg):
|
|
return errors.New("httpapi: a configured role fires this workspace's hooks; " +
|
|
"this server does not run hooks, so take the roles from the store beneath the hook decorator " +
|
|
"((*storage.HookFiringStore).Unwrap)")
|
|
case uow.ProviderFiresHooks(cfg.Provider):
|
|
// The same refusal for the other database source. A provider's roles
|
|
// carry whatever the provider carries, so a hook-firing one would run a
|
|
// user's subprocess per served mutation just as a hook-firing role does.
|
|
return errors.New("httpapi: the configured provider fires this workspace's hooks; " +
|
|
"this server does not run hooks, so pass the provider beneath the hook layer " +
|
|
"(uow.UnwrapProvider)")
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// Addr is the bound address, which is the only way to discover the port under
|
|
// the ephemeral default.
|
|
func (s *Server) Addr() string { return s.listener.Addr().String() }
|
|
|
|
// drainBudget computes how long Serve should wait for in-flight work to
|
|
// finish once shutdown begins. It is a pure function, pulled out of Serve so
|
|
// the EXTENSION behavior described on Serve's doc comment — a large apply
|
|
// still in flight gets AT LEAST the remainder of its own deadline, not just
|
|
// the ordinary drainTimeout — can be pinned directly by a fast unit test
|
|
// (TestDrainBudget) instead of only indirectly, via a real multi-second wait.
|
|
//
|
|
// base is the floor Serve uses when no large apply is in flight — ordinarily
|
|
// the package-level drainTimeout constant, but overridable per-Server (see
|
|
// Server.drainTimeout) so a test can shrink it and observe Serve's ACTUAL
|
|
// drain genuinely outlasting that floor for an in-flight large apply, rather
|
|
// than only asserting this helper's return value in isolation.
|
|
//
|
|
// largeApplyDeadline is s.largeApplyDeadline.Load()'s result: nil when no
|
|
// large apply holds the slot at the moment shutdown starts.
|
|
func drainBudget(base time.Duration, largeApplyDeadline *time.Time) time.Duration {
|
|
budget := base
|
|
if largeApplyDeadline != nil {
|
|
if remaining := time.Until(*largeApplyDeadline); remaining > budget {
|
|
budget = remaining + drainGrace
|
|
}
|
|
}
|
|
return budget
|
|
}
|
|
|
|
// Serve accepts requests until ctx is canceled, then drains. It returns nil
|
|
// on a clean shutdown; a listener failure is returned as-is.
|
|
//
|
|
// The drain budget covers a committing request that is mid-retry, because
|
|
// Shutdown does not cancel in-flight handler contexts: killing such a
|
|
// connection early would leave the client unable to tell whether its write
|
|
// landed.
|
|
//
|
|
// A large batch-apply can legitimately still be running at up to its own
|
|
// largeApplyCeiling (an operator flag, 5 minutes by default) when shutdown
|
|
// begins — far past the ordinary drainTimeout — so the budget here EXTENDS to
|
|
// that request's own deadline when one is in flight (s.largeApplyDeadline),
|
|
// instead of killing a transaction that may already be minutes into
|
|
// committing. An ordinary shutdown with no large apply in flight is
|
|
// unaffected: it still drains in drainTimeout.
|
|
//
|
|
// s.closing is closed here, BEFORE the budget is computed, rather than left
|
|
// to fire only when http.Server.Shutdown is called below: RegisterOnShutdown
|
|
// hooks (closeStreams among them) run when Shutdown is CALLED, not before, so
|
|
// computing the budget first would leave a window where a large apply could
|
|
// still start — and admitLargeApply's own post-acquisition recheck of
|
|
// s.closing is the second gate that closes that window from the other side.
|
|
//
|
|
// This is safe to compute ONCE, at shutdown start, because
|
|
// acquireLargeApply refuses every NEW acquisition of the large-apply slot
|
|
// once s.closing fires: the in-flight request this budget was sized for is
|
|
// the only one that can still be holding the slot by now, never a later
|
|
// arrival that queued after the snapshot.
|
|
//
|
|
// The operator's orchestrator stop grace must be AT LEAST largeApplyCeiling
|
|
// plus drainGrace, or a SIGKILL from outside this process can still cut off
|
|
// an in-flight large apply this drain would otherwise have waited out; see
|
|
// engdocs/SERVE_RUNBOOK.md.
|
|
func (s *Server) Serve(ctx context.Context) error {
|
|
errCh := make(chan error, 1)
|
|
go func() { errCh <- s.http.Serve(s.listener) }()
|
|
|
|
select {
|
|
case err := <-errCh:
|
|
if errors.Is(err, http.ErrServerClosed) {
|
|
return nil
|
|
}
|
|
return err
|
|
case <-ctx.Done():
|
|
}
|
|
|
|
// Close s.closing before computing the drain budget (see doc comment
|
|
// above): closeStreams is idempotent (sync.Once), so it is safe to call
|
|
// here AND still leave it registered via RegisterOnShutdown for when
|
|
// Shutdown runs below.
|
|
s.closeStreams()
|
|
|
|
budget := drainBudget(orDefault(s.drainTimeout, drainTimeout), s.largeApplyDeadline.Load())
|
|
s.event("shutdown_start", "drain_timeout", budget.String(), "conns", s.liveConns.Load())
|
|
|
|
// Detached: ctx is already canceled, and the drain is the point.
|
|
drainCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), budget)
|
|
defer cancel()
|
|
|
|
if err := s.http.Shutdown(drainCtx); err != nil {
|
|
killed := s.liveConns.Load()
|
|
_ = s.http.Close()
|
|
s.event("shutdown_forced", "conns_killed", killed, "reason", err.Error())
|
|
} else {
|
|
s.event("shutdown_complete")
|
|
}
|
|
<-errCh
|
|
return nil
|
|
}
|
|
|
|
// connState tracks accepted connections, and says out loud when the cap is
|
|
// reached.
|
|
//
|
|
// It has to: netutil.LimitListener simply stops calling Accept at the cap, so
|
|
// further connections wait in the kernel backlog with nothing on stderr — and
|
|
// /healthz needs a fresh accept too, so an exhausted cap is indistinguishable
|
|
// from no traffic at all. Request lines just stop. This is the connection
|
|
// tier's version of the semaphore's saturation event, and the one wedge mode
|
|
// this slice can actually exhibit.
|
|
//
|
|
// The event is edge-triggered: once when the cap is reached, again only after
|
|
// it has cleared. The conns gauge on every request line is what shows it
|
|
// climbing beforehand.
|
|
func (s *Server) connState(_ net.Conn, state http.ConnState) {
|
|
switch state {
|
|
case http.StateNew:
|
|
n := s.liveConns.Add(1)
|
|
if s.maxConns > 0 && n >= int64(s.maxConns) && s.connCapWarned.CompareAndSwap(false, true) {
|
|
s.event("conn_cap_saturated", "conns", n, "max_conns", s.maxConns)
|
|
}
|
|
case http.StateHijacked, http.StateClosed:
|
|
if s.liveConns.Add(-1) < int64(s.maxConns) {
|
|
s.connCapWarned.Store(false)
|
|
}
|
|
}
|
|
}
|
|
|
|
// reader returns the issue-query surface for one request.
|
|
//
|
|
// On the ROLES source it is the configured role. There is nothing to build: a
|
|
// store's accessor already answered for its whole decorator chain when the
|
|
// caller called it, and this server opens no units of work on that path.
|
|
//
|
|
// On the PROVIDER source it is built per request rather than once at startup so
|
|
// that the units of work it opens are timed into THIS request's log line. That
|
|
// is the only reason: the role itself is stateless, and the accessor is the API
|
|
// on this seam exactly as it is on a store.
|
|
//
|
|
// The source is held by INTERFACE, not by the concrete wrapper. That is what
|
|
// makes uow.IssueReaderSource load-bearing rather than decorative: this call
|
|
// site type-checks against the accessor the provider seam publishes, so
|
|
// renaming or dropping it is a compile error here.
|
|
//
|
|
// EITHER WAY it goes out through checkedReader, which is what makes
|
|
// handleGetIssue's dereference of the detail view safe by construction again —
|
|
// see roles.go.
|
|
func (s *Server) reader(r *http.Request) (issueops.Reader, error) {
|
|
if s.provider == nil {
|
|
return checkedReader{inner: s.issueReader}, nil
|
|
}
|
|
var src uow.IssueReaderSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
rd, err := src.IssueReader()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return checkedReader{inner: rd}, nil
|
|
}
|
|
|
|
// statsReporter returns the guarded summary-statistics surface for one request.
|
|
//
|
|
// Same two sources as reader() and claimer(), for the same reasons, and held by
|
|
// INTERFACE so uow.StatsReporterSource is load-bearing rather than decorative.
|
|
// No checked wrapper: issueops.StatsResult carries a VALUE, so there is no
|
|
// nil-with-nil-error answer for a handler to dereference. checkedReader exists
|
|
// because Reader.Get hands back a pointer.
|
|
func (s *Server) statsReporter(r *http.Request) (issueops.StatsReporter, error) {
|
|
if s.provider == nil {
|
|
return s.issueStats, nil
|
|
}
|
|
var src uow.StatsReporterSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.StatsReporter()
|
|
}
|
|
|
|
// cycleDetector returns the guarded cycle-report surface for one request.
|
|
//
|
|
// Same two sources as reader() and claimer(), for the same reasons, and held by
|
|
// INTERFACE so uow.CycleDetectorSource is load-bearing rather than decorative.
|
|
// No checked wrapper: this report is a value whose slice a nil-safe range
|
|
// walks, so there is no dereference for a misbehaving implementation to turn
|
|
// into a panic.
|
|
func (s *Server) cycleDetector(r *http.Request) (issueops.CycleDetector, error) {
|
|
if s.provider == nil {
|
|
return s.issueCycles, nil
|
|
}
|
|
var src uow.CycleDetectorSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.CycleDetector()
|
|
}
|
|
|
|
// claimer returns the guarded atomic-claim surface for one request.
|
|
//
|
|
// It is the write-side twin of reader above, for all the same reasons: the
|
|
// configured role on the roles source, and on the provider source one built per
|
|
// request so its units of work are timed into THIS request's log line, held by
|
|
// INTERFACE so uow.IssueClaimerSource is load-bearing rather than decorative —
|
|
// and, from either source, wrapped in checkedClaimer.
|
|
func (s *Server) claimer(r *http.Request) (issueops.Claimer, error) {
|
|
if s.provider == nil {
|
|
return checkedClaimer{inner: s.issueClaimer}, nil
|
|
}
|
|
var src uow.IssueClaimerSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
cl, err := src.IssueClaimer()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return checkedClaimer{inner: cl}, nil
|
|
}
|
|
|
|
// batchCloser returns the many-issue close surface for one request, on the same
|
|
// terms as every role above and held by INTERFACE so uow.BatchCloserSource is
|
|
// load-bearing rather than decorative.
|
|
//
|
|
// Wrapped in checkedBatchCloser from either source, and the hazard it folds is
|
|
// not the one the other wrappers exist for. Nothing here dereferences the issue
|
|
// pointer an outcome carries; what this role owns instead is a POSITIONAL array
|
|
// the client reads against its own argument list, and checkedBatchCloser says
|
|
// what a miscounted or contentless entry in it costs.
|
|
func (s *Server) batchCloser(r *http.Request) (issueops.BatchCloser, error) {
|
|
if s.provider == nil {
|
|
return checkedBatchCloser{inner: s.issueBatchCloser}, nil
|
|
}
|
|
var src uow.BatchCloserSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
closer, err := src.BatchCloser()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return checkedBatchCloser{inner: closer}, nil
|
|
}
|
|
|
|
// readyClaimer returns the take-ready-work surface for one request.
|
|
//
|
|
// Built the same two ways as claimer above and for the same reasons, and held
|
|
// by INTERFACE so uow.ReadyClaimerSource is load-bearing rather than
|
|
// decorative.
|
|
//
|
|
// IT GOES OUT UNWRAPPED, and the difference from checkedClaimer is the whole
|
|
// reason that wrapper exists. That one folds a nil issue because handleClaim
|
|
// DEREFERENCES the pointer the role returned; this handler forwards it, and a
|
|
// nil is not even a fault here — it is the documented answer for an empty ready
|
|
// front. A wrapper would be ceremony that reads like a guarantee.
|
|
func (s *Server) readyClaimer(r *http.Request) (issueops.ReadyClaimer, error) {
|
|
if s.provider == nil {
|
|
return s.issueReadyClaimer, nil
|
|
}
|
|
var src uow.ReadyClaimerSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.ReadyClaimer()
|
|
}
|
|
|
|
// releaser returns the claim-release surface for one request.
|
|
//
|
|
// Built the same two ways as claimer above and for the same reasons: the
|
|
// configured role on the roles source, and on the provider source one built per
|
|
// request so its units of work are timed into THIS request's log line, held by
|
|
// INTERFACE so uow.ReleaserSource is load-bearing rather than decorative — and,
|
|
// from either source, wrapped in checkedReleaser, because the handler
|
|
// dereferences the pointer the result carries.
|
|
func (s *Server) releaser(r *http.Request) (issueops.Releaser, error) {
|
|
if s.provider == nil {
|
|
return checkedReleaser{inner: s.issueReleaser}, nil
|
|
}
|
|
var src uow.ReleaserSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
rel, err := src.Releaser()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return checkedReleaser{inner: rel}, nil
|
|
}
|
|
|
|
// lifecycle returns the guarded issue-mutation surface for one request.
|
|
//
|
|
// Built the same two ways as claimer above and for the same reasons: the
|
|
// configured role on the roles source, and on the provider source one built per
|
|
// request so its units of work are timed into THIS request's log line, held by
|
|
// INTERFACE so uow.IssueLifecycleSource is load-bearing rather than decorative
|
|
// — and, from either source, wrapped in checkedLifecycle.
|
|
func (s *Server) lifecycle(r *http.Request) (issueops.Lifecycle, error) {
|
|
if s.provider == nil {
|
|
return checkedLifecycle{inner: s.issueLifecycle}, nil
|
|
}
|
|
var src uow.IssueLifecycleSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
lc, err := src.IssueLifecycle()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return checkedLifecycle{inner: lc}, nil
|
|
}
|
|
|
|
// workspaceConfig returns the guarded workspace-settings surface for one
|
|
// request.
|
|
//
|
|
// Same two sources as reader and claimer above, for the same reasons, and held
|
|
// by INTERFACE so uow.WorkspaceConfigSource is load-bearing rather than
|
|
// decorative. No checked wrapper: both settings handlers read VALUES out of the
|
|
// result, so there is no pointer for a caller-supplied role to hand back nil
|
|
// in.
|
|
func (s *Server) workspaceConfig(r *http.Request) (issueops.WorkspaceConfig, error) {
|
|
if s.provider == nil {
|
|
return s.settings, nil
|
|
}
|
|
var src uow.WorkspaceConfigSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.WorkspaceConfig()
|
|
}
|
|
|
|
// edgeReader returns the guarded stored-edge surface for one request.
|
|
//
|
|
// Same two sources as reader() and claimer(), for the same reasons, and held by
|
|
// INTERFACE so uow.EdgeReaderSource is load-bearing rather than decorative. No
|
|
// checked wrapper: this role answers with a VALUE, so no handler dereferences a
|
|
// pointer it returned — checkedReader exists for Get alone.
|
|
func (s *Server) edgeReader(r *http.Request) (issueops.EdgeReader, error) {
|
|
if s.provider == nil {
|
|
return s.issueEdges, nil
|
|
}
|
|
var src uow.EdgeReaderSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.EdgeReader()
|
|
}
|
|
|
|
// graphCounter returns the edge-count surface for one request, built the same
|
|
// two ways as edgeReader above and held by INTERFACE so
|
|
// uow.GraphCounterSource is load-bearing rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED, for counter's reason: the role answers with a VALUE
|
|
// whose slice a nil-safe range walks, so no handler dereferences a pointer it
|
|
// returned and a checked wrapper would be ceremony that reads like a guarantee.
|
|
func (s *Server) graphCounter(r *http.Request) (issueops.GraphCounter, error) {
|
|
if s.provider == nil {
|
|
return s.issueEdgeCounter, nil
|
|
}
|
|
var src uow.GraphCounterSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.GraphCounter()
|
|
}
|
|
|
|
// batchGetter returns the many-ids read surface for one request, built the
|
|
// same two ways as edgeReader above and held by INTERFACE so
|
|
// uow.BatchGetterSource is load-bearing rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED, for graphCounter's reason: the role answers with a
|
|
// VALUE whose slices a nil-safe range walks, so no handler dereferences a
|
|
// pointer it returned.
|
|
func (s *Server) batchGetter(r *http.Request) (issueops.BatchGetter, error) {
|
|
if s.provider == nil {
|
|
return s.issueBatchGetter, nil
|
|
}
|
|
var src uow.BatchGetterSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.BatchGetter()
|
|
}
|
|
|
|
// leaseReclaimer returns the stale-lease sweep surface for one request, on the
|
|
// same terms as every role above and held by INTERFACE so
|
|
// uow.LeaseReclaimerSource is load-bearing rather than decorative. It goes out
|
|
// unwrapped for the sweeper's reason: ReclaimResult is a VALUE whose slice a
|
|
// nil-safe range walks.
|
|
func (s *Server) leaseReclaimer(r *http.Request) (issueops.LeaseReclaimer, error) {
|
|
if s.provider == nil {
|
|
return s.issueReclaimer, nil
|
|
}
|
|
var src uow.LeaseReclaimerSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.LeaseReclaimer()
|
|
}
|
|
|
|
// relations returns the single-anchor neighbor surface for one request, built
|
|
// the same two ways as edgeReader above and held by INTERFACE so
|
|
// uow.RelationsSource is load-bearing rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED even though the role answers with a slice of POINTERS,
|
|
// which is the one place this differs from checkedReader's argument. That
|
|
// wrapper exists because handleGetIssue DEREFERENCES the pointer a role handed
|
|
// back; here wireRelated drops a nil element the way wireItems and wireEdges
|
|
// already do, so there is nothing for a checked wrapper to make safe.
|
|
func (s *Server) relations(r *http.Request) (issueops.Relations, error) {
|
|
if s.provider == nil {
|
|
return s.issueRelations, nil
|
|
}
|
|
var src uow.RelationsSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.IssueRelations()
|
|
}
|
|
|
|
// commenter returns the append-one-comment surface for one request, built the
|
|
// same two ways as every role above and held by INTERFACE so
|
|
// uow.CommenterSource is load-bearing rather than decorative.
|
|
//
|
|
// It goes out WRAPPED, unlike the reads beside it and for checkedClaimer's
|
|
// reason: handleAddComment dereferences the pointer the role answers with, so a
|
|
// caller-supplied role that reported success without a row would panic on a live
|
|
// server rather than reaching the generic 500 with the fault in the log.
|
|
func (s *Server) commenter(r *http.Request) (issueops.Commenter, error) {
|
|
if s.provider == nil {
|
|
return checkedCommenter{inner: s.issueCommenter}, nil
|
|
}
|
|
var src uow.CommenterSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
c, err := src.Commenter()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return checkedCommenter{inner: c}, nil
|
|
}
|
|
|
|
// blockingAnnotator returns the derived blocking-decoration surface for one
|
|
// request, on the same terms as every role above and held by INTERFACE so
|
|
// uow.BlockingAnnotatorSource is load-bearing rather than decorative. It goes
|
|
// out UNWRAPPED for the reason edgeReader's answer does: this role answers with
|
|
// a VALUE, and checkedReader exists for Get alone.
|
|
func (s *Server) blockingAnnotator(r *http.Request) (issueops.BlockingAnnotator, error) {
|
|
if s.provider == nil {
|
|
return s.issueBlocking, nil
|
|
}
|
|
var src uow.BlockingAnnotatorSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.BlockingAnnotator()
|
|
}
|
|
|
|
// treeWalker returns the guarded dependency-tree surface for one request.
|
|
//
|
|
// Built the same two ways as its siblings and for the same reasons, held by
|
|
// INTERFACE so uow.TreeWalkerSource is load-bearing rather than decorative. No
|
|
// checked wrapper, for the reason cycleDetector gives: this role answers with a
|
|
// VALUE whose slice a nil-safe range walks.
|
|
func (s *Server) treeWalker(r *http.Request) (issueops.TreeWalker, error) {
|
|
if s.provider == nil {
|
|
return s.issueTree, nil
|
|
}
|
|
var src uow.TreeWalkerSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.TreeWalker()
|
|
}
|
|
|
|
// readyCounter returns the ready-count surface for one request, on the same
|
|
// terms as every role above and held by INTERFACE so uow.ReadyCounterSource is
|
|
// load-bearing rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED. checkedReader and checkedClaimer exist because their
|
|
// handlers dereference a POINTER a role returned; CountReady answers with a
|
|
// value, so a wrapper would be ceremony that reads like a guarantee.
|
|
func (s *Server) readyCounter(r *http.Request) (issueops.ReadyCounter, error) {
|
|
if s.provider == nil {
|
|
return s.issueReadyCounter, nil
|
|
}
|
|
var src uow.ReadyCounterSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.ReadyCounter()
|
|
}
|
|
|
|
// counter returns the issue-count surface for one request, on the same terms as
|
|
// readyCounter above and held by INTERFACE so uow.CounterSource is load-bearing
|
|
// rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED, for readyCounter's reason: both of this role's methods
|
|
// answer with a VALUE, so a checked wrapper would be ceremony that reads like a
|
|
// guarantee. The one pointer-shaped thing in its result is CountByGroupResult's
|
|
// map, and the role promises an empty map rather than nil — a promise the
|
|
// handler does not have to trust, because a nil map ranges and marshals as an
|
|
// empty object either way.
|
|
func (s *Server) counter(r *http.Request) (issueops.Counter, error) {
|
|
if s.provider == nil {
|
|
return s.issueCounter, nil
|
|
}
|
|
var src uow.CounterSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.Counter()
|
|
}
|
|
|
|
// querier returns the boolean-query surface for one request, on the same terms
|
|
// as every role above and held by INTERFACE so uow.QuerierSource is
|
|
// load-bearing rather than decorative. It goes out UNWRAPPED, like the counter
|
|
// and unlike checkedReader: a page is a value carrying a slice, so there is
|
|
// nothing for a wrapper to make safe.
|
|
func (s *Server) querier(r *http.Request) (issueops.Querier, error) {
|
|
if s.provider == nil {
|
|
return s.issueQuerier, nil
|
|
}
|
|
var src uow.QuerierSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.Querier()
|
|
}
|
|
|
|
// sweeper returns the guarded bulk-clearance surface for one request, on the
|
|
// same terms as every role above and held by INTERFACE so uow.SweeperSource is
|
|
// load-bearing rather than decorative. It goes out unwrapped: SweepResult is a
|
|
// VALUE, so there is no pointer for a caller-supplied role to hand back nil in.
|
|
//
|
|
// The role this returns is the ONLY thing standing between a POST body and a
|
|
// mass delete — the require-a-filter gate, the pinned protection and the
|
|
// closed_at recheck are all inside it — which is why the Config field it comes
|
|
// from is required rather than optional.
|
|
func (s *Server) sweeper(r *http.Request) (issueops.Sweeper, error) {
|
|
if s.provider == nil {
|
|
return s.issueSweeper, nil
|
|
}
|
|
var src uow.SweeperSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.Sweeper()
|
|
}
|
|
|
|
// deleter returns the named-row erasure surface for one request, on the same
|
|
// terms as every role above and held by INTERFACE so uow.DeleterSource is
|
|
// load-bearing rather than decorative. It goes out unwrapped for the reason the
|
|
// sweeper does: DeleteResult is a VALUE.
|
|
//
|
|
// The role this returns is the only thing standing between a POST body and an
|
|
// orphaned dependency graph — the guard, the id resolution and the reference
|
|
// rewrite are all inside it — which is why the Config field it comes from is
|
|
// required rather than optional.
|
|
func (s *Server) deleter(r *http.Request) (issueops.Deleter, error) {
|
|
if s.provider == nil {
|
|
return s.issueDeleter, nil
|
|
}
|
|
var src uow.DeleterSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.Deleter()
|
|
}
|
|
|
|
// batchCreator returns the batch-create surface for one request, on the same
|
|
// terms as every role above and held by INTERFACE so uow.BatchCreatorSource is
|
|
// load-bearing rather than decorative.
|
|
//
|
|
// It goes out CHECKED, unlike the ready counter. CreateBatchResult carries a
|
|
// slice of POINTERS and the response body carries values, so the handler
|
|
// dereferences every one of them — the checkedClaimer hazard, N times over.
|
|
// See checkedBatchCreator.
|
|
func (s *Server) batchCreator(r *http.Request) (issueops.BatchCreator, error) {
|
|
if s.provider == nil {
|
|
return checkedBatchCreator{inner: s.issueBatchCreator}, nil
|
|
}
|
|
var src uow.BatchCreatorSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
creator, err := src.BatchCreator()
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
return checkedBatchCreator{inner: creator}, nil
|
|
}
|
|
|
|
// dependencyEditor returns the guarded dependency-graph write surface for one
|
|
// request, on the same terms as every role above and held by INTERFACE so
|
|
// uow.DependencyEditorSource is load-bearing rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED, like the sweeper and the deleter: both of this role's
|
|
// results are VALUES, so no handler dereferences a pointer it returned.
|
|
//
|
|
// The role this returns owns every refusal the graph can raise — the cycle
|
|
// gate, the hierarchy rule, the type conflict and the endpoint existence checks
|
|
// — which is why the Config field it comes from is required rather than
|
|
// optional.
|
|
func (s *Server) dependencyEditor(r *http.Request) (issueops.DependencyEditor, error) {
|
|
if s.provider == nil {
|
|
return s.issueDependencies, nil
|
|
}
|
|
var src uow.DependencyEditorSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.DependencyEditor()
|
|
}
|
|
|
|
// metadataCAS returns the conditional metadata write for one request, on the
|
|
// same terms as every role above and held by INTERFACE so
|
|
// uow.MetadataCASSource is load-bearing rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED: the role's result is a VALUE whose only pointer member
|
|
// is an optional raw value the handler passes through without dereferencing.
|
|
func (s *Server) metadataCAS(r *http.Request) (issueops.MetadataCAS, error) {
|
|
if s.provider == nil {
|
|
return s.issueMetadataCAS, nil
|
|
}
|
|
var src uow.MetadataCASSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.MetadataCAS()
|
|
}
|
|
|
|
// batchApplier returns the ordered-plan write surface for one request, on the
|
|
// same terms as every role above and held by INTERFACE so
|
|
// uow.BatchApplierSource is load-bearing rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED, like the dependency editor: ApplyBatchResult is a
|
|
// VALUE, and the one pointer its items carry — the post-item issue snapshot —
|
|
// never reaches the wire, so no handler dereferences anything this role
|
|
// returned.
|
|
//
|
|
// The role owns every refusal this operation can raise: the ref graph, the
|
|
// as-modified preconditions, the close policy, the assignee fence and the end
|
|
// gate. That is why the Config field it comes from is required rather than
|
|
// optional.
|
|
func (s *Server) batchApplier(r *http.Request) (issueops.BatchApplier, error) {
|
|
if s.provider == nil {
|
|
return s.issueBatchApplier, nil
|
|
}
|
|
var src uow.BatchApplierSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.BatchApplier()
|
|
}
|
|
|
|
// memories returns the persistent-memory surface for one request, on the same
|
|
// terms as every role above and held by INTERFACE so uow.MemoriesSource is
|
|
// load-bearing rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED: all four of this role's results are VALUES, so no
|
|
// handler dereferences a pointer it returned, and a miss is a Found field
|
|
// rather than a nil the wire would have to interpret.
|
|
func (s *Server) memories(r *http.Request) (memoryops.Memories, error) {
|
|
if s.provider == nil {
|
|
return s.workspaceMemories, nil
|
|
}
|
|
var src uow.MemoriesSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.Memories()
|
|
}
|
|
|
|
// eventsJournalCursor returns the journal read surface for one request, on the
|
|
// same terms as every role above and held by INTERFACE so
|
|
// uow.EventsJournalCursorSource is load-bearing rather than decorative.
|
|
//
|
|
// It goes out UNWRAPPED: a page is a value carrying a slice, so there is
|
|
// nothing for a checked wrapper to make safe — the querier's argument.
|
|
//
|
|
// The narrow type is the point of this accessor rather than an accident of it.
|
|
// The provider can also hand out uw.EventsJournalUseCase(), which PRUNES; going
|
|
// through a source that only publishes storage.EventsJournalCursor is what
|
|
// keeps the read-only promise a fact about what the handler is holding.
|
|
func (s *Server) eventsJournalCursor(r *http.Request) (storage.EventsJournalCursor, error) {
|
|
if s.provider == nil {
|
|
if s.eventsJournal == nil {
|
|
// Unreachable through the handler, which refuses a disabled
|
|
// workspace before asking for a reader, and Listen refuses an
|
|
// enabled one with no reader. Said out loud rather than returned as
|
|
// a nil interface for the reason WithUOW says its own: a 500 naming
|
|
// the condition beats a panic on a live server if either of those
|
|
// two gates is ever moved.
|
|
return nil, errors.New("httpapi: this server has no events-journal reader; it was configured for a workspace with the journal off")
|
|
}
|
|
return s.eventsJournal, nil
|
|
}
|
|
var src uow.EventsJournalCursorSource = timedProvider{inner: s.provider, rec: requestInfo(r.Context())}
|
|
return src.EventsJournalCursor()
|
|
}
|
|
|
|
// WithUOW runs fn inside one unit of work and guarantees the rollback.
|
|
//
|
|
// The close context is DETACHED on purpose. Close sends ROLLBACK on the pinned
|
|
// connection, and the transaction layer POISONS that connection if the send
|
|
// fails (internal/storage/uow/doltserver_tx.go) — go-sql-driver's session reset
|
|
// does not clear an open transaction, so a session that may still be in one
|
|
// must never go back to the pool. Correctness is therefore safe either way, but
|
|
// closing with the request's own canceled context would fail the ROLLBACK
|
|
// immediately and burn one pinned session on every client disconnect. Reads
|
|
// never commit.
|
|
//
|
|
// It is provider-only, and says so rather than dereferencing nil: a
|
|
// roles-backed server has no unit of work to open, and the roles it does hold
|
|
// own their own transactions.
|
|
func (s *Server) WithUOW(ctx context.Context, rec *reqInfo, fn func(uow.UnitOfWork) error) error {
|
|
if s.provider == nil {
|
|
return errors.New("httpapi: this server has no unit-of-work provider; it answers from configured issue roles")
|
|
}
|
|
start := time.Now()
|
|
uw, err := s.provider.NewUOW(ctx)
|
|
if rec != nil {
|
|
rec.uowWait = time.Since(start)
|
|
}
|
|
if err != nil {
|
|
return err
|
|
}
|
|
defer func() {
|
|
closeCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), uowCloseTimeout)
|
|
defer cancel()
|
|
uw.Close(closeCtx)
|
|
}()
|
|
return fn(uw)
|
|
}
|
|
|
|
// acquire takes a database slot, or gives up. A timed-out wait is ErrBusy, not
|
|
// a request parked for the full deadline and then answered with a
|
|
// non-retryable 500.
|
|
func (s *Server) acquire(ctx context.Context, rec *reqInfo) (release func(), err error) {
|
|
start := time.Now()
|
|
release = func() { <-s.sem }
|
|
|
|
select {
|
|
case s.sem <- struct{}{}:
|
|
rec.semWait = time.Since(start)
|
|
return release, nil
|
|
default:
|
|
}
|
|
|
|
timer := time.NewTimer(orDefault(s.semTimeout, semAcquireTimeout))
|
|
defer timer.Stop()
|
|
select {
|
|
case s.sem <- struct{}{}:
|
|
rec.semWait = time.Since(start)
|
|
s.noteSaturation(rec, "acquired")
|
|
return release, nil
|
|
case <-timer.C:
|
|
rec.semWait = time.Since(start)
|
|
s.event("semaphore_timeout", "request_id", rec.id, "wait_ms", millis(rec.semWait),
|
|
"inflight", maxInflight, "conns", s.liveConns.Load())
|
|
return nil, ErrBusy
|
|
case <-ctx.Done():
|
|
// The client hung up, or the request deadline expired, while queued.
|
|
// Still a saturation datapoint: it is the same wedge, observed from a
|
|
// request that did not live long enough to be shed.
|
|
rec.semWait = time.Since(start)
|
|
s.noteSaturation(rec, "abandoned")
|
|
return nil, ctx.Err()
|
|
}
|
|
}
|
|
|
|
// noteSaturation logs a wait that lasted long enough to matter. This is the
|
|
// signal that separates "wedged" from "no traffic" at 3 a.m., because /healthz
|
|
// stays green either way.
|
|
func (s *Server) noteSaturation(rec *reqInfo, outcome string) {
|
|
if rec.semWait < orDefault(s.semWarn, saturationWarn) {
|
|
return
|
|
}
|
|
s.event("semaphore_saturated",
|
|
"request_id", rec.id, "wait_ms", millis(rec.semWait),
|
|
"inflight", maxInflight, "conns", s.liveConns.Load(), "outcome", outcome)
|
|
}
|
|
|
|
func orDefault(v, fallback time.Duration) time.Duration {
|
|
if v > 0 {
|
|
return v
|
|
}
|
|
return fallback
|
|
}
|
|
|
|
// acquireLargeApply takes the one-wide large-write slot for a POST
|
|
// issues:batchApply request over largeApplyItemThreshold items, and builds
|
|
// that request's run context.
|
|
//
|
|
// THE WAIT IS BOUNDED, on the SAME orDefault(s.semTimeout, semAcquireTimeout)
|
|
// budget acquire uses for the ordinary sem — not the caller's own request
|
|
// deadline, and not unbounded. A goroutine queued here still holds its
|
|
// general sem slot (route() acquires sem before the handler runs, and holds
|
|
// it for the handler's whole lifetime), so an unbounded wait would let enough
|
|
// queued large applies starve every other request behind a saturated sem;
|
|
// see TestLargeApplyQueueDoesNotStarveOrdinaryRequests. Every refusal this
|
|
// function can produce — the bounded wait expiring, s.closing firing because
|
|
// a graceful shutdown began, or the caller's own ctx.Done() firing while
|
|
// queued — answers ErrBusy, never a raw ctx.Err(): nothing was written in any
|
|
// of those cases, so the caller gets a 503 with Retry-After rather than the
|
|
// non-retryable 500 a bare context error would classify as (ClassifyError
|
|
// deliberately excludes context errors from its 503 path for OTHER callers,
|
|
// where a context error can mean a write is in flight — that exclusion does
|
|
// not apply here, because this slot guards nothing but queue position).
|
|
//
|
|
// THE RUN DEADLINE IS STAMPED ONLY ON SUCCESS, and only once the slot is
|
|
// actually held: runCtx is a DETACHED copy of ctx (context.WithoutCancel)
|
|
// given a fresh largeApplyCeiling-wide timeout starting NOW, not narrowed
|
|
// from whatever deadline ctx already carried and not started back when the
|
|
// request first queued. That is what makes this an EXTENSION of the small
|
|
// request's ordinary budget rather than a narrowing of it: every request
|
|
// under largeApplyItemThreshold items runs under route()'s own unconditional
|
|
// requestDeadline and never reaches this function at all, and a request that
|
|
// clears the threshold gets a whole fresh largeApplyCeiling on top of
|
|
// whatever it already spent queued. Detaching from ctx's cancellation means a
|
|
// client disconnect mid-run does not abort an already-admitted large apply —
|
|
// deliberate, because it is a single atomic write that should not be
|
|
// abandoned half-considered merely because the caller stopped reading; the
|
|
// same trade-off WithUOW's own deferred close already makes for a request
|
|
// context.
|
|
//
|
|
// release cancels runCtx, clears s.largeApplyDeadline and frees the slot, in
|
|
// that order, so nothing observes the deadline as cleared while the run
|
|
// context is still live.
|
|
func (s *Server) acquireLargeApply(ctx context.Context) (runCtx context.Context, release func(), err error) {
|
|
select {
|
|
case <-s.closing:
|
|
return nil, nil, ErrBusy
|
|
default:
|
|
}
|
|
select {
|
|
case s.largeApplySem <- struct{}{}:
|
|
return s.admitLargeApply(ctx)
|
|
default:
|
|
}
|
|
|
|
// At most ONE caller may queue for the slot at a time. Without this gate,
|
|
// every large request beyond the one already holding largeApplySem would
|
|
// still wait up to semTimeout below — and each of those waiters holds its
|
|
// own sem slot the whole time (route() acquires sem before the handler
|
|
// runs) — so a burst of large requests could occupy a semTimeout's worth
|
|
// of general concurrency slots merely queueing, starving small requests
|
|
// for that whole window even though only one waiter could ever win the
|
|
// large-apply slot. A second large request arriving while one is already
|
|
// queued is refused immediately: no wait, no sem slot held for it.
|
|
if !s.largeApplyWaiting.CompareAndSwap(false, true) {
|
|
return nil, nil, ErrBusy
|
|
}
|
|
defer s.largeApplyWaiting.Store(false)
|
|
|
|
timer := time.NewTimer(orDefault(s.semTimeout, semAcquireTimeout))
|
|
defer timer.Stop()
|
|
select {
|
|
case s.largeApplySem <- struct{}{}:
|
|
return s.admitLargeApply(ctx)
|
|
case <-timer.C:
|
|
return nil, nil, ErrBusy
|
|
case <-s.closing:
|
|
return nil, nil, ErrBusy
|
|
case <-ctx.Done():
|
|
// The client hung up, or the request's own (unrelated, ordinary)
|
|
// deadline expired, while still queued for the slot. Nothing was
|
|
// written — this path never opened a transaction — so it is the same
|
|
// ErrBusy every other refusal above answers, not ctx.Err().
|
|
return nil, nil, ErrBusy
|
|
}
|
|
}
|
|
|
|
// admitLargeApply runs once acquireLargeApply has actually taken the slot: it
|
|
// builds the run's own detached, freshly-timed context and stamps
|
|
// s.largeApplyDeadline for Serve's graceful-drain budget to read.
|
|
//
|
|
// The run context is also marked with issueops.WithExtendedRetryBudget, and
|
|
// this is the only place that marks one: the marker, not the deadline's size,
|
|
// is what lets a BatchApplier's commit-retry loop scale to the extended
|
|
// budget instead of stopping at its ordinary fixed ceiling. An ordinary
|
|
// request never reaches this function, so it stays unmarked.
|
|
//
|
|
// It re-checks s.closing AFTER taking the slot and storing the deadline,
|
|
// because a select with multiple ready cases (largeApplySem and s.closing
|
|
// both ready at once, in acquireLargeApply above) picks pseudo-randomly: the
|
|
// slot can be won in the same instant a graceful shutdown begins, racing
|
|
// ahead of Serve's drainBudget snapshot. Losing that race here — releasing
|
|
// the slot and refusing with ErrBusy — is what keeps Serve's "the in-flight
|
|
// request this budget was sized for is the only one that can still be
|
|
// holding the slot" invariant true.
|
|
func (s *Server) admitLargeApply(ctx context.Context) (runCtx context.Context, release func(), err error) {
|
|
runCtx, cancel := context.WithTimeout(
|
|
issueops.WithExtendedRetryBudget(context.WithoutCancel(ctx)),
|
|
orDefault(s.largeApplyCeiling, DefaultLargeApplyCeiling),
|
|
)
|
|
deadline, _ := runCtx.Deadline()
|
|
s.largeApplyDeadline.Store(&deadline)
|
|
release = func() {
|
|
cancel()
|
|
s.largeApplyDeadline.Store(nil)
|
|
<-s.largeApplySem
|
|
}
|
|
|
|
select {
|
|
case <-s.closing:
|
|
release()
|
|
return nil, nil, ErrBusy
|
|
default:
|
|
}
|
|
|
|
return runCtx, release, nil
|
|
}
|
|
|
|
// handler builds the whole request path: the route table's registrations, the
|
|
// catch-all that keeps unrouted paths on the same error shape, and the
|
|
// middleware in front of both.
|
|
func (s *Server) handler() http.Handler {
|
|
mux := http.NewServeMux()
|
|
// Rows carrying a customMethod SHARE a pattern, so they get one
|
|
// registration between them and a dispatcher in front. Collected in table
|
|
// order, which is the order customMethodTarget tries the suffixes in.
|
|
shared := map[string][]route{}
|
|
var sharedOrder []string
|
|
for _, rt := range routeTable {
|
|
if rt.customMethod == "" {
|
|
mux.Handle(rt.method+" "+rt.pattern, s.route(rt))
|
|
continue
|
|
}
|
|
key := rt.method + " " + rt.pattern
|
|
if _, seen := shared[key]; !seen {
|
|
sharedOrder = append(sharedOrder, key)
|
|
}
|
|
shared[key] = append(shared[key], rt)
|
|
}
|
|
for _, key := range sharedOrder {
|
|
mux.Handle(key, s.dispatchCustomMethod(shared[key]))
|
|
}
|
|
|
|
// Not an operation and deliberately not in the route table: it exists so
|
|
// that an unrouted path still answers with problem+json rather than
|
|
// net/http's text/plain default, which the document promises for EVERY
|
|
// non-2xx byte. A method mismatch on a known path lands here too and
|
|
// answers 404 rather than 405, because 405 is not in the v0 vocabulary.
|
|
mux.Handle("/", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
s.fail(w, r, newResult(CodeNotFound, "no such route on this server"))
|
|
}))
|
|
|
|
return s.withRequestContext(s.checkHost(mux))
|
|
}
|
|
|
|
// reqInfo is the per-request record the log line is assembled from. Layers fill
|
|
// in what they know; the outermost middleware writes it.
|
|
type reqInfo struct {
|
|
id string
|
|
op string
|
|
status int
|
|
code Code
|
|
semWait time.Duration
|
|
uowWait time.Duration
|
|
// refused is the caller-supplied value a middleware turned down: the Host
|
|
// this server does not answer to, or the unrecognized parameter name. It
|
|
// goes on the request line so a refusal is attributable — a rebinding probe
|
|
// that leaves no server-side trace is a control nobody can investigate.
|
|
// logValue quotes it, which is what makes logging attacker-controlled text
|
|
// safe.
|
|
//
|
|
// A pointer, so that the request line carries the field only when there was
|
|
// a refusal: the empty parameter name in `?=1` is a refusal with an empty
|
|
// value, not the absence of one.
|
|
refused *string
|
|
}
|
|
|
|
// refuse records the value a middleware turned down, for the request line.
|
|
func (rec *reqInfo) refuse(value string) { rec.refused = &value }
|
|
|
|
type reqInfoKey struct{}
|
|
|
|
// requestInfo returns the record for the request in flight. It never returns
|
|
// nil: every request goes through withRequestContext, and handing back a
|
|
// detached record rather than nil means a mis-wired caller loses a log line
|
|
// instead of panicking mid-response.
|
|
func requestInfo(ctx context.Context) *reqInfo {
|
|
if rec, ok := ctx.Value(reqInfoKey{}).(*reqInfo); ok {
|
|
return rec
|
|
}
|
|
return &reqInfo{}
|
|
}
|
|
|
|
// withRequestContext assigns the correlation id, applies response-wide
|
|
// headers, recovers panics, and writes the one log line per request. It is
|
|
// outermost so that a request refused by the Host check is logged like any
|
|
// other.
|
|
func (s *Server) withRequestContext(next http.Handler) http.Handler {
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
rec := &reqInfo{id: s.nextID(), status: http.StatusOK}
|
|
r = r.WithContext(context.WithValue(r.Context(), reqInfoKey{}, rec))
|
|
|
|
// No client or intermediary may cache an answer about live work.
|
|
w.Header().Set("Cache-Control", "no-store")
|
|
|
|
sw := &statusWriter{
|
|
ResponseWriter: w,
|
|
rc: http.NewResponseController(w),
|
|
budget: orDefault(s.writeStall, writeStallTimeout),
|
|
}
|
|
// Arm before the handler runs: a handler that writes nothing still has a
|
|
// response, written by net/http on the way out.
|
|
sw.extendWriteDeadline(sw.budget)
|
|
|
|
start := time.Now()
|
|
|
|
// Deferred, so that the two things this middleware promises — an answer
|
|
// in the documented shape, and one log line per request — survive the
|
|
// one failure where correlating them matters most.
|
|
defer func() {
|
|
p := recover()
|
|
if p != nil && p != http.ErrAbortHandler {
|
|
s.panicked(sw, r, rec, p)
|
|
}
|
|
|
|
// net/http flushes what is left of the buffered response after the
|
|
// handler returns, so extend once more to cover it. The extension has
|
|
// to outlast the idle timeout too: the deadline stays armed while a
|
|
// keep-alive connection waits for its next request, and net/http
|
|
// answers some requests (a malformed request line, oversized headers)
|
|
// without reaching this middleware — an expired deadline would turn
|
|
// those into a dropped connection. Anything that does reach here
|
|
// re-arms above, before a handler writes a byte.
|
|
sw.extendWriteDeadline(sw.budget + idleTimeout)
|
|
|
|
if sw.status != 0 {
|
|
// Still zero means the handler returned without writing
|
|
// anything, in which case net/http has sent the 200 rec already
|
|
// carries.
|
|
rec.status = sw.status
|
|
}
|
|
|
|
fields := []any{
|
|
"request_id", rec.id,
|
|
"op", rec.op,
|
|
"method", r.Method,
|
|
"path", r.URL.Path,
|
|
"status", rec.status,
|
|
"code", string(rec.code),
|
|
"duration_ms", millis(time.Since(start)),
|
|
"sem_wait_ms", millis(rec.semWait),
|
|
"uow_ms", millis(rec.uowWait),
|
|
// The connection gauge belongs on the busiest line in the log:
|
|
// it is the only place an operator watches it climb toward the
|
|
// cap. remote_addr answers "which client", which on loopback
|
|
// means "which local process" via the port.
|
|
"conns", s.liveConns.Load(),
|
|
"remote_addr", r.RemoteAddr,
|
|
}
|
|
if rec.refused != nil {
|
|
fields = append(fields, "refused", *rec.refused)
|
|
}
|
|
s.event("request", fields...)
|
|
|
|
if p == http.ErrAbortHandler {
|
|
// net/http's documented "abandon this response silently" signal.
|
|
// It gets its log line like every other request, and then it gets
|
|
// the abort it asked for.
|
|
panic(p)
|
|
}
|
|
}()
|
|
|
|
next.ServeHTTP(sw, r)
|
|
})
|
|
}
|
|
|
|
// panicked gives a panicking handler the same shape as every other failure: one
|
|
// problem+json response and one log line, both carrying the request id.
|
|
//
|
|
// Without it the panic reaches net/http's per-connection recover, which prints
|
|
// an unstructured stack trace to stderr with nothing on it to tie to a client
|
|
// report, drops the connection with no body at all, and skips the request line —
|
|
// so the one class of failure where correlation matters most is the one class
|
|
// that has none. The panic text stays out of the response for the same reason
|
|
// every other 5xx detail does.
|
|
func (s *Server) panicked(sw *statusWriter, r *http.Request, rec *reqInfo, p any) {
|
|
rec.code = CodeInternal
|
|
s.event("panic",
|
|
"request_id", rec.id,
|
|
"op", rec.op,
|
|
"method", r.Method,
|
|
"path", r.URL.Path,
|
|
"error", fmt.Sprint(p),
|
|
"stack", string(debug.Stack()),
|
|
)
|
|
if sw.status != 0 {
|
|
// The response is already on the wire; a truncated body is all the
|
|
// client can be told, and writing a second header would only add a
|
|
// superfluous-WriteHeader line to the log.
|
|
return
|
|
}
|
|
s.fail(sw, r, newResult(CodeInternal, ""))
|
|
}
|
|
|
|
// checkHost is the DNS-rebinding defense. An unauthenticated service on
|
|
// loopback is reachable from any browser on the host; a page that re-resolves
|
|
// its own name to 127.0.0.1 issues requests the browser treats as same-origin,
|
|
// so no CORS rule stops them. What the browser does preserve is the attacker's
|
|
// hostname in Host, which is what this rejects.
|
|
func (s *Server) checkHost(next http.Handler) http.Handler {
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
if !s.hosts.allows(r.Host) {
|
|
requestInfo(r.Context()).refuse(r.Host)
|
|
s.fail(w, r, InvalidArgument("Host", ReasonInvalidValue,
|
|
"Host header is not one this server answers to"))
|
|
return
|
|
}
|
|
next.ServeHTTP(w, r)
|
|
})
|
|
}
|
|
|
|
// dispatchCustomMethod is the one registration the single-resource custom
|
|
// methods share. It splits the trailing `:verb` off the matched segment, hands
|
|
// the request to the row that claims it, and leaves the id where that row's
|
|
// handler reads it.
|
|
//
|
|
// The split happens BEFORE s.route, which is what makes an unrouted suffix cost
|
|
// nothing: it takes no database slot and books no operation on the request
|
|
// line, exactly as the catch-all's 404 does. Answering it from inside a row's
|
|
// handler — where the claim answered it while it was the only POST here — would
|
|
// attribute every probe of this prefix to whichever operation happened to be
|
|
// first in the table.
|
|
func (s *Server) dispatchCustomMethod(rows []route) http.Handler {
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
rt, id, res := customMethodTarget(rows, r.PathValue(customMethodPathValue))
|
|
if res != nil {
|
|
s.fail(w, r, *res)
|
|
return
|
|
}
|
|
r.SetPathValue(customMethodIDValue, id)
|
|
s.route(rt).ServeHTTP(w, r)
|
|
})
|
|
}
|
|
|
|
// closeStreams signals every streaming handler that this server is shutting
|
|
// down. Idempotent: http.Server calls its registered hooks once per Shutdown,
|
|
// and a second Shutdown must not panic on an already-closed channel.
|
|
func (s *Server) closeStreams() {
|
|
s.closingOnce.Do(func() { close(s.closing) })
|
|
}
|
|
|
|
// route wraps one operation with the limits that apply to it: the per-request
|
|
// deadline, the bearer credential unless the operation is exempt, the
|
|
// Bd-Project-Id stamp check unless the operation is exempt, and — unless the
|
|
// operation is exempt — a database slot.
|
|
//
|
|
// The credential check runs BEFORE the semaphore, which is the load-bearing
|
|
// ordering: a storm of refused requests then costs one SHA-256 each and can
|
|
// never occupy the slots, or the SQL connections pinned to them, that
|
|
// authenticated clients are waiting for. It runs inside withRequestContext, so
|
|
// a 401 gets a request id and a request log line like every other refusal.
|
|
//
|
|
// The stamp check runs AFTER the credential and before the semaphore: the
|
|
// project-mismatch refusal is the one that discloses this server's own project
|
|
// id (server_project_id), so it must sit behind the authentication gate — an
|
|
// unauthenticated caller is turned away by the 401 before the stamp is ever
|
|
// compared, and so learns nothing about the workspace's identity.
|
|
func (s *Server) route(rt route) http.Handler {
|
|
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
|
rec := requestInfo(r.Context())
|
|
rec.op = rt.op
|
|
|
|
// A STREAMING OPERATION GETS NO DEADLINE. For every other row this is
|
|
// the backstop that stops a request from holding resources forever; on a
|
|
// held-open response it would do nothing but sever the stream at sixty
|
|
// seconds. What replaces it is per-read: streamEvents holds no slot
|
|
// between passes and bounds each read on its own.
|
|
if !rt.streaming {
|
|
ctx, cancel := context.WithTimeout(r.Context(), requestDeadline)
|
|
defer cancel()
|
|
r = r.WithContext(ctx)
|
|
}
|
|
|
|
if !rt.authExempt && s.auth != nil && !s.authorize(w, r, rec) {
|
|
return
|
|
}
|
|
|
|
// Identity before resources: a stamp for the wrong workspace is turned
|
|
// away before it can buy a database slot or open a unit of work, so a
|
|
// misdirected read or write costs nothing and mutates nothing. The two
|
|
// exempt routes (liveness, identity handshake) skip it. It runs after
|
|
// the credential check so the server_project_id it discloses stays
|
|
// behind the authentication gate.
|
|
if !rt.projectExempt {
|
|
if res := s.checkProjectStamp(r); res != nil {
|
|
s.fail(w, r, *res)
|
|
return
|
|
}
|
|
}
|
|
|
|
// Wire-floor before resources, same as the project stamp: a client that
|
|
// has declared a revision this build no longer supports is turned away
|
|
// before it can buy a database slot or open a unit of work. Checked on
|
|
// every route but health (see wireRevisionExempt) — including the
|
|
// identity handshake, where catching the mismatch is cheapest.
|
|
if !rt.wireRevisionExempt {
|
|
if res := s.checkWireRevision(r); res != nil {
|
|
s.fail(w, r, *res)
|
|
return
|
|
}
|
|
}
|
|
|
|
if !rt.bypassSemaphore {
|
|
release, err := s.acquire(r.Context(), rec)
|
|
if err != nil {
|
|
s.failErr(w, r, err)
|
|
return
|
|
}
|
|
defer release()
|
|
}
|
|
|
|
rt.handler(s, w, r)
|
|
})
|
|
}
|
|
|
|
// authorize verifies the request's bearer credential, writing the 401 itself
|
|
// and reporting whether the handler may run.
|
|
//
|
|
// The presented credential appears in NO log field and NO response byte. It is
|
|
// deliberately not recorded through rec.refuse, which is defined as an echoed
|
|
// CALLER VALUE and goes on the request line: a Host header or a parameter name
|
|
// is attacker-controlled text worth attributing, and a token is a secret. What
|
|
// the log gets instead is the reason — which of the three client mistakes it
|
|
// was — and the request id that ties it to the response.
|
|
func (s *Server) authorize(w http.ResponseWriter, r *http.Request, rec *reqInfo) bool {
|
|
token, reason := bearerCredential(r.Header.Get("Authorization"))
|
|
if reason == "" {
|
|
ok, reloadErr := s.auth.Verify(token)
|
|
if reloadErr != nil {
|
|
// A reload that failed leaves the last-good token set in force, so
|
|
// this is not a refusal on its own — but it means the file the
|
|
// operator is rotating is unreadable, and nothing else would say so.
|
|
s.event("auth_reload_error", "request_id", rec.id, "error", reloadErr.Error())
|
|
}
|
|
if ok {
|
|
return true
|
|
}
|
|
reason = "unknown_token"
|
|
}
|
|
|
|
s.event("auth_refused", "request_id", rec.id, "op", rec.op,
|
|
"reason", reason, "remote_addr", r.RemoteAddr)
|
|
s.fail(w, r, newResult(CodeUnauthenticated, ""))
|
|
return false
|
|
}
|
|
|
|
// bearerCredential extracts the token from an Authorization header value. It
|
|
// returns a non-empty reason instead when there is nothing to verify, so the
|
|
// log can separate a misconfigured client from a wrong or stale token.
|
|
//
|
|
// The scheme is matched case-insensitively, which RFC 9110 requires.
|
|
func bearerCredential(header string) (token, reason string) {
|
|
if strings.TrimSpace(header) == "" {
|
|
return "", "missing"
|
|
}
|
|
rest, ok := strings.CutPrefix(header, "Bearer ")
|
|
if !ok {
|
|
scheme, tail, split := strings.Cut(header, " ")
|
|
if !split || !strings.EqualFold(scheme, "Bearer") {
|
|
return "", "malformed"
|
|
}
|
|
rest = tail
|
|
}
|
|
if token = strings.TrimSpace(rest); token == "" {
|
|
return "", "malformed"
|
|
}
|
|
return token, ""
|
|
}
|
|
|
|
// authLabel describes the credential posture for the startup line. It names the
|
|
// token FILE, never a token: the path is configuration an operator already
|
|
// knows, and the contents are the secret.
|
|
func (s *Server) authLabel() string {
|
|
if s.auth == nil {
|
|
return "none"
|
|
}
|
|
return "bearer (" + s.auth.path + ")"
|
|
}
|
|
|
|
// checkProjectStamp enforces per-request workspace identity. A client that
|
|
// stamps a request with its intended workspace's project id in the
|
|
// Bd-Project-Id header is asserting "I mean to be talking to THIS workspace"; if
|
|
// the id it names is not the one this server serves, the request is refused
|
|
// before it can read or write the wrong workspace. It returns nil when the
|
|
// request may proceed, or the 400 to write.
|
|
//
|
|
// The comparison is LITERAL, and a server whose own project id is empty refuses
|
|
// any non-empty stamp: it cannot prove it is the workspace the client named, so
|
|
// it does not answer as if it were. That mirrors the identity handshake — a
|
|
// server advertises the id it can assert, and asserts nothing when it has none.
|
|
//
|
|
// Two paths return nil. An ABSENT header is the backward-compatible one: an
|
|
// older client never sends it, and enforcement triggers only when the header
|
|
// arrives, so this adds no precondition to any request already in the field. A
|
|
// stamp equal to the server's own project id is a match.
|
|
//
|
|
// The refusal is recorded on the request line like every other middleware
|
|
// refusal, so a client persistently addressing the wrong server is attributable
|
|
// on loopback down to the local process.
|
|
func (s *Server) checkProjectStamp(r *http.Request) *Result {
|
|
stamp := r.Header.Get(ProjectIDHeader)
|
|
if stamp == "" {
|
|
return nil
|
|
}
|
|
if stamp == s.ctxBody.ProjectId {
|
|
return nil
|
|
}
|
|
requestInfo(r.Context()).refuse(stamp)
|
|
res := ProjectMismatch(stamp, s.ctxBody.ProjectId)
|
|
return &res
|
|
}
|
|
|
|
// fail writes a problem response and records what it was for the log line.
|
|
// Every non-2xx byte this server emits goes through here or through
|
|
// handleNotImplemented.
|
|
func (s *Server) fail(w http.ResponseWriter, r *http.Request, res Result) {
|
|
rec := requestInfo(r.Context())
|
|
rec.code = Code(res.Problem.Code)
|
|
Write(w, res.WithRequestID(rec.id))
|
|
}
|
|
|
|
// failErr maps an error from the storage seam and logs the error text. On a 5xx
|
|
// that text goes to the log and NOWHERE else: driver and dial errors routinely
|
|
// embed the DSN, and the response detail is a fixed string per code. The
|
|
// request_id in both places is what reconnects them.
|
|
func (s *Server) failErr(w http.ResponseWriter, r *http.Request, err error) {
|
|
rec := requestInfo(r.Context())
|
|
res := ClassifyError(err)
|
|
s.fail(w, r, res)
|
|
|
|
// A client that hung up — while queued for a slot, or mid unit of work — is
|
|
// not a server fault, and this is the moment it would be counted as one:
|
|
// context.Canceled has nowhere better to go than the generic 500, and every
|
|
// >=500 emits request_error. On a saturated server, which is exactly when
|
|
// clients time out and disconnect, that turns impatient callers into a spike
|
|
// in the one signal an operator alerts on. The status stays as classified
|
|
// (it is written to a socket nobody is reading either way); only the
|
|
// accounting changes.
|
|
//
|
|
// An EXPIRED request deadline is a different statement and keeps the 500:
|
|
// nothing about it says the client left.
|
|
if errors.Is(err, context.Canceled) {
|
|
rec.code = codeClientClosed
|
|
return
|
|
}
|
|
if res.Problem.Status >= 500 {
|
|
s.event("request_error", "request_id", rec.id, "error", err.Error())
|
|
}
|
|
}
|
|
|
|
// writeJSON emits a success body. The status is always 200: every 2xx on this
|
|
// surface is a 200, and every non-2xx byte goes through Write as problem+json
|
|
// instead — so a status parameter here would only ever be a way to write one
|
|
// that the document does not describe.
|
|
func writeJSON(w http.ResponseWriter, body any) {
|
|
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
|
w.WriteHeader(http.StatusOK)
|
|
_ = json.NewEncoder(w).Encode(body)
|
|
}
|
|
|
|
// statusWriter records the status for the log line and bounds how long any one
|
|
// write may stall. It intentionally does not buffer the body: an unlimited read
|
|
// must stream.
|
|
type statusWriter struct {
|
|
http.ResponseWriter
|
|
rc *http.ResponseController
|
|
budget time.Duration
|
|
status int
|
|
}
|
|
|
|
func (w *statusWriter) WriteHeader(status int) {
|
|
if w.status == 0 {
|
|
w.status = status
|
|
}
|
|
w.extendWriteDeadline(w.budget)
|
|
w.ResponseWriter.WriteHeader(status)
|
|
}
|
|
|
|
func (w *statusWriter) Write(b []byte) (int, error) {
|
|
if w.status == 0 {
|
|
w.status = http.StatusOK
|
|
}
|
|
w.extendWriteDeadline(w.budget)
|
|
return w.ResponseWriter.Write(b)
|
|
}
|
|
|
|
// extendWriteDeadline rolls the connection's write deadline d into the future.
|
|
// Rolling it before every write bounds each write rather than the transfer: a
|
|
// client that keeps reading streams a body of any size, while one that stops
|
|
// reading fails its handler within d — which is what lets the deferred
|
|
// semaphore release and unit-of-work rollback actually run.
|
|
//
|
|
// SetWriteDeadline is unsupported on a ResponseWriter with no connection under
|
|
// it (httptest's recorder), where there is nothing to stall; the error is
|
|
// dropped for that reason and no other.
|
|
func (w *statusWriter) extendWriteDeadline(d time.Duration) {
|
|
if w.rc == nil || d <= 0 {
|
|
return
|
|
}
|
|
_ = w.rc.SetWriteDeadline(time.Now().Add(d))
|
|
}
|
|
|
|
// Unwrap keeps http.ResponseController working through the wrapper, so a
|
|
// handler that needs to flush a large streamed page still can.
|
|
func (w *statusWriter) Unwrap() http.ResponseWriter { return w.ResponseWriter }
|
|
|
|
// hostPolicy is the set of Host header values this server answers to. It is
|
|
// data rather than a closure so the startup line can state the whole policy,
|
|
// and so the wildcard case below is a visible rule instead of an absence.
|
|
type hostPolicy struct {
|
|
// ips are the numeric addresses allowed, matched with net.IP.Equal so every
|
|
// spelling of one address matches: [0:0:0:0:0:0:0:1] and [::ffff:127.0.0.1]
|
|
// are the same hosts as ::1 and 127.0.0.1, and a client that spells one of
|
|
// them the long way is not an attacker.
|
|
ips []net.IP
|
|
// names are the allowed non-numeric Host values, lowercased. "localhost"
|
|
// is always there; an operator may enumerate more with --allowed-host, and
|
|
// nothing else can add one. Matching is EXACT — no wildcard and no suffix
|
|
// syntax — so the allowlist is precisely what was enumerated.
|
|
names map[string]bool
|
|
// anyIP additionally allows ANY numeric Host literal. Only a wildcard bind
|
|
// sets it; see newHostPolicy for why that is still a rebinding defense.
|
|
anyIP bool
|
|
}
|
|
|
|
// newHostPolicy returns the Host policy implied by a bind address.
|
|
//
|
|
// The loopback spellings are always allowed, and the bind's own address is too
|
|
// — including an alternate loopback bind like 127.0.0.2, whose clients dial
|
|
// exactly that address and would otherwise be refused by the defense meant to
|
|
// protect them.
|
|
//
|
|
// A WILDCARD bind (0.0.0.0, ::) has no single configured address to allow, so
|
|
// it allows any numeric IP literal instead — and still refuses foreign DNS
|
|
// names, which is the whole defense. A rebound page cannot produce an IP-literal
|
|
// Host: the browser sends the hostname from the attacker's URL, and fetching an
|
|
// IP URL directly is a direct connection, which is the exposure the operator
|
|
// accepted when they passed --allow-non-loopback. Disabling the check outright
|
|
// would instead surrender the defense on the serving host's own loopback
|
|
// interface, which is rebinding's canonical target, and on every LAN browser
|
|
// behind a firewall the attacker cannot otherwise reach.
|
|
// EXTRA is the operator's enumerated additions (--allowed-host). A deployment
|
|
// where clients dial a service DNS name is refused by the policy above on every
|
|
// single request, so without this the server is unreachable rather than
|
|
// protected. Admitting a name the operator named does not weaken the defense
|
|
// the check exists for: a rebound page still cannot make a browser send that
|
|
// Host to 127.0.0.1, and the in-cluster clients that do send it are not
|
|
// browsers. Numeric values land in ips, so a pod IP can be enumerated too.
|
|
func newHostPolicy(bind net.IP, extra []string) hostPolicy {
|
|
p := hostPolicy{
|
|
ips: []net.IP{net.IPv4(127, 0, 0, 1), net.IPv6loopback},
|
|
names: map[string]bool{"localhost": true},
|
|
anyIP: bind.IsUnspecified(),
|
|
}
|
|
if !p.anyIP && !containsIP(p.ips, bind) {
|
|
p.ips = append(p.ips, bind)
|
|
}
|
|
for _, host := range extra {
|
|
h := hostOnly(host)
|
|
if ip := net.ParseIP(h); ip != nil {
|
|
if !containsIP(p.ips, ip) {
|
|
p.ips = append(p.ips, ip)
|
|
}
|
|
continue
|
|
}
|
|
p.names[h] = true
|
|
}
|
|
return p
|
|
}
|
|
|
|
// ValidateAllowedHost refuses an allowlist entry that is not a bare host.
|
|
//
|
|
// The Host header's port is stripped before matching (hostOnly), so an entry
|
|
// carrying one would silently never match — and an operator who wrote it would
|
|
// reasonably read the startup line as proof that it does. A URL, a path or
|
|
// embedded whitespace is the same mistake in a louder form.
|
|
func ValidateAllowedHost(v string) error {
|
|
if strings.TrimSpace(v) == "" {
|
|
return errors.New("--allowed-host is empty; pass the Host header value clients send, such as bd-myproject.beads.svc.cluster.local")
|
|
}
|
|
if strings.ContainsAny(v, " \t\r\n") {
|
|
return fmt.Errorf("--allowed-host %q contains whitespace; it must be a bare host name or IP", v)
|
|
}
|
|
if strings.ContainsAny(v, "/@") {
|
|
return fmt.Errorf("--allowed-host %q looks like a URL; pass just the host, with no scheme and no path", v)
|
|
}
|
|
// An IPv6 address is spelled in brackets in a Host header, so an operator
|
|
// copying one off the wire types it that way. hostOnly strips them before
|
|
// matching, so the entry works; refusing it here — with a message about a
|
|
// port it does not have — would be the validation lying about the policy.
|
|
if net.ParseIP(strings.TrimSuffix(strings.TrimPrefix(v, "["), "]")) != nil {
|
|
return nil
|
|
}
|
|
if strings.Contains(v, ":") {
|
|
return fmt.Errorf("--allowed-host %q carries a port; the port is stripped from a request's Host before matching, so an entry with one could never match", v)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// allows reports whether a Host header value is one this server answers to.
|
|
func (p hostPolicy) allows(host string) bool {
|
|
h := hostOnly(host)
|
|
if p.names[h] {
|
|
return true
|
|
}
|
|
ip := net.ParseIP(h)
|
|
if ip == nil {
|
|
return false
|
|
}
|
|
return p.anyIP || containsIP(p.ips, ip)
|
|
}
|
|
|
|
// label renders the policy for the startup line, so an operator can read what
|
|
// this server will answer to without deducing it from the bind address.
|
|
func (p hostPolicy) label() string {
|
|
parts := make([]string, 0, len(p.ips)+len(p.names)+1)
|
|
for _, ip := range p.ips {
|
|
parts = append(parts, ip.String())
|
|
}
|
|
parts = append(parts, slices.Sorted(maps.Keys(p.names))...)
|
|
if p.anyIP {
|
|
parts = append(parts, "any-ip-literal (wildcard bind)")
|
|
}
|
|
return strings.Join(parts, ",")
|
|
}
|
|
|
|
func containsIP(ips []net.IP, want net.IP) bool {
|
|
return slices.ContainsFunc(ips, want.Equal)
|
|
}
|
|
|
|
// hostOnly strips the port and any IPv6 brackets from a Host header value.
|
|
func hostOnly(host string) string {
|
|
if h, _, err := net.SplitHostPort(host); err == nil {
|
|
host = h
|
|
}
|
|
host = strings.TrimPrefix(host, "[")
|
|
host = strings.TrimSuffix(host, "]")
|
|
return strings.ToLower(host)
|
|
}
|
|
|
|
// newIDPrefix draws one random prefix per process so ids from two servers, or
|
|
// from two runs, never collide in a shared log.
|
|
func newIDPrefix() (string, error) {
|
|
var b [4]byte
|
|
if _, err := rand.Read(b[:]); err != nil {
|
|
return "", fmt.Errorf("httpapi: request id seed: %w", err)
|
|
}
|
|
return hex.EncodeToString(b[:]), nil
|
|
}
|
|
|
|
func (s *Server) nextID() string {
|
|
return fmt.Sprintf("%s-%06d", s.idPrefix, s.idSeq.Add(1))
|
|
}
|
|
|
|
func (s *Server) logStartup() {
|
|
s.event("startup",
|
|
"addr", s.Addr(),
|
|
"mode", s.cfg.Mode,
|
|
"db", s.dbSource(),
|
|
"workspace", s.cfg.Workspace.RepoRoot,
|
|
"beads_dir", s.cfg.Workspace.BeadsDir,
|
|
"database", s.cfg.Workspace.Database,
|
|
"host_allowlist", s.hosts.label(),
|
|
"capabilities", strings.Join(s.ctxBody.Capabilities, ","),
|
|
// Whether this server requires a credential is the first thing an
|
|
// operator checks after a deploy, and the last thing they should have
|
|
// to infer from the absence of a flag in a process listing.
|
|
"auth", s.authLabel(),
|
|
)
|
|
|
|
limits := []any{
|
|
"max_inflight", maxInflight,
|
|
"max_conns", maxConns,
|
|
"sem_wait", semAcquireTimeout.String(),
|
|
"deadline", requestDeadline.String(),
|
|
// large_apply_ceiling is the PER-ROUTE budget a POST
|
|
// issues:batchApply request over largeApplyItemThreshold items gets,
|
|
// separate from the ordinary "deadline" above: an operator reading
|
|
// this line must not conclude every request here is bounded by the
|
|
// 60s "deadline" figure just because that is the only one printed.
|
|
"large_apply_ceiling", orDefault(s.largeApplyCeiling, DefaultLargeApplyCeiling).String(),
|
|
}
|
|
// The pool bounds are this server's, applied to the provider above. On the
|
|
// roles source there is no pool here to bound, and printing the numbers
|
|
// anyway would report limits nothing enforces.
|
|
if s.provider != nil {
|
|
limits = append(limits,
|
|
"pool_max_open", servePoolLimits.MaxOpenConns,
|
|
"pool_max_idle", servePoolLimits.MaxIdleConns,
|
|
"pool_idle_time", servePoolLimits.ConnMaxIdleTime.String(),
|
|
"pool_lifetime", servePoolLimits.ConnMaxLifetime.String(),
|
|
)
|
|
}
|
|
s.event("limits", limits...)
|
|
}
|
|
|
|
// dbSource names which database source this server was built from, for the
|
|
// startup line.
|
|
//
|
|
// It is there so uow_ms is attributable. That field means "how long this
|
|
// request spent OBTAINING units of work", and a roles-backed server obtains
|
|
// none — so every one of its request lines reads uow_ms=0.000, which is the
|
|
// true value and is indistinguishable, on its own, from instrumentation that
|
|
// broke. This is the line that tells them apart.
|
|
func (s *Server) dbSource() string {
|
|
if s.provider != nil {
|
|
return "provider"
|
|
}
|
|
return "roles"
|
|
}
|
|
|
|
// event writes one structured stderr line. Values are quoted when they are not
|
|
// bare tokens, so a path or an error message can never inject a field — or a
|
|
// whole line — into the log.
|
|
func (s *Server) event(name string, kv ...any) {
|
|
var b strings.Builder
|
|
b.WriteString("event=")
|
|
b.WriteString(name)
|
|
for i := 0; i+1 < len(kv); i += 2 {
|
|
key, _ := kv[i].(string)
|
|
b.WriteByte(' ')
|
|
b.WriteString(key)
|
|
b.WriteByte('=')
|
|
b.WriteString(logValue(kv[i+1]))
|
|
}
|
|
s.log.Print(b.String())
|
|
}
|
|
|
|
// logValue renders one value of the k=v request line, quoting anything that
|
|
// would disturb either the line or the terminal reading it.
|
|
//
|
|
// Two audiences, two rules. The k=v framing needs space, '"' and '=' quoted, or
|
|
// a caller-supplied value forges fields and whole lines. The operator's console
|
|
// needs every CONTROL character quoted, C1 included: an unquoted U+009B is a
|
|
// CSI introducer, so a refusal recorded from a request body member name or a
|
|
// Content-Type header would paint the terminal of whoever tails the log. Bytes
|
|
// 0x80-0xFF are legal obs-text in an HTTP/1 field value and arrive here as
|
|
// invalid UTF-8 rather than as runes, so validity is checked separately —
|
|
// ContainsFunc would see only U+FFFD, which is not a control character.
|
|
func logValue(v any) string {
|
|
str, ok := v.(string)
|
|
if !ok {
|
|
return fmt.Sprint(v)
|
|
}
|
|
if str == "" {
|
|
return `""`
|
|
}
|
|
if !utf8.ValidString(str) || strings.ContainsFunc(str, func(r rune) bool {
|
|
return r <= ' ' || r == '"' || r == '=' || isControlChar(r)
|
|
}) {
|
|
return strconv.Quote(str)
|
|
}
|
|
return str
|
|
}
|
|
|
|
func millis(d time.Duration) string {
|
|
return strconv.FormatFloat(float64(d)/float64(time.Millisecond), 'f', 3, 64)
|
|
}
|