beads/backend/http/credential.go
Julian Knutsen 048f5e5192
fix(httpclient): bound response-header waits and retry stalled reads on another connection (#7553)
* fix(httpclient): bound response-header waits and retry stalled reads on a new connection

The maintainer-city shadow run saw 22 requests sit out the full 60s client
timeout, each on one silent HTTP/1.1 connection while other connections to
the same serve kept answering.

wire.Client now bounds an idempotent request's wait for response headers
(15s default; BEADS_HTTP_RESPONSE_HEADER_TIMEOUT or
DialOptions.ResponseHeaderTimeout; "off" disables), separately from the 60s
overall timeout, and stops the timer once headers arrive so a streaming body
is never cut. A read (GET/HEAD/OPTIONS, or a Request marked Idempotent: the
POST batch get) that stalls or has its connection reset is retried once: the
stalled attempt's cancellation closes its connection and idle connections are
dropped, so the retry dials fresh. Every retry is counted
(wire.StallRetryCount) and reported (OnStallRetry; the dialer's default logs
one stderr line with the count).

Writes are never retried and get no header timer, because bd serve answers
a write only after it commits (a large batch apply can legitimately take
tens of seconds). A write whose overall timeout fires before its headers
fails with a typed *wire.ResponseHeaderTimeoutError (inside *ConnectError)
naming the host and saying the outcome is unknown.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(httpclient): retry a stalled read within the overall timeout (bd-p1wzpi)

Adoption review of #7553 found three defects in the header timer and its
retry. This maintainer fixup sits on top of the contributor commit and
supersedes three of its claims: that the timer bounds a read's every
attempt, that idle connections are dropped before the retry, and that any
write whose overall timeout fires before its headers is typed.

A slow but healthy read is no longer cut short. The 15s timer cannot tell
a silent connection from a serve still working on the read (serve writes a
read's headers only after waiting up to 10s for a database slot and running
the query), and the retry ran under a fresh 15s timer, so a read slower
than 15s failed at about 30s after sending two requests where it used to
succeed within 60s. The timer now bounds only the first attempt. The retry
has no header timer and runs under a deadline of when the first attempt was
sent plus the client's overall Timeout, the moment that Timeout itself
counts from, so it finishes such a read and no read waits longer than the
60s it could before. The request is built and stamped once and each
attempt sends a copy of it, so the credential source is consulted once,
before any clock starts, and a cold token command costs the retry none of
its budget. What is left: a read whose headers take longer than the timer,
and whose retry then needs more than what remains of the 60s (about 45s)
for its headers and body together, now fails at 60s where it used to
succeed, and the server also gets the abandoned first request. A read whose
first attempt already spent the whole Timeout (the timer off) is not
retried at all and keeps its typed timeout.

A request is typed *ResponseHeaderTimeoutError only once it was written.
Any Timeout() error on a write used to be typed, including the baseline
transport's dial and TLS handshake timeouts where nothing reached the
server, and the message then said the write "may or may not have been
applied", gave the overall timeout as the wait, dropped the transport's
error, and Temporary() was true. The timer is now armed from httptrace's
WroteRequest (where http.Transport.ResponseHeaderTimeout starts its own
clock), which also records that the request went out. A timeout is typed
only for a written request whose caller's context is still alive; a dial
or TLS timeout stays a plain *ConnectError. After is the measured wait,
Err is printed, Temporary() reports Replayable (false for a write), and a
retried read says why its first attempt was abandoned (Reason). Arming at
the write also stops a slow credential source (BEADS_HTTP_TOKEN_COMMAND may
take 30s) from being reported, counted and retried as a server stall.

No idle-connection flush. CloseIdleConnections ran on a transport shared
by every no-CA dial in the process (baselineTransport), so one stalled read
dropped idle keep-alives to every host. Canceling the stalled attempt
already closes its connection (net/http never pools a canceled HTTP/1.1
connection), so the retry rides another one.

Also:
- wire.Request.Replayable (GET/HEAD/OPTIONS, or Idempotent) is now the one
  predicate behind both the wire's stall retry and dispatch's bounded 503
  retry, so the batch get, a read on POST, is retried on a 503 too.
  dispatch's doc comment no longer says the wire never resends.
- The docs, the error and the stall line say the retry goes out on
  "another connection", not a "new" one: the idle pool may hand it one
  already open. dialStream's doc says the events:watch connect gets
  neither the timer nor the retry.
- BEADS_HTTP_RESPONSE_HEADER_TIMEOUT takes bare seconds ("30") as
  BEADS_GATE_WAIT_TIMEOUT does, and "off" in any case. "off" or 0 turns off
  only the timer: a reset connection is still retried and a written request
  that times out is still typed.
- README says "each read" and spells out the above, bounding the retry by
  its headers and body together; it no longer says a streaming body is
  never cut off (the header timer never does, the 60s bound still does).
  The CHANGELOG entry moves from Added to Fixed and states the same bound.
- CredentialProvider's doc, in wire and on the public bdhttp alias, says
  Authorize runs once per request, not once per attempt: the stall retry
  resends the stamped headers up to the overall timeout later, so a
  single-use or short-lived credential may draw a 401 there, which reaches
  Refresh like any other.

Tests: a read queued behind a LimitListener backlog finishes once a slot
frees; a read slower than the timer on every connection finishes on its
retry; a slow credential source is not a stall, and one that takes the
whole Timeout to resolve still leaves the retry its budget (starting the
deadline before the stamp, or re-stamping the retry, fails it); a write
whose dial times out is a plain *ConnectError; a stall leaves another
server's idle connection on a shared transport alone; a retry after a
reset names the reset; the stall-twice case is bounded by the overall
timeout and pins After, Reason and the message; dispatch retries a read
on POST on a 503, and GetMany's batch get is sent as one (dropping its
Idempotent fails that); the env knob takes "30", " 1.5 " and "OFF" and
refuses "-3". A read whose first attempt spent the whole Timeout is not
retried and keeps its typed error (dropping the spent-budget check fails
it; no other test did). A stall retry that draws a 401 refreshes once and
authorizes only the re-issued request, so Authorize runs twice, not three
times (re-stamping the retry fails it).

Not covered, as before, and now said in README and the CHANGELOG: the
events:watch stream's connect (dialStream) has no header timer or retry.
Nor is a reset on Windows: stallReason matches syscall.ECONNRESET, which a
Windows reset (WSAECONNRESET) is not, so there a reset connection is
retried only when it surfaces as EOF.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <adopt-pr@gascity.com>
2026-10-14 05:04:19 +00:00

66 lines
3.1 KiB
Go

package bdhttp
import (
"net/url"
"github.com/steveyegge/beads/internal/httpclient"
)
// CredentialProvider authorizes outbound requests to a bd serve.
//
// An implementation must be safe for concurrent use, must never log or
// persist a secret, and must FAIL CLOSED: a configured source that errors
// aborts the request rather than silently downgrading to an unauthenticated
// one.
//
// Authorize is called once per request, not once per attempt: a read whose
// first attempt stalls or has its connection reset is resent once with the
// same headers, up to the client's overall timeout later. A provider whose
// credentials are single-use or short-lived should expect that resend to draw
// a 401, which reaches Refresh like any other.
type CredentialProvider = httpclient.CredentialProvider
const (
// TokenEnv carries the bearer token itself, the highest rung of the
// built-in ladder.
// #nosec G101 -- the NAME of an environment variable, not a credential.
TokenEnv = httpclient.TokenEnv
// TokenCommandEnv names a helper that prints a token, either bare or in
// the kubectl ExecCredential envelope {"token","expirationTimestamp"}.
// #nosec G101 -- the NAME of an environment variable, not a credential.
TokenCommandEnv = httpclient.TokenCommandEnv
)
// BearerProvider is the built-in credential ladder: TokenEnv, then
// TokenCommandEnv, then the credentials file's [host:port] section, then no
// credential at all (the tip OSS server's loopback-trust posture, a
// legitimate answer and not a failure).
type BearerProvider = httpclient.BearerProvider
// NewBearerProvider builds the built-in ladder for a server. Open and
// Handshake already dial with it; it is exported so an embedder that wants to
// WRAP the default (decorate it for one target, delegate to it for the rest)
// does not have to reimplement it.
func NewBearerProvider(base *url.URL) *BearerProvider {
return httpclient.NewBearerProvider(base)
}
// ProvidedCredential adapts a CredentialProvider into the opaque
// backends.Credential marker interface, so it can be carried through
// beads.OpenOptions / beads.OpenBestAvailableWith (and, directly, through
// backends.OpenOptions) to the registered "http" backend's OpenWith hook
// Register installs.
//
// A multi-tenant embedder (for example gc, Gas City: one process serving many
// workspaces, called "cities") that wants a distinct credential per workspace
// wraps that workspace's own provider in a ProvidedCredential when it opens
// through the registry (the direct door takes the provider itself, as
// Options.Credential): the registered dialer Register installs is
// process-wide and cannot hold one credential per tenant, but the per-open
// OpenWith seam can, and this is the opaque wrapper type that seam
// recognizes. Either way the supplied provider is the ONLY credential the
// open uses, and no ambient credential, CA or plaintext grant is read; see the
// package doc's "Credentials" section. See httpclient.ResolveCredential and
// backends.OpenOptions.Credential's own doc comment for the single-tenant
// ambient-ladder default this exists to opt out of.
type ProvidedCredential = httpclient.ProvidedCredential