* fix(httpclient): bound response-header waits and retry stalled reads on a new connection The maintainer-city shadow run saw 22 requests sit out the full 60s client timeout, each on one silent HTTP/1.1 connection while other connections to the same serve kept answering. wire.Client now bounds an idempotent request's wait for response headers (15s default; BEADS_HTTP_RESPONSE_HEADER_TIMEOUT or DialOptions.ResponseHeaderTimeout; "off" disables), separately from the 60s overall timeout, and stops the timer once headers arrive so a streaming body is never cut. A read (GET/HEAD/OPTIONS, or a Request marked Idempotent: the POST batch get) that stalls or has its connection reset is retried once: the stalled attempt's cancellation closes its connection and idle connections are dropped, so the retry dials fresh. Every retry is counted (wire.StallRetryCount) and reported (OnStallRetry; the dialer's default logs one stderr line with the count). Writes are never retried and get no header timer, because bd serve answers a write only after it commits (a large batch apply can legitimately take tens of seconds). A write whose overall timeout fires before its headers fails with a typed *wire.ResponseHeaderTimeoutError (inside *ConnectError) naming the host and saying the outcome is unknown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(httpclient): retry a stalled read within the overall timeout (bd-p1wzpi) Adoption review of #7553 found three defects in the header timer and its retry. This maintainer fixup sits on top of the contributor commit and supersedes three of its claims: that the timer bounds a read's every attempt, that idle connections are dropped before the retry, and that any write whose overall timeout fires before its headers is typed. A slow but healthy read is no longer cut short. The 15s timer cannot tell a silent connection from a serve still working on the read (serve writes a read's headers only after waiting up to 10s for a database slot and running the query), and the retry ran under a fresh 15s timer, so a read slower than 15s failed at about 30s after sending two requests where it used to succeed within 60s. The timer now bounds only the first attempt. The retry has no header timer and runs under a deadline of when the first attempt was sent plus the client's overall Timeout, the moment that Timeout itself counts from, so it finishes such a read and no read waits longer than the 60s it could before. The request is built and stamped once and each attempt sends a copy of it, so the credential source is consulted once, before any clock starts, and a cold token command costs the retry none of its budget. What is left: a read whose headers take longer than the timer, and whose retry then needs more than what remains of the 60s (about 45s) for its headers and body together, now fails at 60s where it used to succeed, and the server also gets the abandoned first request. A read whose first attempt already spent the whole Timeout (the timer off) is not retried at all and keeps its typed timeout. A request is typed *ResponseHeaderTimeoutError only once it was written. Any Timeout() error on a write used to be typed, including the baseline transport's dial and TLS handshake timeouts where nothing reached the server, and the message then said the write "may or may not have been applied", gave the overall timeout as the wait, dropped the transport's error, and Temporary() was true. The timer is now armed from httptrace's WroteRequest (where http.Transport.ResponseHeaderTimeout starts its own clock), which also records that the request went out. A timeout is typed only for a written request whose caller's context is still alive; a dial or TLS timeout stays a plain *ConnectError. After is the measured wait, Err is printed, Temporary() reports Replayable (false for a write), and a retried read says why its first attempt was abandoned (Reason). Arming at the write also stops a slow credential source (BEADS_HTTP_TOKEN_COMMAND may take 30s) from being reported, counted and retried as a server stall. No idle-connection flush. CloseIdleConnections ran on a transport shared by every no-CA dial in the process (baselineTransport), so one stalled read dropped idle keep-alives to every host. Canceling the stalled attempt already closes its connection (net/http never pools a canceled HTTP/1.1 connection), so the retry rides another one. Also: - wire.Request.Replayable (GET/HEAD/OPTIONS, or Idempotent) is now the one predicate behind both the wire's stall retry and dispatch's bounded 503 retry, so the batch get, a read on POST, is retried on a 503 too. dispatch's doc comment no longer says the wire never resends. - The docs, the error and the stall line say the retry goes out on "another connection", not a "new" one: the idle pool may hand it one already open. dialStream's doc says the events:watch connect gets neither the timer nor the retry. - BEADS_HTTP_RESPONSE_HEADER_TIMEOUT takes bare seconds ("30") as BEADS_GATE_WAIT_TIMEOUT does, and "off" in any case. "off" or 0 turns off only the timer: a reset connection is still retried and a written request that times out is still typed. - README says "each read" and spells out the above, bounding the retry by its headers and body together; it no longer says a streaming body is never cut off (the header timer never does, the 60s bound still does). The CHANGELOG entry moves from Added to Fixed and states the same bound. - CredentialProvider's doc, in wire and on the public bdhttp alias, says Authorize runs once per request, not once per attempt: the stall retry resends the stamped headers up to the overall timeout later, so a single-use or short-lived credential may draw a 401 there, which reaches Refresh like any other. Tests: a read queued behind a LimitListener backlog finishes once a slot frees; a read slower than the timer on every connection finishes on its retry; a slow credential source is not a stall, and one that takes the whole Timeout to resolve still leaves the retry its budget (starting the deadline before the stamp, or re-stamping the retry, fails it); a write whose dial times out is a plain *ConnectError; a stall leaves another server's idle connection on a shared transport alone; a retry after a reset names the reset; the stall-twice case is bounded by the overall timeout and pins After, Reason and the message; dispatch retries a read on POST on a 503, and GetMany's batch get is sent as one (dropping its Idempotent fails that); the env knob takes "30", " 1.5 " and "OFF" and refuses "-3". A read whose first attempt spent the whole Timeout is not retried and keeps its typed error (dropping the spent-budget check fails it; no other test did). A stall retry that draws a 401 refreshes once and authorizes only the re-issued request, so Authorize runs twice, not three times (re-stamping the retry fails it). Not covered, as before, and now said in README and the CHANGELOG: the events:watch stream's connect (dialStream) has no header timer or retry. Nor is a reset on Windows: stallReason matches syscall.ECONNRESET, which a Windows reset (WSAECONNRESET) is not, so there a reset connection is retried only when it surfaces as EOF. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Eddie the Engineer <adopt-pr@gascity.com>
187 lines
7 KiB
Go
187 lines
7 KiB
Go
// Contributed by gascity from bd-enterprise (internal/enterprise/httpstore/dispatch_test.go@49d1df2f6)
|
|
// to OSS beads under the MIT license.
|
|
package httpclient
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"net/http"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/steveyegge/beads/internal/httpapi/apigen"
|
|
"github.com/steveyegge/beads/internal/httpclient/wire"
|
|
"github.com/steveyegge/beads/issueops"
|
|
)
|
|
|
|
// busyThenOK answers with a 503 for the first busyCount Do calls and succeeds
|
|
// after, recording how many times it was dialed. It is the dispatch retry's
|
|
// double: a real server's slot pressure clears the same way.
|
|
type busyThenOK struct {
|
|
busyCount int
|
|
retryAfter time.Duration
|
|
calls int
|
|
}
|
|
|
|
func (b *busyThenOK) do(_ context.Context, req wire.Request, _ any) error {
|
|
b.calls++
|
|
if b.calls <= b.busyCount {
|
|
return &wire.ProblemError{
|
|
Op: req.Op,
|
|
Status: http.StatusServiceUnavailable,
|
|
Err: wire.ErrBusy,
|
|
RetryAfter: b.retryAfter,
|
|
}
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func retryTestStore(t *testing.T, do func(context.Context, wire.Request, any) error) *Store {
|
|
t.Helper()
|
|
w := &fakeWire{
|
|
res: &apigen.ContextResponse{},
|
|
preflight: func(context.Context, string) error { return nil },
|
|
do: do,
|
|
}
|
|
return New(testTarget(t), w, nil)
|
|
}
|
|
|
|
func readRequest() wire.Request {
|
|
return wire.Request{Op: wire.OpListReadyWork, Method: http.MethodGet, Path: wire.PathReady}
|
|
}
|
|
|
|
// TestDispatchRetriesIdempotentReadsOnBusy: a GET that a 503 turned away comes
|
|
// back, honors the server's Retry-After, and succeeds once the slot frees.
|
|
func TestDispatchRetriesIdempotentReadsOnBusy(t *testing.T) {
|
|
server := &busyThenOK{busyCount: 2, retryAfter: time.Millisecond}
|
|
s := retryTestStore(t, server.do)
|
|
|
|
if err := s.dispatch(context.Background(), readRequest(), nil); err != nil {
|
|
t.Fatalf("dispatch: %v, want the retry to succeed", err)
|
|
}
|
|
if server.calls != 3 {
|
|
t.Errorf("Do called %d times, want 3 (one dial plus two retries)", server.calls)
|
|
}
|
|
}
|
|
|
|
// TestDispatchCapsIdempotentRetries: a server that stays busy is not dialed
|
|
// forever. The bound is one dial plus maxIdempotentRetries, and the 503 is what
|
|
// the caller finally sees.
|
|
func TestDispatchCapsIdempotentRetries(t *testing.T) {
|
|
server := &busyThenOK{busyCount: 100, retryAfter: time.Millisecond}
|
|
s := retryTestStore(t, server.do)
|
|
|
|
err := s.dispatch(context.Background(), readRequest(), nil)
|
|
if !errors.Is(err, wire.ErrBusy) {
|
|
t.Fatalf("dispatch = %v, want the busy error once the retries are spent", err)
|
|
}
|
|
if server.calls != 1+maxIdempotentRetries {
|
|
t.Errorf("Do called %d times, want %d (one dial plus the capped retries)", server.calls, 1+maxIdempotentRetries)
|
|
}
|
|
}
|
|
|
|
// TestDispatchRetriesAReadOnPostOnBusy: a read carried on POST that marks
|
|
// itself Idempotent comes back after a 503 the way a GET does. That the batch
|
|
// get marks itself is TestGetManyRetriesTheBatchGetOnBusy's.
|
|
func TestDispatchRetriesAReadOnPostOnBusy(t *testing.T) {
|
|
server := &busyThenOK{busyCount: 1, retryAfter: time.Millisecond}
|
|
s := retryTestStore(t, server.do)
|
|
|
|
read := wire.Request{Op: wire.OpBatchGetIssues, Method: http.MethodPost, Path: wire.PathIssuesBatchGet, Idempotent: true}
|
|
if err := s.dispatch(context.Background(), read, nil); err != nil {
|
|
t.Fatalf("dispatch: %v, want the retry to succeed", err)
|
|
}
|
|
if server.calls != 2 {
|
|
t.Errorf("Do called %d times, want 2 (one dial plus one retry)", server.calls)
|
|
}
|
|
}
|
|
|
|
// TestGetManyRetriesTheBatchGetOnBusy: the batch get is a read dialed as a
|
|
// POST, so it is replayable only because it says so (Idempotent). Through
|
|
// GetMany it comes back after a 503, and what it sends is replayable for the
|
|
// wire's stall retry too.
|
|
func TestGetManyRetriesTheBatchGetOnBusy(t *testing.T) {
|
|
token, _ := wire.CapabilityFor(wire.OpBatchGetIssues)
|
|
server := &busyThenOK{busyCount: 1, retryAfter: time.Millisecond}
|
|
var sent []wire.Request
|
|
w := &fakeWire{
|
|
res: &apigen.ContextResponse{Capabilities: []string{token}},
|
|
preflight: func(context.Context, string) error { return nil },
|
|
do: func(ctx context.Context, req wire.Request, out any) error {
|
|
sent = append(sent, req)
|
|
return server.do(ctx, req, out)
|
|
},
|
|
}
|
|
g := &httpBatchGetter{store: New(testTarget(t), w, nil)}
|
|
|
|
if _, err := g.GetMany(context.Background(), issueops.GetManyRequest{IDs: []string{"bd-1"}}); err != nil {
|
|
t.Fatalf("GetMany: %v, want the busy batch get retried", err)
|
|
}
|
|
if server.calls != 2 {
|
|
t.Errorf("Do called %d times, want 2 (one dial plus one retry)", server.calls)
|
|
}
|
|
for _, req := range sent {
|
|
if req.Op != wire.OpBatchGetIssues || req.Method != http.MethodPost || !req.Replayable() {
|
|
t.Errorf("sent %s %s (replayable %v), want the batch get as a replayable POST", req.Method, req.Op, req.Replayable())
|
|
}
|
|
}
|
|
}
|
|
|
|
// TestDispatchDoesNotRetryWrites: a non-idempotent write is dialed exactly once
|
|
// on a 503, because its first attempt may have committed before the server ran
|
|
// out of slots and a resend would double-apply it.
|
|
func TestDispatchDoesNotRetryWrites(t *testing.T) {
|
|
server := &busyThenOK{busyCount: 1, retryAfter: time.Millisecond}
|
|
s := retryTestStore(t, server.do)
|
|
|
|
write := wire.Request{Op: wire.OpClaimIssue, Method: http.MethodPost, Path: "/v0/beads/issues/bd-1:claim"}
|
|
err := s.dispatch(context.Background(), write, nil)
|
|
if !errors.Is(err, wire.ErrBusy) {
|
|
t.Fatalf("dispatch = %v, want the busy error unretried", err)
|
|
}
|
|
if server.calls != 1 {
|
|
t.Errorf("a write was dialed %d times on a 503, want exactly 1", server.calls)
|
|
}
|
|
}
|
|
|
|
// TestDispatchDoesNotRetryNonBusyReads: a read is retried only for the 503 busy
|
|
// class. A 500 fault is the same request earning the same answer, so it is not
|
|
// retried.
|
|
func TestDispatchDoesNotRetryNonBusyReads(t *testing.T) {
|
|
var calls int
|
|
s := retryTestStore(t, func(_ context.Context, req wire.Request, _ any) error {
|
|
calls++
|
|
return &wire.ProblemError{Op: req.Op, Status: http.StatusInternalServerError, Err: wire.ErrServerFault}
|
|
})
|
|
|
|
err := s.dispatch(context.Background(), readRequest(), nil)
|
|
if !errors.Is(err, wire.ErrServerFault) {
|
|
t.Fatalf("dispatch = %v, want the server fault unretried", err)
|
|
}
|
|
if calls != 1 {
|
|
t.Errorf("a 500 was dialed %d times, want exactly 1", calls)
|
|
}
|
|
}
|
|
|
|
// TestDispatchRetryStopsWhenTheContextIsDone: a caller whose deadline expires
|
|
// mid-backoff stops retrying and gets the server's busy answer, not a wait that
|
|
// outlives the request.
|
|
func TestDispatchRetryStopsWhenTheContextIsDone(t *testing.T) {
|
|
server := &busyThenOK{busyCount: 100, retryAfter: time.Hour} // never clears in time
|
|
s := retryTestStore(t, server.do)
|
|
|
|
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
|
|
defer cancel()
|
|
|
|
start := time.Now()
|
|
err := s.dispatch(ctx, readRequest(), nil)
|
|
if !errors.Is(err, wire.ErrBusy) {
|
|
t.Fatalf("dispatch = %v, want the busy error when the deadline cut the wait short", err)
|
|
}
|
|
if elapsed := time.Since(start); elapsed > time.Second {
|
|
t.Errorf("dispatch honored the hour-long Retry-After past the caller's deadline (%s)", elapsed)
|
|
}
|
|
if server.calls != 1 {
|
|
t.Errorf("Do called %d times; the deadline should have stopped the first backoff", server.calls)
|
|
}
|
|
}
|