fix(httpclient): bound response-header waits and retry stalled reads on another connection (#7553)
* fix(httpclient): bound response-header waits and retry stalled reads on a new connection The maintainer-city shadow run saw 22 requests sit out the full 60s client timeout, each on one silent HTTP/1.1 connection while other connections to the same serve kept answering. wire.Client now bounds an idempotent request's wait for response headers (15s default; BEADS_HTTP_RESPONSE_HEADER_TIMEOUT or DialOptions.ResponseHeaderTimeout; "off" disables), separately from the 60s overall timeout, and stops the timer once headers arrive so a streaming body is never cut. A read (GET/HEAD/OPTIONS, or a Request marked Idempotent: the POST batch get) that stalls or has its connection reset is retried once: the stalled attempt's cancellation closes its connection and idle connections are dropped, so the retry dials fresh. Every retry is counted (wire.StallRetryCount) and reported (OnStallRetry; the dialer's default logs one stderr line with the count). Writes are never retried and get no header timer, because bd serve answers a write only after it commits (a large batch apply can legitimately take tens of seconds). A write whose overall timeout fires before its headers fails with a typed *wire.ResponseHeaderTimeoutError (inside *ConnectError) naming the host and saying the outcome is unknown. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(httpclient): retry a stalled read within the overall timeout (bd-p1wzpi) Adoption review of #7553 found three defects in the header timer and its retry. This maintainer fixup sits on top of the contributor commit and supersedes three of its claims: that the timer bounds a read's every attempt, that idle connections are dropped before the retry, and that any write whose overall timeout fires before its headers is typed. A slow but healthy read is no longer cut short. The 15s timer cannot tell a silent connection from a serve still working on the read (serve writes a read's headers only after waiting up to 10s for a database slot and running the query), and the retry ran under a fresh 15s timer, so a read slower than 15s failed at about 30s after sending two requests where it used to succeed within 60s. The timer now bounds only the first attempt. The retry has no header timer and runs under a deadline of when the first attempt was sent plus the client's overall Timeout, the moment that Timeout itself counts from, so it finishes such a read and no read waits longer than the 60s it could before. The request is built and stamped once and each attempt sends a copy of it, so the credential source is consulted once, before any clock starts, and a cold token command costs the retry none of its budget. What is left: a read whose headers take longer than the timer, and whose retry then needs more than what remains of the 60s (about 45s) for its headers and body together, now fails at 60s where it used to succeed, and the server also gets the abandoned first request. A read whose first attempt already spent the whole Timeout (the timer off) is not retried at all and keeps its typed timeout. A request is typed *ResponseHeaderTimeoutError only once it was written. Any Timeout() error on a write used to be typed, including the baseline transport's dial and TLS handshake timeouts where nothing reached the server, and the message then said the write "may or may not have been applied", gave the overall timeout as the wait, dropped the transport's error, and Temporary() was true. The timer is now armed from httptrace's WroteRequest (where http.Transport.ResponseHeaderTimeout starts its own clock), which also records that the request went out. A timeout is typed only for a written request whose caller's context is still alive; a dial or TLS timeout stays a plain *ConnectError. After is the measured wait, Err is printed, Temporary() reports Replayable (false for a write), and a retried read says why its first attempt was abandoned (Reason). Arming at the write also stops a slow credential source (BEADS_HTTP_TOKEN_COMMAND may take 30s) from being reported, counted and retried as a server stall. No idle-connection flush. CloseIdleConnections ran on a transport shared by every no-CA dial in the process (baselineTransport), so one stalled read dropped idle keep-alives to every host. Canceling the stalled attempt already closes its connection (net/http never pools a canceled HTTP/1.1 connection), so the retry rides another one. Also: - wire.Request.Replayable (GET/HEAD/OPTIONS, or Idempotent) is now the one predicate behind both the wire's stall retry and dispatch's bounded 503 retry, so the batch get, a read on POST, is retried on a 503 too. dispatch's doc comment no longer says the wire never resends. - The docs, the error and the stall line say the retry goes out on "another connection", not a "new" one: the idle pool may hand it one already open. dialStream's doc says the events:watch connect gets neither the timer nor the retry. - BEADS_HTTP_RESPONSE_HEADER_TIMEOUT takes bare seconds ("30") as BEADS_GATE_WAIT_TIMEOUT does, and "off" in any case. "off" or 0 turns off only the timer: a reset connection is still retried and a written request that times out is still typed. - README says "each read" and spells out the above, bounding the retry by its headers and body together; it no longer says a streaming body is never cut off (the header timer never does, the 60s bound still does). The CHANGELOG entry moves from Added to Fixed and states the same bound. - CredentialProvider's doc, in wire and on the public bdhttp alias, says Authorize runs once per request, not once per attempt: the stall retry resends the stamped headers up to the overall timeout later, so a single-use or short-lived credential may draw a 401 there, which reaches Refresh like any other. Tests: a read queued behind a LimitListener backlog finishes once a slot frees; a read slower than the timer on every connection finishes on its retry; a slow credential source is not a stall, and one that takes the whole Timeout to resolve still leaves the retry its budget (starting the deadline before the stamp, or re-stamping the retry, fails it); a write whose dial times out is a plain *ConnectError; a stall leaves another server's idle connection on a shared transport alone; a retry after a reset names the reset; the stall-twice case is bounded by the overall timeout and pins After, Reason and the message; dispatch retries a read on POST on a 503, and GetMany's batch get is sent as one (dropping its Idempotent fails that); the env knob takes "30", " 1.5 " and "OFF" and refuses "-3". A read whose first attempt spent the whole Timeout is not retried and keeps its typed error (dropping the spent-budget check fails it; no other test did). A stall retry that draws a 401 refreshes once and authorizes only the re-issued request, so Authorize runs twice, not three times (re-stamping the retry fails it). Not covered, as before, and now said in README and the CHANGELOG: the events:watch stream's connect (dialStream) has no header timer or retry. Nor is a reset on Windows: stallReason matches syscall.ECONNRESET, which a Windows reset (WSAECONNRESET) is not, so there a reset connection is retried only when it surfaces as EOF. Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Julian Knutsen Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Eddie the Engineer <adopt-pr@gascity.com>
This commit is contained in:
parent
e7ad85bcff
commit
048f5e5192
15 changed files with 1480 additions and 18 deletions
15
CHANGELOG.md
15
CHANGELOG.md
|
|
@ -476,6 +476,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
of a remote export puts genuine no-history wisps on the durable issues
|
||||
plane instead of the wisps plane, and gc's JSONL archive keeps them.
|
||||
Nothing is dropped.
|
||||
- **A silent HTTP connection no longer costs a read the full 60s.** The HTTP
|
||||
backend client gives a read's first attempt 15s from when it is sent to get
|
||||
its response headers (`BEADS_HTTP_RESPONSE_HEADER_TIMEOUT`: a duration, a
|
||||
number of seconds, or `off`), then retries it once on another connection
|
||||
with no header timer, on what is left of the same 60s request timeout. A
|
||||
slow but healthy read still finishes if its retry's headers and body
|
||||
together fit in that remainder (about 45s); one that needs longer now fails
|
||||
at 60s. A read whose connection is reset is retried the same way (on Windows
|
||||
only when the reset surfaces as an early EOF), and each retry is logged to
|
||||
stderr with its count. The events:watch stream's connect gets neither the
|
||||
timer nor the retry. A write is never retried; one that was sent and timed
|
||||
out before its headers fails with a typed `wire.ResponseHeaderTimeoutError`
|
||||
naming the server's host, because it may or may not have been applied,
|
||||
while a dial or TLS timeout stays a plain connection error. The batch get,
|
||||
a read on POST, is now also retried on a 503 the way a GET is.
|
||||
- **An update of a template is now refused for every caller, not only
|
||||
`bd update`.** The template guard used to be a pre-read in `cmd/bd`, so an
|
||||
update through `bd serve` (`PATCH /v0/beads/issues/{id}`), an `update` item
|
||||
|
|
|
|||
23
README.md
23
README.md
|
|
@ -251,6 +251,29 @@ credential would cross the network unencrypted) — pass `--allow-plaintext` to
|
|||
override, or use `https://`. Switching a workspace that already selects a
|
||||
different backend needs `--force`.
|
||||
|
||||
Each read (a GET, or the POST batch get) gives its first attempt 15s, from
|
||||
when the request is sent, to get the server's response headers
|
||||
(`BEADS_HTTP_RESPONSE_HEADER_TIMEOUT`: a duration such as `30s`, a number of
|
||||
seconds, or `off`). A read whose headers do not arrive in time, or whose
|
||||
connection is reset, is retried once on another connection, and each retry
|
||||
is logged to stderr with its count. The retry has no header timer, but it
|
||||
runs on what is left of the 60s bound on the whole request, counted from
|
||||
when the first attempt was sent: a slow but healthy read still finishes if
|
||||
the retry's headers and body together fit in that remainder (about 45s after
|
||||
a header timeout), and one that needs longer now fails at 60s. The header
|
||||
timer never cuts off a body that is still streaming; the 60s bound still
|
||||
does. The events:watch stream's connect gets neither the header timer nor
|
||||
the retry. On Windows a reset connection is retried only when it surfaces as
|
||||
an early EOF, because the check matches the Unix reset error (`ECONNRESET`),
|
||||
which a Windows reset (`WSAECONNRESET`) is not. A write gets no
|
||||
header timer (the server answers a write only after it commits, so a large
|
||||
one can be slow and healthy) and is never retried; if the 60s bound expires
|
||||
after it was sent but before its headers, it fails with an error naming the
|
||||
server's host, because it may or may not have been applied. A dial or TLS
|
||||
timeout stays a plain connection error, since the server never saw the
|
||||
request. `off` turns off only the header timer: a reset read is still
|
||||
retried, and a write's timeout is still reported as above.
|
||||
|
||||
`bd search`, `bd show --current` and `bd recompute-blocked` work over HTTP
|
||||
too, each in its own way. `bd search` asks the server's listing with its text (`q`, behind the `issues.list.search` capability) and
|
||||
refuses before dialing against a server that does not advertise it. `bd list
|
||||
|
|
|
|||
|
|
@ -12,6 +12,12 @@ import (
|
|||
// persist a secret, and must FAIL CLOSED: a configured source that errors
|
||||
// aborts the request rather than silently downgrading to an unauthenticated
|
||||
// one.
|
||||
//
|
||||
// Authorize is called once per request, not once per attempt: a read whose
|
||||
// first attempt stalls or has its connection reset is resent once with the
|
||||
// same headers, up to the client's overall timeout later. A provider whose
|
||||
// credentials are single-use or short-lived should expect that resend to draw
|
||||
// a 401, which reaches Refresh like any other.
|
||||
type CredentialProvider = httpclient.CredentialProvider
|
||||
|
||||
const (
|
||||
|
|
|
|||
|
|
@ -107,6 +107,7 @@ go_test(
|
|||
"create_priority_test.go",
|
||||
"credential_source_test.go",
|
||||
"credential_test.go",
|
||||
"dial_header_timeout_test.go",
|
||||
"dial_test.go",
|
||||
"dispatch_test.go",
|
||||
"divergence_citation_gate_test.go",
|
||||
|
|
|
|||
|
|
@ -84,6 +84,8 @@ func (g *httpBatchGetter) GetMany(ctx context.Context, req issueops.GetManyReque
|
|||
Body: apigen.BatchGetIssuesRequest{
|
||||
Ids: append([]string(nil), req.IDs...),
|
||||
},
|
||||
// A read on POST: safe to replay after a stall or a 503, as a GET is.
|
||||
Idempotent: true,
|
||||
}, &body); err != nil {
|
||||
return issueops.GetManyResult{}, err
|
||||
}
|
||||
|
|
|
|||
|
|
@ -7,8 +7,10 @@ import (
|
|||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/httpapi/apigen"
|
||||
"github.com/steveyegge/beads/internal/httpclient/wire"
|
||||
|
|
@ -94,6 +96,58 @@ type DialOptions struct {
|
|||
// Unexported so the only way to set it is the door that also refuses a
|
||||
// nil credential.
|
||||
explicitCredential bool
|
||||
|
||||
// ResponseHeaderTimeout bounds a read's first attempt's wait for response
|
||||
// headers, separately from the client's overall timeout
|
||||
// (wire.DefaultTimeout): see wire.Options.ResponseHeaderTimeout. Zero takes
|
||||
// ResponseHeaderTimeoutEnv, else wire.DefaultResponseHeaderTimeout;
|
||||
// negative disables it.
|
||||
ResponseHeaderTimeout time.Duration
|
||||
// OnStallRetry is told about every retry of a stalled or reset idempotent
|
||||
// request. Nil logs one line per retry to stderr, so a stall is visible
|
||||
// rather than silently absorbed.
|
||||
OnStallRetry func(wire.StallRetry)
|
||||
}
|
||||
|
||||
// ResponseHeaderTimeoutEnv overrides the header timeout for every dial that
|
||||
// does not set DialOptions.ResponseHeaderTimeout: a Go duration ("15s",
|
||||
// "2m") or a number of seconds ("30"), as BEADS_GATE_WAIT_TIMEOUT takes, or "0"/
|
||||
// "off" to disable it. Disabling it turns off only the timer: a read whose
|
||||
// connection is reset is still retried once, and a write that times out
|
||||
// awaiting its headers is still reported as one. An unparseable value
|
||||
// refuses the dial.
|
||||
const ResponseHeaderTimeoutEnv = "BEADS_HTTP_RESPONSE_HEADER_TIMEOUT"
|
||||
|
||||
// responseHeaderTimeout resolves the header timeout for one dial: the
|
||||
// explicit option, else ResponseHeaderTimeoutEnv, else 0 (wire's default).
|
||||
func responseHeaderTimeout(opt time.Duration) (time.Duration, error) {
|
||||
if opt != 0 {
|
||||
return opt, nil
|
||||
}
|
||||
raw := strings.TrimSpace(os.Getenv(ResponseHeaderTimeoutEnv))
|
||||
if raw == "" {
|
||||
return 0, nil
|
||||
}
|
||||
if strings.EqualFold(raw, "off") {
|
||||
return -1, nil
|
||||
}
|
||||
d, err := time.ParseDuration(raw)
|
||||
if err != nil {
|
||||
d, err = time.ParseDuration(raw + "s")
|
||||
}
|
||||
if err != nil || d < 0 {
|
||||
return 0, fmt.Errorf("%s=%q: want a non-negative duration such as 15s or 30, or off", ResponseHeaderTimeoutEnv, raw)
|
||||
}
|
||||
if d == 0 {
|
||||
return -1, nil
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// logStallRetry is the default OnStallRetry: one stderr line per retry, with
|
||||
// the retry number and the process-wide count.
|
||||
func logStallRetry(e wire.StallRetry) {
|
||||
fmt.Fprintln(os.Stderr, e.String())
|
||||
}
|
||||
|
||||
// Conn is the store's view of one wire client.
|
||||
|
|
@ -226,10 +280,20 @@ func DialWith(target Target, creds CredentialProvider, opts DialOptions) (*Conn,
|
|||
// SAME flag before the sidecar carrying it even exists yet.
|
||||
creds = guardInsecureCredential(target, creds, opts.AllowInsecureCredential || target.AllowInsecureCredential, !opts.explicitCredential)
|
||||
|
||||
headerTimeout, err := responseHeaderTimeout(opts.ResponseHeaderTimeout)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
onStall := opts.OnStallRetry
|
||||
if onStall == nil {
|
||||
onStall = logStallRetry
|
||||
}
|
||||
client, err := wire.New(target.BaseURL, creds, wire.Options{
|
||||
HTTPClient: opts.HTTPClient,
|
||||
UserAgent: opts.UserAgent,
|
||||
ExpectProjectID: target.ExpectProjectID,
|
||||
HTTPClient: opts.HTTPClient,
|
||||
UserAgent: opts.UserAgent,
|
||||
ExpectProjectID: target.ExpectProjectID,
|
||||
ResponseHeaderTimeout: headerTimeout,
|
||||
OnStallRetry: onStall,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
|
|||
97
internal/httpclient/dial_header_timeout_test.go
Normal file
97
internal/httpclient/dial_header_timeout_test.go
Normal file
|
|
@ -0,0 +1,97 @@
|
|||
package httpclient
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/httpclient/wire"
|
||||
)
|
||||
|
||||
func TestResponseHeaderTimeoutResolution(t *testing.T) {
|
||||
cases := []struct {
|
||||
opt time.Duration
|
||||
env string
|
||||
want time.Duration
|
||||
wantErr bool
|
||||
}{
|
||||
{opt: 0, env: "", want: 0}, // wire's default
|
||||
{opt: 0, env: "5s", want: 5 * time.Second},
|
||||
{opt: 0, env: "30", want: 30 * time.Second}, // seconds, as BEADS_GATE_WAIT_TIMEOUT takes
|
||||
{opt: 0, env: " 1.5 ", want: 1500 * time.Millisecond},
|
||||
{opt: 0, env: "off", want: -1},
|
||||
{opt: 0, env: "OFF", want: -1},
|
||||
{opt: 0, env: "0", want: -1},
|
||||
{opt: 0, env: "0s", want: -1},
|
||||
{opt: 0, env: "soon", wantErr: true},
|
||||
{opt: 0, env: "-3s", wantErr: true},
|
||||
{opt: 0, env: "-3", wantErr: true},
|
||||
{opt: 2 * time.Second, env: "5s", want: 2 * time.Second}, // the option wins
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Setenv(ResponseHeaderTimeoutEnv, tc.env)
|
||||
got, err := responseHeaderTimeout(tc.opt)
|
||||
if (err != nil) != tc.wantErr {
|
||||
t.Errorf("opt=%v env=%q: err = %v, wantErr %v", tc.opt, tc.env, err, tc.wantErr)
|
||||
continue
|
||||
}
|
||||
if !tc.wantErr && got != tc.want {
|
||||
t.Errorf("opt=%v env=%q: got %v, want %v", tc.opt, tc.env, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUnparseableHeaderTimeoutEnvRefusesTheDial(t *testing.T) {
|
||||
clearCAEnvironment(t)
|
||||
t.Setenv(ResponseHeaderTimeoutEnv, "soon")
|
||||
if _, err := Dial(Target{BaseURL: mustParseURL(t, "http://127.0.0.1:1")}, DialOptions{}); err == nil {
|
||||
t.Fatal("Dial succeeded with an unparseable header timeout")
|
||||
}
|
||||
}
|
||||
|
||||
// TestTheProductionDialRetriesAStalledReadOnAnotherConnection is the shadow
|
||||
// run's stall end to end through the production transport DialWith builds:
|
||||
// the first connection never answers, the read is retried once on another one
|
||||
// and succeeds within about the header timeout, not the 60s overall timeout.
|
||||
func TestTheProductionDialRetriesAStalledReadOnAnotherConnection(t *testing.T) {
|
||||
clearCAEnvironment(t)
|
||||
t.Setenv(ResponseHeaderTimeoutEnv, "300ms")
|
||||
var mu sync.Mutex
|
||||
first := ""
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
if first == "" {
|
||||
first = r.RemoteAddr
|
||||
}
|
||||
stuck := r.RemoteAddr == first
|
||||
mu.Unlock()
|
||||
if stuck {
|
||||
<-r.Context().Done()
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_, _ = w.Write([]byte(`{"status":"ok"}`))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
var events []wire.StallRetry
|
||||
conn, err := Dial(Target{BaseURL: mustParseURL(t, srv.URL)}, DialOptions{
|
||||
OnStallRetry: func(e wire.StallRetry) { events = append(events, e) },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Dial: %v", err)
|
||||
}
|
||||
start := time.Now()
|
||||
if err := conn.Health(context.Background()); err != nil {
|
||||
t.Fatalf("Health: %v", err)
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed > 5*time.Second {
|
||||
t.Errorf("elapsed %v, want about one 300ms header timeout", elapsed)
|
||||
}
|
||||
if len(events) != 1 || events[0].Retry != 1 {
|
||||
t.Fatalf("events = %+v, want exactly one retry", events)
|
||||
}
|
||||
}
|
||||
|
|
@ -6,7 +6,6 @@ import (
|
|||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/httpclient/wire"
|
||||
|
|
@ -37,10 +36,12 @@ const (
|
|||
// of a reviewer's memory: a role that wants to reach the server has no other
|
||||
// route to it.
|
||||
//
|
||||
// It is also the one place that knows an operation's HTTP method, so the bounded
|
||||
// 503 retry lives here: the wire layer bounds Retry-After and hands it back but
|
||||
// never sleeps or resends, precisely because whether a 503 is safe to come back
|
||||
// to is a question only the method answers, and only a GET's answer is yes.
|
||||
// The bounded 503 retry lives here too: the wire layer bounds Retry-After and
|
||||
// hands it back but never sleeps on it. Whether a 503 is safe to come back to
|
||||
// is the question wire.Request.Replayable answers, and only a read's answer
|
||||
// is yes. (The wire does resend a replayable request once, at once, when its
|
||||
// first attempt stalled before any headers or had its connection reset; that
|
||||
// retry answers a transport-level failure, a 503 is the server's answer.)
|
||||
//
|
||||
// A store with no transport is a build-wiring fault (ErrNoTransport), not the
|
||||
// user's: the workspace and the server may both be fine.
|
||||
|
|
@ -58,11 +59,12 @@ func (s *Store) dispatch(ctx context.Context, req wire.Request, out any) error {
|
|||
}
|
||||
|
||||
err := s.wire.Do(ctx, req, out)
|
||||
// Only an idempotent GET is retried. A 503 can interrupt a non-idempotent
|
||||
// write whose first attempt already committed before the server ran out of
|
||||
// slots, so resending a POST/PATCH/DELETE could double-apply it — DELETE
|
||||
// included, since forgetMemory is a mutation the wire happens to spell as one.
|
||||
if req.Method != http.MethodGet {
|
||||
// Only a replayable read is retried: a GET, or a read on POST that says so
|
||||
// (the batch get). A 503 can interrupt a non-idempotent write whose first
|
||||
// attempt already committed before the server ran out of slots, so
|
||||
// resending a POST/PATCH/DELETE could double-apply it — DELETE included,
|
||||
// since forgetMemory is a mutation the wire happens to spell as one.
|
||||
if !req.Replayable() {
|
||||
return err
|
||||
}
|
||||
for attempt := 0; attempt < maxIdempotentRetries; attempt++ {
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ import (
|
|||
|
||||
"github.com/steveyegge/beads/internal/httpapi/apigen"
|
||||
"github.com/steveyegge/beads/internal/httpclient/wire"
|
||||
"github.com/steveyegge/beads/issueops"
|
||||
)
|
||||
|
||||
// busyThenOK answers with a 503 for the first busyCount Do calls and succeeds
|
||||
|
|
@ -79,6 +80,53 @@ func TestDispatchCapsIdempotentRetries(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
// TestDispatchRetriesAReadOnPostOnBusy: a read carried on POST that marks
|
||||
// itself Idempotent comes back after a 503 the way a GET does. That the batch
|
||||
// get marks itself is TestGetManyRetriesTheBatchGetOnBusy's.
|
||||
func TestDispatchRetriesAReadOnPostOnBusy(t *testing.T) {
|
||||
server := &busyThenOK{busyCount: 1, retryAfter: time.Millisecond}
|
||||
s := retryTestStore(t, server.do)
|
||||
|
||||
read := wire.Request{Op: wire.OpBatchGetIssues, Method: http.MethodPost, Path: wire.PathIssuesBatchGet, Idempotent: true}
|
||||
if err := s.dispatch(context.Background(), read, nil); err != nil {
|
||||
t.Fatalf("dispatch: %v, want the retry to succeed", err)
|
||||
}
|
||||
if server.calls != 2 {
|
||||
t.Errorf("Do called %d times, want 2 (one dial plus one retry)", server.calls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetManyRetriesTheBatchGetOnBusy: the batch get is a read dialed as a
|
||||
// POST, so it is replayable only because it says so (Idempotent). Through
|
||||
// GetMany it comes back after a 503, and what it sends is replayable for the
|
||||
// wire's stall retry too.
|
||||
func TestGetManyRetriesTheBatchGetOnBusy(t *testing.T) {
|
||||
token, _ := wire.CapabilityFor(wire.OpBatchGetIssues)
|
||||
server := &busyThenOK{busyCount: 1, retryAfter: time.Millisecond}
|
||||
var sent []wire.Request
|
||||
w := &fakeWire{
|
||||
res: &apigen.ContextResponse{Capabilities: []string{token}},
|
||||
preflight: func(context.Context, string) error { return nil },
|
||||
do: func(ctx context.Context, req wire.Request, out any) error {
|
||||
sent = append(sent, req)
|
||||
return server.do(ctx, req, out)
|
||||
},
|
||||
}
|
||||
g := &httpBatchGetter{store: New(testTarget(t), w, nil)}
|
||||
|
||||
if _, err := g.GetMany(context.Background(), issueops.GetManyRequest{IDs: []string{"bd-1"}}); err != nil {
|
||||
t.Fatalf("GetMany: %v, want the busy batch get retried", err)
|
||||
}
|
||||
if server.calls != 2 {
|
||||
t.Errorf("Do called %d times, want 2 (one dial plus one retry)", server.calls)
|
||||
}
|
||||
for _, req := range sent {
|
||||
if req.Op != wire.OpBatchGetIssues || req.Method != http.MethodPost || !req.Replayable() {
|
||||
t.Errorf("sent %s %s (replayable %v), want the batch get as a replayable POST", req.Method, req.Op, req.Replayable())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestDispatchDoesNotRetryWrites: a non-idempotent write is dialed exactly once
|
||||
// on a 503, because its first attempt may have committed before the server ran
|
||||
// out of slots and a resend would double-apply it.
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ go_library(
|
|||
"paths.go",
|
||||
"problem.go",
|
||||
"revision_tolerance.go",
|
||||
"stall_retry.go",
|
||||
"stream.go",
|
||||
"writes.go",
|
||||
],
|
||||
|
|
@ -37,6 +38,7 @@ go_test(
|
|||
"project_stamp_test.go",
|
||||
"revision_tolerance_test.go",
|
||||
"skew_test.go",
|
||||
"stall_retry_test.go",
|
||||
"stream_test.go",
|
||||
"vocabulary_test.go",
|
||||
"writes_batch_cap_test.go",
|
||||
|
|
@ -57,6 +59,7 @@ go_test(
|
|||
"//internal/types",
|
||||
"//issueops",
|
||||
"@in_gopkg_yaml_v3//:yaml_v3",
|
||||
"@org_golang_x_net//netutil",
|
||||
],
|
||||
)
|
||||
|
||||
|
|
|
|||
|
|
@ -73,6 +73,22 @@ type Options struct {
|
|||
// handshake compares it against ContextResponse.project_id; "" skips the
|
||||
// check.
|
||||
ExpectProjectID string
|
||||
// ResponseHeaderTimeout bounds the first attempt's wait for response
|
||||
// headers on an idempotent request (GET, or Request.Idempotent), counted
|
||||
// from when the request has been written and separately from the
|
||||
// HTTPClient's overall Timeout: zero means DefaultResponseHeaderTimeout,
|
||||
// negative disables it. A body that is still streaming once the headers
|
||||
// arrived is never cut off by it. Such a request that hits it, or whose
|
||||
// connection is reset, is retried once on another connection, with no
|
||||
// header timer and only what is left of the overall Timeout, counted from
|
||||
// when the first attempt was sent. A write is never
|
||||
// retried and is bounded by the overall Timeout alone (bd serve answers a
|
||||
// write after it commits). A written request whose headers do not come
|
||||
// back in time fails with *ResponseHeaderTimeoutError.
|
||||
ResponseHeaderTimeout time.Duration
|
||||
// OnStallRetry, when set, is told about every such retry. The process-wide
|
||||
// count is StallRetryCount either way.
|
||||
OnStallRetry func(StallRetry)
|
||||
}
|
||||
|
||||
// Client is one workspace's connection to one bd serve.
|
||||
|
|
@ -87,6 +103,9 @@ type Client struct {
|
|||
maxBytes int64
|
||||
maxRetry time.Duration
|
||||
|
||||
headerTimeout time.Duration
|
||||
onStall func(StallRetry)
|
||||
|
||||
handshake handshakeCache
|
||||
expectID string
|
||||
}
|
||||
|
|
@ -124,6 +143,13 @@ func New(base *url.URL, creds CredentialProvider, opts Options) (*Client, error)
|
|||
maxBytes: opts.MaxResponseBytes,
|
||||
maxRetry: opts.MaxRetryAfter,
|
||||
expectID: opts.ExpectProjectID,
|
||||
onStall: opts.OnStallRetry,
|
||||
}
|
||||
switch {
|
||||
case opts.ResponseHeaderTimeout == 0:
|
||||
c.headerTimeout = DefaultResponseHeaderTimeout
|
||||
case opts.ResponseHeaderTimeout > 0:
|
||||
c.headerTimeout = opts.ResponseHeaderTimeout
|
||||
}
|
||||
if c.userAgent == "" {
|
||||
c.userAgent = DefaultUserAgent
|
||||
|
|
@ -212,6 +238,23 @@ type Request struct {
|
|||
// error names it — the close's not_assignee — when the wire does not
|
||||
// repeat what the request already said.
|
||||
Actor string
|
||||
// Idempotent marks a read carried on a non-GET method (a POST batch get)
|
||||
// as safe to replay, the way a GET is (see Replayable). Never set it on a
|
||||
// write.
|
||||
Idempotent bool
|
||||
}
|
||||
|
||||
// Replayable reports whether r may be sent again after an attempt that got no
|
||||
// usable answer: the safe methods, or a read carried on POST that says so
|
||||
// (Idempotent). It is the one predicate both retries consult: the wire's
|
||||
// retry of a stalled or reset attempt on another connection, and the store's
|
||||
// bounded retry of a 503.
|
||||
func (r Request) Replayable() bool {
|
||||
switch r.Method {
|
||||
case http.MethodGet, http.MethodHead, http.MethodOptions:
|
||||
return true
|
||||
}
|
||||
return r.Idempotent
|
||||
}
|
||||
|
||||
// Do issues r and decodes a 2xx body into out, which may be nil for a response
|
||||
|
|
@ -335,13 +378,70 @@ type response struct {
|
|||
body []byte
|
||||
}
|
||||
|
||||
// roundTrip issues one HTTP request and reads its whole bounded body.
|
||||
// roundTrip issues r, retrying an idempotent request once on another
|
||||
// connection when the first attempt stalls before its headers or has its
|
||||
// connection reset (see Options.ResponseHeaderTimeout).
|
||||
//
|
||||
// Reading the body here rather than streaming it to the caller is what makes
|
||||
// the 401 retry above expressible at all: both attempts are then symmetric, and
|
||||
// the request body is a []byte that can be replayed. The size cap is what makes
|
||||
// that safe.
|
||||
// The request is built and stamped once and each attempt sends a copy, so the
|
||||
// retry replays what the first attempt sent and the credential source is
|
||||
// consulted before any clock starts. The retry gets no header timer, and both
|
||||
// attempts share the one overall budget the client's Timeout gives a request,
|
||||
// counted, as that Timeout is, from when the first attempt is sent: a
|
||||
// credential source slow to resolve (a cold token command) is not the
|
||||
// server's time. The timer cannot tell a connection that went silent from a
|
||||
// serve still working on a slow read, so a retry cut short by it again would
|
||||
// fail a healthy read the client used to wait out. Bounded by what is left of
|
||||
// the overall Timeout instead, such a read still finishes if its headers and
|
||||
// body fit in what is left, and no read waits longer than that Timeout.
|
||||
func (c *Client) roundTrip(ctx context.Context, r Request, u *url.URL, body []byte) (*response, error) {
|
||||
req, err := c.newRequest(ctx, r, u, body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !r.Replayable() {
|
||||
// Only a replayable request gets the header timer. bd serve sends a
|
||||
// write's headers after the write commits, so a slow but healthy write
|
||||
// (a large batch apply) can legitimately take longer than the header
|
||||
// timeout, and cutting it would turn it into an unknown outcome that
|
||||
// cannot be retried. A write stays bounded by the client's overall
|
||||
// timeout, and a timeout awaiting its headers is typed in
|
||||
// roundTripOnce.
|
||||
return c.roundTripOnce(ctx, r, u, req, 0, time.Time{})
|
||||
}
|
||||
sent := time.Now()
|
||||
res, err := c.roundTripOnce(ctx, r, u, req, c.headerTimeout, time.Time{})
|
||||
if err == nil {
|
||||
return res, nil
|
||||
}
|
||||
var deadline time.Time
|
||||
if c.hc.Timeout > 0 {
|
||||
deadline = sent.Add(c.hc.Timeout)
|
||||
}
|
||||
reason := stallReason(err)
|
||||
if reason == "" || ctx.Err() != nil || (!deadline.IsZero() && !time.Now().Before(deadline)) {
|
||||
return nil, err
|
||||
}
|
||||
// No idle-pool flush: canceling the stalled attempt closed its connection
|
||||
// (net/http never pools a canceled HTTP/1.1 connection, which is what bd
|
||||
// serve speaks; behind an HTTP/2 proxy the retry may share the proxy's
|
||||
// connection), so the retry rides another one. The transport may also be
|
||||
// shared by every client in the process, and its idle connections to
|
||||
// other servers are none of this request's business.
|
||||
event := StallRetry{Op: r.Op, Method: r.Method, Host: u.Host, Reason: reason, Retry: 1, Total: stallRetries.Add(1), Err: err}
|
||||
if c.onStall != nil {
|
||||
c.onStall(event)
|
||||
}
|
||||
res, err = c.roundTripOnce(ctx, r, u, req, 0, deadline)
|
||||
var hte *ResponseHeaderTimeoutError
|
||||
if errors.As(err, &hte) {
|
||||
hte.After, hte.Attempts, hte.Retried, hte.Reason = time.Since(sent), 2, true, reason
|
||||
}
|
||||
return res, err
|
||||
}
|
||||
|
||||
// newRequest builds r's HTTP request and stamps it (see stampRequest).
|
||||
// roundTripOnce sends a copy of it per attempt.
|
||||
func (c *Client) newRequest(ctx context.Context, r Request, u *url.URL, body []byte) (*http.Request, error) {
|
||||
var rdr io.Reader
|
||||
if body != nil {
|
||||
rdr = bytes.NewReader(body)
|
||||
|
|
@ -357,10 +457,60 @@ func (c *Client) roundTrip(ctx context.Context, r Request, u *url.URL, body []by
|
|||
if err := c.stampRequest(ctx, r.Op, req); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return req, nil
|
||||
}
|
||||
|
||||
// roundTripOnce sends one copy of the stamped request and reads its whole
|
||||
// bounded body. headerTimeout (0: none) bounds its wait for headers once it is
|
||||
// written, and deadline (zero: none) bounds the whole attempt on top of ctx
|
||||
// and the client's own Timeout.
|
||||
//
|
||||
// Reading the body here rather than streaming it to the caller is what makes
|
||||
// the 401 retry above expressible at all: both attempts are then symmetric, and
|
||||
// the request body is a []byte that can be replayed. The size cap is what makes
|
||||
// that safe.
|
||||
func (c *Client) roundTripOnce(ctx context.Context, r Request, u *url.URL, stamped *http.Request, headerTimeout time.Duration, deadline time.Time) (*response, error) {
|
||||
attemptCtx := ctx
|
||||
if !deadline.IsZero() {
|
||||
var cancel context.CancelFunc
|
||||
attemptCtx, cancel = context.WithDeadline(ctx, deadline)
|
||||
defer cancel()
|
||||
}
|
||||
attemptCtx, ht := startHeaderTimer(attemptCtx, headerTimeout)
|
||||
defer ht.done()
|
||||
req := stamped.Clone(attemptCtx)
|
||||
if stamped.GetBody != nil {
|
||||
// Clone shares the body reader, which an earlier attempt may have
|
||||
// consumed; every attempt sends the body from its start.
|
||||
b, err := stamped.GetBody()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", r.Op, err)
|
||||
}
|
||||
req.Body = b
|
||||
}
|
||||
|
||||
//nolint:gosec // G704: the URL is the workspace's configured server, resolved from the sidecar or --server-url; dialing it IS the feature.
|
||||
resp, err := c.hc.Do(req)
|
||||
if ht.stop() {
|
||||
// The header timer fired first. Headers that raced it are discarded:
|
||||
// the attempt's context is already canceled, so its body is lost.
|
||||
if resp != nil {
|
||||
_ = resp.Body.Close()
|
||||
}
|
||||
hte := &ResponseHeaderTimeoutError{Op: r.Op, Method: r.Method, Host: u.Host, After: headerTimeout, Attempts: 1, Replayable: r.Replayable()}
|
||||
return nil, &ConnectError{Op: r.Op, ServerURL: c.base.Redacted(), Err: hte}
|
||||
}
|
||||
if err != nil {
|
||||
// A timeout after the request was written ran out awaiting its
|
||||
// headers: Do returns once they arrive, so whichever bound fired (the
|
||||
// client's Timeout or this attempt's deadline) fired before them. A
|
||||
// timeout before the request was written (a dial or TLS handshake)
|
||||
// stays a plain ConnectError, because the server cannot have seen
|
||||
// the request. The caller's own context expiring is the caller's.
|
||||
if wroteAt, ok := ht.written(); ok && ctx.Err() == nil && isTimeout(err) {
|
||||
hte := &ResponseHeaderTimeoutError{Op: r.Op, Method: r.Method, Host: u.Host, After: time.Since(wroteAt), Attempts: 1, Replayable: r.Replayable(), Err: err}
|
||||
return nil, &ConnectError{Op: r.Op, ServerURL: c.base.Redacted(), Err: hte}
|
||||
}
|
||||
return nil, &ConnectError{Op: r.Op, ServerURL: c.base.Redacted(), Err: err}
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
|
|
|||
|
|
@ -25,6 +25,13 @@ type CredentialProvider interface {
|
|||
// A no-auth deployment — the loopback-trust OSS server, which has no token
|
||||
// file at all — returns nil without modifying req.
|
||||
//
|
||||
// It is called once per request the client builds, not once per attempt:
|
||||
// a replayable request whose first attempt stalls or has its connection
|
||||
// reset is resent once with the headers Authorize set, up to the client's
|
||||
// overall timeout later (see Client.roundTrip). A provider whose
|
||||
// credentials are single-use or expire within that window should expect
|
||||
// the resend to draw a 401, which reaches Refresh like any other.
|
||||
//
|
||||
// The error must not carry the credential: it travels into the client's own
|
||||
// error text, which reaches logs and terminals.
|
||||
Authorize(ctx context.Context, req *http.Request) error
|
||||
|
|
|
|||
261
internal/httpclient/wire/stall_retry.go
Normal file
261
internal/httpclient/wire/stall_retry.go
Normal file
|
|
@ -0,0 +1,261 @@
|
|||
package wire
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http/httptrace"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DefaultResponseHeaderTimeout bounds how long the first attempt of a
|
||||
// replayable request waits for the server's response HEADERS once the
|
||||
// request has been written, separately from DefaultTimeout's bound on the
|
||||
// whole exchange.
|
||||
//
|
||||
// It exists for the stuck connection the maintainer-city shadow run found: a
|
||||
// single HTTP/1.1 connection that went silent while other connections to the
|
||||
// same serve kept answering, so each request on it sat out the full 60s
|
||||
// overall timeout. The timer cannot tell that connection from a serve that is
|
||||
// still working: serve writes a read's headers only once the handler has its
|
||||
// answer, after waiting up to 10s for a database slot (a wait that ends in a
|
||||
// 503, which is headers). 15s sits above that wait, but a large read can take
|
||||
// longer and still be healthy. That is why the retry gets no header timer of
|
||||
// its own and runs on what is left of the overall timeout (see
|
||||
// Client.roundTrip): such a read still finishes, later and at the cost of one
|
||||
// abandoned request, unless the retry's headers and body together take longer
|
||||
// than what is left, about DefaultTimeout minus this.
|
||||
//
|
||||
// Only the headers are bounded: once they arrive the timer is stopped, and a
|
||||
// slow body that keeps streaming is governed by DefaultTimeout (and the
|
||||
// caller's context) alone.
|
||||
//
|
||||
// The timer applies to replayable requests (see Request.Replayable). A write is
|
||||
// bounded by the client's overall timeout instead, because bd serve answers a
|
||||
// write only after it commits; see roundTrip.
|
||||
const DefaultResponseHeaderTimeout = 15 * time.Second
|
||||
|
||||
// ErrResponseHeaderTimeout is the sentinel a *ResponseHeaderTimeoutError
|
||||
// unwraps to.
|
||||
var ErrResponseHeaderTimeout = errors.New("bd serve sent no response headers in time")
|
||||
|
||||
// ResponseHeaderTimeoutError is a request that was written to the server and
|
||||
// got no response headers in time: its first attempt hit the header timer, or
|
||||
// the overall timeout ran out while it waited. A request that never left the
|
||||
// machine (a dial or TLS handshake that timed out) is not one; it stays a
|
||||
// plain *ConnectError, because the server cannot have seen it. It is returned
|
||||
// inside a *ConnectError (the server said nothing), so callers that classify
|
||||
// an unreachable server keep working; errors.As reaches this type for the
|
||||
// detail.
|
||||
type ResponseHeaderTimeoutError struct {
|
||||
Op string
|
||||
Method string
|
||||
// Host is the server's host[:port], named so the operator knows which
|
||||
// server to look at.
|
||||
Host string
|
||||
// After is how long the request waited: the header timeout when its timer
|
||||
// fired, else the measured wait (from when the first attempt was sent, for
|
||||
// a retried read).
|
||||
After time.Duration
|
||||
// Attempts is how many requests went out: 1, or 2 for a read whose retry
|
||||
// on another connection got no headers either.
|
||||
Attempts int
|
||||
// Retried reports whether a retry on another connection was made.
|
||||
Retried bool
|
||||
// Reason is why the first attempt was abandoned, when Retried:
|
||||
// StallHeaderTimeout or StallConnectionReset.
|
||||
Reason string
|
||||
// Replayable reports whether the request was idempotent. One that is not
|
||||
// was never retried and may or may not have been applied.
|
||||
Replayable bool
|
||||
// Err is the transport's own error, when the overall timeout (not the
|
||||
// header timer) is what fired.
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e *ResponseHeaderTimeoutError) Error() string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "%s: no response headers from %s within %s", e.Op, e.Host, e.After.Round(10*time.Millisecond))
|
||||
if e.Retried {
|
||||
first := "got no headers within the header timeout"
|
||||
if e.Reason == StallConnectionReset {
|
||||
first = "had its connection reset"
|
||||
}
|
||||
fmt.Fprintf(&b, " on %d attempts (the first %s; the retry used another connection)", e.Attempts, first)
|
||||
}
|
||||
if e.Err != nil {
|
||||
fmt.Fprintf(&b, " (%v)", e.Err)
|
||||
}
|
||||
if !e.Replayable {
|
||||
fmt.Fprintf(&b, "; the %s was not retried because it is not idempotent (it may or may not have been applied)", e.Method)
|
||||
}
|
||||
fmt.Fprintf(&b, "; check the server at %s (is it at its connection limit?)", e.Host)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func (e *ResponseHeaderTimeoutError) Unwrap() []error {
|
||||
if e.Err != nil {
|
||||
return []error{ErrResponseHeaderTimeout, e.Err}
|
||||
}
|
||||
return []error{ErrResponseHeaderTimeout}
|
||||
}
|
||||
|
||||
// Timeout reports true, so net.Error-style timeout checks classify it.
|
||||
func (e *ResponseHeaderTimeoutError) Timeout() bool { return true }
|
||||
|
||||
// Temporary is part of the net.Error interface. It is false for a request
|
||||
// that is not replayable, so a caller that retries whatever reports itself
|
||||
// temporary never resends a write that may already have been applied.
|
||||
func (e *ResponseHeaderTimeoutError) Temporary() bool { return e.Replayable }
|
||||
|
||||
// isTimeout reports whether err is a timeout (net.Error, or the client's own
|
||||
// Timeout surfacing through *url.Error).
|
||||
func isTimeout(err error) bool {
|
||||
var te interface{ Timeout() bool }
|
||||
return errors.As(err, &te) && te.Timeout()
|
||||
}
|
||||
|
||||
// Reasons a StallRetry names.
|
||||
const (
|
||||
StallHeaderTimeout = "response_header_timeout"
|
||||
StallConnectionReset = "connection_reset"
|
||||
)
|
||||
|
||||
// StallRetry describes one retry of an idempotent request on another
|
||||
// connection after the first attempt stalled (no headers within the header
|
||||
// timeout) or had its connection reset. It is reported, never swallowed: a retry that
|
||||
// hides a sick server is a stall nobody sees.
|
||||
type StallRetry struct {
|
||||
Op string
|
||||
Method string
|
||||
Host string
|
||||
Reason string
|
||||
// Retry is the 1-based retry number for this request (always 1 today:
|
||||
// one retry per request).
|
||||
Retry int
|
||||
// Total is the process-wide count of stall retries, this one included.
|
||||
Total uint64
|
||||
// Err is the first attempt's failure.
|
||||
Err error
|
||||
}
|
||||
|
||||
func (e StallRetry) String() string {
|
||||
return fmt.Sprintf("bd: http %s %s to %s stalled (%s); retry %d on another connection (%d stall retries in this process): %v",
|
||||
e.Op, e.Method, e.Host, e.Reason, e.Retry, e.Total, e.Err)
|
||||
}
|
||||
|
||||
var stallRetries atomic.Uint64
|
||||
|
||||
// StallRetryCount is the number of stall retries this process has made.
|
||||
func StallRetryCount() uint64 { return stallRetries.Load() }
|
||||
|
||||
// stallReason classifies a failed attempt as retryable on another
|
||||
// connection, returning "" when it is not.
|
||||
func stallReason(err error) string {
|
||||
var ce *ConnectError
|
||||
if !errors.As(err, &ce) {
|
||||
// Only a failure to get an answer is a stall; a credential source's
|
||||
// own error that happens to wrap io.EOF is not.
|
||||
return ""
|
||||
}
|
||||
switch {
|
||||
case errors.Is(err, ErrResponseHeaderTimeout):
|
||||
return StallHeaderTimeout
|
||||
case errors.Is(err, syscall.ECONNRESET), errors.Is(err, io.ErrUnexpectedEOF), errors.Is(err, io.EOF):
|
||||
return StallConnectionReset
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// headerTimer bounds one attempt's wait for response headers. It starts once
|
||||
// the request has been written (httptrace's WroteRequest, where
|
||||
// http.Transport.ResponseHeaderTimeout starts its own clock), so a slow dial,
|
||||
// TLS handshake or credential source is never counted as a server that went
|
||||
// silent. When it fires it cancels the attempt's context, which makes
|
||||
// net/http close that connection rather than return it to the idle pool, so
|
||||
// a retry can never ride it. It also records when the request was written,
|
||||
// which is what tells a write that timed out awaiting its headers from one
|
||||
// that never left the machine.
|
||||
type headerTimer struct {
|
||||
d time.Duration
|
||||
cancel context.CancelCauseFunc
|
||||
|
||||
mu sync.Mutex
|
||||
state int // headerWaiting, headerSettled or headerTimedOut
|
||||
wroteAt time.Time // zero until the request has been written
|
||||
timer *time.Timer
|
||||
}
|
||||
|
||||
const (
|
||||
headerWaiting = iota
|
||||
headerSettled // Do returned before the timer fired
|
||||
headerTimedOut // the timer fired first
|
||||
)
|
||||
|
||||
// startHeaderTimer returns the attempt's context, which carries the trace hook
|
||||
// that arms a timer of d (none when d <= 0) once the request is written.
|
||||
func startHeaderTimer(ctx context.Context, d time.Duration) (context.Context, *headerTimer) {
|
||||
attemptCtx, cancel := context.WithCancelCause(ctx)
|
||||
h := &headerTimer{d: d, cancel: cancel}
|
||||
return httptrace.WithClientTrace(attemptCtx, &httptrace.ClientTrace{WroteRequest: h.wroteRequest}), h
|
||||
}
|
||||
|
||||
// wroteRequest runs on the transport's writing goroutine. A failed write
|
||||
// records nothing (the attempt fails on its own), and only the first
|
||||
// successful one counts: the transport may resend a request it got no answer
|
||||
// to on a connection it found dead, and the wait is measured from the first.
|
||||
func (h *headerTimer) wroteRequest(info httptrace.WroteRequestInfo) {
|
||||
if info.Err != nil {
|
||||
return
|
||||
}
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if !h.wroteAt.IsZero() {
|
||||
return
|
||||
}
|
||||
h.wroteAt = time.Now()
|
||||
if h.d > 0 && h.state == headerWaiting {
|
||||
h.timer = time.AfterFunc(h.d, h.fire)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *headerTimer) fire() {
|
||||
h.mu.Lock()
|
||||
fired := h.state == headerWaiting
|
||||
if fired {
|
||||
h.state = headerTimedOut
|
||||
}
|
||||
h.mu.Unlock()
|
||||
if fired {
|
||||
h.cancel(ErrResponseHeaderTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
// stop must be called as soon as Do returns. It disarms the timer and reports
|
||||
// whether it had already fired (the headers lost the race).
|
||||
func (h *headerTimer) stop() (timedOut bool) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
if h.timer != nil {
|
||||
h.timer.Stop()
|
||||
}
|
||||
if h.state == headerWaiting {
|
||||
h.state = headerSettled
|
||||
}
|
||||
return h.state == headerTimedOut
|
||||
}
|
||||
|
||||
// written reports when the request was written, if it was.
|
||||
func (h *headerTimer) written() (time.Time, bool) {
|
||||
h.mu.Lock()
|
||||
defer h.mu.Unlock()
|
||||
return h.wroteAt, !h.wroteAt.IsZero()
|
||||
}
|
||||
|
||||
// done releases the attempt's context. Call it once the body has been read.
|
||||
func (h *headerTimer) done() { h.cancel(nil) }
|
||||
775
internal/httpclient/wire/stall_retry_test.go
Normal file
775
internal/httpclient/wire/stall_retry_test.go
Normal file
|
|
@ -0,0 +1,775 @@
|
|||
package wire
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"golang.org/x/net/netutil"
|
||||
)
|
||||
|
||||
// stallServer answers every request normally except those that arrive on the
|
||||
// FIRST connection it ever sees: those get no response headers at all, the
|
||||
// shape of the shadow run's stuck HTTP/1.1 connection (a silent connection
|
||||
// while other connections to the same serve kept answering). The hung handler
|
||||
// returns when the client gives up on the connection.
|
||||
type stallServer struct {
|
||||
srv *httptest.Server
|
||||
|
||||
mu sync.Mutex
|
||||
first string // RemoteAddr of the first connection
|
||||
remotes []string // RemoteAddr of every request, in order
|
||||
methods []string
|
||||
hang func(w http.ResponseWriter, r *http.Request)
|
||||
}
|
||||
|
||||
func newStallServer(t *testing.T, answer http.HandlerFunc) *stallServer {
|
||||
t.Helper()
|
||||
s := &stallServer{}
|
||||
s.hang = func(_ http.ResponseWriter, r *http.Request) {
|
||||
// Drain the body first: net/http only notices the client closing the
|
||||
// connection (and cancels r.Context) once the body has been read.
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
<-r.Context().Done()
|
||||
}
|
||||
s.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
s.mu.Lock()
|
||||
if s.first == "" {
|
||||
s.first = r.RemoteAddr
|
||||
}
|
||||
stuck := r.RemoteAddr == s.first
|
||||
s.remotes = append(s.remotes, r.RemoteAddr)
|
||||
s.methods = append(s.methods, r.Method)
|
||||
hang := s.hang
|
||||
s.mu.Unlock()
|
||||
if stuck {
|
||||
hang(w, r)
|
||||
return
|
||||
}
|
||||
answer(w, r)
|
||||
}))
|
||||
t.Cleanup(s.srv.Close)
|
||||
return s
|
||||
}
|
||||
|
||||
func (s *stallServer) seen() []string {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return append([]string(nil), s.remotes...)
|
||||
}
|
||||
|
||||
func (s *stallServer) client(t *testing.T, opts Options) *Client {
|
||||
t.Helper()
|
||||
base, err := url.Parse(s.srv.URL)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if opts.HTTPClient == nil {
|
||||
// The production shape: a pooled transport with the package's own 60s
|
||||
// overall ceiling, which the header timeout must undercut.
|
||||
opts.HTTPClient = &http.Client{Transport: s.srv.Client().Transport, Timeout: DefaultTimeout}
|
||||
}
|
||||
c, err := New(base, nil, opts)
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
const testHeaderTimeout = 300 * time.Millisecond
|
||||
|
||||
func TestAStalledIdempotentReadRetriesOnceOnAnotherConnection(t *testing.T) {
|
||||
s := newStallServer(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"ok":true}`))
|
||||
})
|
||||
var events []StallRetry
|
||||
c := s.client(t, Options{
|
||||
ResponseHeaderTimeout: testHeaderTimeout,
|
||||
OnStallRetry: func(e StallRetry) { events = append(events, e) },
|
||||
})
|
||||
before := StallRetryCount()
|
||||
|
||||
start := time.Now()
|
||||
var out map[string]any
|
||||
if err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, &out); err != nil {
|
||||
t.Fatalf("Do: %v", err)
|
||||
}
|
||||
elapsed := time.Since(start)
|
||||
|
||||
if out["ok"] != true {
|
||||
t.Fatalf("body = %v, want the second connection's answer", out)
|
||||
}
|
||||
// One header timeout, then one prompt retry: well under two timeouts, and
|
||||
// nowhere near the 60s overall ceiling the shadow run hit.
|
||||
if elapsed < testHeaderTimeout || elapsed > 2*testHeaderTimeout+time.Second {
|
||||
t.Errorf("elapsed %v, want about one header timeout (%v)", elapsed, testHeaderTimeout)
|
||||
}
|
||||
seen := s.seen()
|
||||
if len(seen) != 2 {
|
||||
t.Fatalf("server saw %d requests, want 2 (the stalled one and one retry): %v", len(seen), seen)
|
||||
}
|
||||
if seen[0] == seen[1] {
|
||||
t.Errorf("the retry rode the stuck connection %s; it must use another one", seen[0])
|
||||
}
|
||||
if len(events) != 1 {
|
||||
t.Fatalf("OnStallRetry fired %d times, want 1: %+v", len(events), events)
|
||||
}
|
||||
e := events[0]
|
||||
if e.Op != OpGetContext || e.Method != http.MethodGet || e.Retry != 1 || e.Reason != StallHeaderTimeout {
|
||||
t.Errorf("event = %+v, want op %s, GET, retry 1, reason %q", e, OpGetContext, StallHeaderTimeout)
|
||||
}
|
||||
if !strings.Contains(e.String(), "retry 1") || !strings.Contains(e.String(), e.Host) {
|
||||
t.Errorf("event line %q should carry the retry count and the host", e.String())
|
||||
}
|
||||
if got := StallRetryCount() - before; got != 1 {
|
||||
t.Errorf("StallRetryCount advanced by %d, want 1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnExplicitlyIdempotentPostReadRetriesToo(t *testing.T) {
|
||||
s := newStallServer(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
})
|
||||
c := s.client(t, Options{ResponseHeaderTimeout: testHeaderTimeout})
|
||||
err := c.Do(ctx(t), Request{Op: OpBatchGetIssues, Method: http.MethodPost, Path: PathIssuesBatchGet,
|
||||
Body: map[string]any{"ids": []string{"x"}}, Idempotent: true}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Do: %v", err)
|
||||
}
|
||||
if n := len(s.seen()); n != 2 {
|
||||
t.Fatalf("server saw %d requests, want 2", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStalledWriteIsNeverRetriedAndSurfacesATypedTimeout(t *testing.T) {
|
||||
s := newStallServer(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
})
|
||||
fired := 0
|
||||
// A write is bounded by the client's overall timeout, not the header
|
||||
// timer; shrink the overall one so the stall surfaces quickly.
|
||||
c := s.client(t, Options{
|
||||
HTTPClient: &http.Client{Transport: s.srv.Client().Transport, Timeout: 2 * testHeaderTimeout},
|
||||
ResponseHeaderTimeout: testHeaderTimeout,
|
||||
OnStallRetry: func(StallRetry) { fired++ },
|
||||
})
|
||||
|
||||
start := time.Now()
|
||||
err := c.Do(ctx(t), Request{Op: OpCreateIssue, Method: http.MethodPost, Path: PathIssues,
|
||||
Body: map[string]any{"title": "t"}}, nil)
|
||||
elapsed := time.Since(start)
|
||||
if err == nil {
|
||||
t.Fatal("Do succeeded; want the header timeout")
|
||||
}
|
||||
var hte *ResponseHeaderTimeoutError
|
||||
if !errors.As(err, &hte) {
|
||||
t.Fatalf("err = %T %v, want *ResponseHeaderTimeoutError", err, err)
|
||||
}
|
||||
if !errors.Is(err, ErrResponseHeaderTimeout) {
|
||||
t.Errorf("err does not unwrap to ErrResponseHeaderTimeout: %v", err)
|
||||
}
|
||||
var ce *ConnectError
|
||||
if !errors.As(err, &ce) {
|
||||
t.Errorf("err should still be a *ConnectError for callers that classify unreachable servers: %v", err)
|
||||
}
|
||||
var ne net.Error
|
||||
if !errors.As(err, &ne) || !ne.Timeout() {
|
||||
t.Errorf("err should report Timeout() = true: %v", err)
|
||||
}
|
||||
if hte.Retried || hte.Attempts != 1 || hte.Method != http.MethodPost {
|
||||
t.Errorf("hte = %+v, want one unretried POST attempt", hte)
|
||||
}
|
||||
if hte.Temporary() {
|
||||
t.Error("a write's timeout reports Temporary() = true, so a generic net.Error retry would resend it")
|
||||
}
|
||||
// The measured wait since the request was written, not a configured
|
||||
// number, and the transport's own error rather than nothing.
|
||||
if hte.After < testHeaderTimeout || hte.After > elapsed || hte.Err == nil {
|
||||
t.Errorf("hte.After = %v, hte.Err = %v; want the measured wait (at most %v) and the transport's error", hte.After, hte.Err, elapsed)
|
||||
}
|
||||
host := s.srv.Listener.Addr().String()
|
||||
msg := err.Error()
|
||||
for _, want := range []string{host, "not retried", "may or may not have been applied", "Client.Timeout exceeded", "check the server"} {
|
||||
if !strings.Contains(msg, want) {
|
||||
t.Errorf("error %q should contain %q", msg, want)
|
||||
}
|
||||
}
|
||||
if n := len(s.seen()); n != 1 {
|
||||
t.Fatalf("server saw %d requests, want exactly 1 (a write is never replayed)", n)
|
||||
}
|
||||
if fired != 0 {
|
||||
t.Errorf("OnStallRetry fired %d times for a write", fired)
|
||||
}
|
||||
if elapsed < 2*testHeaderTimeout || elapsed > 2*testHeaderTimeout+time.Second {
|
||||
t.Errorf("elapsed %v; the write should fail at the overall timeout (%v)", elapsed, 2*testHeaderTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
// TestASlowWriteIsNotCutOffByTheHeaderTimeout: bd serve answers a write only
|
||||
// after it commits, so a write whose headers take longer than the header
|
||||
// timeout is a slow but healthy write, and must complete.
|
||||
func TestASlowWriteIsNotCutOffByTheHeaderTimeout(t *testing.T) {
|
||||
c, rec := newTestClient(t, Options{ResponseHeaderTimeout: testHeaderTimeout}, nil, func(w http.ResponseWriter, r *http.Request) {
|
||||
_, _ = io.Copy(io.Discard, r.Body)
|
||||
time.Sleep(3 * testHeaderTimeout)
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
})
|
||||
err := c.Do(ctx(t), Request{Op: OpCreateIssue, Method: http.MethodPost, Path: PathIssues,
|
||||
Body: map[string]any{"title": "t"}}, nil)
|
||||
if err != nil {
|
||||
t.Fatalf("Do: %v", err)
|
||||
}
|
||||
if n := rec.count(); n != 1 {
|
||||
t.Errorf("server saw %d requests, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAReadThatStallsTwiceGivesUpAfterOneRetry: the retry has no header timer
|
||||
// of its own, so a read whose retry gets no headers either fails once the
|
||||
// overall timeout is spent, counted from the first attempt, and not later.
|
||||
func TestAReadThatStallsTwiceGivesUpAfterOneRetry(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
n := 0
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
n++
|
||||
mu.Unlock()
|
||||
<-r.Context().Done()
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
base, _ := url.Parse(srv.URL)
|
||||
budget := 4 * testHeaderTimeout
|
||||
c, err := New(base, nil, Options{
|
||||
HTTPClient: &http.Client{Transport: srv.Client().Transport, Timeout: budget},
|
||||
ResponseHeaderTimeout: testHeaderTimeout,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
start := time.Now()
|
||||
err = c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, nil)
|
||||
elapsed := time.Since(start)
|
||||
var hte *ResponseHeaderTimeoutError
|
||||
if !errors.As(err, &hte) {
|
||||
t.Fatalf("err = %v, want *ResponseHeaderTimeoutError", err)
|
||||
}
|
||||
if !hte.Retried || hte.Attempts != 2 || hte.Reason != StallHeaderTimeout || !hte.Temporary() {
|
||||
t.Errorf("hte = %+v, want 2 attempts with the retry and its reason recorded", hte)
|
||||
}
|
||||
if elapsed < budget || elapsed > budget+time.Second {
|
||||
t.Errorf("elapsed %v, want the overall timeout (%v) and no more", elapsed, budget)
|
||||
}
|
||||
if hte.After < budget || hte.After > elapsed {
|
||||
t.Errorf("hte.After = %v, want the whole wait (between %v and %v)", hte.After, budget, elapsed)
|
||||
}
|
||||
if msg := err.Error(); !strings.Contains(msg, "on 2 attempts (the first got no headers within the header timeout") {
|
||||
t.Errorf("error %q should say both attempts went unanswered", msg)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if n != 2 {
|
||||
t.Errorf("server saw %d requests, want 2", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAResetConnectionOnAReadRetriesOnAnotherConnection(t *testing.T) {
|
||||
s := newStallServer(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"ok":true}`))
|
||||
})
|
||||
// The first connection is reset (RST, not FIN) instead of hanging.
|
||||
s.hang = resetConnection
|
||||
var events []StallRetry
|
||||
c := s.client(t, Options{ResponseHeaderTimeout: testHeaderTimeout, OnStallRetry: func(e StallRetry) { events = append(events, e) }})
|
||||
var out map[string]any
|
||||
if err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, &out); err != nil {
|
||||
t.Fatalf("Do: %v", err)
|
||||
}
|
||||
if out["ok"] != true {
|
||||
t.Fatalf("body = %v", out)
|
||||
}
|
||||
if len(events) != 1 || events[0].Reason != StallConnectionReset {
|
||||
t.Fatalf("events = %+v, want one %q retry", events, StallConnectionReset)
|
||||
}
|
||||
}
|
||||
|
||||
// resetConnection is a hang that resets the connection (RST, not FIN).
|
||||
func resetConnection(w http.ResponseWriter, _ *http.Request) {
|
||||
conn, _, err := w.(http.Hijacker).Hijack()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if tc, ok := conn.(*net.TCPConn); ok {
|
||||
_ = tc.SetLinger(0)
|
||||
}
|
||||
_ = conn.Close()
|
||||
}
|
||||
|
||||
func TestARetryAfterAResetNamesTheReset(t *testing.T) {
|
||||
s := newStallServer(t, func(_ http.ResponseWriter, r *http.Request) {
|
||||
<-r.Context().Done() // the retry's connection never answers either
|
||||
})
|
||||
s.hang = resetConnection
|
||||
c := s.client(t, Options{
|
||||
HTTPClient: &http.Client{Transport: s.srv.Client().Transport, Timeout: 2 * testHeaderTimeout},
|
||||
ResponseHeaderTimeout: testHeaderTimeout,
|
||||
})
|
||||
err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, nil)
|
||||
var hte *ResponseHeaderTimeoutError
|
||||
if !errors.As(err, &hte) {
|
||||
t.Fatalf("err = %v, want *ResponseHeaderTimeoutError", err)
|
||||
}
|
||||
if !hte.Retried || hte.Reason != StallConnectionReset {
|
||||
t.Errorf("hte = %+v, want the retry recorded with reason %q", hte, StallConnectionReset)
|
||||
}
|
||||
if msg := err.Error(); !strings.Contains(msg, "the first had its connection reset") {
|
||||
t.Errorf("error %q should name the first attempt's reset", msg)
|
||||
}
|
||||
}
|
||||
|
||||
// TestASlowReadFinishesOnItsRetry: the timer cannot tell a silent connection
|
||||
// from a serve still working on a slow read. A read slower than the timer on
|
||||
// every connection is retried once and, with no timer on the retry, still
|
||||
// finishes, as it did before there was a timer.
|
||||
func TestASlowReadFinishesOnItsRetry(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
n := 0
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
n++
|
||||
mu.Unlock()
|
||||
select {
|
||||
case <-time.After(2 * testHeaderTimeout):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"ok":true}`))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
base, _ := url.Parse(srv.URL)
|
||||
var events []StallRetry
|
||||
c, err := New(base, nil, Options{
|
||||
HTTPClient: &http.Client{Transport: srv.Client().Transport, Timeout: DefaultTimeout},
|
||||
ResponseHeaderTimeout: testHeaderTimeout,
|
||||
OnStallRetry: func(e StallRetry) { events = append(events, e) },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
start := time.Now()
|
||||
var out map[string]any
|
||||
if err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, &out); err != nil {
|
||||
t.Fatalf("Do: %v; a read slower than the header timeout must still finish", err)
|
||||
}
|
||||
if out["ok"] != true {
|
||||
t.Fatalf("body = %v", out)
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed < 3*testHeaderTimeout {
|
||||
t.Errorf("elapsed %v, want one header timeout plus the slow answer (%v)", elapsed, 3*testHeaderTimeout)
|
||||
}
|
||||
if len(events) != 1 || events[0].Reason != StallHeaderTimeout {
|
||||
t.Errorf("events = %+v, want one %q retry", events, StallHeaderTimeout)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if n != 2 {
|
||||
t.Errorf("server saw %d requests, want 2 (the abandoned attempt and the retry)", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAReadQueuedBehindTheConnectionLimitFinishesOnceASlotFrees is serve at its
|
||||
// connection limit (a netutil.LimitListener, as serve's own): the kernel
|
||||
// completes the handshake and takes the request, but serve accepts the
|
||||
// connection only once a slot frees. The retry waits for that within the
|
||||
// overall timeout instead of failing a read serve was about to answer.
|
||||
func TestAReadQueuedBehindTheConnectionLimitFinishesOnceASlotFrees(t *testing.T) {
|
||||
occupied := make(chan struct{})
|
||||
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/occupy" {
|
||||
close(occupied)
|
||||
time.Sleep(3 * testHeaderTimeout)
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"ok":true}`))
|
||||
}))
|
||||
srv.Listener = netutil.LimitListener(srv.Listener, 1)
|
||||
// Every answer closes its connection, so the slot frees once it is sent.
|
||||
srv.Config.SetKeepAlivesEnabled(false)
|
||||
srv.Start()
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
occupier := make(chan error, 1)
|
||||
go func() {
|
||||
resp, err := srv.Client().Get(srv.URL + "/occupy")
|
||||
if err == nil {
|
||||
_ = resp.Body.Close()
|
||||
}
|
||||
occupier <- err
|
||||
}()
|
||||
<-occupied
|
||||
|
||||
base, _ := url.Parse(srv.URL)
|
||||
var events []StallRetry
|
||||
c, err := New(base, nil, Options{
|
||||
HTTPClient: &http.Client{Transport: srv.Client().Transport, Timeout: DefaultTimeout},
|
||||
ResponseHeaderTimeout: testHeaderTimeout,
|
||||
OnStallRetry: func(e StallRetry) { events = append(events, e) },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out map[string]any
|
||||
if err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, &out); err != nil {
|
||||
t.Fatalf("Do: %v; a read queued behind the connection limit must finish once a slot frees", err)
|
||||
}
|
||||
if out["ok"] != true {
|
||||
t.Fatalf("body = %v", out)
|
||||
}
|
||||
if len(events) != 1 || events[0].Reason != StallHeaderTimeout {
|
||||
t.Errorf("events = %+v, want one %q retry", events, StallHeaderTimeout)
|
||||
}
|
||||
if err := <-occupier; err != nil {
|
||||
t.Fatalf("the request holding the slot failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// slowCredentials is a credential source slower than the header timer, such as
|
||||
// a cold BEADS_HTTP_TOKEN_COMMAND.
|
||||
type slowCredentials struct{ d time.Duration }
|
||||
|
||||
func (s slowCredentials) Authorize(ctx context.Context, _ *http.Request) error {
|
||||
select {
|
||||
case <-time.After(s.d):
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func (slowCredentials) Refresh(context.Context) (bool, error) { return false, nil }
|
||||
|
||||
// TestASlowCredentialSourceIsNotAStall: the timer starts once the request has
|
||||
// been written, so the time a credential source takes is not counted as a
|
||||
// server that went silent.
|
||||
func TestASlowCredentialSourceIsNotAStall(t *testing.T) {
|
||||
fired := 0
|
||||
c, rec := newTestClient(t, Options{ResponseHeaderTimeout: testHeaderTimeout, OnStallRetry: func(StallRetry) { fired++ }},
|
||||
slowCredentials{d: 2 * testHeaderTimeout}, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
})
|
||||
if err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, nil); err != nil {
|
||||
t.Fatalf("Do: %v", err)
|
||||
}
|
||||
if fired != 0 {
|
||||
t.Errorf("OnStallRetry fired %d times for a slow credential source", fired)
|
||||
}
|
||||
if n := rec.count(); n != 1 {
|
||||
t.Errorf("server saw %d requests, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
// TestASlowCredentialSourceDoesNotShortenTheRetry: both attempts share the
|
||||
// client's Timeout counted, as that Timeout is, from when the first attempt is
|
||||
// sent, so a credential source that takes the whole Timeout to resolve (a cold
|
||||
// token command) leaves the retry of a slow but healthy read all of what the
|
||||
// first attempt did not use. The retry replays the stamped request rather than
|
||||
// consulting the source again, so a source this slow on every call does not
|
||||
// cost the retry its budget either.
|
||||
func TestASlowCredentialSourceDoesNotShortenTheRetry(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var arrived []time.Time
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
arrived = append(arrived, time.Now())
|
||||
mu.Unlock()
|
||||
select {
|
||||
case <-time.After(2 * testHeaderTimeout):
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"ok":true}`))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
base, _ := url.Parse(srv.URL)
|
||||
budget := 5 * testHeaderTimeout
|
||||
var events []StallRetry
|
||||
c, err := New(base, slowCredentials{d: budget}, Options{
|
||||
HTTPClient: &http.Client{Transport: srv.Client().Transport, Timeout: budget},
|
||||
ResponseHeaderTimeout: testHeaderTimeout,
|
||||
OnStallRetry: func(e StallRetry) { events = append(events, e) },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out map[string]any
|
||||
if err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, &out); err != nil {
|
||||
t.Fatalf("Do: %v; the credential source's time is not the server's, and must not cut the retry short", err)
|
||||
}
|
||||
done := time.Now()
|
||||
if out["ok"] != true {
|
||||
t.Fatalf("body = %v", out)
|
||||
}
|
||||
if len(events) != 1 || events[0].Reason != StallHeaderTimeout {
|
||||
t.Errorf("events = %+v, want one %q retry", events, StallHeaderTimeout)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(arrived) != 2 {
|
||||
t.Fatalf("server saw %d requests, want 2 (the abandoned attempt and the retry)", len(arrived))
|
||||
}
|
||||
if took := done.Sub(arrived[0]); took > budget {
|
||||
t.Errorf("the read finished %v after its first attempt arrived, want within one Timeout (%v)", took, budget)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAWriteWhoseDialTimesOutIsAPlainConnectError: a timeout before the request
|
||||
// was written (here the dial) cannot have reached the server, so it is not
|
||||
// reported as a header timeout that may or may not have been applied.
|
||||
func TestAWriteWhoseDialTimesOutIsAPlainConnectError(t *testing.T) {
|
||||
base, _ := url.Parse("http://127.0.0.1:1")
|
||||
transport := &http.Transport{DialContext: func(context.Context, string, string) (net.Conn, error) {
|
||||
return nil, &net.OpError{Op: "dial", Net: "tcp", Err: os.ErrDeadlineExceeded}
|
||||
}}
|
||||
c, err := New(base, nil, Options{
|
||||
HTTPClient: &http.Client{Transport: transport, Timeout: DefaultTimeout},
|
||||
ResponseHeaderTimeout: testHeaderTimeout,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
err = c.Do(ctx(t), Request{Op: OpCreateIssue, Method: http.MethodPost, Path: PathIssues,
|
||||
Body: map[string]any{"title": "t"}}, nil)
|
||||
var ce *ConnectError
|
||||
if !errors.As(err, &ce) {
|
||||
t.Fatalf("err = %T %v, want *ConnectError", err, err)
|
||||
}
|
||||
var hte *ResponseHeaderTimeoutError
|
||||
if errors.As(err, &hte) {
|
||||
t.Fatalf("a write whose dial timed out was reported as a header timeout: %v", err)
|
||||
}
|
||||
if msg := err.Error(); !strings.Contains(msg, "i/o timeout") || strings.Contains(msg, "may or may not") {
|
||||
t.Errorf("error %q should be the dial timeout, not a maybe-applied write", msg)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAStallLeavesOtherServersIdleConnectionsAlone: a transport may be shared by
|
||||
// every client in the process (DialWith's baseline transport), so a stall
|
||||
// retry must not flush its idle pool: another server's client keeps its
|
||||
// kept-alive connection.
|
||||
func TestAStallLeavesOtherServersIdleConnectionsAlone(t *testing.T) {
|
||||
stalled := newStallServer(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
})
|
||||
var mu sync.Mutex
|
||||
var remotes []string
|
||||
other := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
remotes = append(remotes, r.RemoteAddr)
|
||||
mu.Unlock()
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
}))
|
||||
t.Cleanup(other.Close)
|
||||
|
||||
shared := &http.Transport{}
|
||||
t.Cleanup(shared.CloseIdleConnections)
|
||||
hc := &http.Client{Transport: shared, Timeout: DefaultTimeout}
|
||||
stalledClient := stalled.client(t, Options{HTTPClient: hc, ResponseHeaderTimeout: testHeaderTimeout})
|
||||
otherBase, _ := url.Parse(other.URL)
|
||||
otherClient, err := New(otherBase, nil, Options{HTTPClient: hc, ResponseHeaderTimeout: testHeaderTimeout})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
read := Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}
|
||||
if err := otherClient.Do(ctx(t), read, nil); err != nil {
|
||||
t.Fatalf("other server: %v", err)
|
||||
}
|
||||
if err := stalledClient.Do(ctx(t), read, nil); err != nil {
|
||||
t.Fatalf("stalled server: %v", err)
|
||||
}
|
||||
if n := len(stalled.seen()); n != 2 {
|
||||
t.Fatalf("stalled server saw %d requests, want the stall and its retry", n)
|
||||
}
|
||||
if err := otherClient.Do(ctx(t), read, nil); err != nil {
|
||||
t.Fatalf("other server again: %v", err)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(remotes) != 2 || remotes[0] != remotes[1] {
|
||||
t.Errorf("other server saw connections %v; its kept-alive connection should have survived the stall retry", remotes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASlowStreamingBodyIsNotCutOffByTheHeaderTimeout(t *testing.T) {
|
||||
c, rec := newTestClient(t, Options{ResponseHeaderTimeout: testHeaderTimeout}, nil, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = io.WriteString(w, `{"ok":true`)
|
||||
w.(http.Flusher).Flush()
|
||||
// Keep streaming for well past the header timeout.
|
||||
for range 8 {
|
||||
time.Sleep(testHeaderTimeout / 2)
|
||||
_, _ = io.WriteString(w, " ")
|
||||
w.(http.Flusher).Flush()
|
||||
}
|
||||
_, _ = io.WriteString(w, `}`)
|
||||
})
|
||||
start := time.Now()
|
||||
var out map[string]any
|
||||
if err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, &out); err != nil {
|
||||
t.Fatalf("Do: %v", err)
|
||||
}
|
||||
if out["ok"] != true {
|
||||
t.Fatalf("body = %v", out)
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed < 3*testHeaderTimeout {
|
||||
t.Errorf("elapsed %v: the body should have streamed past the header timeout", elapsed)
|
||||
}
|
||||
if n := rec.count(); n != 1 {
|
||||
t.Errorf("server saw %d requests, want 1 (no retry for a slow body)", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheDefaultHeaderTimeoutUndercutsTheOverallTimeout(t *testing.T) {
|
||||
if DefaultResponseHeaderTimeout <= 0 || DefaultResponseHeaderTimeout >= DefaultTimeout {
|
||||
t.Fatalf("DefaultResponseHeaderTimeout = %v, want it positive and below DefaultTimeout (%v)",
|
||||
DefaultResponseHeaderTimeout, DefaultTimeout)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANegativeHeaderTimeoutDisablesIt(t *testing.T) {
|
||||
c, _ := newTestClient(t, Options{ResponseHeaderTimeout: -1}, nil, func(w http.ResponseWriter, _ *http.Request) {
|
||||
time.Sleep(2 * testHeaderTimeout)
|
||||
_, _ = w.Write([]byte(`{}`))
|
||||
})
|
||||
if c.headerTimeout != 0 {
|
||||
t.Fatalf("headerTimeout = %v, want disabled", c.headerTimeout)
|
||||
}
|
||||
if err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, nil); err != nil {
|
||||
t.Fatalf("Do: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestAReadThatSpentTheWholeTimeoutIsNotRetried: the retry runs on what the
|
||||
// first attempt left of the overall timeout, so a read whose first attempt used
|
||||
// all of it (here with the header timer off, as
|
||||
// BEADS_HTTP_RESPONSE_HEADER_TIMEOUT=off leaves it) fails with its typed
|
||||
// timeout, and reports no retry it has no time left to make.
|
||||
func TestAReadThatSpentTheWholeTimeoutIsNotRetried(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
n := 0
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
n++
|
||||
mu.Unlock()
|
||||
<-r.Context().Done()
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
base, _ := url.Parse(srv.URL)
|
||||
budget := testHeaderTimeout
|
||||
fired := 0
|
||||
c, err := New(base, nil, Options{
|
||||
HTTPClient: &http.Client{Transport: srv.Client().Transport, Timeout: budget},
|
||||
ResponseHeaderTimeout: -1,
|
||||
OnStallRetry: func(StallRetry) { fired++ },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
start := time.Now()
|
||||
err = c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, nil)
|
||||
elapsed := time.Since(start)
|
||||
var hte *ResponseHeaderTimeoutError
|
||||
if !errors.As(err, &hte) {
|
||||
t.Fatalf("err = %T %v, want *ResponseHeaderTimeoutError", err, err)
|
||||
}
|
||||
if hte.Retried || hte.Attempts != 1 || hte.Err == nil {
|
||||
t.Errorf("hte = %+v, want one unretried attempt and the transport's timeout", hte)
|
||||
}
|
||||
if fired != 0 {
|
||||
t.Errorf("OnStallRetry fired %d times with none of the overall timeout left", fired)
|
||||
}
|
||||
if elapsed < budget || elapsed > budget+time.Second {
|
||||
t.Errorf("elapsed %v, want the overall timeout (%v) and no more", elapsed, budget)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if n != 1 {
|
||||
t.Errorf("server saw %d requests, want 1", n)
|
||||
}
|
||||
}
|
||||
|
||||
// countingToken is a staticToken that counts its Authorize calls.
|
||||
type countingToken struct {
|
||||
staticToken
|
||||
authorized int
|
||||
}
|
||||
|
||||
func (c *countingToken) Authorize(ctx context.Context, req *http.Request) error {
|
||||
c.authorized++
|
||||
return c.staticToken.Authorize(ctx, req)
|
||||
}
|
||||
|
||||
// TestAStallRetryReplaysTheCredentialAndA401StillRefreshes: a read whose first
|
||||
// attempt stalls is resent with the headers Authorize set for it, not
|
||||
// authorized again, so a credential that went stale meanwhile draws a 401 on
|
||||
// the retry. That 401 reaches Refresh like any other, and the request Refresh
|
||||
// re-issues is authorized afresh.
|
||||
func TestAStallRetryReplaysTheCredentialAndA401StillRefreshes(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var answered []string
|
||||
s := newStallServer(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
auth := r.Header.Get("Authorization")
|
||||
mu.Lock()
|
||||
answered = append(answered, auth)
|
||||
mu.Unlock()
|
||||
if auth != "Bearer rotated" {
|
||||
unauthenticated(w)
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(`{"ok":true}`))
|
||||
})
|
||||
creds := &countingToken{staticToken: staticToken{tokens: []string{"stale", "rotated"}, retry: true}}
|
||||
base, _ := url.Parse(s.srv.URL)
|
||||
var events []StallRetry
|
||||
c, err := New(base, creds, Options{
|
||||
HTTPClient: &http.Client{Transport: s.srv.Client().Transport, Timeout: DefaultTimeout},
|
||||
ResponseHeaderTimeout: testHeaderTimeout,
|
||||
OnStallRetry: func(e StallRetry) { events = append(events, e) },
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var out map[string]any
|
||||
if err := c.Do(ctx(t), Request{Op: OpGetContext, Method: http.MethodGet, Path: PathContext}, &out); err != nil {
|
||||
t.Fatalf("Do: %v", err)
|
||||
}
|
||||
if out["ok"] != true {
|
||||
t.Fatalf("body = %v, want the answer to the refreshed credential", out)
|
||||
}
|
||||
if len(events) != 1 || events[0].Reason != StallHeaderTimeout {
|
||||
t.Errorf("events = %+v, want one %q retry", events, StallHeaderTimeout)
|
||||
}
|
||||
if creds.authorized != 2 {
|
||||
t.Errorf("Authorize called %d times, want 2 (once per request built, not once per attempt)", creds.authorized)
|
||||
}
|
||||
if creds.refreshN != 1 {
|
||||
t.Errorf("Refresh called %d times, want 1", creds.refreshN)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if want := []string{"Bearer stale", "Bearer rotated"}; !slices.Equal(answered, want) {
|
||||
t.Errorf("answered requests carried %q, want %q (the stall retry, then the refreshed request)", answered, want)
|
||||
}
|
||||
if n := len(s.seen()); n != 3 {
|
||||
t.Errorf("server saw %d requests, want 3 (the stall, its retry, and the refreshed request)", n)
|
||||
}
|
||||
}
|
||||
|
|
@ -35,6 +35,14 @@ import (
|
|||
// pieces: the shared request stamp, redirectRefused, the bounded error-body
|
||||
// read, mapProblem. Only once the stream is open does the vocabulary become
|
||||
// SSE.
|
||||
//
|
||||
// What the connect does NOT share is roundTrip's stall handling: it gets no
|
||||
// response-header timer and no retry on another connection (see
|
||||
// Options.ResponseHeaderTimeout). With the timeout removed, a connect that
|
||||
// lands on a connection gone silent waits until the caller's context ends, as
|
||||
// every connect did before the timer existed. The timer was scoped to Do's
|
||||
// reads, where the shadow run's stall cost each read the full 60s; bounding
|
||||
// this connect is a separate change.
|
||||
|
||||
// watchIdleTimeout is how long a stream may go silent before this client
|
||||
// declares it dead and fails loudly.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue