fix(proxy): fail fast with a clear MySQL error when an external upstream is unreachable (#7181)
* fix(proxy): fail fast with a MySQL error when the external upstream is unreachable A proxied-server command against a stopped external Dolt upstream stalled ~20-30s before failing with a bare "invalid connection". The local db proxy stays up, so on a refused backend dial (or a front such as stunnel/socat that accepts and closes because its own upstream is gone) it simply closed the client connection. The MySQL driver reports that as invalid connection / unexpected EOF, which the uow bootstrap ping (#6003) classifies as a transient drop and retries with exponential backoff for its whole 30s budget. Retrying cannot help: the proxy already proved the upstream at its own startup, and the outage is the upstream's. The proxy now answers such a connection itself with a MySQL ERR packet (2003, HY000) in place of the server greeting, naming the upstream and the cause, exactly as a real server refuses a connection before the handshake. The driver surfaces it as a *mysql.MySQLError, which the bootstrap retry already treats as permanent, so the command fails in ~0.1s with an actionable message. Only shapes that prove the upstream is not serving qualify: ECONNREFUSED/ENOENT/EHOSTUNREACH/ENETUNREACH on dial, or a backend that reaches EOF having sent zero bytes (a MySQL server always speaks first). A dial timeout or a drop after the greeting keeps the plain close and the client's transient retry. External backends only: a managed (local sidecar) backend is owned by the proxy and its dial can be refused transiently while bd drives it (the full proxied lane caught TestProxiedServerInitExplicitBeadsDirIsAuthoritative hitting exactly that mid-init when this applied to every backend), so managed backends keep the bare close and the client's bootstrap retry. TestExternalProxyOutageAndRestoreReDialsTCPAndUnix asserted the old silent close during an outage; it now asserts the ERR packet. With the latency fixed, TestProxiedOutageReconnectAcceptanceMatrix is renamed into the proxied discovery pattern as TestProxiedServerOutageReconnectAcceptanceMatrix and placed in the Bazel 30-shard block by gen_proxied_shard_manifest.py's incremental --write (shard 8). The frozen legacy 15-shard block is untouched; the test hash-distributes there like every other test added since it was frozen. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(proxy,uow): retry the proxy's upstream-outage report for ~1s; hedge the zero-byte case Review follow-up to the fail-fast change. A zero-byte close from the upstream is not proof it is down: Dolt closes queued connections without a byte when max_connections/back_log are full, when a queued connection times out, and while it shuts down for a restart; an L4 front does the same while its target is replaced. Its report now says "closed the connection before the MySQL greeting (down, restarting, or at its connection limit)" instead of "unreachable", and the comment no longer claims proof. Refusals and endpoint churn also lost #6003's retry entirely, so an external Dolt restart or a Service with no ready endpoints failed a command at once. The client now retries the proxy's report (proxy.IsUpstreamOutageError: 2003 with the "beads db proxy: " prefix) for upstreamOutageRetryWindow (1s) from the first time it sees it, with the backoff clamped so the last attempt lands at the window's end; other transient errors keep the 30s budget. A real outage still fails in about a second, inside the acceptance test's 3s bound. Also: the dial report drops Go call-site wrapping ("dial tcp H:P: connection refused"), truncation keeps UTF-8 intact, the comment states how the packet differs from a real server's pre-handshake ERR (keeps the '#'+SQLSTATE marker for vitess; 2003 is a client code chosen on purpose), a Stats counter for reported outages, and the isTransientPingError paragraph is reflowed and corrected. Tests: pingWithRetry retries the report inside the window and gives up at its end without using the full budget; openDB against a front that plays the proxy (its real packet) and comes back 300ms after the first connection succeeds in one call, and fails in about the window when it never does; NewProxyServer enables the report for external backends only; the dial message carries no Go type names; rune-safe truncation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(proxied): fail, not skip, socat-bridged subtests in the Bazel lane The external-unix front-door matrix subtest and both outage/reconnect matrix subtests bridge an external endpoint with socat and skipped when it was missing, so they could stop running without anyone noticing (the parent test still passes). requireSocat now fails them when BEADS_TEST_REQUIRE_SOCAT=1, which //cmd/bd:bd_proxied_test sets (its remote workers have socat; a recent main run's external-unix subtest passed there). The legacy GitHub 15-shard jobs have no socat and leave it unset, so the subtests keep skipping there. The check now runs before the upstream Dolt server is started, so a skip no longer pays for one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(proxied): measured weight for the outage matrix; pin it to legacy shard 1 Give TestProxiedServerOutageReconnectAcceptanceMatrix a duration entry (80s: local runs measured 55-79s alone, 60-70s in its shard, and review 68-109s under load) and re-place it with the generator's incremental --write: it moves from 30/8 to 30/11, one line, nothing else moved. The heaviest estimated shard is now 435s against a ~357s median; a repack would even that out but moves most assignments, so it is left to a separate CI-only change that refreshes all durations together. Legacy 15-shard block: the test would hash onto shard 14, already ~13.5 of the 15 min timeout on main. It is pinned to shard 1 (~6.5 min in recent main runs) under the block's reviewed-pin rule; it skips there today (no socat), and the pin keeps shard 14 safe if that changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
b8a9545cbe
commit
683993660f
19 changed files with 821 additions and 41 deletions
8
.github/scripts/proxied-cmd-test-shards.txt
vendored
8
.github/scripts/proxied-cmd-test-shards.txt
vendored
|
|
@ -28,6 +28,12 @@
|
|||
# incremental --write keeps pinned lines where they are; a pin is a
|
||||
# reviewed choice, not a regeneration of this block.
|
||||
#
|
||||
# TestProxiedServerOutageReconnectAcceptanceMatrix is pinned to shard 1
|
||||
# (the lightest in recent Main runs, ~6.5 min) instead of hashing onto
|
||||
# shard 14, which already runs ~13.5 min of the 15m timeout. These jobs
|
||||
# have no socat, so today it skips here; the pin keeps shard 14 safe if
|
||||
# socat is ever installed.
|
||||
#
|
||||
# This file is generated: notes added here are erased by the next
|
||||
# regeneration. Add them to the generator instead. Known blind spots
|
||||
# of the cost proxy (it counts inits, not the work done per init):
|
||||
|
|
@ -51,6 +57,7 @@
|
|||
15 1 TestProxiedServerDepConcurrent
|
||||
15 1 TestProxiedServerListSortRetiresTheWalk
|
||||
15 1 TestProxiedServerMolLastActivity
|
||||
15 1 TestProxiedServerOutageReconnectAcceptanceMatrix
|
||||
15 1 TestProxiedServerReopenConcurrent
|
||||
15 1 TestProxiedServerServeReadParity
|
||||
15 1 TestProxiedServerServeRefusesAStreamOnADisabledJournal
|
||||
|
|
@ -309,6 +316,7 @@
|
|||
30 10 TestProxiedServerServeEdgeCounts
|
||||
|
||||
30 11 TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce
|
||||
30 11 TestProxiedServerOutageReconnectAcceptanceMatrix
|
||||
30 11 TestProxiedServerReady2
|
||||
30 11 TestProxiedServerServeReadsTheJournalItJustWrote
|
||||
30 11 TestProxiedServerServeRefusesTheJournalWhenItIsDisabled
|
||||
|
|
|
|||
13
CHANGELOG.md
13
CHANGELOG.md
|
|
@ -20,6 +20,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
### Fixed
|
||||
|
||||
- **A proxied-server command against an unreachable external Dolt upstream
|
||||
now fails within about a second with a clear error instead of stalling
|
||||
~20-30s.** The local db proxy stayed up, so the client only saw a bare
|
||||
close, which the bootstrap ping retried as a transient drop for its whole
|
||||
30s budget before reporting `invalid connection`. The proxy now answers such
|
||||
a connection with a MySQL error (2003, prefixed `beads db proxy:`) saying
|
||||
what happened: the dial was refused / the socket is missing / the host is
|
||||
unreachable, or the upstream closed the connection before the MySQL greeting
|
||||
(down, restarting, or at its connection limit). The client retries that
|
||||
error only for about a second, so an endpoint that is flapping or rebinding
|
||||
is still ridden out within one command. A backend that drops a connection
|
||||
after the greeting, a dial timeout, and every refusal from a managed (local
|
||||
sidecar) backend keep the full transient retry.
|
||||
- **`bd list` no longer silently drops all but the last repeated filter flag.**
|
||||
`--status`, `--state`, and `--id` were plain string flags, so
|
||||
`bd list --status open --status closed --status pinned` kept only `pinned` —
|
||||
|
|
|
|||
|
|
@ -1456,6 +1456,10 @@ sh_test(
|
|||
"BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)",
|
||||
"BEADS_TEST_PROXIED_SERVER": "1",
|
||||
"BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1",
|
||||
# The external-unix / outage-matrix subtests bridge endpoints with
|
||||
# socat, which the workers have: fail rather than skip if it is
|
||||
# missing (requireSocat). The legacy GitHub lane leaves this unset.
|
||||
"BEADS_TEST_REQUIRE_SOCAT": "1",
|
||||
# The 4-vCPU runner shape: -test.parallel defaults to GOMAXPROCS, and
|
||||
# the job runs 4 tests at once, not one per worker core (39
|
||||
# concurrent `bd init --proxied-server` against one server hung a
|
||||
|
|
|
|||
|
|
@ -46,6 +46,7 @@ func TestProxiedServerHistoryRemoteRefusalFrontDoorMatrix(t *testing.T) {
|
|||
}},
|
||||
fixture{name: "external-unix", make: func(t *testing.T) proxiedProject {
|
||||
requireProxiedServerEnv(t)
|
||||
requireSocat(t)
|
||||
upstream := testutil.StartIsolatedDoltContainerHandle(t)
|
||||
socket := shortSocketPath(t, "dolt.sock")
|
||||
bridge := startHistoryUnixBridge(t, socket, upstream.Port)
|
||||
|
|
@ -316,9 +317,7 @@ func openDirectHistoryDB(t *testing.T, p directHistoryProject) *sql.DB {
|
|||
|
||||
func startHistoryUnixBridge(t *testing.T, endpoint, upstreamPort string) *exec.Cmd {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("socat"); err != nil {
|
||||
t.Skipf("socat is required for external Unix topology: %v", err)
|
||||
}
|
||||
requireSocat(t)
|
||||
if err := os.Remove(endpoint); err != nil && !os.IsNotExist(err) {
|
||||
t.Fatalf("remove stale Unix socket: %v", err)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -34,6 +34,21 @@ func requireProxiedServerEnv(t *testing.T) {
|
|||
testutil.RequireDoltBinary(t)
|
||||
}
|
||||
|
||||
// requireSocat skips a test that needs socat as an external-endpoint bridge
|
||||
// when it is not installed, or fails it when BEADS_TEST_REQUIRE_SOCAT=1. The
|
||||
// Bazel proxied lane sets that (its workers have socat), so these subtests
|
||||
// cannot silently stop running there; the legacy GitHub 15-shard lane has no
|
||||
// socat and leaves it unset, so they skip there.
|
||||
func requireSocat(t *testing.T) {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("socat"); err != nil {
|
||||
if os.Getenv("BEADS_TEST_REQUIRE_SOCAT") == "1" {
|
||||
t.Fatalf("socat is required (BEADS_TEST_REQUIRE_SOCAT=1): %v", err)
|
||||
}
|
||||
t.Skipf("socat is required: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func bdProxiedEnv(dir string) []string {
|
||||
var env []string
|
||||
for _, e := range os.Environ() {
|
||||
|
|
|
|||
|
|
@ -26,14 +26,7 @@ import (
|
|||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// Deliberately not named TestProxiedServer...: this would make it discovered
|
||||
// by the shard scripts' ^Test(ProxiedServer|ServerMode) regex, but both its
|
||||
// subtests still fail a real "exceeded 3s bound" assertion against a stopped
|
||||
// external upstream (bead filed: outage command exceeds 3s bound against
|
||||
// stopped external upstream, tcp and unix). Renaming it into a required lane
|
||||
// before that's fixed would turn Bazel + the legacy 15-shard hash-fallback
|
||||
// red. Rename it once the underlying latency issue is resolved.
|
||||
func TestProxiedOutageReconnectAcceptanceMatrix(t *testing.T) {
|
||||
func TestProxiedServerOutageReconnectAcceptanceMatrix(t *testing.T) {
|
||||
requireProxiedServerEnv(t)
|
||||
bd := buildEmbeddedBD(t)
|
||||
for _, topology := range []struct {
|
||||
|
|
@ -44,10 +37,8 @@ func TestProxiedOutageReconnectAcceptanceMatrix(t *testing.T) {
|
|||
{name: "external-unix-socket", socket: true},
|
||||
} {
|
||||
t.Run(topology.name, func(t *testing.T) {
|
||||
requireSocat(t)
|
||||
upstream := testutil.StartIsolatedDoltContainerHandle(t)
|
||||
if _, err := exec.LookPath("socat"); err != nil {
|
||||
t.Skip("socat is required")
|
||||
}
|
||||
gatePort, err := proxy.PickFreePort()
|
||||
require.NoError(t, err)
|
||||
var endpoint string
|
||||
|
|
|
|||
|
|
@ -97,7 +97,10 @@ sit behind that API, selected by `BEADS_TEST_DOLT_SERVER`:
|
|||
|
||||
`BEADS_TEST_REQUIRE_DOLT_CONTAINER=1` turns an unavailable backend into a
|
||||
failure (per test and in every `TestMain`) instead of a skip; lanes that
|
||||
exist to run the Dolt suites set it.
|
||||
exist to run the Dolt suites set it. `BEADS_TEST_REQUIRE_SOCAT=1` does the
|
||||
same for the proxied subtests that bridge an external endpoint with `socat`
|
||||
(external-unix, the outage/reconnect matrix); `//cmd/bd:bd_proxied_test`
|
||||
sets it, and the legacy GitHub proxied jobs, which have no `socat`, do not.
|
||||
|
||||
Under Bazel, `bazel test //... --config=doltserver` runs the Dolt-backed
|
||||
suites of pr.yml's "Test (storage domain + uow)" and "Contract corpus" jobs
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ go_library(
|
|||
"unverified_process_linux.go",
|
||||
"unverified_process_unsupported.go",
|
||||
"unverified_process_windows.go",
|
||||
"upstream_error.go",
|
||||
],
|
||||
importpath = "github.com/steveyegge/beads/internal/storage/dbproxy/proxy",
|
||||
visibility = ["//:__subpackages__"],
|
||||
|
|
@ -36,6 +37,7 @@ go_library(
|
|||
"//internal/storage/dbproxy/server",
|
||||
"//internal/storage/dbproxy/util",
|
||||
"@com_github_cenkalti_backoff_v4//:backoff",
|
||||
"@com_github_go_sql_driver_mysql//:mysql",
|
||||
"@org_golang_x_sync//errgroup",
|
||||
] + select({
|
||||
"@rules_go//go/platform:android": [
|
||||
|
|
@ -70,6 +72,7 @@ go_test(
|
|||
"force_stop_linux_test.go",
|
||||
"server_epoch_internal_test.go",
|
||||
"server_test.go",
|
||||
"upstream_error_test.go",
|
||||
],
|
||||
embed = [":proxy"],
|
||||
deps = [
|
||||
|
|
@ -79,6 +82,7 @@ go_test(
|
|||
"//internal/storage/dbproxy/pidfile",
|
||||
"//internal/storage/dbproxy/server",
|
||||
"//internal/storage/dbproxy/util",
|
||||
"@com_github_go_sql_driver_mysql//:mysql",
|
||||
"@com_github_stretchr_testify//assert",
|
||||
"@com_github_stretchr_testify//require",
|
||||
] + select({
|
||||
|
|
|
|||
|
|
@ -380,12 +380,16 @@ func TestExternalProxyOutageAndRestoreReDialsTCPAndUnix(t *testing.T) {
|
|||
// Close the restored listener to create a deterministic outage, then
|
||||
// rebind it in the same test so the proxy must re-dial the endpoint.
|
||||
fixture.Close()
|
||||
// The outage is reported to the client as a MySQL ERR packet in
|
||||
// place of the greeting (upstream_error.go), then the connection
|
||||
// closes; no upstream data is ever relayed.
|
||||
failed := dialInternalProxy(t, running.port)
|
||||
_ = failed.SetReadDeadline(time.Now().Add(2 * time.Second))
|
||||
var b [1]byte
|
||||
if _, err := failed.Read(b[:]); err == nil {
|
||||
t.Fatal("outage client unexpectedly received data")
|
||||
outage, err := io.ReadAll(failed)
|
||||
if err != nil {
|
||||
t.Fatalf("read outage response: %v", err)
|
||||
}
|
||||
requireUpstreamErrorPacket(t, outage)
|
||||
_ = failed.Close()
|
||||
eventuallyInternal(t, 2*time.Second, func() bool {
|
||||
return running.proxy.stats.Snapshot().BackendDialErrors >= 1
|
||||
|
|
|
|||
|
|
@ -56,6 +56,11 @@ type proxyServer struct {
|
|||
stats *Stats
|
||||
stopEpoch string
|
||||
|
||||
// reportUpstreamOutage answers a connection whose upstream is not
|
||||
// serving with a MySQL error instead of a bare close; see
|
||||
// upstream_error.go. Set for external backends only.
|
||||
reportUpstreamOutage bool
|
||||
|
||||
logger *log.Logger
|
||||
listener net.Listener
|
||||
activeConns atomic.Int64
|
||||
|
|
@ -101,6 +106,8 @@ func NewProxyServer(opts ProxyOpts) *proxyServer {
|
|||
server: opts.Server,
|
||||
stats: opts.Stats,
|
||||
stopEpoch: opts.StopEpoch,
|
||||
|
||||
reportUpstreamOutage: reportsUpstreamOutage(opts.Server),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -459,6 +466,9 @@ func (p *proxyServer) handleConn(ctx context.Context, client net.Conn) error {
|
|||
if err != nil {
|
||||
p.tracef("handleConn(%s) backend dial error: %v", addr, err)
|
||||
p.stats.IncBackendDialError()
|
||||
if p.reportUpstreamOutage && isUpstreamUnreachableDialError(err) {
|
||||
p.writeUpstreamOutage(client, dialFailureMessage(err))
|
||||
}
|
||||
_ = client.Close()
|
||||
return err
|
||||
}
|
||||
|
|
@ -497,6 +507,16 @@ func (p *proxyServer) handleConn(ctx context.Context, client net.Conn) error {
|
|||
n, err := io.Copy(client, backend)
|
||||
p.stats.AddBytesBackendToClient(n)
|
||||
p.tracef("handleConn(%s) backend→client done (n=%d, err=%v)", addr, n, err)
|
||||
// A MySQL server speaks first, so a backend that reaches EOF having
|
||||
// sent nothing never served this connection: a front whose own
|
||||
// target is gone, or a server at its connection limit or shutting
|
||||
// down. That is not proof of an outage (see upstream_error.go), so
|
||||
// the client retries the report briefly rather than failing on it.
|
||||
// A client that hung up first closes backend, which makes this Copy
|
||||
// fail rather than return a clean EOF, so that case stays silent.
|
||||
if p.reportUpstreamOutage && n == 0 && err == nil {
|
||||
p.writeUpstreamOutage(client, closedBeforeGreetingMessage(backend))
|
||||
}
|
||||
return err
|
||||
})
|
||||
return g.Wait()
|
||||
|
|
|
|||
|
|
@ -3,19 +3,22 @@ package proxy
|
|||
import "sync"
|
||||
|
||||
type Counters struct {
|
||||
ListenAndServeCalls int64
|
||||
BackendStartCalls int64
|
||||
BackendStopCalls int64
|
||||
IdleTimeouts int64
|
||||
SignalsReceived int64
|
||||
AcceptCalls int64
|
||||
AcceptErrors int64
|
||||
BackendDialAttempts int64
|
||||
BackendDialSuccess int64
|
||||
BackendDialErrors int64
|
||||
HandledConns int64
|
||||
BytesClientToBackend int64
|
||||
BytesBackendToClient int64
|
||||
ListenAndServeCalls int64
|
||||
BackendStartCalls int64
|
||||
BackendStopCalls int64
|
||||
IdleTimeouts int64
|
||||
SignalsReceived int64
|
||||
AcceptCalls int64
|
||||
AcceptErrors int64
|
||||
BackendDialAttempts int64
|
||||
BackendDialSuccess int64
|
||||
BackendDialErrors int64
|
||||
// UpstreamErrorsReported counts connections answered with the
|
||||
// upstream-outage ERR packet instead of a bare close.
|
||||
UpstreamErrorsReported int64
|
||||
HandledConns int64
|
||||
BytesClientToBackend int64
|
||||
BytesBackendToClient int64
|
||||
}
|
||||
|
||||
type Stats struct {
|
||||
|
|
@ -51,7 +54,10 @@ func (s *Stats) IncAcceptError() { s.update(func(c *Counters) { c.AcceptE
|
|||
func (s *Stats) IncBackendDialAttempt() { s.update(func(c *Counters) { c.BackendDialAttempts++ }) }
|
||||
func (s *Stats) IncBackendDialSuccess() { s.update(func(c *Counters) { c.BackendDialSuccess++ }) }
|
||||
func (s *Stats) IncBackendDialError() { s.update(func(c *Counters) { c.BackendDialErrors++ }) }
|
||||
func (s *Stats) IncHandledConn() { s.update(func(c *Counters) { c.HandledConns++ }) }
|
||||
func (s *Stats) IncUpstreamErrorReported() {
|
||||
s.update(func(c *Counters) { c.UpstreamErrorsReported++ })
|
||||
}
|
||||
func (s *Stats) IncHandledConn() { s.update(func(c *Counters) { c.HandledConns++ }) }
|
||||
func (s *Stats) AddBytesClientToBackend(n int64) {
|
||||
s.update(func(c *Counters) { c.BytesClientToBackend += n })
|
||||
}
|
||||
|
|
|
|||
161
internal/storage/dbproxy/proxy/upstream_error.go
Normal file
161
internal/storage/dbproxy/proxy/upstream_error.go
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
package proxy
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
mysql "github.com/go-sql-driver/mysql"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
|
||||
)
|
||||
|
||||
// A client of the proxy only ever sees the proxy's own listener, which is up,
|
||||
// so a plain close is all it learns when the external upstream behind it is
|
||||
// not serving: the MySQL driver reports "invalid connection" / unexpected
|
||||
// EOF, and the uow bootstrap ping classifies that as a transient drop and
|
||||
// retries it for its whole 30s budget (#6003). For an upstream that is down
|
||||
// that turns every bd command into a ~20-30s stall ending in an unhelpful
|
||||
// message.
|
||||
//
|
||||
// Instead the proxy answers such a connection itself with a MySQL ERR packet
|
||||
// in place of the server greeting, naming the upstream and what happened.
|
||||
// The client (internal/storage/uow, via IsUpstreamOutageError) retries this
|
||||
// error only for a short window (about a second), long enough to ride out an
|
||||
// endpoint that is flapping or rebinding, short enough that a real outage
|
||||
// fails within a couple of seconds instead of thirty. Two shapes qualify:
|
||||
//
|
||||
// - The dial was refused, the unix socket path is gone, or the host or
|
||||
// network is unreachable. Nothing is accepting connections there right
|
||||
// now.
|
||||
// - The upstream accepted and then closed without sending a byte. A MySQL
|
||||
// server speaks first, so it did not serve this connection. That is NOT
|
||||
// proof it is down: Dolt does exactly this when max_connections and
|
||||
// back_log are full, when a queued connection times out, and while it
|
||||
// shuts down for a restart; an L4 front (socat, stunnel, a TCP load
|
||||
// balancer, kubectl port-forward) does it when its own target is gone.
|
||||
// The message says so, and the client's short retry window is what
|
||||
// covers the transient cases.
|
||||
//
|
||||
// A dial timeout, a reset, or a connection dropped after the greeting keeps
|
||||
// the plain close, and with it the client's full transient retry.
|
||||
//
|
||||
// External backends only. A managed backend (local Dolt sidecar) is owned by
|
||||
// the proxy, and its dial can be refused transiently while bd is still
|
||||
// driving it, which the client's full bootstrap retry absorbs; those keep the
|
||||
// plain close.
|
||||
//
|
||||
// The packet is not byte-for-byte what a real server sends before the
|
||||
// handshake. MySQL omits the '#'+SQLSTATE marker there (no capabilities are
|
||||
// negotiated yet); this packet keeps it because dolthub/vitess's client
|
||||
// always expects it, while go-sql-driver accepts either form. The mysql CLI
|
||||
// will therefore show "#HY000" at the start of the message text. 2003
|
||||
// (CR_CONN_HOST_ERROR, "Can't connect to MySQL server") is a client-side
|
||||
// code that no real server sends; it is chosen because it describes the
|
||||
// condition, and the "beads db proxy:" prefix is what marks it as ours.
|
||||
|
||||
const (
|
||||
// UpstreamErrorNumber and UpstreamErrorPrefix identify the ERR packet
|
||||
// the proxy sends for an unreachable external upstream; see
|
||||
// IsUpstreamOutageError.
|
||||
UpstreamErrorNumber = 2003
|
||||
UpstreamErrorPrefix = "beads db proxy: "
|
||||
|
||||
upstreamErrorSQLState = "HY000"
|
||||
upstreamErrorMaxMessage = 1024
|
||||
upstreamErrorWriteTimeout = time.Second
|
||||
)
|
||||
|
||||
// IsUpstreamOutageError reports whether err is the proxy's own report that
|
||||
// its external upstream is unreachable (see the comment above), as opposed to
|
||||
// an error from the Dolt server itself.
|
||||
func IsUpstreamOutageError(err error) bool {
|
||||
var me *mysql.MySQLError
|
||||
return errors.As(err, &me) && me.Number == UpstreamErrorNumber &&
|
||||
strings.HasPrefix(me.Message, UpstreamErrorPrefix)
|
||||
}
|
||||
|
||||
// reportsUpstreamOutage reports whether the proxy should answer an
|
||||
// unreachable upstream with a MySQL error for this backend.
|
||||
func reportsUpstreamOutage(s server.DatabaseServer) bool {
|
||||
_, external := s.(*server.ExternalDoltServer)
|
||||
return external
|
||||
}
|
||||
|
||||
// isUpstreamUnreachableDialError reports whether a backend dial failure means
|
||||
// nothing is accepting connections at the upstream right now, as opposed to
|
||||
// slow (timeout) or a local resource problem.
|
||||
func isUpstreamUnreachableDialError(err error) bool {
|
||||
return errors.Is(err, syscall.ECONNREFUSED) ||
|
||||
errors.Is(err, syscall.ENOENT) ||
|
||||
errors.Is(err, syscall.EHOSTUNREACH) ||
|
||||
errors.Is(err, syscall.ENETUNREACH)
|
||||
}
|
||||
|
||||
// dialFailureMessage describes a refused/unreachable dial as
|
||||
// "dial <network> <address>: <reason>", without the Go call-site wrapping the
|
||||
// error carries.
|
||||
func dialFailureMessage(err error) string {
|
||||
var opErr *net.OpError
|
||||
var errno syscall.Errno
|
||||
if errors.As(err, &opErr) && opErr.Addr != nil && errors.As(err, &errno) {
|
||||
return fmt.Sprintf("upstream Dolt server unreachable: %s %s %s: %v", opErr.Op, opErr.Net, opErr.Addr, errno)
|
||||
}
|
||||
return "upstream Dolt server unreachable: " + err.Error()
|
||||
}
|
||||
|
||||
// closedBeforeGreetingMessage describes a backend connection that reached EOF
|
||||
// before the server greeting, naming the endpoint the proxy dialed.
|
||||
func closedBeforeGreetingMessage(backend net.Conn) string {
|
||||
where := ""
|
||||
if ra := backend.RemoteAddr(); ra != nil && ra.String() != "" {
|
||||
where = fmt.Sprintf(" at %s %s", ra.Network(), ra)
|
||||
}
|
||||
return "upstream Dolt server" + where + " closed the connection before the MySQL greeting " +
|
||||
"(down, restarting, or at its connection limit)"
|
||||
}
|
||||
|
||||
// upstreamErrorPacket encodes a MySQL ERR packet (sequence id 0, with the
|
||||
// '#'+SQLSTATE marker) carrying msg, truncated on a UTF-8 boundary.
|
||||
func upstreamErrorPacket(msg string) []byte {
|
||||
if len(msg) > upstreamErrorMaxMessage {
|
||||
cut := upstreamErrorMaxMessage
|
||||
for cut > 0 && !utf8.RuneStart(msg[cut]) {
|
||||
cut--
|
||||
}
|
||||
msg = msg[:cut]
|
||||
}
|
||||
payload := make([]byte, 0, 9+len(msg))
|
||||
payload = append(payload, 0xff)
|
||||
payload = binary.LittleEndian.AppendUint16(payload, UpstreamErrorNumber)
|
||||
payload = append(payload, '#')
|
||||
payload = append(payload, upstreamErrorSQLState...)
|
||||
payload = append(payload, msg...)
|
||||
pkt := make([]byte, 4, 4+len(payload))
|
||||
pkt[0] = byte(len(payload))
|
||||
pkt[1] = byte(len(payload) >> 8)
|
||||
pkt[2] = byte(len(payload) >> 16)
|
||||
pkt[3] = 0 // sequence id: the server's first packet
|
||||
return append(pkt, payload...)
|
||||
}
|
||||
|
||||
// UpstreamOutagePacket returns the ERR packet the proxy sends for an
|
||||
// unreachable external upstream, with detail as the message after
|
||||
// UpstreamErrorPrefix. Exported so client-side tests can play the proxy.
|
||||
func UpstreamOutagePacket(detail string) []byte {
|
||||
return upstreamErrorPacket(UpstreamErrorPrefix + detail)
|
||||
}
|
||||
|
||||
// writeUpstreamOutage sends the client the proxy's upstream-outage ERR
|
||||
// packet. Best effort: the caller closes the connection either way.
|
||||
func (p *proxyServer) writeUpstreamOutage(client net.Conn, detail string) {
|
||||
p.stats.IncUpstreamErrorReported()
|
||||
p.tracef("handleConn(%s) reporting upstream outage: %s", client.RemoteAddr(), detail)
|
||||
_ = client.SetWriteDeadline(time.Now().Add(upstreamErrorWriteTimeout))
|
||||
_, _ = client.Write(UpstreamOutagePacket(detail))
|
||||
}
|
||||
280
internal/storage/dbproxy/proxy/upstream_error_test.go
Normal file
280
internal/storage/dbproxy/proxy/upstream_error_test.go
Normal file
|
|
@ -0,0 +1,280 @@
|
|||
package proxy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
mysql "github.com/go-sql-driver/mysql"
|
||||
|
||||
"github.com/steveyegge/beads/internal/configfile"
|
||||
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
|
||||
)
|
||||
|
||||
// serveThroughProxy runs handleConn for every connection accepted on a fresh
|
||||
// loopback listener and returns its address. reportOutage stands in for an
|
||||
// external backend (the fakes here are not *server.ExternalDoltServer).
|
||||
func serveThroughProxy(t *testing.T, backend tcpBlackholeBackend, reportOutage bool) string {
|
||||
t.Helper()
|
||||
p := NewProxyServer(ProxyOpts{Server: backend, Stats: &Stats{}})
|
||||
p.logger = log.Default()
|
||||
p.reportUpstreamOutage = reportOutage
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = ln.Close() })
|
||||
go func() {
|
||||
for {
|
||||
conn, err := ln.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
go func() { _ = p.handleConn(context.Background(), conn) }()
|
||||
}
|
||||
}()
|
||||
return ln.Addr().String()
|
||||
}
|
||||
|
||||
// pingThroughProxy pings the proxy with the real MySQL driver and returns the
|
||||
// error and how long it took.
|
||||
func pingThroughProxy(t *testing.T, addr string) (time.Duration, error) {
|
||||
t.Helper()
|
||||
db, err := sql.Open("mysql", "root@tcp("+addr+")/?timeout=5s")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
err = db.PingContext(ctx)
|
||||
return time.Since(start), err
|
||||
}
|
||||
|
||||
func requireUpstreamUnreachable(t *testing.T, err error, elapsed time.Duration, wantInMessage string) {
|
||||
t.Helper()
|
||||
var me *mysql.MySQLError
|
||||
if !errors.As(err, &me) {
|
||||
t.Fatalf("ping error = %v (%T), want *mysql.MySQLError", err, err)
|
||||
}
|
||||
if !IsUpstreamOutageError(err) || string(me.SQLState[:]) != upstreamErrorSQLState {
|
||||
t.Fatalf("MySQL error = %d/%s %q, want the proxy's %d/%s upstream report", me.Number, me.SQLState, me.Message, UpstreamErrorNumber, upstreamErrorSQLState)
|
||||
}
|
||||
if !strings.Contains(me.Message, wantInMessage) {
|
||||
t.Fatalf("message %q lacks upstream context %q", me.Message, wantInMessage)
|
||||
}
|
||||
if elapsed > 2*time.Second {
|
||||
t.Fatalf("unreachable upstream took %s to report", elapsed)
|
||||
}
|
||||
}
|
||||
|
||||
// A refused upstream (nothing listening) is answered with a MySQL error the
|
||||
// driver reports as permanent, instead of a bare close the uow bootstrap
|
||||
// would retry for 30s.
|
||||
func TestHandleConnRefusedUpstreamAnswersWithMySQLError(t *testing.T) {
|
||||
dead, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deadAddr := dead.Addr().String()
|
||||
_ = dead.Close()
|
||||
|
||||
addr := serveThroughProxy(t, tcpBlackholeBackend{address: deadAddr}, true)
|
||||
elapsed, err := pingThroughProxy(t, addr)
|
||||
requireUpstreamUnreachable(t, err, elapsed, "connection refused")
|
||||
}
|
||||
|
||||
// A front that accepts and immediately closes (its own upstream is gone)
|
||||
// never sends the greeting; the proxy names the endpoint instead of letting
|
||||
// the client see an unexplained EOF.
|
||||
func TestHandleConnUpstreamClosingBeforeGreetingAnswersWithMySQLError(t *testing.T) {
|
||||
front, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = front.Close() })
|
||||
go func() {
|
||||
for {
|
||||
conn, err := front.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
_ = conn.Close()
|
||||
}
|
||||
}()
|
||||
|
||||
addr := serveThroughProxy(t, tcpBlackholeBackend{address: front.Addr().String()}, true)
|
||||
elapsed, err := pingThroughProxy(t, addr)
|
||||
requireUpstreamUnreachable(t, err, elapsed, front.Addr().String())
|
||||
if !strings.Contains(err.Error(), "closed the connection before the MySQL greeting (down, restarting, or at its connection limit)") {
|
||||
t.Fatalf("zero-byte close must not be reported as a proven outage: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A backend that spoke and then dropped the connection is not an unreachable
|
||||
// upstream: the client must see exactly the backend's bytes and a plain EOF,
|
||||
// so the bootstrap ping still treats the drop as transient (#6003).
|
||||
func TestHandleConnUpstreamDropAfterGreetingStaysPlainClose(t *testing.T) {
|
||||
greeting := []byte("\x01\x00\x00\x00\x0a")
|
||||
front, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = front.Close() })
|
||||
go func() {
|
||||
conn, err := front.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
_, _ = conn.Write(greeting)
|
||||
_ = conn.Close()
|
||||
}()
|
||||
|
||||
addr := serveThroughProxy(t, tcpBlackholeBackend{address: front.Addr().String()}, true)
|
||||
conn, err := net.Dial("tcp", addr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
_ = conn.SetReadDeadline(time.Now().Add(5 * time.Second))
|
||||
got, err := io.ReadAll(conn)
|
||||
if err != nil {
|
||||
t.Fatalf("read: %v", err)
|
||||
}
|
||||
if string(got) != string(greeting) {
|
||||
t.Fatalf("client received %q, want only the backend's %q", got, greeting)
|
||||
}
|
||||
}
|
||||
|
||||
// A managed (non-external) backend keeps the bare close on a refused dial, so
|
||||
// the client's bootstrap retry still covers a sidecar that is refused
|
||||
// transiently.
|
||||
func TestHandleConnManagedBackendRefusalStaysPlainClose(t *testing.T) {
|
||||
dead, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deadAddr := dead.Addr().String()
|
||||
_ = dead.Close()
|
||||
if reportsUpstreamOutage(tcpBlackholeBackend{address: deadAddr}) {
|
||||
t.Fatal("a non-external backend must not report upstream outages")
|
||||
}
|
||||
|
||||
addr := serveThroughProxy(t, tcpBlackholeBackend{address: deadAddr}, false)
|
||||
conn, err := net.Dial("tcp", addr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
_ = conn.SetReadDeadline(time.Now().Add(5 * time.Second))
|
||||
got, err := io.ReadAll(conn)
|
||||
if err != nil {
|
||||
t.Fatalf("read: %v", err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("managed backend refusal sent %q, want a bare close", got)
|
||||
}
|
||||
}
|
||||
|
||||
// requireUpstreamErrorPacket asserts that got is exactly one MySQL ERR packet
|
||||
// carrying the proxy's upstream-unreachable error.
|
||||
func requireUpstreamErrorPacket(t *testing.T, got []byte) {
|
||||
t.Helper()
|
||||
if len(got) < 13 {
|
||||
t.Fatalf("response %q is too short for an ERR packet", got)
|
||||
}
|
||||
payloadLen := int(got[0]) | int(got[1])<<8 | int(got[2])<<16
|
||||
if payloadLen != len(got)-4 || got[3] != 0 || got[4] != 0xff {
|
||||
t.Fatalf("response is not a single sequence-0 ERR packet: % x", got)
|
||||
}
|
||||
if errno := int(got[5]) | int(got[6])<<8; errno != UpstreamErrorNumber {
|
||||
t.Fatalf("errno = %d, want %d", errno, UpstreamErrorNumber)
|
||||
}
|
||||
if string(got[7:13]) != "#"+upstreamErrorSQLState {
|
||||
t.Fatalf("SQL state marker = %q, want #%s", got[7:13], upstreamErrorSQLState)
|
||||
}
|
||||
if msg := string(got[13:]); !strings.HasPrefix(msg, UpstreamErrorPrefix+"upstream Dolt server") {
|
||||
t.Fatalf("message %q lacks the proxy prefix and upstream context", msg)
|
||||
}
|
||||
}
|
||||
|
||||
// External backends enable the outage report through NewProxyServer; the
|
||||
// handleConn tests above set the flag by hand on fakes.
|
||||
func TestNewProxyServerReportsOutagesForExternalBackendsOnly(t *testing.T) {
|
||||
ext, err := server.NewExternalDoltServer(configfile.ExternalDoltConfig{Host: "127.0.0.1", Port: 3306})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !NewProxyServer(ProxyOpts{Server: ext}).reportUpstreamOutage {
|
||||
t.Fatal("an external backend must report upstream outages")
|
||||
}
|
||||
if NewProxyServer(ProxyOpts{Server: blockingBackend{}}).reportUpstreamOutage {
|
||||
t.Fatal("a non-external backend must not report upstream outages")
|
||||
}
|
||||
}
|
||||
|
||||
// The dial report names network, address and reason, without the Go
|
||||
// call-site prefixes ExternalDoltServer.Dial wraps the error in.
|
||||
func TestDialFailureMessageDropsGoCallSites(t *testing.T) {
|
||||
dead, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
host, portStr, _ := net.SplitHostPort(dead.Addr().String())
|
||||
_ = dead.Close()
|
||||
port, _ := strconv.Atoi(portStr)
|
||||
ext, err := server.NewExternalDoltServer(configfile.ExternalDoltConfig{Host: host, Port: port})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_, dialErr := ext.Dial(context.Background())
|
||||
if dialErr == nil || !isUpstreamUnreachableDialError(dialErr) {
|
||||
t.Fatalf("dial error = %v, want a refusal", dialErr)
|
||||
}
|
||||
got := dialFailureMessage(dialErr)
|
||||
want := "upstream Dolt server unreachable: dial tcp " + dead.Addr().String() + ": connection refused"
|
||||
if got != want {
|
||||
t.Fatalf("dialFailureMessage = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsUpstreamOutageErrorRequiresTheProxyPrefix(t *testing.T) {
|
||||
if IsUpstreamOutageError(&mysql.MySQLError{Number: UpstreamErrorNumber, Message: "Can't connect to MySQL server"}) {
|
||||
t.Fatal("a 2003 without the proxy prefix is not the proxy's report")
|
||||
}
|
||||
if IsUpstreamOutageError(&mysql.MySQLError{Number: 1045, Message: UpstreamErrorPrefix + "x"}) {
|
||||
t.Fatal("only 2003 carries the proxy's report")
|
||||
}
|
||||
if !IsUpstreamOutageError(fmt.Errorf("wrapped: %w", &mysql.MySQLError{Number: UpstreamErrorNumber, Message: UpstreamErrorPrefix + "x"})) {
|
||||
t.Fatal("a wrapped proxy report must be recognized")
|
||||
}
|
||||
}
|
||||
|
||||
func TestUpstreamErrorPacketTruncatesOnARuneBoundary(t *testing.T) {
|
||||
// "é" is two bytes; an odd prefix puts the byte limit mid-rune.
|
||||
pkt := upstreamErrorPacket("x" + strings.Repeat("é", upstreamErrorMaxMessage))
|
||||
if msg := pkt[13:]; !utf8.Valid(msg) || len(msg) > upstreamErrorMaxMessage {
|
||||
t.Fatalf("truncated message is %d bytes, valid UTF-8 = %v", len(msg), utf8.Valid(msg))
|
||||
}
|
||||
}
|
||||
|
||||
func TestUpstreamErrorPacketTruncatesMessage(t *testing.T) {
|
||||
pkt := upstreamErrorPacket(strings.Repeat("x", 4*upstreamErrorMaxMessage))
|
||||
payloadLen := int(pkt[0]) | int(pkt[1])<<8 | int(pkt[2])<<16
|
||||
if payloadLen != len(pkt)-4 || payloadLen != 9+upstreamErrorMaxMessage {
|
||||
t.Fatalf("payload length %d (packet %d), want %d", payloadLen, len(pkt), 9+upstreamErrorMaxMessage)
|
||||
}
|
||||
if pkt[3] != 0 || pkt[4] != 0xff || pkt[7] != '#' {
|
||||
t.Fatalf("malformed ERR packet header % x", pkt[:8])
|
||||
}
|
||||
}
|
||||
|
|
@ -148,6 +148,7 @@ go_test(
|
|||
"testmain_test.go",
|
||||
"tree_walker_contract_test.go",
|
||||
"tx_test.go",
|
||||
"upstream_outage_window_test.go",
|
||||
"version_reconcile_boot_test.go",
|
||||
"version_reconcile_integration_test.go",
|
||||
"version_reconciler_contract_test.go",
|
||||
|
|
|
|||
|
|
@ -607,9 +607,13 @@ const pingAttemptTimeout = 10 * time.Second
|
|||
// application-level rejection that can reach us as one, so at boot an OpError
|
||||
// is always the connection itself: refused while the server restarts, reset,
|
||||
// or broken pipe. MySQL's own rejections arrive as *mysql.MySQLError and stay
|
||||
// permanent. A name that does not resolve is the exception — it will not start
|
||||
// resolving, so retrying only spends the whole budget before reporting the
|
||||
// same misconfiguration.
|
||||
// permanent here. A name that does not resolve is the exception — it will not
|
||||
// start resolving, so retrying only spends the whole budget before reporting
|
||||
// the same misconfiguration.
|
||||
//
|
||||
// The db proxy's own upstream-outage report (proxy.IsUpstreamOutageError) is
|
||||
// a *mysql.MySQLError too, so this function calls it permanent; pingWithRetry
|
||||
// handles it first and retries it only for upstreamOutageRetryWindow.
|
||||
//
|
||||
// context.DeadlineExceeded counts only because pingWithRetry applies a
|
||||
// per-attempt deadline and checks the caller's context *before* consulting
|
||||
|
|
@ -640,12 +644,45 @@ func isTransientPingError(err error) bool {
|
|||
errors.As(err, &opErr)
|
||||
}
|
||||
|
||||
// upstreamOutageRetryWindow is how long pingWithRetry keeps retrying the db
|
||||
// proxy's report that its external upstream is unreachable (refused, socket
|
||||
// gone, or closed before the greeting; see dbproxy/proxy/upstream_error.go).
|
||||
// Long enough to ride out an endpoint that is flapping or rebinding (a Dolt
|
||||
// restart, a front whose target is being replaced, a server briefly at its
|
||||
// connection limit), short enough that a real outage fails within a couple of
|
||||
// seconds instead of spending the full 30s bootstrap budget on a dead
|
||||
// upstream. A var so tests can shorten it.
|
||||
var upstreamOutageRetryWindow = time.Second
|
||||
|
||||
// outageWindowBackOff clamps the wrapped backoff so that, once the first
|
||||
// upstream-outage report has been seen, no wait runs past the end of
|
||||
// upstreamOutageRetryWindow: the last attempt lands at the window's end
|
||||
// rather than up to one backoff interval after it.
|
||||
type outageWindowBackOff struct {
|
||||
backoff.BackOff
|
||||
deadline time.Time // zero until the first upstream-outage report
|
||||
}
|
||||
|
||||
func (b *outageWindowBackOff) NextBackOff() time.Duration {
|
||||
d := b.BackOff.NextBackOff()
|
||||
if d == backoff.Stop || b.deadline.IsZero() {
|
||||
return d
|
||||
}
|
||||
if remaining := time.Until(b.deadline); remaining < d {
|
||||
return max(remaining, 0)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// pingWithRetry pings until the server answers, the error proves durable, or
|
||||
// bo's MaxElapsedTime is spent. Each attempt gets its own deadline:
|
||||
// backoff.Retry bounds the gap between attempts but cannot interrupt one
|
||||
// already in flight, so without this cap a server that accepts TCP and then
|
||||
// stalls blocks for the caller's entire context.
|
||||
// stalls blocks for the caller's entire context. The db proxy's
|
||||
// upstream-outage report is retried only for upstreamOutageRetryWindow from
|
||||
// the first time it is seen.
|
||||
func pingWithRetry(ctx context.Context, p pinger, bo *backoff.ExponentialBackOff, attemptTimeout time.Duration) error {
|
||||
window := &outageWindowBackOff{BackOff: bo}
|
||||
return backoff.Retry(func() error {
|
||||
attemptCtx, cancel := context.WithTimeout(ctx, attemptTimeout)
|
||||
defer cancel()
|
||||
|
|
@ -661,11 +698,20 @@ func pingWithRetry(ctx context.Context, p pinger, bo *backoff.ExponentialBackOff
|
|||
if ctx.Err() != nil {
|
||||
return backoff.Permanent(err)
|
||||
}
|
||||
if proxy.IsUpstreamOutageError(err) {
|
||||
if window.deadline.IsZero() {
|
||||
window.deadline = time.Now().Add(upstreamOutageRetryWindow)
|
||||
}
|
||||
if !time.Now().Before(window.deadline) {
|
||||
return backoff.Permanent(err)
|
||||
}
|
||||
return err
|
||||
}
|
||||
if isTransientPingError(err) {
|
||||
return err
|
||||
}
|
||||
return backoff.Permanent(err)
|
||||
}, backoff.WithContext(bo, ctx))
|
||||
}, backoff.WithContext(window, ctx))
|
||||
}
|
||||
|
||||
// assertSessionDatabaseOnPool runs assertSessionDatabase on a connection pinned
|
||||
|
|
|
|||
215
internal/storage/uow/upstream_outage_window_test.go
Normal file
215
internal/storage/uow/upstream_outage_window_test.go
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
package uow
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"io"
|
||||
"net"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage/dbproxy/proxy"
|
||||
)
|
||||
|
||||
func proxyOutageErr() error {
|
||||
return &mysql.MySQLError{Number: proxy.UpstreamErrorNumber, Message: proxy.UpstreamErrorPrefix + "upstream Dolt server unreachable: dial tcp 127.0.0.1:1: connection refused"}
|
||||
}
|
||||
|
||||
func shortOutageWindow(t *testing.T, d time.Duration) {
|
||||
t.Helper()
|
||||
prev := upstreamOutageRetryWindow
|
||||
upstreamOutageRetryWindow = d
|
||||
t.Cleanup(func() { upstreamOutageRetryWindow = prev })
|
||||
}
|
||||
|
||||
func TestPingWithRetryRetriesTheProxyOutageReportOnlyBriefly(t *testing.T) {
|
||||
t.Run("upstream back inside the window recovers", func(t *testing.T) {
|
||||
shortOutageWindow(t, 2*time.Second)
|
||||
p := &fakePinger{t: t, errs: []error{proxyOutageErr(), proxyOutageErr(), nil}}
|
||||
if err := pingWithRetry(context.Background(), p, testPingBackOff(), testPingAttemptTimeout); err != nil {
|
||||
t.Fatalf("pingWithRetry() error = %v, want recovery inside the outage window", err)
|
||||
}
|
||||
if p.calls != 3 {
|
||||
t.Fatalf("PingContext called %d times, want 3", p.calls)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("outage for the whole window fails at its end", func(t *testing.T) {
|
||||
const window = 300 * time.Millisecond
|
||||
shortOutageWindow(t, window)
|
||||
var calls atomic.Int32
|
||||
p := pingerFunc(func(context.Context) error { calls.Add(1); return proxyOutageErr() })
|
||||
bo := testPingBackOff()
|
||||
bo.InitialInterval = 50 * time.Millisecond
|
||||
bo.MaxElapsedTime = 30 * time.Second // the full bootstrap budget must not apply
|
||||
start := time.Now()
|
||||
err := pingWithRetry(context.Background(), p, bo, testPingAttemptTimeout)
|
||||
elapsed := time.Since(start)
|
||||
if !proxy.IsUpstreamOutageError(err) {
|
||||
t.Fatalf("pingWithRetry() error = %v, want the proxy's outage report", err)
|
||||
}
|
||||
if elapsed < window || elapsed > window+500*time.Millisecond {
|
||||
t.Fatalf("gave up after %s, want just past the %s window", elapsed, window)
|
||||
}
|
||||
if calls.Load() < 2 {
|
||||
t.Fatalf("PingContext called %d times, want the report retried inside the window", calls.Load())
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("other transient errors keep the full budget", func(t *testing.T) {
|
||||
shortOutageWindow(t, time.Nanosecond)
|
||||
p := &fakePinger{t: t, errs: []error{mysql.ErrInvalidConn, mysql.ErrInvalidConn, nil}}
|
||||
if err := pingWithRetry(context.Background(), p, testPingBackOff(), testPingAttemptTimeout); err != nil {
|
||||
t.Fatalf("pingWithRetry() error = %v, want nil", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
type pingerFunc func(context.Context) error
|
||||
|
||||
func (f pingerFunc) PingContext(ctx context.Context) error { return f(ctx) }
|
||||
|
||||
// TestOpenDBRidesOutAnUpstreamThatReturnsInsideTheOutageWindow plays the db
|
||||
// proxy in front of an external upstream that is briefly unreachable: the
|
||||
// first connections get the proxy's real outage packet, and 300ms after the
|
||||
// first one the upstream "comes back" (a minimal MySQL server answers). One
|
||||
// openDB call — one bd command's bootstrap — must succeed without the caller
|
||||
// retrying.
|
||||
func TestOpenDBRidesOutAnUpstreamThatReturnsInsideTheOutageWindow(t *testing.T) {
|
||||
front := newFlappingFront(t, 300*time.Millisecond)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
db, err := openDB(ctx, "root@tcp("+front.addr+")/?timeout=5s")
|
||||
if err != nil {
|
||||
t.Fatalf("openDB() error = %v, want recovery once the upstream returned", err)
|
||||
}
|
||||
_ = db.Close()
|
||||
if front.refused.Load() == 0 {
|
||||
t.Fatal("the command never hit the outage; the test proved nothing")
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed > upstreamOutageRetryWindow+time.Second {
|
||||
t.Fatalf("recovery took %s", elapsed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenDBFailsFastWhenTheUpstreamStaysUnreachable(t *testing.T) {
|
||||
front := newFlappingFront(t, time.Hour)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
start := time.Now()
|
||||
_, err := openDB(ctx, "root@tcp("+front.addr+")/?timeout=5s")
|
||||
elapsed := time.Since(start)
|
||||
if !proxy.IsUpstreamOutageError(err) {
|
||||
t.Fatalf("openDB() error = %v, want the proxy's outage report", err)
|
||||
}
|
||||
if elapsed > upstreamOutageRetryWindow+time.Second {
|
||||
t.Fatalf("an unreachable upstream took %s to fail, want about %s", elapsed, upstreamOutageRetryWindow)
|
||||
}
|
||||
}
|
||||
|
||||
// flappingFront answers with the db proxy's outage packet until downFor has
|
||||
// passed since its first connection, then serves as a minimal MySQL server.
|
||||
type flappingFront struct {
|
||||
addr string
|
||||
refused atomic.Int32
|
||||
}
|
||||
|
||||
func newFlappingFront(t *testing.T, downFor time.Duration) *flappingFront {
|
||||
t.Helper()
|
||||
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f := &flappingFront{addr: ln.Addr().String()}
|
||||
var wg sync.WaitGroup
|
||||
t.Cleanup(func() { _ = ln.Close(); wg.Wait() })
|
||||
var once sync.Once
|
||||
var upAt time.Time
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for {
|
||||
conn, err := ln.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
once.Do(func() { upAt = time.Now().Add(downFor) })
|
||||
if time.Now().Before(upAt) {
|
||||
f.refused.Add(1)
|
||||
_, _ = conn.Write(proxy.UpstreamOutagePacket("upstream Dolt server unreachable: dial tcp 127.0.0.1:1: connection refused"))
|
||||
_ = conn.Close()
|
||||
continue
|
||||
}
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); serveMinimalMySQL(conn) }()
|
||||
}
|
||||
}()
|
||||
return f
|
||||
}
|
||||
|
||||
// serveMinimalMySQL speaks just enough of the server side of the MySQL
|
||||
// protocol for go-sql-driver to connect and ping: a greeting, OK to any
|
||||
// authentication, and OK to every command until COM_QUIT or EOF.
|
||||
func serveMinimalMySQL(conn net.Conn) {
|
||||
defer conn.Close()
|
||||
_ = conn.SetDeadline(time.Now().Add(10 * time.Second))
|
||||
const (
|
||||
clientLongPassword = 1 << 0
|
||||
clientProtocol41 = 1 << 9
|
||||
clientTransactions = 1 << 13
|
||||
clientSecureConn = 1 << 15
|
||||
clientPluginAuth = 1 << 19
|
||||
)
|
||||
caps := uint32(clientLongPassword | clientProtocol41 | clientTransactions | clientSecureConn | clientPluginAuth)
|
||||
g := []byte{0x0a}
|
||||
g = append(g, "8.0.0-fake\x00"...)
|
||||
g = binary.LittleEndian.AppendUint32(g, 1)
|
||||
g = append(g, "abcdefgh"...) // auth data part 1
|
||||
g = append(g, 0)
|
||||
g = binary.LittleEndian.AppendUint16(g, uint16(caps))
|
||||
g = append(g, 0x21) // utf8_general_ci
|
||||
g = binary.LittleEndian.AppendUint16(g, 0x0002)
|
||||
g = binary.LittleEndian.AppendUint16(g, uint16(caps>>16))
|
||||
g = append(g, 21)
|
||||
g = append(g, make([]byte, 10)...)
|
||||
g = append(g, "ijklmnopqrst\x00"...) // auth data part 2
|
||||
g = append(g, "mysql_native_password\x00"...)
|
||||
ok := []byte{0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00}
|
||||
|
||||
write := func(seq byte, payload []byte) error {
|
||||
hdr := []byte{byte(len(payload)), byte(len(payload) >> 8), byte(len(payload) >> 16), seq}
|
||||
_, err := conn.Write(append(hdr, payload...))
|
||||
return err
|
||||
}
|
||||
read := func() (byte, []byte, error) {
|
||||
var hdr [4]byte
|
||||
if _, err := io.ReadFull(conn, hdr[:]); err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
body := make([]byte, int(hdr[0])|int(hdr[1])<<8|int(hdr[2])<<16)
|
||||
_, err := io.ReadFull(conn, body)
|
||||
return hdr[3], body, err
|
||||
}
|
||||
|
||||
if write(0, g) != nil {
|
||||
return
|
||||
}
|
||||
seq, _, err := read() // handshake response
|
||||
if err != nil || write(seq+1, ok) != nil {
|
||||
return
|
||||
}
|
||||
for {
|
||||
seq, body, err := read()
|
||||
if err != nil || len(body) == 0 || body[0] == 0x01 { // EOF or COM_QUIT
|
||||
return
|
||||
}
|
||||
if write(seq+1, ok) != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -217,7 +217,9 @@ def render(total, shards, weights):
|
|||
out.append('# Oct 2026 for TestProxiedServerStorageClass,')
|
||||
out.append('# TestProxiedServerServeRefusesAStaleCheckpointWithGone,')
|
||||
out.append('# TestProxiedServerListComments and')
|
||||
out.append('# TestProxiedServerServeRefusesAStreamOnADisabledJournal).')
|
||||
out.append('# TestProxiedServerServeRefusesAStreamOnADisabledJournal; and for')
|
||||
out.append('# TestProxiedServerOutageReconnectAcceptanceMatrix, pinned to shard 1')
|
||||
out.append('# instead of hashing onto shard 14).')
|
||||
else:
|
||||
out.append(f'# {total}-shard split for the Bazel-only proxied-server tier')
|
||||
out.append('# (bazel-proxied in .github/workflows/bazel.yml), bin-packed')
|
||||
|
|
|
|||
|
|
@ -49,9 +49,16 @@
|
|||
"//cmd/bd:bd_proxied_test --build_event_json_file=<path>` sweep, then pull",
|
||||
"each shard's bazel-testlogs test.xml <testsuite time> per top-level test --",
|
||||
"not from downloaded CI artifacts -- so the parallel=4, CI-representative",
|
||||
"methodology above is applied consistently."
|
||||
"methodology above is applied consistently.",
|
||||
"",
|
||||
"Added after that sweep: TestProxiedServerOutageReconnectAcceptanceMatrix (80.0,",
|
||||
"renamed into the discovery pattern once its outage bound was fixed). Not a",
|
||||
"single sweep's value: local --config=doltserver-proxied runs measured it at",
|
||||
"55-79 s alone, 60-70 s inside its shard, and a reviewer's 68-109 s under load;",
|
||||
"80 s is a rounded middle of that spread. seconds_per_init_fallback was",
|
||||
"recomputed to include it."
|
||||
],
|
||||
"seconds_per_init_fallback": 17.1946,
|
||||
"seconds_per_init_fallback": 17.2952,
|
||||
"tests": {
|
||||
"TestProxiedServerAssign": 38.67,
|
||||
"TestProxiedServerAutoPrune": 67.5,
|
||||
|
|
@ -142,6 +149,7 @@
|
|||
"TestProxiedServerMultiStatementSQL": 6.76,
|
||||
"TestProxiedServerNote": 11.21,
|
||||
"TestProxiedServerOrphans": 12.11,
|
||||
"TestProxiedServerOutageReconnectAcceptanceMatrix": 80.0,
|
||||
"TestProxiedServerPathHelpers": 0.0,
|
||||
"TestProxiedServerPing": 11.73,
|
||||
"TestProxiedServerPour": 253.65,
|
||||
|
|
|
|||
|
|
@ -1739,7 +1739,7 @@ func TestBazelRetiredLanesCannotBeNarrowed(t *testing.T) {
|
|||
map[string]string{
|
||||
"BEADS_TEST_BD_BINARY": "$(rlocationpath :bd_for_tests)", "BEADS_TEST_DOLT_SERVER": "local", "BEADS_TEST_GIT_IDENTITY": "1",
|
||||
"BEADS_TEST_GOFMT": "$(rlocationpath @go_sdk//:bin/gofmt)", "BEADS_TEST_PROXIED_SERVER": "1",
|
||||
"BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", "GOMAXPROCS": "4",
|
||||
"BEADS_TEST_REQUIRE_DOLT_CONTAINER": "1", "BEADS_TEST_REQUIRE_SOCAT": "1", "GOMAXPROCS": "4",
|
||||
},
|
||||
},
|
||||
"//internal/storage/dolt:dolt_server_conformance_test": {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue