* fix(proxy): fail fast with a MySQL error when the external upstream is unreachable A proxied-server command against a stopped external Dolt upstream stalled ~20-30s before failing with a bare "invalid connection". The local db proxy stays up, so on a refused backend dial (or a front such as stunnel/socat that accepts and closes because its own upstream is gone) it simply closed the client connection. The MySQL driver reports that as invalid connection / unexpected EOF, which the uow bootstrap ping (#6003) classifies as a transient drop and retries with exponential backoff for its whole 30s budget. Retrying cannot help: the proxy already proved the upstream at its own startup, and the outage is the upstream's. The proxy now answers such a connection itself with a MySQL ERR packet (2003, HY000) in place of the server greeting, naming the upstream and the cause, exactly as a real server refuses a connection before the handshake. The driver surfaces it as a *mysql.MySQLError, which the bootstrap retry already treats as permanent, so the command fails in ~0.1s with an actionable message. Only shapes that prove the upstream is not serving qualify: ECONNREFUSED/ENOENT/EHOSTUNREACH/ENETUNREACH on dial, or a backend that reaches EOF having sent zero bytes (a MySQL server always speaks first). A dial timeout or a drop after the greeting keeps the plain close and the client's transient retry. External backends only: a managed (local sidecar) backend is owned by the proxy and its dial can be refused transiently while bd drives it (the full proxied lane caught TestProxiedServerInitExplicitBeadsDirIsAuthoritative hitting exactly that mid-init when this applied to every backend), so managed backends keep the bare close and the client's bootstrap retry. TestExternalProxyOutageAndRestoreReDialsTCPAndUnix asserted the old silent close during an outage; it now asserts the ERR packet. With the latency fixed, TestProxiedOutageReconnectAcceptanceMatrix is renamed into the proxied discovery pattern as TestProxiedServerOutageReconnectAcceptanceMatrix and placed in the Bazel 30-shard block by gen_proxied_shard_manifest.py's incremental --write (shard 8). The frozen legacy 15-shard block is untouched; the test hash-distributes there like every other test added since it was frozen. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(proxy,uow): retry the proxy's upstream-outage report for ~1s; hedge the zero-byte case Review follow-up to the fail-fast change. A zero-byte close from the upstream is not proof it is down: Dolt closes queued connections without a byte when max_connections/back_log are full, when a queued connection times out, and while it shuts down for a restart; an L4 front does the same while its target is replaced. Its report now says "closed the connection before the MySQL greeting (down, restarting, or at its connection limit)" instead of "unreachable", and the comment no longer claims proof. Refusals and endpoint churn also lost #6003's retry entirely, so an external Dolt restart or a Service with no ready endpoints failed a command at once. The client now retries the proxy's report (proxy.IsUpstreamOutageError: 2003 with the "beads db proxy: " prefix) for upstreamOutageRetryWindow (1s) from the first time it sees it, with the backoff clamped so the last attempt lands at the window's end; other transient errors keep the 30s budget. A real outage still fails in about a second, inside the acceptance test's 3s bound. Also: the dial report drops Go call-site wrapping ("dial tcp H:P: connection refused"), truncation keeps UTF-8 intact, the comment states how the packet differs from a real server's pre-handshake ERR (keeps the '#'+SQLSTATE marker for vitess; 2003 is a client code chosen on purpose), a Stats counter for reported outages, and the isTransientPingError paragraph is reflowed and corrected. Tests: pingWithRetry retries the report inside the window and gives up at its end without using the full budget; openDB against a front that plays the proxy (its real packet) and comes back 300ms after the first connection succeeds in one call, and fails in about the window when it never does; NewProxyServer enables the report for external backends only; the dial message carries no Go type names; rune-safe truncation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(proxied): fail, not skip, socat-bridged subtests in the Bazel lane The external-unix front-door matrix subtest and both outage/reconnect matrix subtests bridge an external endpoint with socat and skipped when it was missing, so they could stop running without anyone noticing (the parent test still passes). requireSocat now fails them when BEADS_TEST_REQUIRE_SOCAT=1, which //cmd/bd:bd_proxied_test sets (its remote workers have socat; a recent main run's external-unix subtest passed there). The legacy GitHub 15-shard jobs have no socat and leave it unset, so the subtests keep skipping there. The check now runs before the upstream Dolt server is started, so a skip no longer pays for one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(proxied): measured weight for the outage matrix; pin it to legacy shard 1 Give TestProxiedServerOutageReconnectAcceptanceMatrix a duration entry (80s: local runs measured 55-79s alone, 60-70s in its shard, and review 68-109s under load) and re-place it with the generator's incremental --write: it moves from 30/8 to 30/11, one line, nothing else moved. The heaviest estimated shard is now 435s against a ~357s median; a repack would even that out but moves most assignments, so it is left to a separate CI-only change that refreshes all durations together. Legacy 15-shard block: the test would hash onto shard 14, already ~13.5 of the 15 min timeout on main. It is pinned to shard 1 (~6.5 min in recent main runs) under the block's reviewed-pin rule; it skips there today (no socat), and the pin keeps shard 14 safe if that changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
66 lines
2.2 KiB
Go
66 lines
2.2 KiB
Go
package proxy
|
|
|
|
import "sync"
|
|
|
|
type Counters struct {
|
|
ListenAndServeCalls int64
|
|
BackendStartCalls int64
|
|
BackendStopCalls int64
|
|
IdleTimeouts int64
|
|
SignalsReceived int64
|
|
AcceptCalls int64
|
|
AcceptErrors int64
|
|
BackendDialAttempts int64
|
|
BackendDialSuccess int64
|
|
BackendDialErrors int64
|
|
// UpstreamErrorsReported counts connections answered with the
|
|
// upstream-outage ERR packet instead of a bare close.
|
|
UpstreamErrorsReported int64
|
|
HandledConns int64
|
|
BytesClientToBackend int64
|
|
BytesBackendToClient int64
|
|
}
|
|
|
|
type Stats struct {
|
|
mu sync.Mutex
|
|
counters Counters
|
|
}
|
|
|
|
func (s *Stats) Snapshot() Counters {
|
|
if s == nil {
|
|
return Counters{}
|
|
}
|
|
s.mu.Lock()
|
|
defer s.mu.Unlock()
|
|
return s.counters
|
|
}
|
|
|
|
func (s *Stats) update(fn func(*Counters)) {
|
|
if s == nil {
|
|
return
|
|
}
|
|
s.mu.Lock()
|
|
fn(&s.counters)
|
|
s.mu.Unlock()
|
|
}
|
|
|
|
func (s *Stats) IncListenAndServe() { s.update(func(c *Counters) { c.ListenAndServeCalls++ }) }
|
|
func (s *Stats) IncBackendStart() { s.update(func(c *Counters) { c.BackendStartCalls++ }) }
|
|
func (s *Stats) IncBackendStop() { s.update(func(c *Counters) { c.BackendStopCalls++ }) }
|
|
func (s *Stats) IncIdleTimeout() { s.update(func(c *Counters) { c.IdleTimeouts++ }) }
|
|
func (s *Stats) IncSignalReceived() { s.update(func(c *Counters) { c.SignalsReceived++ }) }
|
|
func (s *Stats) IncAccept() { s.update(func(c *Counters) { c.AcceptCalls++ }) }
|
|
func (s *Stats) IncAcceptError() { s.update(func(c *Counters) { c.AcceptErrors++ }) }
|
|
func (s *Stats) IncBackendDialAttempt() { s.update(func(c *Counters) { c.BackendDialAttempts++ }) }
|
|
func (s *Stats) IncBackendDialSuccess() { s.update(func(c *Counters) { c.BackendDialSuccess++ }) }
|
|
func (s *Stats) IncBackendDialError() { s.update(func(c *Counters) { c.BackendDialErrors++ }) }
|
|
func (s *Stats) IncUpstreamErrorReported() {
|
|
s.update(func(c *Counters) { c.UpstreamErrorsReported++ })
|
|
}
|
|
func (s *Stats) IncHandledConn() { s.update(func(c *Counters) { c.HandledConns++ }) }
|
|
func (s *Stats) AddBytesClientToBackend(n int64) {
|
|
s.update(func(c *Counters) { c.BytesClientToBackend += n })
|
|
}
|
|
func (s *Stats) AddBytesBackendToClient(n int64) {
|
|
s.update(func(c *Counters) { c.BytesBackendToClient += n })
|
|
}
|