beads/internal/storage/dbproxy/proxy/upstream_error.go
Julian Knutsen 683993660f
fix(proxy): fail fast with a clear MySQL error when an external upstream is unreachable (#7181)
* fix(proxy): fail fast with a MySQL error when the external upstream is unreachable

A proxied-server command against a stopped external Dolt upstream stalled
~20-30s before failing with a bare "invalid connection". The local db proxy
stays up, so on a refused backend dial (or a front such as stunnel/socat
that accepts and closes because its own upstream is gone) it simply closed
the client connection. The MySQL driver reports that as invalid
connection / unexpected EOF, which the uow bootstrap ping (#6003) classifies
as a transient drop and retries with exponential backoff for its whole 30s
budget. Retrying cannot help: the proxy already proved the upstream at its
own startup, and the outage is the upstream's.

The proxy now answers such a connection itself with a MySQL ERR packet
(2003, HY000) in place of the server greeting, naming the upstream and the
cause, exactly as a real server refuses a connection before the handshake.
The driver surfaces it as a *mysql.MySQLError, which the bootstrap retry
already treats as permanent, so the command fails in ~0.1s with an
actionable message. Only shapes that prove the upstream is not serving
qualify: ECONNREFUSED/ENOENT/EHOSTUNREACH/ENETUNREACH on dial, or a backend
that reaches EOF having sent zero bytes (a MySQL server always speaks
first). A dial timeout or a drop after the greeting keeps the plain close
and the client's transient retry.

External backends only: a managed (local sidecar) backend is owned by the
proxy and its dial can be refused transiently while bd drives it (the
full proxied lane caught TestProxiedServerInitExplicitBeadsDirIsAuthoritative
hitting exactly that mid-init when this applied to every backend), so
managed backends keep the bare close and the client's bootstrap retry.
TestExternalProxyOutageAndRestoreReDialsTCPAndUnix asserted the old
silent close during an outage; it now asserts the ERR packet.

With the latency fixed, TestProxiedOutageReconnectAcceptanceMatrix is
renamed into the proxied discovery pattern as
TestProxiedServerOutageReconnectAcceptanceMatrix and placed in the Bazel
30-shard block by gen_proxied_shard_manifest.py's incremental --write
(shard 8). The frozen legacy 15-shard block is untouched; the test
hash-distributes there like every other test added since it was frozen.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(proxy,uow): retry the proxy's upstream-outage report for ~1s; hedge the zero-byte case

Review follow-up to the fail-fast change.

A zero-byte close from the upstream is not proof it is down: Dolt closes
queued connections without a byte when max_connections/back_log are full,
when a queued connection times out, and while it shuts down for a
restart; an L4 front does the same while its target is replaced. Its
report now says "closed the connection before the MySQL greeting (down,
restarting, or at its connection limit)" instead of "unreachable", and
the comment no longer claims proof.

Refusals and endpoint churn also lost #6003's retry entirely, so an
external Dolt restart or a Service with no ready endpoints failed a
command at once. The client now retries the proxy's report
(proxy.IsUpstreamOutageError: 2003 with the "beads db proxy: " prefix)
for upstreamOutageRetryWindow (1s) from the first time it sees it, with
the backoff clamped so the last attempt lands at the window's end; other
transient errors keep the 30s budget. A real outage still fails in about
a second, inside the acceptance test's 3s bound.

Also: the dial report drops Go call-site wrapping ("dial tcp H:P:
connection refused"), truncation keeps UTF-8 intact, the comment states
how the packet differs from a real server's pre-handshake ERR (keeps the
'#'+SQLSTATE marker for vitess; 2003 is a client code chosen on purpose),
a Stats counter for reported outages, and the isTransientPingError
paragraph is reflowed and corrected.

Tests: pingWithRetry retries the report inside the window and gives up
at its end without using the full budget; openDB against a front that
plays the proxy (its real packet) and comes back 300ms after the first
connection succeeds in one call, and fails in about the window when it
never does; NewProxyServer enables the report for external backends
only; the dial message carries no Go type names; rune-safe truncation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(proxied): fail, not skip, socat-bridged subtests in the Bazel lane

The external-unix front-door matrix subtest and both outage/reconnect
matrix subtests bridge an external endpoint with socat and skipped when
it was missing, so they could stop running without anyone noticing (the
parent test still passes). requireSocat now fails them when
BEADS_TEST_REQUIRE_SOCAT=1, which //cmd/bd:bd_proxied_test sets (its
remote workers have socat; a recent main run's external-unix subtest
passed there). The legacy GitHub 15-shard jobs have no socat and leave
it unset, so the subtests keep skipping there. The check now runs before
the upstream Dolt server is started, so a skip no longer pays for one.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* ci(proxied): measured weight for the outage matrix; pin it to legacy shard 1

Give TestProxiedServerOutageReconnectAcceptanceMatrix a duration entry
(80s: local runs measured 55-79s alone, 60-70s in its shard, and review
68-109s under load) and re-place it with the generator's incremental
--write: it moves from 30/8 to 30/11, one line, nothing else moved. The
heaviest estimated shard is now 435s against a ~357s median; a repack
would even that out but moves most assignments, so it is left to a
separate CI-only change that refreshes all durations together.

Legacy 15-shard block: the test would hash onto shard 14, already ~13.5
of the 15 min timeout on main. It is pinned to shard 1 (~6.5 min in
recent main runs) under the block's reviewed-pin rule; it skips there
today (no socat), and the pin keeps shard 14 safe if that changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 02:37:58 -07:00

161 lines
6.7 KiB
Go

package proxy
import (
"encoding/binary"
"errors"
"fmt"
"net"
"strings"
"syscall"
"time"
"unicode/utf8"
mysql "github.com/go-sql-driver/mysql"
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
)
// A client of the proxy only ever sees the proxy's own listener, which is up,
// so a plain close is all it learns when the external upstream behind it is
// not serving: the MySQL driver reports "invalid connection" / unexpected
// EOF, and the uow bootstrap ping classifies that as a transient drop and
// retries it for its whole 30s budget (#6003). For an upstream that is down
// that turns every bd command into a ~20-30s stall ending in an unhelpful
// message.
//
// Instead the proxy answers such a connection itself with a MySQL ERR packet
// in place of the server greeting, naming the upstream and what happened.
// The client (internal/storage/uow, via IsUpstreamOutageError) retries this
// error only for a short window (about a second), long enough to ride out an
// endpoint that is flapping or rebinding, short enough that a real outage
// fails within a couple of seconds instead of thirty. Two shapes qualify:
//
// - The dial was refused, the unix socket path is gone, or the host or
// network is unreachable. Nothing is accepting connections there right
// now.
// - The upstream accepted and then closed without sending a byte. A MySQL
// server speaks first, so it did not serve this connection. That is NOT
// proof it is down: Dolt does exactly this when max_connections and
// back_log are full, when a queued connection times out, and while it
// shuts down for a restart; an L4 front (socat, stunnel, a TCP load
// balancer, kubectl port-forward) does it when its own target is gone.
// The message says so, and the client's short retry window is what
// covers the transient cases.
//
// A dial timeout, a reset, or a connection dropped after the greeting keeps
// the plain close, and with it the client's full transient retry.
//
// External backends only. A managed backend (local Dolt sidecar) is owned by
// the proxy, and its dial can be refused transiently while bd is still
// driving it, which the client's full bootstrap retry absorbs; those keep the
// plain close.
//
// The packet is not byte-for-byte what a real server sends before the
// handshake. MySQL omits the '#'+SQLSTATE marker there (no capabilities are
// negotiated yet); this packet keeps it because dolthub/vitess's client
// always expects it, while go-sql-driver accepts either form. The mysql CLI
// will therefore show "#HY000" at the start of the message text. 2003
// (CR_CONN_HOST_ERROR, "Can't connect to MySQL server") is a client-side
// code that no real server sends; it is chosen because it describes the
// condition, and the "beads db proxy:" prefix is what marks it as ours.
const (
// UpstreamErrorNumber and UpstreamErrorPrefix identify the ERR packet
// the proxy sends for an unreachable external upstream; see
// IsUpstreamOutageError.
UpstreamErrorNumber = 2003
UpstreamErrorPrefix = "beads db proxy: "
upstreamErrorSQLState = "HY000"
upstreamErrorMaxMessage = 1024
upstreamErrorWriteTimeout = time.Second
)
// IsUpstreamOutageError reports whether err is the proxy's own report that
// its external upstream is unreachable (see the comment above), as opposed to
// an error from the Dolt server itself.
func IsUpstreamOutageError(err error) bool {
var me *mysql.MySQLError
return errors.As(err, &me) && me.Number == UpstreamErrorNumber &&
strings.HasPrefix(me.Message, UpstreamErrorPrefix)
}
// reportsUpstreamOutage reports whether the proxy should answer an
// unreachable upstream with a MySQL error for this backend.
func reportsUpstreamOutage(s server.DatabaseServer) bool {
_, external := s.(*server.ExternalDoltServer)
return external
}
// isUpstreamUnreachableDialError reports whether a backend dial failure means
// nothing is accepting connections at the upstream right now, as opposed to
// slow (timeout) or a local resource problem.
func isUpstreamUnreachableDialError(err error) bool {
return errors.Is(err, syscall.ECONNREFUSED) ||
errors.Is(err, syscall.ENOENT) ||
errors.Is(err, syscall.EHOSTUNREACH) ||
errors.Is(err, syscall.ENETUNREACH)
}
// dialFailureMessage describes a refused/unreachable dial as
// "dial <network> <address>: <reason>", without the Go call-site wrapping the
// error carries.
func dialFailureMessage(err error) string {
var opErr *net.OpError
var errno syscall.Errno
if errors.As(err, &opErr) && opErr.Addr != nil && errors.As(err, &errno) {
return fmt.Sprintf("upstream Dolt server unreachable: %s %s %s: %v", opErr.Op, opErr.Net, opErr.Addr, errno)
}
return "upstream Dolt server unreachable: " + err.Error()
}
// closedBeforeGreetingMessage describes a backend connection that reached EOF
// before the server greeting, naming the endpoint the proxy dialed.
func closedBeforeGreetingMessage(backend net.Conn) string {
where := ""
if ra := backend.RemoteAddr(); ra != nil && ra.String() != "" {
where = fmt.Sprintf(" at %s %s", ra.Network(), ra)
}
return "upstream Dolt server" + where + " closed the connection before the MySQL greeting " +
"(down, restarting, or at its connection limit)"
}
// upstreamErrorPacket encodes a MySQL ERR packet (sequence id 0, with the
// '#'+SQLSTATE marker) carrying msg, truncated on a UTF-8 boundary.
func upstreamErrorPacket(msg string) []byte {
if len(msg) > upstreamErrorMaxMessage {
cut := upstreamErrorMaxMessage
for cut > 0 && !utf8.RuneStart(msg[cut]) {
cut--
}
msg = msg[:cut]
}
payload := make([]byte, 0, 9+len(msg))
payload = append(payload, 0xff)
payload = binary.LittleEndian.AppendUint16(payload, UpstreamErrorNumber)
payload = append(payload, '#')
payload = append(payload, upstreamErrorSQLState...)
payload = append(payload, msg...)
pkt := make([]byte, 4, 4+len(payload))
pkt[0] = byte(len(payload))
pkt[1] = byte(len(payload) >> 8)
pkt[2] = byte(len(payload) >> 16)
pkt[3] = 0 // sequence id: the server's first packet
return append(pkt, payload...)
}
// UpstreamOutagePacket returns the ERR packet the proxy sends for an
// unreachable external upstream, with detail as the message after
// UpstreamErrorPrefix. Exported so client-side tests can play the proxy.
func UpstreamOutagePacket(detail string) []byte {
return upstreamErrorPacket(UpstreamErrorPrefix + detail)
}
// writeUpstreamOutage sends the client the proxy's upstream-outage ERR
// packet. Best effort: the caller closes the connection either way.
func (p *proxyServer) writeUpstreamOutage(client net.Conn, detail string) {
p.stats.IncUpstreamErrorReported()
p.tracef("handleConn(%s) reporting upstream outage: %s", client.RemoteAddr(), detail)
_ = client.SetWriteDeadline(time.Now().Add(upstreamErrorWriteTimeout))
_, _ = client.Write(UpstreamOutagePacket(detail))
}