* fix(proxy): fail fast with a MySQL error when the external upstream is unreachable A proxied-server command against a stopped external Dolt upstream stalled ~20-30s before failing with a bare "invalid connection". The local db proxy stays up, so on a refused backend dial (or a front such as stunnel/socat that accepts and closes because its own upstream is gone) it simply closed the client connection. The MySQL driver reports that as invalid connection / unexpected EOF, which the uow bootstrap ping (#6003) classifies as a transient drop and retries with exponential backoff for its whole 30s budget. Retrying cannot help: the proxy already proved the upstream at its own startup, and the outage is the upstream's. The proxy now answers such a connection itself with a MySQL ERR packet (2003, HY000) in place of the server greeting, naming the upstream and the cause, exactly as a real server refuses a connection before the handshake. The driver surfaces it as a *mysql.MySQLError, which the bootstrap retry already treats as permanent, so the command fails in ~0.1s with an actionable message. Only shapes that prove the upstream is not serving qualify: ECONNREFUSED/ENOENT/EHOSTUNREACH/ENETUNREACH on dial, or a backend that reaches EOF having sent zero bytes (a MySQL server always speaks first). A dial timeout or a drop after the greeting keeps the plain close and the client's transient retry. External backends only: a managed (local sidecar) backend is owned by the proxy and its dial can be refused transiently while bd drives it (the full proxied lane caught TestProxiedServerInitExplicitBeadsDirIsAuthoritative hitting exactly that mid-init when this applied to every backend), so managed backends keep the bare close and the client's bootstrap retry. TestExternalProxyOutageAndRestoreReDialsTCPAndUnix asserted the old silent close during an outage; it now asserts the ERR packet. With the latency fixed, TestProxiedOutageReconnectAcceptanceMatrix is renamed into the proxied discovery pattern as TestProxiedServerOutageReconnectAcceptanceMatrix and placed in the Bazel 30-shard block by gen_proxied_shard_manifest.py's incremental --write (shard 8). The frozen legacy 15-shard block is untouched; the test hash-distributes there like every other test added since it was frozen. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(proxy,uow): retry the proxy's upstream-outage report for ~1s; hedge the zero-byte case Review follow-up to the fail-fast change. A zero-byte close from the upstream is not proof it is down: Dolt closes queued connections without a byte when max_connections/back_log are full, when a queued connection times out, and while it shuts down for a restart; an L4 front does the same while its target is replaced. Its report now says "closed the connection before the MySQL greeting (down, restarting, or at its connection limit)" instead of "unreachable", and the comment no longer claims proof. Refusals and endpoint churn also lost #6003's retry entirely, so an external Dolt restart or a Service with no ready endpoints failed a command at once. The client now retries the proxy's report (proxy.IsUpstreamOutageError: 2003 with the "beads db proxy: " prefix) for upstreamOutageRetryWindow (1s) from the first time it sees it, with the backoff clamped so the last attempt lands at the window's end; other transient errors keep the 30s budget. A real outage still fails in about a second, inside the acceptance test's 3s bound. Also: the dial report drops Go call-site wrapping ("dial tcp H:P: connection refused"), truncation keeps UTF-8 intact, the comment states how the packet differs from a real server's pre-handshake ERR (keeps the '#'+SQLSTATE marker for vitess; 2003 is a client code chosen on purpose), a Stats counter for reported outages, and the isTransientPingError paragraph is reflowed and corrected. Tests: pingWithRetry retries the report inside the window and gives up at its end without using the full budget; openDB against a front that plays the proxy (its real packet) and comes back 300ms after the first connection succeeds in one call, and fails in about the window when it never does; NewProxyServer enables the report for external backends only; the dial message carries no Go type names; rune-safe truncation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(proxied): fail, not skip, socat-bridged subtests in the Bazel lane The external-unix front-door matrix subtest and both outage/reconnect matrix subtests bridge an external endpoint with socat and skipped when it was missing, so they could stop running without anyone noticing (the parent test still passes). requireSocat now fails them when BEADS_TEST_REQUIRE_SOCAT=1, which //cmd/bd:bd_proxied_test sets (its remote workers have socat; a recent main run's external-unix subtest passed there). The legacy GitHub 15-shard jobs have no socat and leave it unset, so the subtests keep skipping there. The check now runs before the upstream Dolt server is started, so a skip no longer pays for one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * ci(proxied): measured weight for the outage matrix; pin it to legacy shard 1 Give TestProxiedServerOutageReconnectAcceptanceMatrix a duration entry (80s: local runs measured 55-79s alone, 60-70s in its shard, and review 68-109s under load) and re-place it with the generator's incremental --write: it moves from 30/8 to 30/11, one line, nothing else moved. The heaviest estimated shard is now 435s against a ~357s median; a repack would even that out but moves most assignments, so it is left to a separate CI-only change that refreshes all durations together. Legacy 15-shard block: the test would hash onto shard 14, already ~13.5 of the 15 min timeout on main. It is pinned to shard 1 (~6.5 min in recent main runs) under the block's reviewed-pin rule; it skips there today (no socat), and the pin keeps shard 14 safe if that changes. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
161 lines
6.7 KiB
Go
161 lines
6.7 KiB
Go
package proxy
|
|
|
|
import (
|
|
"encoding/binary"
|
|
"errors"
|
|
"fmt"
|
|
"net"
|
|
"strings"
|
|
"syscall"
|
|
"time"
|
|
"unicode/utf8"
|
|
|
|
mysql "github.com/go-sql-driver/mysql"
|
|
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
|
|
)
|
|
|
|
// A client of the proxy only ever sees the proxy's own listener, which is up,
|
|
// so a plain close is all it learns when the external upstream behind it is
|
|
// not serving: the MySQL driver reports "invalid connection" / unexpected
|
|
// EOF, and the uow bootstrap ping classifies that as a transient drop and
|
|
// retries it for its whole 30s budget (#6003). For an upstream that is down
|
|
// that turns every bd command into a ~20-30s stall ending in an unhelpful
|
|
// message.
|
|
//
|
|
// Instead the proxy answers such a connection itself with a MySQL ERR packet
|
|
// in place of the server greeting, naming the upstream and what happened.
|
|
// The client (internal/storage/uow, via IsUpstreamOutageError) retries this
|
|
// error only for a short window (about a second), long enough to ride out an
|
|
// endpoint that is flapping or rebinding, short enough that a real outage
|
|
// fails within a couple of seconds instead of thirty. Two shapes qualify:
|
|
//
|
|
// - The dial was refused, the unix socket path is gone, or the host or
|
|
// network is unreachable. Nothing is accepting connections there right
|
|
// now.
|
|
// - The upstream accepted and then closed without sending a byte. A MySQL
|
|
// server speaks first, so it did not serve this connection. That is NOT
|
|
// proof it is down: Dolt does exactly this when max_connections and
|
|
// back_log are full, when a queued connection times out, and while it
|
|
// shuts down for a restart; an L4 front (socat, stunnel, a TCP load
|
|
// balancer, kubectl port-forward) does it when its own target is gone.
|
|
// The message says so, and the client's short retry window is what
|
|
// covers the transient cases.
|
|
//
|
|
// A dial timeout, a reset, or a connection dropped after the greeting keeps
|
|
// the plain close, and with it the client's full transient retry.
|
|
//
|
|
// External backends only. A managed backend (local Dolt sidecar) is owned by
|
|
// the proxy, and its dial can be refused transiently while bd is still
|
|
// driving it, which the client's full bootstrap retry absorbs; those keep the
|
|
// plain close.
|
|
//
|
|
// The packet is not byte-for-byte what a real server sends before the
|
|
// handshake. MySQL omits the '#'+SQLSTATE marker there (no capabilities are
|
|
// negotiated yet); this packet keeps it because dolthub/vitess's client
|
|
// always expects it, while go-sql-driver accepts either form. The mysql CLI
|
|
// will therefore show "#HY000" at the start of the message text. 2003
|
|
// (CR_CONN_HOST_ERROR, "Can't connect to MySQL server") is a client-side
|
|
// code that no real server sends; it is chosen because it describes the
|
|
// condition, and the "beads db proxy:" prefix is what marks it as ours.
|
|
|
|
const (
|
|
// UpstreamErrorNumber and UpstreamErrorPrefix identify the ERR packet
|
|
// the proxy sends for an unreachable external upstream; see
|
|
// IsUpstreamOutageError.
|
|
UpstreamErrorNumber = 2003
|
|
UpstreamErrorPrefix = "beads db proxy: "
|
|
|
|
upstreamErrorSQLState = "HY000"
|
|
upstreamErrorMaxMessage = 1024
|
|
upstreamErrorWriteTimeout = time.Second
|
|
)
|
|
|
|
// IsUpstreamOutageError reports whether err is the proxy's own report that
|
|
// its external upstream is unreachable (see the comment above), as opposed to
|
|
// an error from the Dolt server itself.
|
|
func IsUpstreamOutageError(err error) bool {
|
|
var me *mysql.MySQLError
|
|
return errors.As(err, &me) && me.Number == UpstreamErrorNumber &&
|
|
strings.HasPrefix(me.Message, UpstreamErrorPrefix)
|
|
}
|
|
|
|
// reportsUpstreamOutage reports whether the proxy should answer an
|
|
// unreachable upstream with a MySQL error for this backend.
|
|
func reportsUpstreamOutage(s server.DatabaseServer) bool {
|
|
_, external := s.(*server.ExternalDoltServer)
|
|
return external
|
|
}
|
|
|
|
// isUpstreamUnreachableDialError reports whether a backend dial failure means
|
|
// nothing is accepting connections at the upstream right now, as opposed to
|
|
// slow (timeout) or a local resource problem.
|
|
func isUpstreamUnreachableDialError(err error) bool {
|
|
return errors.Is(err, syscall.ECONNREFUSED) ||
|
|
errors.Is(err, syscall.ENOENT) ||
|
|
errors.Is(err, syscall.EHOSTUNREACH) ||
|
|
errors.Is(err, syscall.ENETUNREACH)
|
|
}
|
|
|
|
// dialFailureMessage describes a refused/unreachable dial as
|
|
// "dial <network> <address>: <reason>", without the Go call-site wrapping the
|
|
// error carries.
|
|
func dialFailureMessage(err error) string {
|
|
var opErr *net.OpError
|
|
var errno syscall.Errno
|
|
if errors.As(err, &opErr) && opErr.Addr != nil && errors.As(err, &errno) {
|
|
return fmt.Sprintf("upstream Dolt server unreachable: %s %s %s: %v", opErr.Op, opErr.Net, opErr.Addr, errno)
|
|
}
|
|
return "upstream Dolt server unreachable: " + err.Error()
|
|
}
|
|
|
|
// closedBeforeGreetingMessage describes a backend connection that reached EOF
|
|
// before the server greeting, naming the endpoint the proxy dialed.
|
|
func closedBeforeGreetingMessage(backend net.Conn) string {
|
|
where := ""
|
|
if ra := backend.RemoteAddr(); ra != nil && ra.String() != "" {
|
|
where = fmt.Sprintf(" at %s %s", ra.Network(), ra)
|
|
}
|
|
return "upstream Dolt server" + where + " closed the connection before the MySQL greeting " +
|
|
"(down, restarting, or at its connection limit)"
|
|
}
|
|
|
|
// upstreamErrorPacket encodes a MySQL ERR packet (sequence id 0, with the
|
|
// '#'+SQLSTATE marker) carrying msg, truncated on a UTF-8 boundary.
|
|
func upstreamErrorPacket(msg string) []byte {
|
|
if len(msg) > upstreamErrorMaxMessage {
|
|
cut := upstreamErrorMaxMessage
|
|
for cut > 0 && !utf8.RuneStart(msg[cut]) {
|
|
cut--
|
|
}
|
|
msg = msg[:cut]
|
|
}
|
|
payload := make([]byte, 0, 9+len(msg))
|
|
payload = append(payload, 0xff)
|
|
payload = binary.LittleEndian.AppendUint16(payload, UpstreamErrorNumber)
|
|
payload = append(payload, '#')
|
|
payload = append(payload, upstreamErrorSQLState...)
|
|
payload = append(payload, msg...)
|
|
pkt := make([]byte, 4, 4+len(payload))
|
|
pkt[0] = byte(len(payload))
|
|
pkt[1] = byte(len(payload) >> 8)
|
|
pkt[2] = byte(len(payload) >> 16)
|
|
pkt[3] = 0 // sequence id: the server's first packet
|
|
return append(pkt, payload...)
|
|
}
|
|
|
|
// UpstreamOutagePacket returns the ERR packet the proxy sends for an
|
|
// unreachable external upstream, with detail as the message after
|
|
// UpstreamErrorPrefix. Exported so client-side tests can play the proxy.
|
|
func UpstreamOutagePacket(detail string) []byte {
|
|
return upstreamErrorPacket(UpstreamErrorPrefix + detail)
|
|
}
|
|
|
|
// writeUpstreamOutage sends the client the proxy's upstream-outage ERR
|
|
// packet. Best effort: the caller closes the connection either way.
|
|
func (p *proxyServer) writeUpstreamOutage(client net.Conn, detail string) {
|
|
p.stats.IncUpstreamErrorReported()
|
|
p.tracef("handleConn(%s) reporting upstream outage: %s", client.RemoteAddr(), detail)
|
|
_ = client.SetWriteDeadline(time.Now().Add(upstreamErrorWriteTimeout))
|
|
_, _ = client.Write(UpstreamOutagePacket(detail))
|
|
}
|