* fix(dbproxy): prove the managed dolt owns its port; move a taken Beads-chosen port
The managed-local backend's dolt port comes from a bind-and-close
(proxy.PickFreePort) when bd writes the generated config. The dolt
sql-server binds it only later, after the proxy child has run dolt
config and init and booted the server. If another process takes the port
in that gap, dolt logs "Port N already in use." and exits. But
DoltServer.waitReady only dialed the port, so it accepted the other
process's listener as ready. The proxy then published proxy.pid, its
backend health check saw dolt gone, and it shut down. bd init's first
ping got "unexpected EOF" and every retry after that got "connection
refused". Under parallel tests this hit any test that starts its own
managed backend.
Readiness now requires proof that the process Start launched owns the
port:
- DoltServer sends the child's stdout/stderr through a watcher that
forwards to the server log. A dial counts only after dolt's own
"Server ready. Accepting connections." line has appeared on that
output. That line is info level, and every generated config uses
info. A config with a quieter log_level has no such signal and keeps
the dial-only check.
- A child that exits first is reported as the new server.ErrPortInUse
when its output said its port was taken.
Recovery: DoltServer.SetPortRepicker lets Start move the config to a
fresh port, re-read it, and relaunch, up to 5 ports. dolt config/init
run only on the first attempt. bd's repicker,
repickManagedProxiedServerPort, needs to tell a port Beads chose from
one an operator set, so generated configs now record the chosen port
on a "# beads-chosen-listener-port: N" line after the managed marker.
The repicker moves the port only when the file is Beads-generated,
carries that record, and its listener.port is still the recorded port.
Otherwise the port is treated as pinned and Start fails with
ErrPortInUse. That includes configs generated before this change. It
rewrites only listener.port and the record, through a yaml.Node so the
rest of the file is kept, and writes the file atomically.
db-proxy-child installs it for the local backend.
The proxy's waitForServerReady also re-checks Running after its dial.
Start has already proved ownership, and a running dolt holds its
listener, so only a backend that exited between the two checks could
let a dial reach another process.
Tests:
- server: a held port is not accepted as ready (ErrPortInUse; this
test fails on main), a repicker brings the server up on a new port
that the pid record and DSN both use, and a declining repicker ends
Start with ErrPortInUse.
- cmd/bd: the generated config records its port, a Beads-chosen port
moves while hand edits survive, and a hand-set port, a config
without the record, an operator config, and a stale in-use port are
all refused with the file left untouched.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(dbproxy): move a taken managed port via a runtime config, never config.yaml
This reworks cd00c5acb after review. That commit recovered from a taken
Beads-chosen port by rewriting the managed config.yaml. That broke the
file's own contract ("Never rewritten afterward; hand edits are
preserved", #4986), and #5024 had deferred that contract change to an
RFC.
dolt sql-server has no command-line way around the file: with --config,
-P/--port is silently ignored (checked against dolt 2.1.8 and 2.2.0, in
both argument orders). So on ErrPortInUse the proxy child now writes
proxy-child.runtime-config.yaml (server.RuntimeConfigFileName) in the
server root. That file is the operator config with only listener.port
moved to a free port. The child relaunches dolt from it and records the
effective port where runtime readers already look: the backend pid
record and the DoltServer's dial/DSN config. The runtime config is
removed when the server stops, when Start fails, and by the next Start.
config.yaml is never written.
- server.PortRepicker becomes server.PortConflictPolicy. It only decides
whether the port may move; the server does the move.
- bd's policy (managedPortConflictPolicy) allows a move when the file
carries the managed marker and either:
- has the port record and listener.port still equals it, or
- is byte-identical to the config bd generated before the record
existed, so existing unedited workspaces recover too.
- Everything else is pinned: operator configs, hand-set ports,
hand-edited legacy files, and configs with ${...} placeholders (a
runtime copy would put the expanded values on disk).
- The record line now says plainly that Beads runs on a free port
without editing the file, and that deleting the line or changing
listener.port pins the port. The record parser now requires the
digits to end at a space or line end and to be in 1..65535.
- ErrPortInUse messages say what to do: free port P, set listener.port
in <path> to a free port, or (for a generated file) delete it so
Beads regenerates it. The give-up message names the last port tried.
- A dolt that never logs the ready line is still not accepted. A
grace-period fallback would let a foreign listener through while dolt
is still starting. The timeout error now names the missing line, says
whether anything answered, and points at the escape hatch: at
log_level warning, readiness is dial-only.
- Port-conflict detection matches go-mysql-server's "Port host:port
already in use." and the Windows bind error, and requires the SQL port
in the kernel's bind error, so a metrics or MCP listener failure is
not taken for the SQL port.
- Spawn error paths reap with cmd.Wait(). cmd.WaitDelay is set, and a
comment notes that the backend's output is now a pipe into the proxy
child.
Tests:
- Output-watcher table tests: split, one-byte and CRLF writes, output
past the 4 KiB tail, the MCP ready line, port boundaries, the
go-mysql-server and Windows forms, another listener's bind error, the
unix-socket warning, and nil log.
- A real dolt at log_level warning starts through dial-only readiness.
- With a stand-in dolt (the test binary in fake mode, wired in
TestMain): Start gives up after maxStartPortAttempts, a missing ready
line is diagnosed, plus a positive control.
- With real dolt: a move keeps config.yaml byte-identical, runs from the
runtime config, and the runtime config is removed on Stop. A declining
policy and the no-policy remedy are covered.
- The bd policy's allow and pin cases.
- The managed-local lifecycle test's config/pid port check accepts the
runtime config.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(dbproxy): ask the port policy once, about the operator's port; copy raw config text
Re-review of the runtime-config rework found a regression. After the
first move, Start asked the PortConflictPolicy about the runtime port.
bd's policy re-reads config.yaml, whose listener.port is still the
operator's port, so it refused with "listener.port is A, not the in-use
port B". The "up to 5 ports" retry therefore never ran in production;
the fake-dolt test missed it because its policy always allowed.
Start now asks the policy once per Start, on the first conflict, about
the operator config's port. Ports Start picked itself are its own to move
again. Start also re-reads the operator config up front, so every Start
begins from that config's port.
The runtime config is now built from config.yaml's raw text
(servercfg.NewYamlConfig) instead of the env-interpolated
YamlConfigFromFile result. dolt interpolates the runtime file when it
reads it, so the old way expanded placeholders twice: "$$" escapes lost a
level and placeholder values landed on disk. The server reads the
runtime file back with interpolation for its own Dial/DSN view. Since
placeholders no longer expand on disk, bd's policy stops refusing
configs that contain them.
Tests:
- bd's real policy driving a real DoltServer with a stand-in dolt that
reports every port taken: Start makes five launches on five distinct
ports, starting with config.yaml's, then gives up. It is not refused
on the second move, and config.yaml stays unchanged.
- A frozen byte-for-byte literal of the config bd v1.3.0/v1.3.1
generates. The legacy render must equal it, and the policy must find
it eligible.
- The runtime config keeps placeholders and "$$" verbatim and is mode
0600, while the server dials the interpolated host.
- The give-up test asserts the policy is asked once, about the
operator's port.
- The missing-ready-line test timeout is raised to 30s for the slow
fake-dolt exec under race.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(dbproxy): classify a child that exits before its identity is captured
TestManagedPortConflictPolicy_ConsecutiveCollisions failed on the farm
with "capture child birth identity: procid: process is no longer
running", after 2 launches instead of 5. Its stand-in dolt reports
"Port N already in use." and exits immediately, and it exited before
Start captured its birth identity. Start treated that capture failure as
fatal, so an ErrPortInUse that Start could recover from became a hard
error. A real dolt that finds its port taken can exit just as fast, so
this race exists in the product, not only in the test.
Every spawn step before the child is handed to its supervising
goroutine (identity capture, root identity, pid record) now fails
through failSpawn. failSpawn kills and reaps the child; reaping also
drains its output into the watcher. The child's own exit then decides
the error:
- the child reported its port taken: ErrPortInUse, so the normal policy
and retry path runs;
- the child exited by itself for another reason: "exited (status N)
before startup completed";
- the child was alive and we killed it: the step's own error, as
before.
On Windows, a killed child reports an ordinary exit code, so there an
exit cannot be told from a kill. Only the port-in-use classification
applies there.
procid.Capture is reached through a package var so tests can make the
capture lose the race deterministically.
Tests, using a shell-script stand-in dolt that starts in milliseconds:
- a fast port-in-use exit that beats the capture is ErrPortInUse, and
with a policy it is retried up to the 5-port budget;
- a fast exit for another reason reports its status;
- a capture failure while the child is alive stays fatal and kills the
child;
- ten immediate port-in-use exits with the real capture are all
ErrPortInUse.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
553 lines
17 KiB
Go
553 lines
17 KiB
Go
package proxy
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
"log"
|
|
"net"
|
|
"os"
|
|
"os/signal"
|
|
"path/filepath"
|
|
"sync"
|
|
"sync/atomic"
|
|
"syscall"
|
|
"time"
|
|
|
|
"github.com/cenkalti/backoff/v4"
|
|
"golang.org/x/sync/errgroup"
|
|
|
|
"github.com/steveyegge/beads/internal/lockfile"
|
|
"github.com/steveyegge/beads/internal/procid"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/identity"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/pidfile"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/util"
|
|
)
|
|
|
|
const IdleTimeoutNever time.Duration = -1
|
|
|
|
type ProxyOpts struct {
|
|
RootDir string
|
|
Port int
|
|
IdleTimeout time.Duration
|
|
Server server.DatabaseServer
|
|
// StopEpoch is the proxy stop epoch the spawning parent observed under
|
|
// proxy.lock before forking this proxy. ListenAndServe re-reads the
|
|
// epoch immediately before publishing proxy.pid and aborts if it has
|
|
// advanced, so a slow backend start cannot outlast a concurrent
|
|
// `bd dolt stop` and publish a running proxy after the stop returned.
|
|
// Empty means "no stop had ever run", which readStopEpoch also reports
|
|
// for a missing epoch file, so the zero value stays correct for direct
|
|
// (non-forked) callers such as tests.
|
|
StopEpoch string
|
|
// Stats is optional. When non-nil, the proxy records per-event counters
|
|
// against it; tests use Snapshot() to assert. Production code should
|
|
// leave this nil.
|
|
Stats *Stats
|
|
}
|
|
|
|
type proxyServer struct {
|
|
rootDir string
|
|
port int
|
|
idleTimeout time.Duration
|
|
server server.DatabaseServer
|
|
stats *Stats
|
|
stopEpoch string
|
|
|
|
// reportUpstreamOutage answers a connection whose upstream is not
|
|
// serving with a MySQL error instead of a bare close; see
|
|
// upstream_error.go. Set for external backends only.
|
|
reportUpstreamOutage bool
|
|
|
|
logger *log.Logger
|
|
listener net.Listener
|
|
activeConns atomic.Int64
|
|
conns errgroup.Group
|
|
}
|
|
|
|
const (
|
|
PIDFileName = "proxy.pid"
|
|
LogFileName = "proxy.log"
|
|
LockFileName = "proxy.lock"
|
|
)
|
|
|
|
// LockHeldExitCode is the exit code a child proxy should use when
|
|
// ListenAndServe returns ErrLockHeld. The spawning parent treats this
|
|
// (EX_TEMPFAIL) as "lost the spawn race" and retries via readAndDial.
|
|
const LockHeldExitCode = 75
|
|
|
|
// ErrLockHeld is returned from ListenAndServe when another proxy already
|
|
// holds proxy.lock for the same rootDir. It is a normal "lost the race"
|
|
// outcome, not a failure: callers spawned as children should map it to
|
|
// LockHeldExitCode and exit cleanly.
|
|
var ErrLockHeld = errors.New("proxy lock held by another proxy on this rootDir")
|
|
|
|
const (
|
|
serverReadyTimeout = 30 * time.Second
|
|
readyDialTimeout = 2 * time.Second
|
|
readyInitialBackoff = 50 * time.Millisecond
|
|
readyMaxBackoff = 1 * time.Second
|
|
backendDialTimeout = 5 * time.Second
|
|
idleWatcherMinInterval = 1 * time.Second
|
|
backendStopTimeout = 5 * time.Minute
|
|
tcpKeepAlivePeriod = 30 * time.Second
|
|
backendHealthInterval = 100 * time.Millisecond
|
|
)
|
|
|
|
var errIdleTimeout = errors.New("idle timeout reached")
|
|
|
|
func NewProxyServer(opts ProxyOpts) *proxyServer {
|
|
return &proxyServer{
|
|
rootDir: opts.RootDir,
|
|
port: opts.Port,
|
|
idleTimeout: opts.IdleTimeout,
|
|
server: opts.Server,
|
|
stats: opts.Stats,
|
|
stopEpoch: opts.StopEpoch,
|
|
|
|
reportUpstreamOutage: reportsUpstreamOutage(opts.Server),
|
|
}
|
|
}
|
|
|
|
func (p *proxyServer) tracef(format string, args ...any) {
|
|
p.logger.Printf(format, args...)
|
|
}
|
|
|
|
func (p *proxyServer) ListenAndServe(parentCtx context.Context) error {
|
|
lock, err := util.TryLock(filepath.Join(p.rootDir, LockFileName))
|
|
if err != nil {
|
|
if lockfile.IsLocked(err) {
|
|
return ErrLockHeld
|
|
}
|
|
return fmt.Errorf("acquire %s: %w", LockFileName, err)
|
|
}
|
|
// proxy.lock is held for the proxy's whole lifetime, but a doomed start
|
|
// must be able to release it early (before its backend teardown) without
|
|
// the deferred release double-unlocking. Only the main goroutine touches
|
|
// this.
|
|
lockHeld := true
|
|
releaseLock := func() {
|
|
if lockHeld {
|
|
lockHeld = false
|
|
lock.Unlock()
|
|
}
|
|
}
|
|
defer releaseLock()
|
|
if err := clearSpawnMarkerAfterLock(p.rootDir); err != nil {
|
|
return fmt.Errorf("clear proxy spawn marker: %w", err)
|
|
}
|
|
|
|
// Fast-abort: a concurrent `bd dolt stop` advances the stop epoch before
|
|
// waiting (briefly) for proxy.lock, so an epoch that moved between the
|
|
// spawning parent's read and this child taking the lock dooms this start.
|
|
// Abort before opening any listener or booting the backend: the stopper
|
|
// then observes a free lock within milliseconds instead of after a full
|
|
// doomed boot-and-teardown cycle.
|
|
if changed, err := stopEpochChanged(p.rootDir, p.stopEpoch); err != nil {
|
|
return fmt.Errorf("check proxy stop epoch after acquiring %s: %w", LockFileName, err)
|
|
} else if changed {
|
|
return fmt.Errorf("%w for %s: stop epoch advanced before startup", errStartInterrupted, p.rootDir)
|
|
}
|
|
|
|
logPath := filepath.Join(p.rootDir, LogFileName)
|
|
f, err := os.OpenFile(logPath, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0o600) // #nosec G304 -- logPath is derived from operator-supplied config, not untrusted request input
|
|
if err != nil {
|
|
return fmt.Errorf("open proxy log %q: %w", logPath, err)
|
|
}
|
|
p.logger = log.New(f, "[proxy] ", log.LstdFlags|log.Lmicroseconds)
|
|
defer func() { _ = f.Close() }()
|
|
|
|
ctx, cancel := context.WithCancel(parentCtx)
|
|
defer cancel()
|
|
|
|
// Install signal handlers BEFORE Listen. Without this, Go's default
|
|
// SIGTERM action terminates the process during the startup window
|
|
// (Listen, pidfile write, backend Start, readiness wait), bypassing all
|
|
// deferred cleanup including RemoveDatabaseProxyPidFile.
|
|
sigCh := make(chan os.Signal, 1)
|
|
signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
|
|
defer signal.Stop(sigCh)
|
|
|
|
var sigReceived atomic.Bool
|
|
go func() {
|
|
select {
|
|
case <-ctx.Done():
|
|
case <-sigCh:
|
|
sigReceived.Store(true)
|
|
p.stats.IncSignalReceived()
|
|
cancel()
|
|
}
|
|
}()
|
|
|
|
// Watch the stop epoch across the whole startup window (listen, backend
|
|
// Start, readiness wait). A stop that begins mid-boot advances the epoch
|
|
// first and then waits only shutdownConfirmDeadline for proxy.lock, which
|
|
// this child holds throughout the boot; without a watcher the child would
|
|
// notice the doomed start only at the pre-publish fence, potentially tens
|
|
// of seconds later. Canceling ctx aborts the backend Start / ready wait
|
|
// within about one poll interval. Transient epoch read errors are ignored
|
|
// here; the pre-publish fence remains the authoritative check.
|
|
epochWatchCtx, epochWatchCancel := context.WithCancel(context.Background())
|
|
epochWatchDone := make(chan struct{})
|
|
go func() {
|
|
defer close(epochWatchDone)
|
|
ticker := time.NewTicker(openPollInterval)
|
|
defer ticker.Stop()
|
|
for {
|
|
select {
|
|
case <-epochWatchCtx.Done():
|
|
return
|
|
case <-ticker.C:
|
|
changed, err := stopEpochChanged(p.rootDir, p.stopEpoch)
|
|
if err != nil {
|
|
continue
|
|
}
|
|
if changed {
|
|
cancel()
|
|
return
|
|
}
|
|
}
|
|
}
|
|
}()
|
|
stopEpochWatch := func() {
|
|
epochWatchCancel()
|
|
<-epochWatchDone
|
|
}
|
|
defer stopEpochWatch()
|
|
|
|
// abortInterruptedStart tears down a doomed start whose stop epoch
|
|
// advanced mid-boot. The interrupting stop is polling proxy.lock under a
|
|
// budget (shutdownConfirmDeadline) far smaller than a backend stop can
|
|
// take, and nothing has been published, so release the lock BEFORE the
|
|
// backend teardown instead of starving the stopper into its timeout.
|
|
abortInterruptedStart := func() error {
|
|
p.stats.IncBackendStop()
|
|
releaseLock()
|
|
_ = stopBackendBounded(p.server)
|
|
return fmt.Errorf("%w for %s: stop epoch advanced during startup", errStartInterrupted, p.rootDir)
|
|
}
|
|
|
|
addr := fmt.Sprintf("127.0.0.1:%d", p.port)
|
|
|
|
ln, err := net.Listen("tcp", addr)
|
|
if err != nil {
|
|
return fmt.Errorf("listen on %s: %w", addr, err)
|
|
}
|
|
|
|
p.listener = ln
|
|
defer func() { _ = ln.Close() }()
|
|
p.stats.IncListenAndServe()
|
|
dataPort, ok := ln.Addr().(*net.TCPAddr)
|
|
if !ok {
|
|
return fmt.Errorf("proxy: unexpected data listener address %T", ln.Addr())
|
|
}
|
|
|
|
if _, err := identity.WriteSecret(p.rootDir); err != nil {
|
|
return fmt.Errorf("write proxy secret: %w", err)
|
|
}
|
|
|
|
var identMu sync.RWMutex
|
|
identReply := identity.IdentReply{
|
|
Schema: pidfile.SchemaV2,
|
|
Role: pidfile.KindProxy,
|
|
DataPort: dataPort.Port,
|
|
}
|
|
control, err := startControl(p.rootDir, func() identity.IdentReply {
|
|
identMu.RLock()
|
|
defer identMu.RUnlock()
|
|
return identReply
|
|
})
|
|
if err != nil {
|
|
return fmt.Errorf("start control listener: %w", err)
|
|
}
|
|
defer func() { _ = control.Close() }()
|
|
|
|
p.stats.IncBackendStart()
|
|
if err := p.server.Start(ctx); err != nil {
|
|
// Start failed with no backend left running (Start cleans up its own
|
|
// failure), so there is no teardown to move off the lock; classifying
|
|
// the epoch-watcher cancellation just keeps the child's exit reason
|
|
// precise for the spawning parent.
|
|
if changed, cerr := stopEpochChanged(p.rootDir, p.stopEpoch); cerr == nil && changed {
|
|
return fmt.Errorf("%w for %s: stop epoch advanced during backend start (%v)", errStartInterrupted, p.rootDir, err)
|
|
}
|
|
return fmt.Errorf("start database server: %w", err)
|
|
}
|
|
|
|
if err := waitForServerReady(ctx, p.server, serverReadyTimeout); err != nil {
|
|
if changed, cerr := stopEpochChanged(p.rootDir, p.stopEpoch); cerr == nil && changed {
|
|
return abortInterruptedStart()
|
|
}
|
|
p.stats.IncBackendStop()
|
|
_ = stopBackendBounded(p.server)
|
|
return fmt.Errorf("database server not ready: %w", err)
|
|
}
|
|
birth, err := procid.Capture(os.Getpid())
|
|
if err != nil {
|
|
p.stats.IncBackendStop()
|
|
_ = stopBackendBounded(p.server)
|
|
return fmt.Errorf("capture proxy birth identity: %w", err)
|
|
}
|
|
rootID, err := identity.RootID(p.rootDir)
|
|
if err != nil {
|
|
p.stats.IncBackendStop()
|
|
_ = stopBackendBounded(p.server)
|
|
return fmt.Errorf("resolve proxy root identity: %w", err)
|
|
}
|
|
upstreamID := p.server.ID(ctx)
|
|
identMu.Lock()
|
|
identReply.RootID = rootID
|
|
identReply.UpstreamID = upstreamID
|
|
identReply.PID = os.Getpid()
|
|
identReply.Birth = string(birth)
|
|
identReply.ControlPort = control.Port()
|
|
identMu.Unlock()
|
|
|
|
// Last fence before publishing: the spawn marker was cleared when this
|
|
// process took proxy.lock, so a `bd dolt stop` that began during a slow
|
|
// backend start has no record of this attempt. It did advance the stop
|
|
// epoch first, so re-check it here and abort instead of publishing a
|
|
// running proxy after that stop returned. The startup epoch watcher is
|
|
// stopped (synchronously) first: past this fence a stop finds the
|
|
// published proxy.pid and stops the proxy through it, so a watcher
|
|
// cancellation must not race the publish.
|
|
stopEpochWatch()
|
|
if changed, err := stopEpochChanged(p.rootDir, p.stopEpoch); err != nil {
|
|
p.stats.IncBackendStop()
|
|
_ = stopBackendBounded(p.server)
|
|
return fmt.Errorf("re-check proxy stop epoch before publish: %w", err)
|
|
} else if changed {
|
|
return abortInterruptedStart()
|
|
}
|
|
|
|
if err := pidfile.Write(p.rootDir, PIDFileName, pidfile.PidFile{
|
|
Pid: os.Getpid(),
|
|
Port: dataPort.Port,
|
|
UpstreamID: upstreamID,
|
|
Schema: pidfile.SchemaV2,
|
|
Kind: pidfile.KindProxy,
|
|
Birth: string(birth),
|
|
RootID: rootID,
|
|
ControlPort: control.Port(),
|
|
}); err != nil {
|
|
p.stats.IncBackendStop()
|
|
_ = stopBackendBounded(p.server)
|
|
return fmt.Errorf("write pid file: %w", err)
|
|
}
|
|
defer func() { _ = pidfile.Remove(p.rootDir, PIDFileName) }()
|
|
|
|
g, gctx := errgroup.WithContext(ctx)
|
|
g.Go(func() error {
|
|
<-gctx.Done()
|
|
_ = p.listener.Close()
|
|
_ = control.Close()
|
|
return nil
|
|
})
|
|
g.Go(func() error { return p.idleWatcher(gctx) })
|
|
g.Go(func() error {
|
|
t := time.NewTicker(backendHealthInterval)
|
|
defer t.Stop()
|
|
for {
|
|
select {
|
|
case <-gctx.Done():
|
|
return nil
|
|
case <-t.C:
|
|
if !p.server.Running(gctx) {
|
|
return errors.New("database server exited")
|
|
}
|
|
}
|
|
}
|
|
})
|
|
g.Go(func() error { return p.acceptLoop(gctx) })
|
|
|
|
runErr := g.Wait()
|
|
_ = p.conns.Wait()
|
|
p.stats.IncBackendStop()
|
|
stopErr := stopBackendBounded(p.server)
|
|
if stopErr != nil {
|
|
stopErr = fmt.Errorf("stop database server: %w", stopErr)
|
|
}
|
|
if errors.Is(runErr, errIdleTimeout) || sigReceived.Load() {
|
|
runErr = nil
|
|
}
|
|
return errors.Join(runErr, stopErr)
|
|
}
|
|
|
|
func stopBackendBounded(s server.DatabaseServer) error {
|
|
ctx, cancel := context.WithTimeout(context.Background(), backendStopTimeout)
|
|
defer cancel()
|
|
return s.Stop(ctx)
|
|
}
|
|
|
|
func (p *proxyServer) idleWatcher(ctx context.Context) error {
|
|
if p.idleTimeout <= 0 {
|
|
<-ctx.Done()
|
|
return nil
|
|
}
|
|
interval := p.idleTimeout / 4
|
|
if interval < idleWatcherMinInterval {
|
|
interval = idleWatcherMinInterval
|
|
}
|
|
p.tracef("idleWatcher start (timeout=%s, tick=%s)", p.idleTimeout, interval)
|
|
tick := time.NewTicker(interval)
|
|
defer tick.Stop()
|
|
var idleSince time.Time
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
p.tracef("idleWatcher exit (ctx done)")
|
|
return nil
|
|
case <-tick.C:
|
|
if n := p.activeConns.Load(); n > 0 {
|
|
if !idleSince.IsZero() {
|
|
p.tracef("idleWatcher cleared (active=%d)", n)
|
|
idleSince = time.Time{}
|
|
}
|
|
continue
|
|
}
|
|
if idleSince.IsZero() {
|
|
p.tracef("idleWatcher armed")
|
|
idleSince = time.Now()
|
|
continue
|
|
}
|
|
if time.Since(idleSince) >= p.idleTimeout {
|
|
p.tracef("idleWatcher expired after %s, shutting down", p.idleTimeout)
|
|
p.stats.IncIdleTimeout()
|
|
return errIdleTimeout
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
func (p *proxyServer) acceptLoop(ctx context.Context) error {
|
|
p.tracef("acceptLoop start (addr=%s)", p.listener.Addr())
|
|
for {
|
|
conn, err := p.listener.Accept()
|
|
if err != nil {
|
|
if errors.Is(err, net.ErrClosed) || ctx.Err() != nil {
|
|
p.tracef("acceptLoop exit (ctx=%v)", ctx.Err())
|
|
return nil
|
|
}
|
|
// Surface non-shutdown accept errors to the errgroup so the
|
|
// proxy fails fast instead of busy-looping. Specific errors that
|
|
// warrant retry (e.g. transient EMFILE under load) can be added
|
|
// here as the need arises.
|
|
p.tracef("acceptLoop error: %v", err)
|
|
p.stats.IncAcceptError()
|
|
return fmt.Errorf("accept: %w", err)
|
|
}
|
|
if tc, ok := conn.(*net.TCPConn); ok {
|
|
_ = tc.SetKeepAlive(true)
|
|
_ = tc.SetKeepAlivePeriod(tcpKeepAlivePeriod)
|
|
}
|
|
p.tracef("acceptLoop accepted (remote=%s)", conn.RemoteAddr())
|
|
p.stats.IncAccept()
|
|
p.conns.Go(func() error {
|
|
return p.handleConn(ctx, conn)
|
|
})
|
|
}
|
|
}
|
|
|
|
func (p *proxyServer) handleConn(ctx context.Context, client net.Conn) error {
|
|
addr := client.RemoteAddr()
|
|
p.tracef("handleConn(%s) start", addr)
|
|
p.activeConns.Add(1)
|
|
defer func() {
|
|
p.activeConns.Add(-1)
|
|
p.tracef("handleConn(%s) end (active=%d)", addr, p.activeConns.Load())
|
|
}()
|
|
|
|
p.stats.IncBackendDialAttempt()
|
|
dialCtx, cancelDial := context.WithTimeout(ctx, backendDialTimeout)
|
|
backend, err := p.server.Dial(dialCtx)
|
|
cancelDial()
|
|
if err != nil {
|
|
p.tracef("handleConn(%s) backend dial error: %v", addr, err)
|
|
p.stats.IncBackendDialError()
|
|
if p.reportUpstreamOutage && isUpstreamUnreachableDialError(err) {
|
|
p.writeUpstreamOutage(client, dialFailureMessage(err))
|
|
}
|
|
_ = client.Close()
|
|
return err
|
|
}
|
|
p.tracef("handleConn(%s) backend dial ok", addr)
|
|
p.stats.IncBackendDialSuccess()
|
|
p.stats.IncHandledConn()
|
|
|
|
done := make(chan struct{})
|
|
var doneOnce sync.Once
|
|
finish := func() { doneOnce.Do(func() { close(done) }) }
|
|
|
|
var g errgroup.Group
|
|
g.Go(func() error {
|
|
select {
|
|
case <-ctx.Done():
|
|
p.tracef("handleConn(%s) ctx canceled, force-closing", addr)
|
|
_ = client.Close()
|
|
_ = backend.Close()
|
|
case <-done:
|
|
}
|
|
return nil
|
|
})
|
|
g.Go(func() error {
|
|
defer finish()
|
|
defer func() { _ = backend.Close() }()
|
|
defer func() { _ = client.Close() }()
|
|
n, err := io.Copy(backend, client)
|
|
p.stats.AddBytesClientToBackend(n)
|
|
p.tracef("handleConn(%s) client→backend done (n=%d, err=%v)", addr, n, err)
|
|
return err
|
|
})
|
|
g.Go(func() error {
|
|
defer finish()
|
|
defer func() { _ = backend.Close() }()
|
|
defer func() { _ = client.Close() }()
|
|
n, err := io.Copy(client, backend)
|
|
p.stats.AddBytesBackendToClient(n)
|
|
p.tracef("handleConn(%s) backend→client done (n=%d, err=%v)", addr, n, err)
|
|
// A MySQL server speaks first, so a backend that reaches EOF having
|
|
// sent nothing never served this connection: a front whose own
|
|
// target is gone, or a server at its connection limit or shutting
|
|
// down. That is not proof of an outage (see upstream_error.go), so
|
|
// the client retries the report briefly rather than failing on it.
|
|
// A client that hung up first closes backend, which makes this Copy
|
|
// fail rather than return a clean EOF, so that case stays silent.
|
|
if p.reportUpstreamOutage && n == 0 && err == nil {
|
|
p.writeUpstreamOutage(client, closedBeforeGreetingMessage(backend))
|
|
}
|
|
return err
|
|
})
|
|
return g.Wait()
|
|
}
|
|
|
|
func waitForServerReady(ctx context.Context, s server.DatabaseServer, timeout time.Duration) error {
|
|
bo := backoff.NewExponentialBackOff()
|
|
bo.InitialInterval = readyInitialBackoff
|
|
bo.MaxInterval = readyMaxBackoff
|
|
bo.MaxElapsedTime = timeout
|
|
|
|
return backoff.Retry(func() error {
|
|
if !s.Running(ctx) {
|
|
return errors.New("database server not running")
|
|
}
|
|
dialCtx, cancel := context.WithTimeout(ctx, readyDialTimeout)
|
|
defer cancel()
|
|
conn, err := s.Dial(dialCtx)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
_ = conn.Close()
|
|
// The dial proves something answers on the backend's address, not
|
|
// that the backend does. A local backend's Start has already proved
|
|
// its own process owns the port (see server.DoltServer.waitReady),
|
|
// and while it runs nobody else can bind it, so re-checking Running
|
|
// after the dial closes the gap where the backend exited in between
|
|
// and another process took the port.
|
|
if !s.Running(ctx) {
|
|
return errors.New("database server exited during readiness check")
|
|
}
|
|
return nil
|
|
}, backoff.WithContext(bo, ctx))
|
|
}
|