beads/internal/storage/dbproxy/proxy/endpoint.go
Steve Yegge f56632adcf
perf(storage): stop burning two wasted MySQL sessions per bd invocation (wy-s8ytnw) (#6122)
* perf(storage): stop burning two wasted MySQL sessions per bd invocation (wy-s8ytnw)

Every bd store-open against a shared dolt sql-server cost two sessions that
did no work:

1. readAndDial's data-port probe (internal/storage/dbproxy/proxy/endpoint.go)
   dialed the proxy data port and closed it to "prove" it accepts. The proxy's
   handleConn dials the BACKEND for every accepted client connection before a
   byte flows, so the zero-byte probe was a full backend MySQL session on every
   invocation from a proxied clone — and it proved nothing the authenticated
   control-port identity reply had not: ListenAndServe binds the data listener
   before the control listener exists. Drop the probe; the identity reply is
   the liveness proof. (Deferring the backend dial until the first client byte
   was considered and rejected: MySQL is server-speaks-first, so a real client
   waits for the greeting and would deadlock.)

2. The no-database init connection. On the proxied/server uow path
   (openAndInitSchema) and the dolt.New server path (openServerConnection) bd
   opened a second, database-less pool per store-open purely to probe
   existence (SHOW DATABASES / the schema-init preparation) before opening
   the real pool. Both sites now try the database-scoped connection FIRST —
   keyed the way the Gateway case already is, not by deleting the probe: a
   successful connect is the existence proof, so the steady-state open (every
   open but the very first) costs one session. Any connect failure (1049
   Unknown database, server down, bad credentials) falls through to the
   historical no-database path, which still owns creation, the #5042
   ownership signal that arms FreshBootstrapHeal, databaseNotFoundError, and
   every existing error message. `created` is honestly false on the fast
   path; the uow provider reuses the init pool (nothing in the migration path
   mutates session state).

New test TestReadAndDialAdoptsWithoutDialingDataPort pins the probe removal;
it fails (1 accept) against the previous endpoint.go and passes now.

Why the live server still saw a bare CREATE DATABASE per open despite
openServerConnection's SHOW DATABASES guard (the question on wy-s8ytnw): that
guard is not on the proxied path at all — the CREATE came from the uow
schema-init preparation, already fixed upstream by #6022 (alreadyConverged);
the deployed binary (71e5ad700) predates it (wy-7sx6cj).

* fix(storage): cap the fast-path probe budget; correct the pool-reuse premise

Maintainer review fixes for PR #6122. All four are non-gating minors (plus
one nit) from the review synthesis; the quality scorecard's decision was
approve with no required changes.

- uow fast path no longer costs a second retry budget (finding 1). openDB's
  30s transient-ping budget was spent twice against an unreachable server:
  once on the fast-path probe, whose error was discarded, and again on the
  fall-through open. Measured 0s + 19.8s where it was 19.8s twice; the
  returned "uow: ping db: ..." text is unchanged. The probe is now a single
  bounded ping, mirroring the sibling fast path in internal/storage/dolt,
  and the historical open keeps the retry budget it owned at base. Nothing
  is lost by not retrying here: the endpoint is already live when
  openAndInitSchema runs (GetCreateDatabaseProxyServerEndpoint), so the
  probe was never the thing waiting out a starting server.
- Replace the pool-reuse comment's false premise (finding 2). "Nothing in
  the migration path mutates session state" is wrong: six shipped
  migrations clear FOREIGN_KEY_CHECKS and restore it, and the guarded ones
  leave user variables set. State the real invariant instead — one pinned
  session per attempt, paired restores inside each script, pool closed on
  every initSchema failure — and record the forward requirement in
  migrations/README.md, where migration authors will meet it.
- Log why either fast path fell through (finding 3), matching lock.go's
  alreadyConverged idiom, so a fast path that has quietly stopped firing is
  visible under BD_DEBUG instead of silently costing the second session.
- Amend both stale empty-DSN premise comments (finding 5, nit):
  selectProbeDatabase and alreadyConverged now describe both entry shapes.

Finding 4 (a server-gated assertion on open-path selection) is left as the
follow-up that the synthesis and the scorecard both marked optional.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: plato <plato@ghosttrack.com>
Co-authored-by: Eddie the Engineer <julianknutsen@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-12 09:38:01 -07:00

1077 lines
32 KiB
Go

package proxy
import (
"encoding/json"
"errors"
"fmt"
"io/fs"
"log"
"net"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/steveyegge/beads/internal/atomicfile"
"github.com/steveyegge/beads/internal/configfile"
"github.com/steveyegge/beads/internal/debug"
"github.com/steveyegge/beads/internal/fdhygiene"
"github.com/steveyegge/beads/internal/lockfile"
"github.com/steveyegge/beads/internal/procid"
"github.com/steveyegge/beads/internal/storage/dbproxy/identity"
"github.com/steveyegge/beads/internal/storage/dbproxy/pidfile"
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
"github.com/steveyegge/beads/internal/storage/dbproxy/util"
)
type ErrUpstreamMismatch struct {
RootDir string
Want string
Have string
}
func (e *ErrUpstreamMismatch) Error() string {
return fmt.Sprintf("proxy at %s fronts upstream %s, not %s", e.RootDir, e.Have, e.Want)
}
func IsUpstreamMismatch(err error) bool {
var m *ErrUpstreamMismatch
return errors.As(err, &m)
}
func intendedUpstreamID(opts OpenOpts) string {
if opts.Backend == BackendExternal {
return server.ExternalDoltServerID(opts.External)
}
return ""
}
type Endpoint struct {
Host string
Port int
}
func (e Endpoint) Address() string {
return net.JoinHostPort(e.Host, strconv.Itoa(e.Port))
}
type OpenOpts struct {
IdleTimeout time.Duration
Backend Backend
ConfigFilePath string
LogFilePath string
DoltBinPath string
Database string
Port int
External configfile.ExternalDoltConfig
}
const (
openDeadline = 15 * time.Second
spawnReadyHardTimeout = 2 * time.Minute
openPollInterval = 100 * time.Millisecond
identityProbeTimeout = 500 * time.Millisecond
backendExitTimeout = 5 * time.Second
// quarantineRetention bounds forensic pidfile accumulation; successful
// spawns sweep only records older than 30 days.
quarantineRetention = 30 * 24 * time.Hour
spawnMarkerFileName = "proxy.spawn"
stopEpochFileName = "proxy.stop-epoch"
)
var ResolveExecutable = os.Executable
var (
verifyProcessIdentity = procid.Verify
resolveRootIdentity = identity.RootID
readControlSecret = identity.ReadSecret
)
// ErrUnverifiableProcess marks lifecycle operations that refuse to signal a
// process because its workspace-scoped identity cannot be verified. Callers
// may use errors.Is to offer a narrower recovery path without treating
// unrelated shutdown failures as identity failures.
var ErrUnverifiableProcess = errors.New("proxy process identity is unverifiable")
type unverifiableLifecycleError struct {
message string
}
func (e *unverifiableLifecycleError) Error() string {
return e.message
}
func (e *unverifiableLifecycleError) Unwrap() error {
return ErrUnverifiableProcess
}
var stopEpochSequence atomic.Uint64
// beforeProxyChildStart is a deterministic test hook for the otherwise tiny
// release-lock-before-exec scheduling window. Production leaves it as a no-op.
var beforeProxyChildStart = func() {}
type adoptionStatus uint8
const (
adoptionAdopted adoptionStatus = iota
adoptionNoRecord
adoptionStaleDead
adoptionIdentityMismatch
adoptionUnverifiable
adoptionLegacy
adoptionMalformed
adoptionIOErr
)
func (s adoptionStatus) String() string {
switch s {
case adoptionAdopted:
return "adopted"
case adoptionNoRecord:
return "no record"
case adoptionStaleDead:
return "stale dead"
case adoptionIdentityMismatch:
return "identity mismatch"
case adoptionUnverifiable:
return "identity unverifiable"
case adoptionLegacy:
return "legacy"
case adoptionMalformed:
return "malformed"
case adoptionIOErr:
return "I/O error"
default:
return fmt.Sprintf("unknown adoption status %d", s)
}
}
type adoptionResult struct {
status adoptionStatus
endpoint Endpoint
pidfile *pidfile.PidFile
err error
}
type spawnMarker struct {
Schema int `json:"schema"`
PID int `json:"pid"`
Birth string `json:"birth"`
StopEpoch string `json:"stop_epoch"`
StartedUnix int64 `json:"started_unix"`
}
var errStartInterrupted = errors.New("proxy startup interrupted by concurrent shutdown")
func PickFreePort() (int, error) {
// The managed proxy no longer uses this bind-close allocator: its child
// binds port 0 and publishes the kernel-assigned port. The remaining
// production caller allocates the Dolt config port; that race requires
// the managed-config ownership/retry contract deferred to the PR-C RFC.
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
return 0, err
}
port := ln.Addr().(*net.TCPAddr).Port
_ = ln.Close()
return port, nil
}
func GetCreateDatabaseProxyServerEndpoint(rootDir string, opts OpenOpts) (Endpoint, error) {
if err := opts.Backend.Validate(); err != nil {
return Endpoint{}, fmt.Errorf("OpenOpts.Backend: %w", err)
}
if opts.Port != 0 && (opts.Port < 1 || opts.Port > 65535) {
return Endpoint{}, fmt.Errorf("OpenOpts.Port: must be 0 or 1-65535, got %d", opts.Port)
}
switch opts.Backend {
case BackendLocalServer:
if opts.ConfigFilePath == "" {
return Endpoint{}, fmt.Errorf("OpenOpts.ConfigFilePath is required for backend %q", opts.Backend)
}
if opts.LogFilePath == "" {
return Endpoint{}, fmt.Errorf("OpenOpts.LogFilePath is required for backend %q", opts.Backend)
}
if opts.DoltBinPath == "" {
return Endpoint{}, fmt.Errorf("OpenOpts.DoltBinPath is required for backend %q", opts.Backend)
}
case BackendExternal:
if opts.LogFilePath == "" {
return Endpoint{}, fmt.Errorf("OpenOpts.LogFilePath is required for backend %q", opts.Backend)
}
if err := opts.External.Validate(); err != nil {
return Endpoint{}, fmt.Errorf("OpenOpts.External: %w", err)
}
}
stopEpoch, err := readStopEpoch(rootDir)
if err != nil {
return Endpoint{}, fmt.Errorf("read proxy stop epoch: %w", err)
}
deadline := time.Now().Add(openDeadline)
timeout := time.NewTimer(openDeadline)
defer timeout.Stop()
poll := time.NewTicker(openPollInterval)
defer poll.Stop()
want := intendedUpstreamID(opts)
var lastSpawnErr error
for {
discovery := readAndDial(rootDir)
switch discovery.status {
case adoptionAdopted:
return adoptedEndpoint(rootDir, want, discovery)
case adoptionIOErr:
return Endpoint{}, fmt.Errorf("discover proxy from %s: %w", pidfile.Path(rootDir, PIDFileName), discovery.err)
}
lock, err := util.TryLock(filepath.Join(rootDir, LockFileName))
switch {
case err == nil:
// Re-read under proxy.lock. Another opener may have replaced the
// record after our lock-free discovery.
discovery = readAndDial(rootDir)
if discovery.status == adoptionAdopted {
lock.Unlock()
return adoptedEndpoint(rootDir, want, discovery)
}
if discovery.status == adoptionIOErr {
lock.Unlock()
return Endpoint{}, fmt.Errorf("discover proxy from %s under lock: %w", pidfile.Path(rootDir, PIDFileName), discovery.err)
}
markerActive, markerErr := inspectSpawnMarkerLocked(rootDir)
if markerErr != nil {
lock.Unlock()
return Endpoint{}, markerErr
}
if markerActive {
lock.Unlock()
lastSpawnErr = nil
// This break exits the switch only; the outer discovery loop
// continues with its normal bounded poll.
break
}
var ep Endpoint
ep, lastSpawnErr = spawnAndHandoff(rootDir, opts, deadline, stopEpoch, lock, discovery)
if lastSpawnErr == nil {
return ep, nil
}
if errors.Is(lastSpawnErr, errStartInterrupted) {
return Endpoint{}, lastSpawnErr
}
case !lockfile.IsLocked(err):
return Endpoint{}, fmt.Errorf("probe proxy lock: %w", err)
case discovery.status == adoptionLegacy:
recordPath := pidfile.Path(rootDir, PIDFileName)
return Endpoint{}, fmt.Errorf(
"legacy proxy record %s is protected by held lock %s; stop the pre-upgrade proxy with the old bd binary or wait for its idle exit, then quarantine the record manually by renaming %s to %s.stale-<unix-timestamp> before retrying",
recordPath,
filepath.Join(rootDir, LockFileName),
recordPath,
recordPath,
)
}
select {
case <-timeout.C:
if lastSpawnErr != nil {
return Endpoint{}, lastSpawnErr
}
return Endpoint{}, fmt.Errorf("timeout waiting for proxy on %s", rootDir)
case <-poll.C:
}
}
}
func adoptedEndpoint(rootDir, want string, discovery adoptionResult) (Endpoint, error) {
if want != "" && discovery.pidfile.UpstreamID != "" && discovery.pidfile.UpstreamID != want {
return Endpoint{}, &ErrUpstreamMismatch{
RootDir: rootDir,
Want: want,
Have: discovery.pidfile.UpstreamID,
}
}
return discovery.endpoint, nil
}
func spawnAndHandoff(
rootDir string,
opts OpenOpts,
deadline time.Time,
stopEpoch string,
lock *util.Lock,
discovery adoptionResult,
) (Endpoint, error) {
handedOff := false
defer func() {
if !handedOff {
lock.Unlock()
}
}()
currentEpoch, err := readStopEpoch(rootDir)
if err != nil {
return Endpoint{}, fmt.Errorf("read proxy stop epoch under lock: %w", err)
}
if currentEpoch != stopEpoch {
return Endpoint{}, fmt.Errorf("%w for %s", errStartInterrupted, rootDir)
}
if err := quarantineForSpawn(rootDir, discovery); err != nil {
return Endpoint{}, err
}
if err := cleanupOrphanBackend(rootDir); err != nil {
return Endpoint{}, err
}
handedOff = true
child, err := forkExecChild(rootDir, opts, opts.Port, stopEpoch, lock)
if err != nil {
return Endpoint{}, fmt.Errorf("fork child: %w", err)
}
defer func() { _ = clearOwnSpawnMarker(rootDir, child.marker) }()
defer func() {
if child.handle != nil {
_ = child.handle.Close()
}
}()
hard := time.NewTimer(spawnReadyHardTimeout)
defer hard.Stop()
poll := time.NewTicker(openPollInterval)
defer poll.Stop()
for {
discovered := readAndDial(rootDir)
if discovered.status == adoptionAdopted {
if err := sweepOldQuarantines(rootDir, time.Now()); err != nil {
log.Printf("dbproxy: could not sweep old quarantined records in %s: %v", rootDir, err)
}
return discovered.endpoint, nil
}
if discovered.status == adoptionIOErr {
return Endpoint{}, fmt.Errorf("discover spawned proxy: %w", discovered.err)
}
select {
case childErr := <-child.done:
if interrupted, ierr := stopEpochChanged(rootDir, stopEpoch); ierr != nil {
return Endpoint{}, ierr
} else if interrupted {
return Endpoint{}, fmt.Errorf("%w for %s", errStartInterrupted, rootDir)
}
if childErr == nil {
childErr = errors.New("child exited without reporting an error")
}
// A LockHeldExitCode exit is a lost spawn race, not a listen
// failure; any other exit gets the child's log path so the real
// error (listen, backend start, ...) is findable.
var exitErr *exec.ExitError
if errors.As(childErr, &exitErr) && exitErr.ExitCode() == LockHeldExitCode {
return Endpoint{}, fmt.Errorf(
"proxy child lost the proxy.lock spawn race for %s: %w",
rootDir, childErr,
)
}
if opts.Port != 0 {
return Endpoint{}, fmt.Errorf(
"proxy child exited before becoming ready on explicitly configured port %d (see %s): %w",
opts.Port, opts.LogFilePath, childErr,
)
}
return Endpoint{}, fmt.Errorf(
"proxy child exited before publishing its OS-assigned port (see %s): %w",
opts.LogFilePath, childErr,
)
case <-hard.C:
if err := killSpawnedChild(child); err != nil {
return Endpoint{}, fmt.Errorf("hard timeout waiting for proxy on %s; safe child kill failed: %w", describeSpawnPort(opts.Port), err)
}
return Endpoint{}, fmt.Errorf("hard timeout (%s) waiting for proxy on %s", spawnReadyHardTimeout, describeSpawnPort(opts.Port))
case <-poll.C:
}
if time.Now().After(deadline) {
if err := killSpawnedChild(child); err != nil {
return Endpoint{}, fmt.Errorf("timeout waiting for proxy on %s; safe child kill failed: %w", describeSpawnPort(opts.Port), err)
}
return Endpoint{}, fmt.Errorf("timeout waiting for proxy to become ready on %s", describeSpawnPort(opts.Port))
}
}
}
// describeSpawnPort renders a requested spawn port for wait/timeout
// messages: 0 is the default OS-assigned path, not a literal "port 0".
func describeSpawnPort(port int) string {
if port == 0 {
return "its OS-assigned port"
}
return fmt.Sprintf("port %d", port)
}
type spawnedProxyChild struct {
cmd *exec.Cmd
done <-chan error
handle *procid.Handle
marker spawnMarker
}
func forkExecChild(rootDir string, opts OpenOpts, port int, stopEpoch string, lock *util.Lock) (*spawnedProxyChild, error) {
released := false
defer func() {
if !released {
lock.Unlock()
}
}()
self, err := ResolveExecutable()
if err != nil {
return nil, fmt.Errorf("locate bd executable: %w", err)
}
idleTimeout := opts.IdleTimeout
if idleTimeout < 0 {
idleTimeout = IdleTimeoutNever
}
args := []string{
"db-proxy-child",
"--root", rootDir,
"--port", strconv.Itoa(port),
"--idle-timeout", idleTimeout.String(),
"--backend", string(opts.Backend),
"--stop-epoch", stopEpoch,
}
if opts.ConfigFilePath != "" {
args = append(args, "--config", opts.ConfigFilePath)
}
if opts.LogFilePath != "" {
args = append(args, "--logpath", opts.LogFilePath)
}
if opts.DoltBinPath != "" {
args = append(args, "--dolt-bin", opts.DoltBinPath)
}
if opts.Database != "" {
args = append(args, "--database", opts.Database)
}
if opts.Backend == BackendExternal {
ext := opts.External
if ext.Host != "" {
args = append(args, "--external-host", ext.Host)
}
if ext.Port != 0 {
args = append(args, "--external-port", strconv.Itoa(ext.Port))
}
if ext.Socket != "" {
args = append(args, "--external-socket-path", ext.Socket)
}
if ext.KeepAlivePeriod != 0 {
args = append(args, "--external-keep-alive", ext.KeepAlivePeriod.String())
}
}
logFile, err := os.OpenFile(opts.LogFilePath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600) //nolint:gosec // G304: logFilePath is caller-derived (workspace path), not user-request input
if err != nil {
return nil, fmt.Errorf("open log file %q: %w", opts.LogFilePath, err)
}
cmd := exec.Command(self, args...)
cmd.Stdin = nil
cmd.Stdout = logFile
cmd.Stderr = logFile
cmd.SysProcAttr = procAttrDetached()
birth, err := procid.Capture(os.Getpid())
if err != nil {
_ = logFile.Close()
return nil, fmt.Errorf("capture spawning process identity: %w", err)
}
marker := spawnMarker{
Schema: 1,
PID: os.Getpid(),
Birth: string(birth),
StopEpoch: stopEpoch,
StartedUnix: time.Now().Unix(),
}
if err := writeSpawnMarker(rootDir, marker); err != nil {
_ = logFile.Close()
return nil, err
}
// The marker is durable before proxy.lock is released. Shutdown treats a
// matching live owner as an in-progress start and waits; the child removes
// it only after acquiring proxy.lock. This closes the release-before-Start
// window without making the child deadlock on the parent's flock.
released = true
lock.Unlock()
beforeProxyChildStart()
// GH#4634: same hazard as the direct sql-server spawn, one hop further
// out. The proxy child is detached and long-lived, and it starts the
// sql-server itself, so a caller's non-CLOEXEC descriptor would otherwise
// be inherited twice over and pinned for the proxy's whole lifetime.
if leaked := fdhygiene.MarkInheritedCloexec(); len(leaked) > 0 {
// debug.Logf, not log.Printf: this fires in normal operation whenever
// the caller's environment leaves any fd open, and the parent's stderr
// may be parsed script output.
debug.Logf("dbproxy: marked %d inherited fd(s) close-on-exec before starting proxy child: %v", len(leaked), leaked)
}
if err := cmd.Start(); err != nil {
_ = logFile.Close()
_ = clearOwnSpawnMarker(rootDir, marker)
return nil, fmt.Errorf("start proxy child: %w", err)
}
done := make(chan error, 1)
go func() {
waitErr := cmd.Wait()
_ = logFile.Close()
done <- waitErr
close(done)
}()
// Theoretical race: the birth token is captured after Start, so the child
// could exit and its PID be recycled before Capture runs, making the
// handle describe the unrelated replacement. The window is a few
// milliseconds against OS PID-reuse latency, and the OS has no primitive
// to atomically capture identity at spawn, so this is accepted and
// documented rather than defended.
var handle *procid.Handle
childBirth, captureErr := procid.Capture(cmd.Process.Pid)
if captureErr == nil {
handle, captureErr = procid.Open(cmd.Process.Pid, childBirth)
}
if captureErr != nil && !procid.IsProcessGone(captureErr) {
log.Printf("dbproxy: could not open verified handle for newly spawned proxy pid %d: %v", cmd.Process.Pid, captureErr)
}
return &spawnedProxyChild{
cmd: cmd,
done: done,
handle: handle,
marker: marker,
}, nil
}
func IsRunning(rootDir string) (bool, int) {
discovery := readAndDial(rootDir)
if discovery.status != adoptionAdopted {
return false, 0
}
return true, discovery.pidfile.Pid
}
func readAndDial(rootDir string) adoptionResult {
pf, err := pidfile.Read(rootDir, PIDFileName)
if err != nil {
if isMalformedPIDFileError(err) {
return adoptionResult{status: adoptionMalformed, err: err}
}
return adoptionResult{status: adoptionIOErr, err: err}
}
if pf == nil {
return adoptionResult{status: adoptionNoRecord}
}
if err := pf.ValidateV2(pidfile.KindProxy); err != nil {
if errors.Is(err, pidfile.ErrLegacySchema) {
return adoptionResult{status: adoptionLegacy, pidfile: pf, err: err}
}
return adoptionResult{status: adoptionMalformed, pidfile: pf, err: err}
}
matched, err := verifyProcessIdentity(pf.Pid, procid.Token(pf.Birth))
if err != nil {
// Discovery must not turn an identity-probe failure into a fatal
// pidfile I/O error. In particular, Windows ERROR_ACCESS_DENIED on a
// recycled PID belongs in this non-adopting path; proxy.lock still
// gates quarantine and replacement.
return adoptionResult{
status: adoptionUnverifiable,
pidfile: pf,
err: fmt.Errorf("verify proxy pid %d: %w", pf.Pid, err),
}
}
if !matched {
return adoptionResult{status: adoptionStaleDead, pidfile: pf}
}
expectedRootID, err := resolveRootIdentity(rootDir)
if err != nil {
return adoptionResult{status: adoptionUnverifiable, pidfile: pf, err: err}
}
secret, err := readControlSecret(rootDir)
if err != nil {
return adoptionResult{status: adoptionUnverifiable, pidfile: pf, err: err}
}
reply, err := identity.Identify("127.0.0.1", pf.ControlPort, secret, identityProbeTimeout)
if err != nil {
return adoptionResult{status: adoptionIdentityMismatch, pidfile: pf, err: err}
}
// Accept schema v2 or newer, matching pidfile.ValidateV2's forward-compat
// policy for records.
if reply.Schema < pidfile.SchemaV2 ||
reply.Role != pidfile.KindProxy ||
reply.RootID != expectedRootID ||
reply.RootID != pf.RootID ||
reply.PID != pf.Pid ||
reply.Birth != pf.Birth ||
reply.DataPort != pf.Port ||
reply.ControlPort != pf.ControlPort ||
reply.UpstreamID != pf.UpstreamID {
return adoptionResult{
status: adoptionIdentityMismatch,
pidfile: pf,
err: errors.New("authenticated proxy identity does not match its pidfile or workspace"),
}
}
// The authenticated identity reply above IS the liveness proof. Do not
// add a dial-and-close TCP probe of the data port here: the proxy dials
// its backend for every accepted client connection BEFORE any bytes flow
// (handleConn), so a zero-byte probe costs the shared dolt server a full
// MySQL session on every bd invocation while proving nothing the control
// port did not — ListenAndServe binds the data listener before the
// control listener exists, so an identity reply carrying this DataPort
// already implies the data listener is bound (wy-s8ytnw).
ep := Endpoint{Host: "127.0.0.1", Port: pf.Port}
return adoptionResult{status: adoptionAdopted, endpoint: ep, pidfile: pf}
}
func isMalformedPIDFileError(err error) bool {
var syntaxErr *json.SyntaxError
var typeErr *json.UnmarshalTypeError
return errors.As(err, &syntaxErr) || errors.As(err, &typeErr)
}
func quarantineForSpawn(rootDir string, discovery adoptionResult) error {
switch discovery.status {
case adoptionNoRecord:
return nil
case adoptionStaleDead:
log.Printf("dbproxy: quarantining stale dead proxy record %s before restart", pidfile.Path(rootDir, PIDFileName))
case adoptionIdentityMismatch:
log.Printf(
"dbproxy: refusing proxy identity at %s (%v); quarantining only its record and starting a fresh proxy",
pidfile.Path(rootDir, PIDFileName), discovery.err,
)
case adoptionUnverifiable:
log.Printf(
"dbproxy: proxy identity at %s could not be verified (%v); quarantining only its record under proxy.lock and starting a fresh proxy",
pidfile.Path(rootDir, PIDFileName), discovery.err,
)
case adoptionLegacy:
log.Printf(
"dbproxy: quarantining legacy proxy record %s; a pre-upgrade proxy may still be running and must be stopped with the old bd binary if it does not idle-exit",
pidfile.Path(rootDir, PIDFileName),
)
case adoptionMalformed:
log.Printf("dbproxy: quarantining malformed proxy record %s (%v) before restart", pidfile.Path(rootDir, PIDFileName), discovery.err)
default:
return fmt.Errorf("cannot spawn from proxy discovery status %s", discovery.status)
}
target, err := quarantineRecord(rootDir, PIDFileName, time.Now())
if err != nil {
return fmt.Errorf("quarantine proxy record: %w", err)
}
log.Printf("dbproxy: preserved proxy record as %s", target)
return nil
}
func quarantineRecord(rootDir, name string, now time.Time) (string, error) {
source := pidfile.Path(rootDir, name)
for stamp := now.Unix(); ; stamp++ {
target := filepath.Join(rootDir, name+".stale-"+strconv.FormatInt(stamp, 10))
if _, err := os.Lstat(target); err == nil {
continue
} else if !errors.Is(err, fs.ErrNotExist) {
return "", fmt.Errorf("inspect quarantine target %s: %w", target, err)
}
if err := os.Rename(source, target); err != nil {
return "", fmt.Errorf("rename %s to %s: %w", source, target, err)
}
return target, nil
}
}
func sweepOldQuarantines(rootDir string, now time.Time) error {
entries, err := os.ReadDir(rootDir)
if err != nil {
return err
}
cutoff := now.Add(-quarantineRetention).Unix()
prefixes := []string{
PIDFileName + ".stale-",
server.PIDFileName + ".stale-",
}
var errs []error
for _, entry := range entries {
if entry.IsDir() {
continue
}
var stampText string
for _, prefix := range prefixes {
if strings.HasPrefix(entry.Name(), prefix) {
stampText = strings.TrimPrefix(entry.Name(), prefix)
break
}
}
if stampText == "" {
continue
}
stamp, err := strconv.ParseInt(stampText, 10, 64)
if err != nil || stamp >= cutoff {
continue
}
if err := os.Remove(filepath.Join(rootDir, entry.Name())); err != nil && !errors.Is(err, fs.ErrNotExist) {
errs = append(errs, fmt.Errorf("remove expired quarantine %s: %w", entry.Name(), err))
}
}
return errors.Join(errs...)
}
func cleanupOrphanBackend(rootDir string) error {
recordPath := pidfile.Path(rootDir, server.PIDFileName)
pf, readErr := pidfile.Read(rootDir, server.PIDFileName)
childLockPath := filepath.Join(rootDir, server.LockFileName)
childLock, lockErr := util.TryLock(childLockPath)
switch {
case lockErr == nil:
childLock.Unlock()
case lockfile.IsLocked(lockErr):
pid := 0
if pf != nil {
pid = pf.Pid
}
return fmt.Errorf(
"refusing backend cleanup: %s is held while proxy lock is free (recorded pid %d at %s); stop the process holding the child lock or remove the stale lock owner before retrying",
childLockPath, pid, recordPath,
)
default:
return fmt.Errorf("probe backend lock %s: %w", childLockPath, lockErr)
}
if readErr != nil {
if isMalformedPIDFileError(readErr) {
return unverifiableProcessError(
"backend cleanup",
recordPath,
0,
readErr,
unverifiableProcessChecks{},
)
}
return fmt.Errorf("read backend record %s: %w", recordPath, readErr)
}
if pf == nil {
return nil
}
if err := pf.ValidateV2(pidfile.KindDoltBackend); err != nil {
dead, live, probeErr := probeUnverifiablePID(pf.Pid)
if dead {
if _, quarantineErr := quarantineRecord(rootDir, server.PIDFileName, time.Now()); quarantineErr != nil {
return fmt.Errorf("quarantine dead unverifiable backend record: %w", quarantineErr)
}
return nil
}
if probeErr != nil {
err = errors.Join(err, fmt.Errorf("probe recorded pid: %w", probeErr))
}
return unverifiableProcessError(
"backend cleanup",
recordPath,
pf.Pid,
err,
unverifiableProcessChecks{
LiveEstablished: live,
},
)
}
expectedRootID, err := identity.RootID(rootDir)
if err != nil {
return fmt.Errorf("resolve backend root identity: %w", err)
}
if pf.RootID != expectedRootID {
return unverifiableProcessError(
"backend cleanup",
recordPath,
pf.Pid,
fmt.Errorf("root identity mismatch (record has %q, workspace has %q)", pf.RootID, expectedRootID),
unverifiableProcessChecks{},
)
}
handle, dead, err := openRecordedProcess(pf)
if err != nil {
return unverifiableProcessError(
"backend cleanup",
recordPath,
pf.Pid,
err,
unverifiableProcessChecks{},
)
}
if dead {
if _, err := quarantineRecord(rootDir, server.PIDFileName, time.Now()); err != nil {
return fmt.Errorf("quarantine dead backend record: %w", err)
}
return nil
}
if err := handle.Kill(); err != nil {
_ = handle.Close()
return fmt.Errorf("kill verified orphan backend pid %d from %s: %w", pf.Pid, recordPath, err)
}
// Close before waiting: an open handle on Windows keeps the dead PID
// allocated, so procid.Verify would keep matching it and the exit wait
// below would always time out.
if err := handle.Close(); err != nil {
return fmt.Errorf("close verified backend process handle for pid %d: %w", pf.Pid, err)
}
if err := waitForRecordedProcessExit(pf, backendExitTimeout); err != nil {
return fmt.Errorf("wait for verified orphan backend pid %d: %w", pf.Pid, err)
}
if _, err := quarantineRecord(rootDir, server.PIDFileName, time.Now()); err != nil {
return fmt.Errorf("quarantine orphan backend record: %w", err)
}
return nil
}
type unverifiableProcessChecks struct {
LiveEstablished bool
LegacyProxy bool
}
func unverifiableProcessError(
operation string,
recordPath string,
pid int,
cause error,
checks unverifiableProcessChecks,
) error {
liveness := ""
if checks.LiveEstablished {
liveness = " live"
}
stopGuidance := "stop the recorded process with the binary that started it"
if checks.LegacyProxy && checks.LiveEstablished {
stopGuidance = "stop the pre-upgrade proxy with the old bd binary"
}
return &unverifiableLifecycleError{message: fmt.Sprintf(
"%s refused for unverifiable%s process pid %d recorded at %s: %v; %s, then quarantine the record manually by renaming %s to %s.stale-<unix-timestamp> before retrying",
operation,
liveness,
pid,
recordPath,
cause,
stopGuidance,
recordPath,
recordPath,
)}
}
func probeUnverifiablePID(pid int) (dead bool, live bool, err error) {
if pid <= 0 {
return false, false, errors.New("record has no valid pid")
}
_, err = procid.Capture(pid)
if err == nil {
return false, true, nil
}
if procid.IsProcessGone(err) {
return true, false, nil
}
return false, false, err
}
func openRecordedProcess(pf *pidfile.PidFile) (*procid.Handle, bool, error) {
handle, err := procid.Open(pf.Pid, procid.Token(pf.Birth))
if err == nil {
return handle, false, nil
}
if procid.IsProcessGone(err) {
return nil, true, nil
}
current, captureErr := procid.Capture(pf.Pid)
if captureErr != nil {
if procid.IsProcessGone(captureErr) {
return nil, true, nil
}
return nil, false, fmt.Errorf("open process identity: %w (recheck: %v)", err, captureErr)
}
if current != procid.Token(pf.Birth) {
// A different birth token proves the recorded process exited, even
// when the numeric PID has since been recycled.
return nil, true, nil
}
return nil, false, fmt.Errorf("open verified process handle: %w", err)
}
func waitForRecordedProcessExit(pf *pidfile.PidFile, timeout time.Duration) error {
deadline := time.Now().Add(timeout)
for {
matched, err := procid.Verify(pf.Pid, procid.Token(pf.Birth))
if err != nil {
return err
}
if !matched {
return nil
}
if time.Now().After(deadline) {
return fmt.Errorf("timeout (%s) waiting for process exit", timeout)
}
time.Sleep(shutdownConfirmPoll)
}
}
func killSpawnedChild(child *spawnedProxyChild) error {
select {
case <-child.done:
return nil
default:
}
if child.handle == nil {
// No verified handle could be opened at spawn time. The child is our
// own un-reaped process, so its PID cannot have been recycled and a
// direct kill is safe; the alternative is leaking a live proxy child.
if err := child.cmd.Process.Kill(); err != nil && !errors.Is(err, os.ErrProcessDone) {
return fmt.Errorf("kill spawned proxy child pid %d: %w", child.cmd.Process.Pid, err)
}
} else if err := child.handle.Kill(); err != nil {
return err
}
timer := time.NewTimer(shutdownConfirmDeadline)
defer timer.Stop()
select {
case <-child.done:
return nil
case <-timer.C:
return fmt.Errorf("timeout (%s) waiting for pid %d", shutdownConfirmDeadline, child.cmd.Process.Pid)
}
}
func readStopEpoch(rootDir string) (string, error) {
path := filepath.Join(rootDir, stopEpochFileName)
data, err := os.ReadFile(path) // #nosec G304 - path is a fixed control filename under the workspace proxy root
if errors.Is(err, fs.ErrNotExist) {
return "", nil
}
if err != nil {
return "", err
}
return strings.TrimSpace(string(data)), nil
}
func advanceStopEpoch(rootDir string) error {
epoch := strconv.FormatInt(time.Now().UnixNano(), 10) +
"-" + strconv.Itoa(os.Getpid()) +
"-" + strconv.FormatUint(stopEpochSequence.Add(1), 10)
if err := atomicfile.WriteFile(filepath.Join(rootDir, stopEpochFileName), []byte(epoch+"\n"), 0o600); err != nil {
return err
}
return nil
}
func stopEpochChanged(rootDir, expected string) (bool, error) {
current, err := readStopEpoch(rootDir)
if err != nil {
return false, fmt.Errorf("read proxy stop epoch: %w", err)
}
return current != expected, nil
}
func writeSpawnMarker(rootDir string, marker spawnMarker) error {
data, err := json.Marshal(marker)
if err != nil {
return err
}
if err := atomicfile.WriteFile(filepath.Join(rootDir, spawnMarkerFileName), data, 0o600); err != nil {
return fmt.Errorf("write spawn marker: %w", err)
}
return nil
}
func readSpawnMarker(rootDir string) (*spawnMarker, error) {
path := filepath.Join(rootDir, spawnMarkerFileName)
data, err := os.ReadFile(path) // #nosec G304 - path is a fixed control filename under the workspace proxy root
if errors.Is(err, fs.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, err
}
var marker spawnMarker
if err := json.Unmarshal(data, &marker); err != nil {
return nil, fmt.Errorf("decode %s: %w", path, err)
}
if marker.Schema != 1 || marker.PID <= 0 || marker.Birth == "" {
return nil, fmt.Errorf("invalid spawn marker %s", path)
}
return &marker, nil
}
func inspectSpawnMarkerLocked(rootDir string) (bool, error) {
marker, err := readSpawnMarker(rootDir)
if err != nil {
return false, fmt.Errorf("inspect proxy spawn marker: %w", err)
}
if marker == nil {
return false, nil
}
matched, err := procid.Verify(marker.PID, procid.Token(marker.Birth))
if err != nil {
return false, fmt.Errorf("verify proxy spawn owner pid %d: %w", marker.PID, err)
}
if matched {
return true, nil
}
if err := os.Remove(filepath.Join(rootDir, spawnMarkerFileName)); err != nil && !errors.Is(err, fs.ErrNotExist) {
return false, fmt.Errorf("remove dead spawn marker: %w", err)
}
return false, nil
}
func clearSpawnMarkerAfterLock(rootDir string) error {
err := os.Remove(filepath.Join(rootDir, spawnMarkerFileName))
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return err
}
return nil
}
func clearOwnSpawnMarker(rootDir string, own spawnMarker) error {
deadline := time.Now().Add(shutdownConfirmDeadline)
for {
marker, err := readSpawnMarker(rootDir)
if err != nil || marker == nil {
return err
}
if *marker != own {
return nil
}
lock, err := util.TryLock(filepath.Join(rootDir, LockFileName))
if err == nil {
current, readErr := readSpawnMarker(rootDir)
if readErr == nil && current != nil && *current == own {
readErr = clearSpawnMarkerAfterLock(rootDir)
}
lock.Unlock()
return readErr
}
if !lockfile.IsLocked(err) {
return err
}
if time.Now().After(deadline) {
return fmt.Errorf("timeout clearing spawn marker %s", filepath.Join(rootDir, spawnMarkerFileName))
}
time.Sleep(shutdownConfirmPoll)
}
}