beads/cmd/bd/db_proxy_child.go
Julian Knutsen b184dae412
fix(dbproxy): managed-local dolt readiness proves port ownership; recover onto a free port (#7184)
* fix(dbproxy): prove the managed dolt owns its port; move a taken Beads-chosen port

The managed-local backend's dolt port comes from a bind-and-close
(proxy.PickFreePort) when bd writes the generated config. The dolt
sql-server binds it only later, after the proxy child has run dolt
config and init and booted the server. If another process takes the port
in that gap, dolt logs "Port N already in use." and exits. But
DoltServer.waitReady only dialed the port, so it accepted the other
process's listener as ready. The proxy then published proxy.pid, its
backend health check saw dolt gone, and it shut down. bd init's first
ping got "unexpected EOF" and every retry after that got "connection
refused". Under parallel tests this hit any test that starts its own
managed backend.

Readiness now requires proof that the process Start launched owns the
port:
- DoltServer sends the child's stdout/stderr through a watcher that
  forwards to the server log. A dial counts only after dolt's own
  "Server ready. Accepting connections." line has appeared on that
  output. That line is info level, and every generated config uses
  info. A config with a quieter log_level has no such signal and keeps
  the dial-only check.
- A child that exits first is reported as the new server.ErrPortInUse
  when its output said its port was taken.

Recovery: DoltServer.SetPortRepicker lets Start move the config to a
fresh port, re-read it, and relaunch, up to 5 ports. dolt config/init
run only on the first attempt. bd's repicker,
repickManagedProxiedServerPort, needs to tell a port Beads chose from
one an operator set, so generated configs now record the chosen port
on a "# beads-chosen-listener-port: N" line after the managed marker.
The repicker moves the port only when the file is Beads-generated,
carries that record, and its listener.port is still the recorded port.
Otherwise the port is treated as pinned and Start fails with
ErrPortInUse. That includes configs generated before this change. It
rewrites only listener.port and the record, through a yaml.Node so the
rest of the file is kept, and writes the file atomically.
db-proxy-child installs it for the local backend.

The proxy's waitForServerReady also re-checks Running after its dial.
Start has already proved ownership, and a running dolt holds its
listener, so only a backend that exited between the two checks could
let a dial reach another process.

Tests:
- server: a held port is not accepted as ready (ErrPortInUse; this
  test fails on main), a repicker brings the server up on a new port
  that the pid record and DSN both use, and a declining repicker ends
  Start with ErrPortInUse.
- cmd/bd: the generated config records its port, a Beads-chosen port
  moves while hand edits survive, and a hand-set port, a config
  without the record, an operator config, and a stale in-use port are
  all refused with the file left untouched.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(dbproxy): move a taken managed port via a runtime config, never config.yaml

This reworks cd00c5acb after review. That commit recovered from a taken
Beads-chosen port by rewriting the managed config.yaml. That broke the
file's own contract ("Never rewritten afterward; hand edits are
preserved", #4986), and #5024 had deferred that contract change to an
RFC.

dolt sql-server has no command-line way around the file: with --config,
-P/--port is silently ignored (checked against dolt 2.1.8 and 2.2.0, in
both argument orders). So on ErrPortInUse the proxy child now writes
proxy-child.runtime-config.yaml (server.RuntimeConfigFileName) in the
server root. That file is the operator config with only listener.port
moved to a free port. The child relaunches dolt from it and records the
effective port where runtime readers already look: the backend pid
record and the DoltServer's dial/DSN config. The runtime config is
removed when the server stops, when Start fails, and by the next Start.
config.yaml is never written.

- server.PortRepicker becomes server.PortConflictPolicy. It only decides
  whether the port may move; the server does the move.
- bd's policy (managedPortConflictPolicy) allows a move when the file
  carries the managed marker and either:
  - has the port record and listener.port still equals it, or
  - is byte-identical to the config bd generated before the record
    existed, so existing unedited workspaces recover too.
- Everything else is pinned: operator configs, hand-set ports,
  hand-edited legacy files, and configs with ${...} placeholders (a
  runtime copy would put the expanded values on disk).
- The record line now says plainly that Beads runs on a free port
  without editing the file, and that deleting the line or changing
  listener.port pins the port. The record parser now requires the
  digits to end at a space or line end and to be in 1..65535.
- ErrPortInUse messages say what to do: free port P, set listener.port
  in <path> to a free port, or (for a generated file) delete it so
  Beads regenerates it. The give-up message names the last port tried.
- A dolt that never logs the ready line is still not accepted. A
  grace-period fallback would let a foreign listener through while dolt
  is still starting. The timeout error now names the missing line, says
  whether anything answered, and points at the escape hatch: at
  log_level warning, readiness is dial-only.
- Port-conflict detection matches go-mysql-server's "Port host:port
  already in use." and the Windows bind error, and requires the SQL port
  in the kernel's bind error, so a metrics or MCP listener failure is
  not taken for the SQL port.
- Spawn error paths reap with cmd.Wait(). cmd.WaitDelay is set, and a
  comment notes that the backend's output is now a pipe into the proxy
  child.

Tests:
- Output-watcher table tests: split, one-byte and CRLF writes, output
  past the 4 KiB tail, the MCP ready line, port boundaries, the
  go-mysql-server and Windows forms, another listener's bind error, the
  unix-socket warning, and nil log.
- A real dolt at log_level warning starts through dial-only readiness.
- With a stand-in dolt (the test binary in fake mode, wired in
  TestMain): Start gives up after maxStartPortAttempts, a missing ready
  line is diagnosed, plus a positive control.
- With real dolt: a move keeps config.yaml byte-identical, runs from the
  runtime config, and the runtime config is removed on Stop. A declining
  policy and the no-policy remedy are covered.
- The bd policy's allow and pin cases.
- The managed-local lifecycle test's config/pid port check accepts the
  runtime config.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(dbproxy): ask the port policy once, about the operator's port; copy raw config text

Re-review of the runtime-config rework found a regression. After the
first move, Start asked the PortConflictPolicy about the runtime port.
bd's policy re-reads config.yaml, whose listener.port is still the
operator's port, so it refused with "listener.port is A, not the in-use
port B". The "up to 5 ports" retry therefore never ran in production;
the fake-dolt test missed it because its policy always allowed.

Start now asks the policy once per Start, on the first conflict, about
the operator config's port. Ports Start picked itself are its own to move
again. Start also re-reads the operator config up front, so every Start
begins from that config's port.

The runtime config is now built from config.yaml's raw text
(servercfg.NewYamlConfig) instead of the env-interpolated
YamlConfigFromFile result. dolt interpolates the runtime file when it
reads it, so the old way expanded placeholders twice: "$$" escapes lost a
level and placeholder values landed on disk. The server reads the
runtime file back with interpolation for its own Dial/DSN view. Since
placeholders no longer expand on disk, bd's policy stops refusing
configs that contain them.

Tests:
- bd's real policy driving a real DoltServer with a stand-in dolt that
  reports every port taken: Start makes five launches on five distinct
  ports, starting with config.yaml's, then gives up. It is not refused
  on the second move, and config.yaml stays unchanged.
- A frozen byte-for-byte literal of the config bd v1.3.0/v1.3.1
  generates. The legacy render must equal it, and the policy must find
  it eligible.
- The runtime config keeps placeholders and "$$" verbatim and is mode
  0600, while the server dials the interpolated host.
- The give-up test asserts the policy is asked once, about the
  operator's port.
- The missing-ready-line test timeout is raised to 30s for the slow
  fake-dolt exec under race.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(dbproxy): classify a child that exits before its identity is captured

TestManagedPortConflictPolicy_ConsecutiveCollisions failed on the farm
with "capture child birth identity: procid: process is no longer
running", after 2 launches instead of 5. Its stand-in dolt reports
"Port N already in use." and exits immediately, and it exited before
Start captured its birth identity. Start treated that capture failure as
fatal, so an ErrPortInUse that Start could recover from became a hard
error. A real dolt that finds its port taken can exit just as fast, so
this race exists in the product, not only in the test.

Every spawn step before the child is handed to its supervising
goroutine (identity capture, root identity, pid record) now fails
through failSpawn. failSpawn kills and reaps the child; reaping also
drains its output into the watcher. The child's own exit then decides
the error:
- the child reported its port taken: ErrPortInUse, so the normal policy
  and retry path runs;
- the child exited by itself for another reason: "exited (status N)
  before startup completed";
- the child was alive and we killed it: the step's own error, as
  before.

On Windows, a killed child reports an ordinary exit code, so there an
exit cannot be told from a kill. Only the port-in-use classification
applies there.

procid.Capture is reached through a package var so tests can make the
capture lose the race deterministically.

Tests, using a shell-script stand-in dolt that starts in milliseconds:
- a fast port-in-use exit that beats the capture is ErrPortInUse, and
  with a policy it is retried up to the 5-port budget;
- a fast exit for another reason reports its status;
- a capture failure while the child is alive stays fatal and kills the
  child;
- ten immediate port-in-use exits with the real capture are all
  ErrPortInUse.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 04:24:32 -07:00

118 lines
5 KiB
Go

package main
import (
"context"
"errors"
"fmt"
"strings"
"time"
"github.com/spf13/cobra"
"github.com/steveyegge/beads/internal/configfile"
"github.com/steveyegge/beads/internal/storage/dbproxy/proxy"
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
)
var (
dbProxyChildRoot string
dbProxyChildPort int
dbProxyChildIdleTimeout time.Duration
dbProxyChildBackend string
dbProxyChildConfig string
dbProxyChildLogPath string
dbProxyChildDoltBin string
dbProxyChildDatabase string
dbProxyChildExternalHost string
dbProxyChildExternalPort int
dbProxyChildExternalSocketPath string
dbProxyChildExternalKeepAlive time.Duration
dbProxyChildStopEpoch string
)
var dbProxyChildCmd = &cobra.Command{
Use: "db-proxy-child",
Hidden: true,
Short: "Internal: run as the database proxy child process",
Long: `db-proxy-child runs the long-lived per-rootDir TCP proxy that fronts a
DatabaseServer. It is spawned by the parent bd process via fork+exec and is
not intended to be invoked directly by users.`,
PersistentPreRun: func(cmd *cobra.Command, args []string) {},
PersistentPostRun: func(cmd *cobra.Command, args []string) {},
RunE: func(cmd *cobra.Command, _ []string) error {
backend := proxy.Backend(dbProxyChildBackend)
if err := backend.Validate(); err != nil {
return err
}
external := configfile.ExternalDoltConfig{
Host: dbProxyChildExternalHost,
Port: dbProxyChildExternalPort,
Socket: dbProxyChildExternalSocketPath,
KeepAlivePeriod: dbProxyChildExternalKeepAlive,
}
srv, err := newDatabaseServer(backend, dbProxyChildRoot, dbProxyChildConfig, dbProxyChildLogPath, dbProxyChildDoltBin, dbProxyChildDatabase, external)
if err != nil {
return err
}
defer func() { _ = srv.Stop(context.Background()) }()
p := proxy.NewProxyServer(proxy.ProxyOpts{
RootDir: dbProxyChildRoot,
Port: dbProxyChildPort,
IdleTimeout: dbProxyChildIdleTimeout,
Server: srv,
StopEpoch: dbProxyChildStopEpoch,
})
if err := p.ListenAndServe(cmd.Context()); err != nil {
if errors.Is(err, proxy.ErrLockHeld) {
return &exitError{Code: proxy.LockHeldExitCode}
}
return err
}
return nil
},
}
func newDatabaseServer(backend proxy.Backend, rootDir, configPath, logPath, doltBin, database string, external configfile.ExternalDoltConfig) (server.DatabaseServer, error) {
switch backend {
case proxy.BackendLocalServer:
s, err := server.NewDoltServer(doltBin, rootDir, configPath, logPath, 0, database)
if err != nil {
return nil, err
}
// A Beads-generated config's port was picked before dolt binds it;
// if another process got there first, run on a free port instead.
s.SetPortConflictPolicy(managedPortConflictPolicy)
return s, nil
case proxy.BackendExternal:
return server.NewExternalDoltServer(external)
case proxy.BackendLocalSharedServer:
return nil, fmt.Errorf("backend %q: not yet implemented", backend)
}
return nil, fmt.Errorf("unknown backend %q", backend)
}
func init() {
dbProxyChildCmd.Flags().StringVar(&dbProxyChildRoot, "root", "", "root directory holding proxy.lock, proxy.pid, proxy.log")
dbProxyChildCmd.Flags().IntVar(&dbProxyChildPort, "port", 0, "port to listen on")
dbProxyChildCmd.Flags().DurationVar(&dbProxyChildIdleTimeout, "idle-timeout", 0, "idle timeout before shutdown (0 or negative = never shut down)")
dbProxyChildCmd.Flags().StringVar(&dbProxyChildBackend, "backend", "",
"backend kind: "+strings.Join(proxy.KnownBackendNames(), " | "))
dbProxyChildCmd.Flags().StringVar(&dbProxyChildConfig, "config", "", "path to backend server config (e.g. dolt sql-server YAML)")
dbProxyChildCmd.Flags().StringVar(&dbProxyChildLogPath, "logpath", "", "path the backend server should write its stdout/stderr to")
dbProxyChildCmd.Flags().StringVar(&dbProxyChildDoltBin, "dolt-bin", "", "path to the dolt executable")
dbProxyChildCmd.Flags().StringVar(&dbProxyChildDatabase, "database", "", "database to select when running shutdown maintenance (local-server backend)")
dbProxyChildCmd.Flags().StringVar(&dbProxyChildExternalHost, "external-host", "", "external backend: hostname or IP of the dolt sql-server")
dbProxyChildCmd.Flags().IntVar(&dbProxyChildExternalPort, "external-port", 0, "external backend: TCP port of the dolt sql-server")
dbProxyChildCmd.Flags().StringVar(&dbProxyChildExternalSocketPath, "external-socket-path", "", "external backend: absolute path to a unix domain socket (overrides host/port)")
dbProxyChildCmd.Flags().DurationVar(&dbProxyChildExternalKeepAlive, "external-keep-alive", 0, "external backend: TCP keepalive period (default 30s)")
dbProxyChildCmd.Flags().StringVar(&dbProxyChildStopEpoch, "stop-epoch", "", "stop epoch the parent observed at spawn; the proxy aborts before publishing if it advances")
_ = dbProxyChildCmd.MarkFlagRequired("root")
_ = dbProxyChildCmd.MarkFlagRequired("port")
_ = dbProxyChildCmd.MarkFlagRequired("backend")
rootCmd.AddCommand(dbProxyChildCmd)
}