beads/cmd/bd/proxied_local_helpers_test.go
matt wilkie 4c63e15107
fix(dbproxy): race-free proxy port allocation and managed loopback listener policy (#5024)
* feat(dbproxy): add procid birth-token package and pidfile schema v2 fields

Foundation for verified proxy/backend adoption and safe stale-process
cleanup (mybd-psxg.5, PR-A stage 1): internal/procid provides versioned
process-birth tokens (Linux boot_id+starttime, Windows creation FILETIME,
Darwin p_starttime) with TOCTOU-safe signaling handles (pidfd on Linux,
re-verified process handle on Windows); pidfile gains additive schema-v2
fields (schema, kind, birth, root_id, control_port) and sentinel-error
validation. No behavior changes; existing v1 pidfiles still parse.

The procid Token/Verify shape intentionally matches the API requested in
gastownhall/beads#4513 so internal/doltserver can adopt it later.

Agent-Signature: codex-gpt-5.6-terra-medium on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* feat(dbproxy): publish verifiable proxy identity (secret, control handshake, v2 pidfiles)

Stage 2 of mybd-psxg.5 (PR-A): the managed proxy now proves who it is.
New leaf package internal/storage/dbproxy/identity provides the workspace
RootID (sha256 of the resolved root path), a rotated 0600 proxy.secret,
and the IDENT control protocol (client Identify + IdentReply). The proxy
opens a loopback control listener that answers only to the workspace
secret, and both pidfile writers publish schema-v2 records (kind, birth
token, root_id, control_port) captured via procid. A proxy or backend
that cannot prove its own identity fails startup instead of publishing.

Adoption, cleanup, and kill behavior are unchanged in this stage; the
consumers land next so each step stays reviewable.

Agent-Signature: codex-gpt-5.6-terra-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): verified adoption, fail-closed cleanup, and handle-safe kills

Stage 3 of mybd-psxg.5 (PR-A), the security core. Endpoint discovery now
classifies records with typed statuses instead of adopting any listener
that answers a TCP dial: adoption requires a schema-v2 pidfile, a live
birth-token match, the authenticated IDENT handshake, and root/pid/birth/
port cross-checks. Legacy and malformed records fail closed (quarantined
under proxy.lock, never adopted, never killed); quarantines are
timestamped renames with a 30-day sweep, so no forensic record is ever
unlinked.

Every kill site (spawn-path backend cleanup and bd dolt stop's Shutdown)
signals only through a procid.Handle opened against the recorded birth
token, refusing unverifiable live processes with an actionable error.
Orphaned Dolt backends left by a SIGKILLed proxy supervisor are now
detected regardless of child-lock state and reaped only after
verification. A proxy.spawn marker plus a persistent stop epoch closes
the window where bd dolt stop could report success while a fork already
in flight published a listener afterwards.

Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): harden identity fixes from cross-vendor adversarial review

Fixes every accepted finding from paired independent reviews of the
first three stages (reports in the coordination repo, 2026-07-24):

- darwin: classify wrapped ESRCH and the zero-length/EIO missing-pid
  sysctl result as process-gone; stale pidfiles self-heal again.
- fallback signaling (darwin, non-pidfd linux): a target that exits
  after a fatal signal is a successful kill, not an error.
- Shutdown verifies root_id on the proxy record too (copied-workspace
  kill hazard); positional bool pair replaced by killRecordChecks.
- birth-token mismatch now classifies as provably-dead everywhere;
  dead legacy/malformed records quarantine and self-heal; refusal
  messages name the record file and manual remedy.
- discovery-time verify/secret/root errors map to a non-adopting
  status routed through quarantine-under-lock instead of wedging
  before the lock probe.
- boot_id read failures are environmental errors, never process-gone
  (closes the one fail-open path found in review).
- ForceStopUnverified: executable-identity-checked operator escape for
  pre-upgrade/unverifiable records, quarantines after, reports
  truthfully; CLI wiring intentionally deferred.
- handshake replies are HMAC'd against a client nonce; constant-time
  secret compare; bounded accept failures and handshake concurrency;
  migration purge covers proxy.secret and quarantine files; spawn
  marker cleared via defer; unix-only constraints on sleep tests.

Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* feat(dbproxy): wire bd dolt stop --force and prove the lifecycle lane end to end

Stage 4 of mybd-psxg.5 (PR-A). bd dolt stop gains --force, routed through
proxy.ForceStopUnverified only after the backend is confirmed stopped:
it verifies executable identity before signaling, quarantines the record
afterward, refuses foreign executables and verifiable v2 records, and
reports text/JSON truthfully. The managed-local lane
(BEADS_TEST_PROXIED_LOCAL=1) now exercises the whole security model
against a real dolt sql-server: foreign-listener rejection, reused-PID
no-kill, legacy fail-closed (free lock, held lock refusing a foreign
executable, held lock force-stopping a genuine bd), orphan-backend
reaping after a SIGKILLed supervisor, 5-process cold-start convergence,
and stop/start interleaving. Windows/macOS lane equivalents are defined
in the lane header, implementation staged later.

Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): close cross-vendor review findings — handle lifetimes, pidfd races, force-stop scoping

- B1: cleanupOrphanBackend closes the verified handle before waiting for
  exit, mirroring shutdown.go; an open Windows handle kept the dead PID
  allocated and made every orphan-backend recovery time out.
- B2: Windows procid checks GetExitCodeProcess after OpenProcess, so a
  terminated process kept resolvable by a third-party handle no longer
  verifies; "Verify == true implies running" now holds on all platforms.
- B3: Linux procid.Open opens the pidfd BEFORE token verification; the
  pidfd pins the PID number, closing the reuse window where a later
  Signal could kill an unrelated replacement process.
- B4: PidfdSendSignal ESRCH on a fatal signal is treated as successful
  termination, matching the fallback path (same for Windows terminate).
- B5: ForceStopUnverified now covers the backend (proxy-child) record
  with the same safety checks, and CanForceStopUnverified accepts
  refusals where the backend is also (or only) unverifiable — the
  paired-legacy pre-v2 topology was locked out of its own advertised
  recovery. bd dolt stop --force reports the backend record too.
- B6a: an unverified live PID is only signaled when its command line
  ties it to this workspace (Linux /proc cmdline, Darwin ps args);
  Windows cannot read another process's command line and now refuses,
  which beats basename-only killing. Clear errors name the manual remedy.
- B6b: one stable OS handle (Linux pidfd, Windows process handle) spans
  inspection and signaling, removing the inspect/kill TOCTOU.
- S1: control-listener accept errors are non-fatal — log and retry with
  capped exponential backoff instead of tearing down a healthy proxy.
- S2: requireUnverifiableRecord surfaces RootID resolution errors
  instead of failing open into the destructive branch.
- S3: killSpawnedChild falls back to cmd.Process.Kill for our own
  un-reaped child when no verified handle exists, instead of leaking it.
- S4: adoption handshake accepts reply schema v2 or newer, matching
  pidfile.ValidateV2 forward-compat policy.
- S5: errors.Is for ERROR_INVALID_PARAMETER.
- Tests: pidfd ESRCH-after-exit and Open-after-reap classification;
  paired-legacy force recovery end to end; foreign-workspace bd process
  refused; Windows Verify-after-terminate (compile-checked); schema-v3
  handshake adoption; accept-loop survival under persistent errors.

Agent-Signature: claude-fable-5-medium on behalf of maphew

* fix(dbproxy): race-free proxy port allocation and managed loopback listener policy

Companion to the identity PR (stacks on it). The parent no longer
pre-picks the proxy data port: port 0 flows to the child, which binds
127.0.0.1:0 and publishes the OS-assigned port through its verified
pidfile, eliminating the bind-close-launch race; explicitly configured
ports keep fixed-port failure semantics. PickFreePort survives only for
the dolt-config path, annotated for the follow-up RFC on backend port
collisions.

Managed dolt configs are now validated by a single numeric-loopback
listener policy at init and at runtime resolution, covering all config
paths (generated, BEADS_PROXIED_SERVER_CONFIG, sidecar, pre-marker,
hand-edited marker, interpolation fallback). Hostnames including
localhost are rejected with remediation text naming the
--proxied-server-external-* flags; an omitted host is accepted because
dolt servercfg defaults it. There is deliberately no bypass env:
managed mode assumes loopback for its root/empty-password account.
External proxied topology is exempt by construction.

Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): whole-config managed listener policy, epoch-fenced startup, diagnosis fixes

Review findings from two independent passes over the listener-policy
commit (gastownhall/beads#5024), rebased onto the rewritten #5013 head:

- F1: the managed policy now covers the whole sql-server YAML, not just
  listener.host. validateManagedListenerHost is renamed
  validateManagedServerConfigPolicy and additionally rejects
  listener.socket (incl. the empty-string form, which maps to Dolt's
  default socket path), remotesapi.port, and any cluster block — all of
  which open non-loopback listeners for the root/empty-password child.
- F2: managed mode requires an explicit numeric loopback listener.host.
  An omitted host is no longer trusted: servercfg resolves it to
  "localhost", which CheckForUnixSocket also treats as license to open
  the shared-namespace default unix socket. This removes the dead
  net.ParseIP("localhost") branch and the omitted/explicit asymmetry.
- F3: one interpolation-aware policy check instead of five scattered
  raw-bytes ones. ensureProxiedServerConfig now delegates resolution to
  resolveOrCreateProxiedServerConfig and runs a single
  validateProxiedServerConfig (YamlConfigFromFile, ${VAR}-expanding) on
  the resolved path, so a marker config with an interpolated host is
  judged on what it expands to.
- F4: the proxy child re-checks the stop epoch immediately before
  publishing proxy.pid and aborts if it advanced (the spawn marker is
  cleared at lock acquisition, so a `bd dolt stop` racing a slow backend
  start otherwise has no record to wait on). The parent's observed epoch
  is threaded through a new db-proxy-child --stop-epoch flag into
  ProxyOpts.StopEpoch.
- F5a/b: child-exit diagnosis distinguishes a LockHeldExitCode spawn
  race from real failures, points at the child's log path otherwise, and
  timeout messages say "its OS-assigned port" instead of "port 0".
- F5c/d: validateManagedProxiedServerConfigAtInit now honors
  --proxied-server-root-path (the sidecar is not persisted yet at
  validation time) and drops the redundant explicit-config re-validation
  already done at flag parse.
- F5e: --proxied-server-config-path flag help states the hard managed
  constraint (docs/cli-reference regenerates from the release pin, so
  only the source help string changes here).
- F5f: the listener-policy lifecycle test documents its accepted
  PickFreePort EADDRINUSE flake window.

Tests: whole-config policy table (socket/remotesapi/cluster/omitted/
localhost/loopback forms), marker ${VAR} host interpolation accept and
reject, root-path-flag validation at init (unit + gated integration),
and a deterministic stop-epoch-advance-during-startup abort test.

Agent-Signature: claude-fable-5-medium on behalf of maphew

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 22:29:11 -07:00

243 lines
9 KiB
Go

//go:build cgo && unix
package main
// Helpers for the MANAGED-LOCAL proxied-server lane: bd launches its own
// loopback proxy (a detached `bd db-proxy-child` process) and that proxy
// launches a locally installed `dolt sql-server`. No external host/port and
// no testcontainer are involved, which is the production local-first
// topology that the external-harness shards (BEADS_TEST_PROXIED_SERVER)
// deliberately do not cover.
//
// The lane is gated by its own environment variable so CI can run the two
// proxied lanes independently. These helpers are written to be reusable by
// lifecycle/failure-injection tests (stale artifacts, process identity,
// port safety) beyond the initial smoke test.
import (
"context"
"database/sql"
"fmt"
"os"
"os/exec"
"path/filepath"
"syscall"
"testing"
"time"
"github.com/steveyegge/beads/internal/configfile"
"github.com/steveyegge/beads/internal/storage/dbproxy/pidfile"
"github.com/steveyegge/beads/internal/storage/dbproxy/proxy"
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
)
const managedLocalProxiedEnvVar = "BEADS_TEST_PROXIED_LOCAL"
// requireManagedLocalProxiedEnv gates the managed-local proxied lane.
// Unlike requireProxiedServerEnv, a missing dolt binary FAILS the test when
// the lane is explicitly requested: this lane exists to prove that bd can
// launch and supervise a local Dolt child, so skipping on a broken
// prerequisite would report a green run without testing anything.
func requireManagedLocalProxiedEnv(t *testing.T) {
t.Helper()
if os.Getenv(managedLocalProxiedEnvVar) != "1" {
t.Skipf("set %s=1 to run managed-local proxied lifecycle tests", managedLocalProxiedEnvVar)
}
if _, err := exec.LookPath("dolt"); err != nil {
t.Fatalf("%s=1 but dolt is not in PATH; this lane must fail rather than skip: %v",
managedLocalProxiedEnvVar, err)
}
}
// bdManagedLocalInit initializes a disposable managed-local proxied project:
// `bd init --proxied-server` with NO external host/port flags, so command
// dispatch launches the loopback proxy and the proxy launches dolt.
// idleTimeout is passed through --proxied-server-idle-timeout so tests can
// observe idle shutdown without waiting out the 30s production default.
func bdManagedLocalInit(t *testing.T, bd, prefix string, idleTimeout time.Duration, extraInitArgs ...string) proxiedProject {
t.Helper()
args := append([]string{"--proxied-server-idle-timeout", idleTimeout.String()}, extraInitArgs...)
p := bdProxiedInit(t, bd, prefix, args...)
info, err := configfile.LoadProxiedServerClientInfo(p.beadsDir)
if err != nil {
t.Fatalf("LoadProxiedServerClientInfo(%s): %v", p.beadsDir, err)
}
if info == nil {
t.Fatalf("missing %s in %s after managed-local init", configfile.ProxiedServerClientInfoFileName, p.beadsDir)
}
if info.External != nil {
t.Fatalf("expected managed-local topology (no External block), got %+v", info.External)
}
if info.IdleTimeout != idleTimeout {
t.Fatalf("persisted managed-local idle timeout: got %s, want %s", info.IdleTimeout, idleTimeout)
}
if info.Port != 0 {
t.Fatalf("default managed proxy port: got %d, want 0 so the child binds an OS-assigned port", info.Port)
}
return p
}
// readManagedProxyPidFile returns the loopback proxy's pidfile (proxy.pid:
// the detached `bd db-proxy-child` process and the port the proxy listens
// on), or nil when it does not exist.
func readManagedProxyPidFile(t *testing.T, p proxiedProject) *pidfile.PidFile {
t.Helper()
pf, err := pidfile.Read(p.proxyRoot, proxy.PIDFileName)
if err != nil && !os.IsNotExist(err) {
t.Fatalf("read %s in %s: %v", proxy.PIDFileName, p.proxyRoot, err)
}
return pf
}
// readManagedBackendPidFile returns the supervised Dolt backend's pidfile
// (proxy-child.pid: the `dolt sql-server` process and its listener port),
// or nil when it does not exist.
func readManagedBackendPidFile(t *testing.T, p proxiedProject) *pidfile.PidFile {
t.Helper()
pf, err := pidfile.Read(p.proxyRoot, server.PIDFileName)
if err != nil && !os.IsNotExist(err) {
t.Fatalf("read %s in %s: %v", server.PIDFileName, p.proxyRoot, err)
}
return pf
}
// processAlive reports whether pid refers to a live process we can signal
// (kill -0). A pid that exists but is owned by another user would report
// alive=false here; in this lane every process is spawned by the test user.
func processAlive(pid int) bool {
if pid <= 0 {
return false
}
proc, err := os.FindProcess(pid)
if err != nil {
return false
}
return proc.Signal(syscall.Signal(0)) == nil
}
// managedProxiedArtifacts is a point-in-time snapshot of which lifecycle
// artifacts exist under the proxy root. Lock files legitimately persist on
// disk after a clean shutdown (the flock is what matters, not the file), so
// assertions should usually key on the pid files.
type managedProxiedArtifacts struct {
ProxyLock bool
ProxyPid bool
BackendLock bool
BackendPid bool
ConfigYAML bool
}
func snapshotManagedProxiedArtifacts(p proxiedProject) managedProxiedArtifacts {
exists := func(name string) bool {
_, err := os.Stat(filepath.Join(p.proxyRoot, name))
return err == nil
}
return managedProxiedArtifacts{
ProxyLock: exists(proxy.LockFileName),
ProxyPid: exists(proxy.PIDFileName),
BackendLock: exists(server.LockFileName),
BackendPid: exists(server.PIDFileName),
ConfigYAML: exists(proxiedServerConfigName),
}
}
// waitForManagedProxied polls cond until it reports done or timeout elapses,
// failing the test with the condition's last reported state.
func waitForManagedProxied(t *testing.T, timeout time.Duration, desc string, cond func() (bool, string)) {
t.Helper()
deadline := time.Now().Add(timeout)
last := "(condition never evaluated)"
for {
done, state := cond()
if done {
return
}
last = state
if time.Now().After(deadline) {
t.Fatalf("timed out after %s waiting for %s; last state: %s", timeout, desc, last)
}
time.Sleep(100 * time.Millisecond)
}
}
// waitForManagedProxiedShutdown waits until both pid files are gone and the
// previously observed proxy and backend processes are dead — the observable
// contract of a clean idle shutdown.
func waitForManagedProxiedShutdown(t *testing.T, p proxiedProject, proxyPID, backendPID int, timeout time.Duration) {
t.Helper()
waitForManagedProxied(t, timeout, "idle shutdown of managed proxy and dolt backend", func() (bool, string) {
arts := snapshotManagedProxiedArtifacts(p)
proxyDead := !processAlive(proxyPID)
backendDead := !processAlive(backendPID)
if !arts.ProxyPid && !arts.BackendPid && proxyDead && backendDead {
return true, ""
}
return false, fmt.Sprintf(
"proxy.pid exists=%v proxy-child.pid exists=%v proxy(pid %d) alive=%v dolt(pid %d) alive=%v",
arts.ProxyPid, arts.BackendPid, proxyPID, !proxyDead, backendPID, !backendDead)
})
}
// heldProxiedConn is a single checked-out SQL connection through the
// managed proxy. While it is held, the proxy's idle watcher sees an active
// connection and will not shut the topology down — the intended way to keep
// the process tree stable while inspecting it. Release() must close the
// entire pool, not just return the conn to it: a pooled-but-idle connection
// keeps its TCP session to the proxy open, which still counts as active and
// blocks idle shutdown indefinitely.
type heldProxiedConn struct {
Conn *sql.Conn
db *sql.DB
}
// Release drops every TCP connection this holder has to the proxy, allowing
// the idle countdown to begin. Safe to call more than once.
func (h *heldProxiedConn) Release() {
if h.Conn != nil {
_ = h.Conn.Close()
h.Conn = nil
}
if h.db != nil {
_ = h.db.Close()
h.db = nil
}
}
// openHeldManagedProxiedConn ensures the managed proxy is running (issuing a
// cheap bd command to start or restart it when needed), then returns a held
// connection through it. Callers must Release() it to let the topology idle
// out; a t.Cleanup Release is registered as a backstop.
func openHeldManagedProxiedConn(t *testing.T, bd string, p proxiedProject) *heldProxiedConn {
t.Helper()
var held *heldProxiedConn
waitForManagedProxied(t, 60*time.Second, "held connection through managed proxy", func() (bool, string) {
pf := readManagedProxyPidFile(t, p)
if pf == nil || !processAlive(pf.Pid) {
// Proxy not up (yet, or idled out between commands): any data
// command restarts it transparently.
if out, err := bdProxiedRun(t, bd, p.dir, "list", "--json"); err != nil {
return false, fmt.Sprintf("bd list to (re)start proxy failed: %v\n%s", err, out)
}
return false, "proxy pidfile absent; issued bd list to start it"
}
dsn := fmt.Sprintf("root:@tcp(127.0.0.1:%d)/%s?parseTime=true", pf.Port, p.database)
db, err := sql.Open("mysql", dsn)
if err != nil {
return false, fmt.Sprintf("sql.Open %s: %v", dsn, err)
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
c, err := db.Conn(ctx)
cancel()
if err != nil {
_ = db.Close()
return false, fmt.Sprintf("checkout conn via %s: %v", dsn, err)
}
held = &heldProxiedConn{Conn: c, db: db}
t.Cleanup(held.Release)
return true, ""
})
return held
}