beads/internal/doltserver/procattr_linux.go
matt wilkie 993ca8d7c6
fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592)
* fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep)

A 2026-07-05 dev-machine audit found 13 orphaned `dolt sql-server`
processes, ages up to 3+ days, each serving a now-deleted /tmp test
tempdir. detached dolt servers (Setpgid, no Pdeathsig) are a deliberate
production feature — a shared server must outlive the `bd` process that
started it — but that same detachment means a SIGKILLed test binary (or
its Relay wrapper) leaves its embedded server running forever with
nothing to reap it.

Two defense-in-depth pieces, both scoped to Linux (the leak evidence)
and gated so production behavior is untouched:

1. Test-gated Pdeathsig (internal/doltserver/procattr_linux.go). Split
   procAttrDetached() out of doltserver_unix.go into a Linux file and a
   non-Linux-unix file, since syscall.SysProcAttr.Pdeathsig only exists
   on Linux. When BEADS_TEST_MODE=1, the detached server now also gets
   Pdeathsig: SIGTERM, so it exits when the thread that spawned it dies.
   Production (BEADS_TEST_MODE unset) stays byte-identical: Setpgid:
   true only. Windows keeps its existing procAttrDetached untouched.
   Pdeathsig fires on death of the spawning *thread*, which the Go
   runtime can retire independently of the process — so this is
   documented as a best-effort test-only backstop, not a guarantee.

2. Suite-exit sweep (internal/doltserver/sweep*.go). A pure selection
   function (selectOrphanTestServerPIDs, unit tested) decides which
   `dolt sql-server` processes are provably test debris: cmdline matches
   dolt+sql-server AND either its working directory has been deleted
   (Linux's /proc/<pid>/cwd grows a " (deleted)" suffix) or sits under a
   suite-scoped root the caller explicitly vouches for as its own (e.g.
   that suite's own testTempRoot). SweepOrphanedTestServers never
   defaults to a global root like os.TempDir(): scripts/test.sh runs
   packages in parallel (-p N), and virtually every suite's data dir
   lives somewhere under os.TempDir() too, so a global root would make
   one suite's TestMain sweep SIGTERM other suites' still-live servers
   mid-run. A production shared server's data directory is never a
   caller-scoped test root and never gets deleted out from under it, so
   it can never match either condition.

   SweepOrphanedTestServers() is wired into the TestMains most likely to
   leak: cmd/bd's main CLI suite (which runs a real `bd` binary with
   auto-start against its own testTempRoot-anchored .beads dir — the
   strongest match for the observed /tmp/beads-bd-tests-* leaks),
   cmd/bd/doctor's e2e suite (its own testBDDir), cmd/bd/doctor/fix and
   internal/storage/dolt (each given a new suite-owned temp root created
   for this purpose), internal/beads (its own suite root), and a new
   TestMain for internal/doltserver's integration-tagged tests, which
   previously had none at all despite calling doltserver.Start directly
   against t.TempDir() (this one passes no root and relies solely on the
   deleted-cwd signal). Non-Linux platforms get a no-op stub so all call
   sites compile unchanged everywhere.

Agent-Signature: claude-sonnet-5-high on behalf of matt wilkie

* fix(doltserver): decouple Pdeathsig from BEADS_TEST_MODE, revalidate PID before sweep kills

Two P2 hardening gaps in the leaked-test-server reap logic added by this
PR:

1. procAttrDetached() gated Pdeathsig on the general-purpose
   BEADS_TEST_MODE=1 flag, which is inherited across exec boundaries.
   When a test execs a short-lived `bd` subprocess with BEADS_TEST_MODE=1
   (e.g. `bd dolt start`, `bd init --server`, the config-apply restart),
   Pdeathsig tied the newly detached dolt sql-server's life to that
   subprocess, SIGTERMing it the moment the subprocess exited by design
   and breaking any later `bd` invocation in the same test run. Split
   this into a narrower BEADS_TEST_PDEATHSIG=1 flag, set only by the two
   TestMains that actually call doltserver.Start directly in-process and
   stay alive for the server's lifetime (internal/doltserver,
   internal/storage/dolt). Auto-started servers were already unaffected
   (resolveAutoStart disables auto-start under test mode).

2. SweepOrphanedTestServers sent SIGKILL to every PID that accepted the
   initial SIGTERM without rechecking it still referred to the same
   process, leaving a PID-reuse race window. Added isDoltServerProcess,
   which re-reads /proc/<pid>/cmdline to revalidate identity immediately
   before both the initial SIGTERM and the follow-up SIGKILL.

Addresses Codex cross-vendor review finding (2026-07-07) on
gastownhall/beads#4592.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
2026-07-07 12:59:07 -07:00

56 lines
2.7 KiB
Go

//go:build linux
package doltserver
import (
"os"
"syscall"
)
// procAttrDetached returns SysProcAttr to detach a child process from the
// parent process group so it survives parent exit. This detachment is a
// deliberate production feature: a shared dolt sql-server must outlive the
// `bd` process that started it, so other `bd` invocations (and other
// terminals) can reuse it.
//
// When BEADS_TEST_PDEATHSIG=1, this additionally sets Pdeathsig: SIGTERM so
// the detached server is asked to exit if its parent (a test binary) dies
// before calling Stop() — e.g. when a test run is SIGKILLed or its process
// group is torn down uncleanly. Without this, a detached test server has no
// way to notice its parent is gone and leaks indefinitely, still holding
// its now-deleted temp data directory open (gastownhall/beads mybd-q6cz).
//
// BEADS_TEST_PDEATHSIG is a narrower, separate flag from BEADS_TEST_MODE —
// set only by test binaries that call Start directly and stay alive for the
// server's whole lifetime (internal/doltserver's own integration TestMain,
// internal/storage/dolt's TestMain). It is deliberately NOT tied to the
// general-purpose BEADS_TEST_MODE=1, which is inherited across exec
// boundaries for many other reasons (deterministic ports, auto-start
// disabling, temp-db naming, ...) and is present in short-lived `bd`
// subprocesses that explicit lifecycle commands spawn on purpose — `bd dolt
// start`, `bd init --server`, the config-apply restart — where the whole
// point is for the detached server to outlive that subprocess. Gating on
// BEADS_TEST_MODE there previously SIGTERMed those servers the moment the
// short-lived subprocess exited, breaking any later `bd` invocation in the
// same test run (see gastownhall/beads#4592 review thread, 2026-07-07).
//
// This is deliberately gated to test mode only: production behavior
// (BEADS_TEST_PDEATHSIG unset) is byte-identical to procAttrDetached before
// this change (Setpgid: true, no Pdeathsig) — the whole point of the
// detached process group is to survive the parent, and Pdeathsig would
// defeat that for real shared servers.
//
// Caveat: Pdeathsig delivers when the specific *thread* that issued the
// clone(2)/fork+exec (i.e., the OS thread running cmd.Start()) exits, not
// necessarily when the whole Go process exits — the Go runtime can retire
// that thread independently of process lifetime. It is therefore a
// best-effort safety net for tests, not a guaranteed kill switch; the
// suite-exit sweep (see sweep_linux.go) is the backstop for cases where
// Pdeathsig does not fire.
func procAttrDetached() *syscall.SysProcAttr {
attr := &syscall.SysProcAttr{Setpgid: true}
if os.Getenv("BEADS_TEST_PDEATHSIG") == "1" {
attr.Pdeathsig = syscall.SIGTERM
}
return attr
}