* fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) A 2026-07-05 dev-machine audit found 13 orphaned `dolt sql-server` processes, ages up to 3+ days, each serving a now-deleted /tmp test tempdir. detached dolt servers (Setpgid, no Pdeathsig) are a deliberate production feature — a shared server must outlive the `bd` process that started it — but that same detachment means a SIGKILLed test binary (or its Relay wrapper) leaves its embedded server running forever with nothing to reap it. Two defense-in-depth pieces, both scoped to Linux (the leak evidence) and gated so production behavior is untouched: 1. Test-gated Pdeathsig (internal/doltserver/procattr_linux.go). Split procAttrDetached() out of doltserver_unix.go into a Linux file and a non-Linux-unix file, since syscall.SysProcAttr.Pdeathsig only exists on Linux. When BEADS_TEST_MODE=1, the detached server now also gets Pdeathsig: SIGTERM, so it exits when the thread that spawned it dies. Production (BEADS_TEST_MODE unset) stays byte-identical: Setpgid: true only. Windows keeps its existing procAttrDetached untouched. Pdeathsig fires on death of the spawning *thread*, which the Go runtime can retire independently of the process — so this is documented as a best-effort test-only backstop, not a guarantee. 2. Suite-exit sweep (internal/doltserver/sweep*.go). A pure selection function (selectOrphanTestServerPIDs, unit tested) decides which `dolt sql-server` processes are provably test debris: cmdline matches dolt+sql-server AND either its working directory has been deleted (Linux's /proc/<pid>/cwd grows a " (deleted)" suffix) or sits under a suite-scoped root the caller explicitly vouches for as its own (e.g. that suite's own testTempRoot). SweepOrphanedTestServers never defaults to a global root like os.TempDir(): scripts/test.sh runs packages in parallel (-p N), and virtually every suite's data dir lives somewhere under os.TempDir() too, so a global root would make one suite's TestMain sweep SIGTERM other suites' still-live servers mid-run. A production shared server's data directory is never a caller-scoped test root and never gets deleted out from under it, so it can never match either condition. SweepOrphanedTestServers() is wired into the TestMains most likely to leak: cmd/bd's main CLI suite (which runs a real `bd` binary with auto-start against its own testTempRoot-anchored .beads dir — the strongest match for the observed /tmp/beads-bd-tests-* leaks), cmd/bd/doctor's e2e suite (its own testBDDir), cmd/bd/doctor/fix and internal/storage/dolt (each given a new suite-owned temp root created for this purpose), internal/beads (its own suite root), and a new TestMain for internal/doltserver's integration-tagged tests, which previously had none at all despite calling doltserver.Start directly against t.TempDir() (this one passes no root and relies solely on the deleted-cwd signal). Non-Linux platforms get a no-op stub so all call sites compile unchanged everywhere. Agent-Signature: claude-sonnet-5-high on behalf of matt wilkie * fix(doltserver): decouple Pdeathsig from BEADS_TEST_MODE, revalidate PID before sweep kills Two P2 hardening gaps in the leaked-test-server reap logic added by this PR: 1. procAttrDetached() gated Pdeathsig on the general-purpose BEADS_TEST_MODE=1 flag, which is inherited across exec boundaries. When a test execs a short-lived `bd` subprocess with BEADS_TEST_MODE=1 (e.g. `bd dolt start`, `bd init --server`, the config-apply restart), Pdeathsig tied the newly detached dolt sql-server's life to that subprocess, SIGTERMing it the moment the subprocess exited by design and breaking any later `bd` invocation in the same test run. Split this into a narrower BEADS_TEST_PDEATHSIG=1 flag, set only by the two TestMains that actually call doltserver.Start directly in-process and stay alive for the server's lifetime (internal/doltserver, internal/storage/dolt). Auto-started servers were already unaffected (resolveAutoStart disables auto-start under test mode). 2. SweepOrphanedTestServers sent SIGKILL to every PID that accepted the initial SIGTERM without rechecking it still referred to the same process, leaving a PID-reuse race window. Added isDoltServerProcess, which re-reads /proc/<pid>/cmdline to revalidate identity immediately before both the initial SIGTERM and the follow-up SIGKILL. Addresses Codex cross-vendor review finding (2026-07-07) on gastownhall/beads#4592. Agent-Signature: claude-fable-5-high on behalf of matt wilkie
56 lines
2.7 KiB
Go
56 lines
2.7 KiB
Go
//go:build linux
|
|
|
|
package doltserver
|
|
|
|
import (
|
|
"os"
|
|
"syscall"
|
|
)
|
|
|
|
// procAttrDetached returns SysProcAttr to detach a child process from the
|
|
// parent process group so it survives parent exit. This detachment is a
|
|
// deliberate production feature: a shared dolt sql-server must outlive the
|
|
// `bd` process that started it, so other `bd` invocations (and other
|
|
// terminals) can reuse it.
|
|
//
|
|
// When BEADS_TEST_PDEATHSIG=1, this additionally sets Pdeathsig: SIGTERM so
|
|
// the detached server is asked to exit if its parent (a test binary) dies
|
|
// before calling Stop() — e.g. when a test run is SIGKILLed or its process
|
|
// group is torn down uncleanly. Without this, a detached test server has no
|
|
// way to notice its parent is gone and leaks indefinitely, still holding
|
|
// its now-deleted temp data directory open (gastownhall/beads mybd-q6cz).
|
|
//
|
|
// BEADS_TEST_PDEATHSIG is a narrower, separate flag from BEADS_TEST_MODE —
|
|
// set only by test binaries that call Start directly and stay alive for the
|
|
// server's whole lifetime (internal/doltserver's own integration TestMain,
|
|
// internal/storage/dolt's TestMain). It is deliberately NOT tied to the
|
|
// general-purpose BEADS_TEST_MODE=1, which is inherited across exec
|
|
// boundaries for many other reasons (deterministic ports, auto-start
|
|
// disabling, temp-db naming, ...) and is present in short-lived `bd`
|
|
// subprocesses that explicit lifecycle commands spawn on purpose — `bd dolt
|
|
// start`, `bd init --server`, the config-apply restart — where the whole
|
|
// point is for the detached server to outlive that subprocess. Gating on
|
|
// BEADS_TEST_MODE there previously SIGTERMed those servers the moment the
|
|
// short-lived subprocess exited, breaking any later `bd` invocation in the
|
|
// same test run (see gastownhall/beads#4592 review thread, 2026-07-07).
|
|
//
|
|
// This is deliberately gated to test mode only: production behavior
|
|
// (BEADS_TEST_PDEATHSIG unset) is byte-identical to procAttrDetached before
|
|
// this change (Setpgid: true, no Pdeathsig) — the whole point of the
|
|
// detached process group is to survive the parent, and Pdeathsig would
|
|
// defeat that for real shared servers.
|
|
//
|
|
// Caveat: Pdeathsig delivers when the specific *thread* that issued the
|
|
// clone(2)/fork+exec (i.e., the OS thread running cmd.Start()) exits, not
|
|
// necessarily when the whole Go process exits — the Go runtime can retire
|
|
// that thread independently of process lifetime. It is therefore a
|
|
// best-effort safety net for tests, not a guaranteed kill switch; the
|
|
// suite-exit sweep (see sweep_linux.go) is the backstop for cases where
|
|
// Pdeathsig does not fire.
|
|
func procAttrDetached() *syscall.SysProcAttr {
|
|
attr := &syscall.SysProcAttr{Setpgid: true}
|
|
if os.Getenv("BEADS_TEST_PDEATHSIG") == "1" {
|
|
attr.Pdeathsig = syscall.SIGTERM
|
|
}
|
|
return attr
|
|
}
|