* fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) A 2026-07-05 dev-machine audit found 13 orphaned `dolt sql-server` processes, ages up to 3+ days, each serving a now-deleted /tmp test tempdir. detached dolt servers (Setpgid, no Pdeathsig) are a deliberate production feature — a shared server must outlive the `bd` process that started it — but that same detachment means a SIGKILLed test binary (or its Relay wrapper) leaves its embedded server running forever with nothing to reap it. Two defense-in-depth pieces, both scoped to Linux (the leak evidence) and gated so production behavior is untouched: 1. Test-gated Pdeathsig (internal/doltserver/procattr_linux.go). Split procAttrDetached() out of doltserver_unix.go into a Linux file and a non-Linux-unix file, since syscall.SysProcAttr.Pdeathsig only exists on Linux. When BEADS_TEST_MODE=1, the detached server now also gets Pdeathsig: SIGTERM, so it exits when the thread that spawned it dies. Production (BEADS_TEST_MODE unset) stays byte-identical: Setpgid: true only. Windows keeps its existing procAttrDetached untouched. Pdeathsig fires on death of the spawning *thread*, which the Go runtime can retire independently of the process — so this is documented as a best-effort test-only backstop, not a guarantee. 2. Suite-exit sweep (internal/doltserver/sweep*.go). A pure selection function (selectOrphanTestServerPIDs, unit tested) decides which `dolt sql-server` processes are provably test debris: cmdline matches dolt+sql-server AND either its working directory has been deleted (Linux's /proc/<pid>/cwd grows a " (deleted)" suffix) or sits under a suite-scoped root the caller explicitly vouches for as its own (e.g. that suite's own testTempRoot). SweepOrphanedTestServers never defaults to a global root like os.TempDir(): scripts/test.sh runs packages in parallel (-p N), and virtually every suite's data dir lives somewhere under os.TempDir() too, so a global root would make one suite's TestMain sweep SIGTERM other suites' still-live servers mid-run. A production shared server's data directory is never a caller-scoped test root and never gets deleted out from under it, so it can never match either condition. SweepOrphanedTestServers() is wired into the TestMains most likely to leak: cmd/bd's main CLI suite (which runs a real `bd` binary with auto-start against its own testTempRoot-anchored .beads dir — the strongest match for the observed /tmp/beads-bd-tests-* leaks), cmd/bd/doctor's e2e suite (its own testBDDir), cmd/bd/doctor/fix and internal/storage/dolt (each given a new suite-owned temp root created for this purpose), internal/beads (its own suite root), and a new TestMain for internal/doltserver's integration-tagged tests, which previously had none at all despite calling doltserver.Start directly against t.TempDir() (this one passes no root and relies solely on the deleted-cwd signal). Non-Linux platforms get a no-op stub so all call sites compile unchanged everywhere. Agent-Signature: claude-sonnet-5-high on behalf of matt wilkie * fix(doltserver): decouple Pdeathsig from BEADS_TEST_MODE, revalidate PID before sweep kills Two P2 hardening gaps in the leaked-test-server reap logic added by this PR: 1. procAttrDetached() gated Pdeathsig on the general-purpose BEADS_TEST_MODE=1 flag, which is inherited across exec boundaries. When a test execs a short-lived `bd` subprocess with BEADS_TEST_MODE=1 (e.g. `bd dolt start`, `bd init --server`, the config-apply restart), Pdeathsig tied the newly detached dolt sql-server's life to that subprocess, SIGTERMing it the moment the subprocess exited by design and breaking any later `bd` invocation in the same test run. Split this into a narrower BEADS_TEST_PDEATHSIG=1 flag, set only by the two TestMains that actually call doltserver.Start directly in-process and stay alive for the server's lifetime (internal/doltserver, internal/storage/dolt). Auto-started servers were already unaffected (resolveAutoStart disables auto-start under test mode). 2. SweepOrphanedTestServers sent SIGKILL to every PID that accepted the initial SIGTERM without rechecking it still referred to the same process, leaving a PID-reuse race window. Added isDoltServerProcess, which re-reads /proc/<pid>/cmdline to revalidate identity immediately before both the initial SIGTERM and the follow-up SIGKILL. Addresses Codex cross-vendor review finding (2026-07-07) on gastownhall/beads#4592. Agent-Signature: claude-fable-5-high on behalf of matt wilkie
58 lines
2 KiB
Go
58 lines
2 KiB
Go
//go:build linux
|
|
|
|
package doltserver
|
|
|
|
import (
|
|
"syscall"
|
|
"testing"
|
|
)
|
|
|
|
// TestProcAttrDetached_Pdeathsig verifies procAttrDetached only sets
|
|
// Pdeathsig when BEADS_TEST_PDEATHSIG=1, and never changes Setpgid — the
|
|
// production detach behavior must stay byte-identical regardless of test
|
|
// mode (gastownhall/beads mybd-q6cz). It also verifies the general-purpose
|
|
// BEADS_TEST_MODE flag no longer controls Pdeathsig on its own: tying it to
|
|
// that broader flag previously SIGTERMed servers started by short-lived `bd`
|
|
// subprocesses (e.g. `bd dolt start`) that inherit BEADS_TEST_MODE=1 from an
|
|
// exec'ing test but are not themselves the process that should own the
|
|
// server's lifetime (gastownhall/beads#4592 review thread, 2026-07-07).
|
|
func TestProcAttrDetached_Pdeathsig(t *testing.T) {
|
|
t.Run("BEADS_TEST_PDEATHSIG=1 sets Pdeathsig=SIGTERM", func(t *testing.T) {
|
|
t.Setenv("BEADS_TEST_PDEATHSIG", "1")
|
|
attr := procAttrDetached()
|
|
if !attr.Setpgid {
|
|
t.Error("Setpgid = false, want true")
|
|
}
|
|
if attr.Pdeathsig != syscall.SIGTERM {
|
|
t.Errorf("Pdeathsig = %v, want SIGTERM", attr.Pdeathsig)
|
|
}
|
|
})
|
|
|
|
t.Run("production (unset) has no Pdeathsig", func(t *testing.T) {
|
|
t.Setenv("BEADS_TEST_PDEATHSIG", "")
|
|
attr := procAttrDetached()
|
|
if !attr.Setpgid {
|
|
t.Error("Setpgid = false, want true")
|
|
}
|
|
if attr.Pdeathsig != 0 {
|
|
t.Errorf("Pdeathsig = %v, want 0 (unset)", attr.Pdeathsig)
|
|
}
|
|
})
|
|
|
|
t.Run("BEADS_TEST_PDEATHSIG=0 has no Pdeathsig", func(t *testing.T) {
|
|
t.Setenv("BEADS_TEST_PDEATHSIG", "0")
|
|
attr := procAttrDetached()
|
|
if attr.Pdeathsig != 0 {
|
|
t.Errorf("Pdeathsig = %v, want 0 (unset)", attr.Pdeathsig)
|
|
}
|
|
})
|
|
|
|
t.Run("BEADS_TEST_MODE=1 alone (BEADS_TEST_PDEATHSIG unset) has no Pdeathsig", func(t *testing.T) {
|
|
t.Setenv("BEADS_TEST_MODE", "1")
|
|
t.Setenv("BEADS_TEST_PDEATHSIG", "")
|
|
attr := procAttrDetached()
|
|
if attr.Pdeathsig != 0 {
|
|
t.Errorf("Pdeathsig = %v, want 0 (unset) — BEADS_TEST_MODE alone must not gate Pdeathsig", attr.Pdeathsig)
|
|
}
|
|
})
|
|
}
|