beads/internal/doltserver/procattr_linux_test.go
matt wilkie 993ca8d7c6
fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592)
* fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep)

A 2026-07-05 dev-machine audit found 13 orphaned `dolt sql-server`
processes, ages up to 3+ days, each serving a now-deleted /tmp test
tempdir. detached dolt servers (Setpgid, no Pdeathsig) are a deliberate
production feature — a shared server must outlive the `bd` process that
started it — but that same detachment means a SIGKILLed test binary (or
its Relay wrapper) leaves its embedded server running forever with
nothing to reap it.

Two defense-in-depth pieces, both scoped to Linux (the leak evidence)
and gated so production behavior is untouched:

1. Test-gated Pdeathsig (internal/doltserver/procattr_linux.go). Split
   procAttrDetached() out of doltserver_unix.go into a Linux file and a
   non-Linux-unix file, since syscall.SysProcAttr.Pdeathsig only exists
   on Linux. When BEADS_TEST_MODE=1, the detached server now also gets
   Pdeathsig: SIGTERM, so it exits when the thread that spawned it dies.
   Production (BEADS_TEST_MODE unset) stays byte-identical: Setpgid:
   true only. Windows keeps its existing procAttrDetached untouched.
   Pdeathsig fires on death of the spawning *thread*, which the Go
   runtime can retire independently of the process — so this is
   documented as a best-effort test-only backstop, not a guarantee.

2. Suite-exit sweep (internal/doltserver/sweep*.go). A pure selection
   function (selectOrphanTestServerPIDs, unit tested) decides which
   `dolt sql-server` processes are provably test debris: cmdline matches
   dolt+sql-server AND either its working directory has been deleted
   (Linux's /proc/<pid>/cwd grows a " (deleted)" suffix) or sits under a
   suite-scoped root the caller explicitly vouches for as its own (e.g.
   that suite's own testTempRoot). SweepOrphanedTestServers never
   defaults to a global root like os.TempDir(): scripts/test.sh runs
   packages in parallel (-p N), and virtually every suite's data dir
   lives somewhere under os.TempDir() too, so a global root would make
   one suite's TestMain sweep SIGTERM other suites' still-live servers
   mid-run. A production shared server's data directory is never a
   caller-scoped test root and never gets deleted out from under it, so
   it can never match either condition.

   SweepOrphanedTestServers() is wired into the TestMains most likely to
   leak: cmd/bd's main CLI suite (which runs a real `bd` binary with
   auto-start against its own testTempRoot-anchored .beads dir — the
   strongest match for the observed /tmp/beads-bd-tests-* leaks),
   cmd/bd/doctor's e2e suite (its own testBDDir), cmd/bd/doctor/fix and
   internal/storage/dolt (each given a new suite-owned temp root created
   for this purpose), internal/beads (its own suite root), and a new
   TestMain for internal/doltserver's integration-tagged tests, which
   previously had none at all despite calling doltserver.Start directly
   against t.TempDir() (this one passes no root and relies solely on the
   deleted-cwd signal). Non-Linux platforms get a no-op stub so all call
   sites compile unchanged everywhere.

Agent-Signature: claude-sonnet-5-high on behalf of matt wilkie

* fix(doltserver): decouple Pdeathsig from BEADS_TEST_MODE, revalidate PID before sweep kills

Two P2 hardening gaps in the leaked-test-server reap logic added by this
PR:

1. procAttrDetached() gated Pdeathsig on the general-purpose
   BEADS_TEST_MODE=1 flag, which is inherited across exec boundaries.
   When a test execs a short-lived `bd` subprocess with BEADS_TEST_MODE=1
   (e.g. `bd dolt start`, `bd init --server`, the config-apply restart),
   Pdeathsig tied the newly detached dolt sql-server's life to that
   subprocess, SIGTERMing it the moment the subprocess exited by design
   and breaking any later `bd` invocation in the same test run. Split
   this into a narrower BEADS_TEST_PDEATHSIG=1 flag, set only by the two
   TestMains that actually call doltserver.Start directly in-process and
   stay alive for the server's lifetime (internal/doltserver,
   internal/storage/dolt). Auto-started servers were already unaffected
   (resolveAutoStart disables auto-start under test mode).

2. SweepOrphanedTestServers sent SIGKILL to every PID that accepted the
   initial SIGTERM without rechecking it still referred to the same
   process, leaving a PID-reuse race window. Added isDoltServerProcess,
   which re-reads /proc/<pid>/cmdline to revalidate identity immediately
   before both the initial SIGTERM and the follow-up SIGKILL.

Addresses Codex cross-vendor review finding (2026-07-07) on
gastownhall/beads#4592.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
2026-07-07 12:59:07 -07:00

58 lines
2 KiB
Go

//go:build linux
package doltserver
import (
"syscall"
"testing"
)
// TestProcAttrDetached_Pdeathsig verifies procAttrDetached only sets
// Pdeathsig when BEADS_TEST_PDEATHSIG=1, and never changes Setpgid — the
// production detach behavior must stay byte-identical regardless of test
// mode (gastownhall/beads mybd-q6cz). It also verifies the general-purpose
// BEADS_TEST_MODE flag no longer controls Pdeathsig on its own: tying it to
// that broader flag previously SIGTERMed servers started by short-lived `bd`
// subprocesses (e.g. `bd dolt start`) that inherit BEADS_TEST_MODE=1 from an
// exec'ing test but are not themselves the process that should own the
// server's lifetime (gastownhall/beads#4592 review thread, 2026-07-07).
func TestProcAttrDetached_Pdeathsig(t *testing.T) {
t.Run("BEADS_TEST_PDEATHSIG=1 sets Pdeathsig=SIGTERM", func(t *testing.T) {
t.Setenv("BEADS_TEST_PDEATHSIG", "1")
attr := procAttrDetached()
if !attr.Setpgid {
t.Error("Setpgid = false, want true")
}
if attr.Pdeathsig != syscall.SIGTERM {
t.Errorf("Pdeathsig = %v, want SIGTERM", attr.Pdeathsig)
}
})
t.Run("production (unset) has no Pdeathsig", func(t *testing.T) {
t.Setenv("BEADS_TEST_PDEATHSIG", "")
attr := procAttrDetached()
if !attr.Setpgid {
t.Error("Setpgid = false, want true")
}
if attr.Pdeathsig != 0 {
t.Errorf("Pdeathsig = %v, want 0 (unset)", attr.Pdeathsig)
}
})
t.Run("BEADS_TEST_PDEATHSIG=0 has no Pdeathsig", func(t *testing.T) {
t.Setenv("BEADS_TEST_PDEATHSIG", "0")
attr := procAttrDetached()
if attr.Pdeathsig != 0 {
t.Errorf("Pdeathsig = %v, want 0 (unset)", attr.Pdeathsig)
}
})
t.Run("BEADS_TEST_MODE=1 alone (BEADS_TEST_PDEATHSIG unset) has no Pdeathsig", func(t *testing.T) {
t.Setenv("BEADS_TEST_MODE", "1")
t.Setenv("BEADS_TEST_PDEATHSIG", "")
attr := procAttrDetached()
if attr.Pdeathsig != 0 {
t.Errorf("Pdeathsig = %v, want 0 (unset) — BEADS_TEST_MODE alone must not gate Pdeathsig", attr.Pdeathsig)
}
})
}