beads/internal/doltserver
matt wilkie 993ca8d7c6
fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592)
* fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep)

A 2026-07-05 dev-machine audit found 13 orphaned `dolt sql-server`
processes, ages up to 3+ days, each serving a now-deleted /tmp test
tempdir. detached dolt servers (Setpgid, no Pdeathsig) are a deliberate
production feature — a shared server must outlive the `bd` process that
started it — but that same detachment means a SIGKILLed test binary (or
its Relay wrapper) leaves its embedded server running forever with
nothing to reap it.

Two defense-in-depth pieces, both scoped to Linux (the leak evidence)
and gated so production behavior is untouched:

1. Test-gated Pdeathsig (internal/doltserver/procattr_linux.go). Split
   procAttrDetached() out of doltserver_unix.go into a Linux file and a
   non-Linux-unix file, since syscall.SysProcAttr.Pdeathsig only exists
   on Linux. When BEADS_TEST_MODE=1, the detached server now also gets
   Pdeathsig: SIGTERM, so it exits when the thread that spawned it dies.
   Production (BEADS_TEST_MODE unset) stays byte-identical: Setpgid:
   true only. Windows keeps its existing procAttrDetached untouched.
   Pdeathsig fires on death of the spawning *thread*, which the Go
   runtime can retire independently of the process — so this is
   documented as a best-effort test-only backstop, not a guarantee.

2. Suite-exit sweep (internal/doltserver/sweep*.go). A pure selection
   function (selectOrphanTestServerPIDs, unit tested) decides which
   `dolt sql-server` processes are provably test debris: cmdline matches
   dolt+sql-server AND either its working directory has been deleted
   (Linux's /proc/<pid>/cwd grows a " (deleted)" suffix) or sits under a
   suite-scoped root the caller explicitly vouches for as its own (e.g.
   that suite's own testTempRoot). SweepOrphanedTestServers never
   defaults to a global root like os.TempDir(): scripts/test.sh runs
   packages in parallel (-p N), and virtually every suite's data dir
   lives somewhere under os.TempDir() too, so a global root would make
   one suite's TestMain sweep SIGTERM other suites' still-live servers
   mid-run. A production shared server's data directory is never a
   caller-scoped test root and never gets deleted out from under it, so
   it can never match either condition.

   SweepOrphanedTestServers() is wired into the TestMains most likely to
   leak: cmd/bd's main CLI suite (which runs a real `bd` binary with
   auto-start against its own testTempRoot-anchored .beads dir — the
   strongest match for the observed /tmp/beads-bd-tests-* leaks),
   cmd/bd/doctor's e2e suite (its own testBDDir), cmd/bd/doctor/fix and
   internal/storage/dolt (each given a new suite-owned temp root created
   for this purpose), internal/beads (its own suite root), and a new
   TestMain for internal/doltserver's integration-tagged tests, which
   previously had none at all despite calling doltserver.Start directly
   against t.TempDir() (this one passes no root and relies solely on the
   deleted-cwd signal). Non-Linux platforms get a no-op stub so all call
   sites compile unchanged everywhere.

Agent-Signature: claude-sonnet-5-high on behalf of matt wilkie

* fix(doltserver): decouple Pdeathsig from BEADS_TEST_MODE, revalidate PID before sweep kills

Two P2 hardening gaps in the leaked-test-server reap logic added by this
PR:

1. procAttrDetached() gated Pdeathsig on the general-purpose
   BEADS_TEST_MODE=1 flag, which is inherited across exec boundaries.
   When a test execs a short-lived `bd` subprocess with BEADS_TEST_MODE=1
   (e.g. `bd dolt start`, `bd init --server`, the config-apply restart),
   Pdeathsig tied the newly detached dolt sql-server's life to that
   subprocess, SIGTERMing it the moment the subprocess exited by design
   and breaking any later `bd` invocation in the same test run. Split
   this into a narrower BEADS_TEST_PDEATHSIG=1 flag, set only by the two
   TestMains that actually call doltserver.Start directly in-process and
   stay alive for the server's lifetime (internal/doltserver,
   internal/storage/dolt). Auto-started servers were already unaffected
   (resolveAutoStart disables auto-start under test mode).

2. SweepOrphanedTestServers sent SIGKILL to every PID that accepted the
   initial SIGTERM without rechecking it still referred to the same
   process, leaving a PID-reuse race window. Added isDoltServerProcess,
   which re-reads /proc/<pid>/cmdline to revalidate identity immediately
   before both the initial SIGTERM and the follow-up SIGKILL.

Addresses Codex cross-vendor review finding (2026-07-07) on
gastownhall/beads#4592.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
2026-07-07 12:59:07 -07:00
..
dirty_state_test.go test: add multi-process concurrency and server lifecycle integration tests 2026-03-22 15:01:45 -07:00
doltserver.go fix: enrich port-conflict diagnostics with docker / external-server hint (#3516) (#3564) 2026-07-07 07:54:31 -07:00
doltserver_test.go fix: enrich port-conflict diagnostics with docker / external-server hint (#3516) (#3564) 2026-07-07 07:54:31 -07:00
doltserver_unix.go fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592) 2026-07-07 12:59:07 -07:00
doltserver_windows.go fix: Windows compatibility for doltserver package 2026-02-24 13:17:23 -05:00
lifecycle_integration_test.go Add PR CI wrapper command surface (#4211) 2026-05-29 18:52:53 -07:00
logrotate.go fix(doltserver): rotate dolt-server.log at startup to cap size (#3161) 2026-04-10 21:52:37 -07:00
logrotate_test.go fix(doltserver): rotate dolt-server.log at startup to cap size (#3161) 2026-04-10 21:52:37 -07:00
manifest_recovery.go fix(doltserver): gate corrupt-manifest .dolt reinit behind explicit bd doctor --fix (bd-6dnrw.6) 2026-06-11 00:30:13 -04:00
manifest_recovery_test.go fix(doltserver): gate corrupt-manifest .dolt reinit behind explicit bd doctor --fix (bd-6dnrw.6) 2026-06-11 00:30:13 -04:00
port_race_test.go test: add multi-process concurrency and server lifecycle integration tests 2026-03-22 15:01:45 -07:00
procattr_linux.go fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592) 2026-07-07 12:59:07 -07:00
procattr_linux_test.go fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592) 2026-07-07 12:59:07 -07:00
procattr_other_unix.go fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592) 2026-07-07 12:59:07 -07:00
servermode.go fix: treat --shared-server as server mode for DoltMode (#2947) 2026-04-02 11:52:20 -07:00
socket_integration_test.go Add PR CI wrapper command surface (#4211) 2026-05-29 18:52:53 -07:00
sweep.go fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592) 2026-07-07 12:59:07 -07:00
sweep_linux.go fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592) 2026-07-07 12:59:07 -07:00
sweep_other.go fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592) 2026-07-07 12:59:07 -07:00
sweep_test.go fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592) 2026-07-07 12:59:07 -07:00
testmain_integration_test.go fix(doltserver): reap leaked test dolt sql-servers (test-gated Pdeathsig + suite-exit sweep) (#4592) 2026-07-07 12:59:07 -07:00