beads/cmd/bd/shared_server_cleanup_test.go
Julian Knutsen 4cf36b0776
fix(doltserver): auto-started dolt must own its port; move a taken bd-chosen port (#7201)
* refactor(doltserver): share dolt's startup-output watcher with the plain server launcher

Move the ready-line / port-in-use watcher that the proxied DoltServer uses
(startupWatch, doltReadyLine, doltPortInUseTexts, logLevelEmitsReadyLine,
ErrPortInUse) into internal/doltserver as StartupWatch and friends, so the
plain server-mode Start can read the same signals. dbproxy/server already
imports internal/doltserver, so it now uses them from there; its
ErrPortInUse is the same error value. No behavior change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(doltserver): prove the auto-started dolt owns its port; move a taken ephemeral port

The plain server-mode Start (bd init --server, auto-start, bd dolt start)
had the same blind readiness check the managed-local proxied backend had
before #7184, plus a blind exit check:

- It picked an ephemeral port, closed the probe listener and launched dolt,
  then slept 200ms and asked whether the child was still alive. dolt only
  reaches its port check after its own startup, which under load takes
  longer than that, so a lost port was never caught there. Worse, the child
  had been Release()d unreaped, and a zombie still answers kill(pid, 0), so
  an exit was invisible for the rest of the bd process.
- It then waited for any MySQL greeting on the port. If another process
  took the port in between, either it was mute (a plain listener) and Start
  ran out its ready timeout -- the direct-local-server auto-start timeout seen
  under port churn -- or it was another dolt, and Start reported success
  against a server it did not start.

Now each attempt launches the child with a goroutine that waits on (and
reaps) it, and awaitOwnedListener accepts a greeting only once the
listener is shown to be the child's: dolt's ready line in the child's
output when its log level emits it (debug), or else the listening socket
belonging to the child's process tree (Linux, via /proc/net/tcp and
/proc/<pid>/fd). The child's output is read from the server log past the
offset it started at, through the StartupWatch shared with dbproxy, so
"Port N already in use." (printed at log_level warning too) ends the
attempt at once as ErrPortInUse, and a child exit ends it as a failure.

Port policy follows #7184's contract: only a port bd chose itself moves.
An ephemeral port that is lost is retried on a fresh one (up to the
existing 10 attempts); an explicitly configured port (env, config.yaml,
metadata, dolt config) stays pinned and fails with ErrPortInUse and a
remedy. Nothing operator-owned is rewritten: the generated
dolt-server-config.yaml is bd's own state file and was already rendered
per attempt. Where ownership cannot be proven (non-Linux, quiet log
level) the greeting decides as before, but the child's exit and
port-in-use report still end the wait.

Tests: a fake dolt (this test binary, behind a PATH shim) that spends
700ms starting up and then binds like dolt does. A foreign greeter on the
port is not accepted (ErrPortInUse); the child's own listener is; a child
exit is reported promptly; and the real Start, handed a taken first
ephemeral port, comes up on a fresh one (fails before this change: Start
adopted the foreign port).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(doltserver): isolate user config in the DefaultConfig tests

Four DefaultConfig tests failed under -tags integration on a host whose
user config sets dolt.shared-server: true. The integration TestMain, unlike
the plain one, does not redirect HOME, so:
- TestDefaultConfigPrecedenceChain and TestDefaultConfigPortSource call
  config.Initialize, which merged ~/.beads/config.yaml.
- TestDefaultConfigReturnsZeroForStandalone and
  TestDefaultConfigPortFileTakesPrecedence read whatever config an earlier
  test had loaded from the real HOME.
Shared mode then sent DefaultConfig to the shared server's state dir and
its port file ("port=3308 source=port_file, want 5002/config_yaml").

isolateUserConfig points HOME, USERPROFILE and XDG_CONFIG_HOME at an empty
dir, the way TestDefaultConfig_NonzeroPortAlwaysHasSource already does.
It also resets the loaded config before the test and again after it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(doltserver): auto-start review fixes: record the server at launch, sharper ownership, port policy by source

Follow-up to 0acb1041b, addressing review.

B1: write the PID and port files again right after each launch, as before
that commit, and remove them on every failure path (port conflict,
not-ready, remotesapi). Previously a bd interrupted during the readiness
wait left a dolt holding the database lock that nothing could find. The
next start, bd dolt stop and killall all saw nothing. Both files carry the
real port, so IsRunning never stops a starting server as "port unknown".

S1: a child gone from /proc is known not to own the port; it is no longer
"unknown", which would let a foreign greeting through. The child's log is
re-read on every check, not only after its exit has been observed, because
the reaper goroutine can lag the death.

S2: if no listener on the port is listed although one just answered, /proc
is incomplete (WSL1, some sandboxes, forwarding). That is now "unknown"
(fallback to the greeting), not "foreign". Also "unknown": a /proc that is
not this PID namespace's.

S3:
- bd's port-file port moves like an ephemeral one outside shared-server
  mode, since bd chose it (GH#4052).
- Operator-configured ports and the shared-server port stay pinned.
- The pinned error now says where the port came from (env, config.yaml,
  dolt config, metadata, --server-port, shared default, shared port file)
  instead of always "configured explicitly".

S4: a killed child is waited for (up to 5s) before the next attempt or the
return, so it has released the database lock. Its process group is killed
too (unix), which reaches a dolt behind a non-exec wrapper.

S5: when /proc proves the listener belongs to another process, the wait
returns ErrPortInUse at once. It no longer sits out a slow child, which
could outlast the timeout and turn a movable port into a hard failure.

NITs:
- Debug-mode timeout names the missing ready line and the way out.
- The log tail restarts if the log is truncated under it.
- waitForReady is removed; its probe tests now drive awaitOwnedListener.
- Stale daemonization comment fixed in cmd/bd.
- CONFIG_PROC_CHILDREN caveat documented.

Tests:
- fake-/proc ownership cases: owned, foreign, child gone, no listener row,
  unreadable fd table;
- /proc/net/tcp parsing;
- proven-foreign ends the wait at once;
- port-in-use in the log counts before the exit is seen;
- ready-line proof, with and without the line;
- Start records PID+port while waiting;
- Start removes them after a failed start;
- pinned env port: one launch, ErrPortInUse, source in the message;
- port-file port moves;
- remedy wording per source.
The Start tests now isolate the user config instead of skipping.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* test(doltserver): de-flake two startup tests under load

- TestStart_RecoversWhenEphemeralPortIsTaken: on Linux, /proc proves the
  taken port foreign as soon as the greeting arrives, so the first child
  can be killed before the fake records its launch. Assert two port
  allocations and the final launch instead of the first launch.
- TestAwaitOwnedListener_ReadyLine: give the "no ready line" case 8s, so
  the fake has time to start and greet on a loaded host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 09:22:03 -07:00

199 lines
7.4 KiB
Go

package main
import (
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/steveyegge/beads/internal/doltserver"
"github.com/steveyegge/beads/internal/procid"
"github.com/steveyegge/beads/internal/storage/dbproxy/proxy"
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
"github.com/steveyegge/beads/internal/testutil"
)
// stopSharedServerCleanup is the cleanup every fixture in this package that
// runs a subprocess `bd init` with BEADS_DOLT_SHARED_SERVER=1 owes.
//
// That init takes the shared-global branch in init.go, which calls
// doltserver.Start(sharedDir) — and Start DAEMONIZES: the server is its own
// process group (Setpgid) with cmd.Dir = <sharedDir>/dolt, and outlives bd (bd
// only reaps it if it exits while bd is still running). The `bd`
// subprocess then exits and the sql-server keeps running, serving a directory
// under the suite's temp HOME that TestMain deletes on the way out. Nothing in
// those fixtures ever stopped it, so a full cmd/bd run finished with a live
// `dolt sql-server --config <suite root>/.beads/shared-server/…` still in the
// process table — the leak this bead is about (wy-j2zc8q).
//
// Because every one of those fixtures resolves the SAME $HOME/.beads/
// shared-server, only the last one to start a server leaves a survivor, which
// is why the process table names one fixture while three carry the defect.
//
// Register it after the fixture's directories are built and BEFORE the first
// subprocess `bd` runs, so it covers the t.Fatalf paths too.
func stopSharedServerCleanup(t *testing.T) {
t.Helper()
// SharedServerPath, not SharedServerDir: resolving must not create the
// tree the command under test is supposed to create.
sharedDir, err := doltserver.SharedServerPath()
if err != nil {
t.Fatalf("resolve shared server dir: %v", err)
}
stopDoltServerCleanup(t, sharedDir)
}
// stopDoltServerCleanup is the same cleanup for a server whose pid file lives
// under an explicit beads directory: a per-project server under
// <repo>/.beads, or a shared server under a HOME the fixture hands ONLY to
// the subprocess (`cmd.Env = ...HOME=<tmp>...`), where this process's own
// SharedServerPath would resolve the wrong tree. Register it BEFORE the first
// subprocess `bd` runs; the directory need not exist yet.
func stopDoltServerCleanup(t *testing.T, beadsDir string) {
t.Helper()
sharedDir := beadsDir
t.Cleanup(func() {
// Read the record BEFORE stopping: a clean Stop removes the pid
// file, so afterwards there is nothing left to verify against.
state, err := doltserver.IsRunning(sharedDir)
if err != nil {
t.Errorf("doltserver.IsRunning(%s): %v", sharedDir, err)
return
}
if state == nil || !state.Running {
return
}
pid := state.PID
// The birth token makes the PID safe to ask about after the stop:
// the server is not this process's child, so its PID is reusable the
// instant it exits.
token, tokenErr := procid.Capture(pid)
if stopErr := doltserver.Stop(sharedDir); stopErr != nil &&
!errors.Is(stopErr, doltserver.ErrServerNotRunning) {
t.Errorf("doltserver.Stop(%s) for pid %d: %v", sharedDir, pid, stopErr)
}
if tokenErr != nil {
t.Logf("procid.Capture(%d): %v; skipping the exit check", pid, tokenErr)
return
}
requireSharedServerExited(t, pid, token)
})
}
// stopProxiedServerCleanup is the same duty for a subprocess `bd init
// --proxied-server`: that init leaves a proxy (proxy.pid) and its backend
// dolt sql-server (proxy-child.pid) running under <beadsDir>/dolt, with no
// dolt-server.pid for stopDoltServerCleanup to find — the post-run sweep
// named the metrics fixture's repo by exactly that cwd. proxy.Shutdown is
// the verified stop `bd dolt stop` performs and covers both records. A root
// that never started (the embedded and per-project cases of the same table)
// has neither record and is not an error.
func stopProxiedServerCleanup(t *testing.T, proxyRoot string) {
t.Helper()
t.Cleanup(func() {
recorded := false
for _, name := range []string{proxy.PIDFileName, server.PIDFileName} {
if _, err := os.Stat(filepath.Join(proxyRoot, name)); err == nil {
recorded = true
}
}
if !recorded {
return
}
if err := proxy.Shutdown(proxyRoot); err != nil {
t.Errorf("proxy.Shutdown(%s): %v", proxyRoot, err)
}
})
}
const (
// sharedServerExitTimeout is how long the cleanup waits for the shared
// dolt sql-server to leave the process table. doltserver.Stop's own
// gracefulStop budget is 5s, so this only ever elapses when the stop did
// not take — the outcome worth reporting.
sharedServerExitTimeout = 30 * time.Second
sharedServerExitPoll = 50 * time.Millisecond
)
// requireSharedServerExited fails the test if the recorded shared server is
// still running after doltserver.Stop, then force-kills it so the leak does
// not outlive the run.
//
// t.Errorf, not t.Logf: a surviving server holds the temp HOME the suite is
// about to delete, and reporting it where it is caused is the whole point —
// the previous machinery printed a stderr line and exited 0, which is how the
// same leak was fixed three times and rediscovered from a process table
// (wy-j2zc8q).
func requireSharedServerExited(t *testing.T, pid int, token procid.Token) {
t.Helper()
deadline := time.Now().Add(sharedServerExitTimeout)
for {
same, err := procid.Verify(pid, token)
if err != nil {
t.Logf("procid.Verify(%d): %v", pid, err)
return
}
if !same {
// Gone, or the PID now belongs to something else. Either way
// our server is not running.
return
}
if time.Now().After(deadline) {
break
}
time.Sleep(sharedServerExitPoll)
}
handle, err := procid.Open(pid, token)
if err != nil {
if sharedServerStillRunning(t, pid, token) {
t.Errorf("shared dolt sql-server pid %d survived Stop by more than %s and could not be opened to kill: %v",
pid, sharedServerExitTimeout, err)
}
return
}
killErr := handle.Kill()
_ = handle.Close()
if killErr != nil {
if sharedServerStillRunning(t, pid, token) {
t.Errorf("shared dolt sql-server pid %d survived Stop by more than %s and could not be killed: %v",
pid, sharedServerExitTimeout, killErr)
}
return
}
t.Errorf("shared dolt sql-server pid %d survived Stop by more than %s (force-killed)",
pid, sharedServerExitTimeout)
}
// sharedServerStillRunning re-checks birth identity after procid.Open or
// Handle.Kill failed. Both verify the token themselves and return a plain
// "does not match token" error when it no longer does — exactly what a
// process that exited between the last poll and the kill attempt produces.
// Without this second look, a server that shut down a few milliseconds late
// would be reported as one that survived Stop entirely.
func sharedServerStillRunning(t *testing.T, pid int, token procid.Token) bool {
t.Helper()
same, err := procid.Verify(pid, token)
if err != nil {
t.Logf("procid.Verify(%d) after a failed kill attempt: %v", pid, err)
return false
}
return same
}
// closedLoopbackPort returns a 127.0.0.1 port nothing listens on right now,
// for tests that need a server connection to fail. A fixed "unlikely" port
// (59999 used to be hard-coded here) is not: any process on the host may
// hold it, and a listener that accepts and resets turns the expected
// "connection refused" into a different error.
func closedLoopbackPort(t *testing.T) int {
t.Helper()
port, err := testutil.FindFreePort()
if err != nil {
t.Fatalf("find free port: %v", err)
}
return port
}