* feat(dbproxy): add procid birth-token package and pidfile schema v2 fields Foundation for verified proxy/backend adoption and safe stale-process cleanup (mybd-psxg.5, PR-A stage 1): internal/procid provides versioned process-birth tokens (Linux boot_id+starttime, Windows creation FILETIME, Darwin p_starttime) with TOCTOU-safe signaling handles (pidfd on Linux, re-verified process handle on Windows); pidfile gains additive schema-v2 fields (schema, kind, birth, root_id, control_port) and sentinel-error validation. No behavior changes; existing v1 pidfiles still parse. The procid Token/Verify shape intentionally matches the API requested in gastownhall/beads#4513 so internal/doltserver can adopt it later. Agent-Signature: codex-gpt-5.6-terra-medium on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * feat(dbproxy): publish verifiable proxy identity (secret, control handshake, v2 pidfiles) Stage 2 of mybd-psxg.5 (PR-A): the managed proxy now proves who it is. New leaf package internal/storage/dbproxy/identity provides the workspace RootID (sha256 of the resolved root path), a rotated 0600 proxy.secret, and the IDENT control protocol (client Identify + IdentReply). The proxy opens a loopback control listener that answers only to the workspace secret, and both pidfile writers publish schema-v2 records (kind, birth token, root_id, control_port) captured via procid. A proxy or backend that cannot prove its own identity fails startup instead of publishing. Adoption, cleanup, and kill behavior are unchanged in this stage; the consumers land next so each step stays reviewable. Agent-Signature: codex-gpt-5.6-terra-high on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): verified adoption, fail-closed cleanup, and handle-safe kills Stage 3 of mybd-psxg.5 (PR-A), the security core. Endpoint discovery now classifies records with typed statuses instead of adopting any listener that answers a TCP dial: adoption requires a schema-v2 pidfile, a live birth-token match, the authenticated IDENT handshake, and root/pid/birth/ port cross-checks. Legacy and malformed records fail closed (quarantined under proxy.lock, never adopted, never killed); quarantines are timestamped renames with a 30-day sweep, so no forensic record is ever unlinked. Every kill site (spawn-path backend cleanup and bd dolt stop's Shutdown) signals only through a procid.Handle opened against the recorded birth token, refusing unverifiable live processes with an actionable error. Orphaned Dolt backends left by a SIGKILLed proxy supervisor are now detected regardless of child-lock state and reaped only after verification. A proxy.spawn marker plus a persistent stop epoch closes the window where bd dolt stop could report success while a fork already in flight published a listener afterwards. Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): harden identity fixes from cross-vendor adversarial review Fixes every accepted finding from paired independent reviews of the first three stages (reports in the coordination repo, 2026-07-24): - darwin: classify wrapped ESRCH and the zero-length/EIO missing-pid sysctl result as process-gone; stale pidfiles self-heal again. - fallback signaling (darwin, non-pidfd linux): a target that exits after a fatal signal is a successful kill, not an error. - Shutdown verifies root_id on the proxy record too (copied-workspace kill hazard); positional bool pair replaced by killRecordChecks. - birth-token mismatch now classifies as provably-dead everywhere; dead legacy/malformed records quarantine and self-heal; refusal messages name the record file and manual remedy. - discovery-time verify/secret/root errors map to a non-adopting status routed through quarantine-under-lock instead of wedging before the lock probe. - boot_id read failures are environmental errors, never process-gone (closes the one fail-open path found in review). - ForceStopUnverified: executable-identity-checked operator escape for pre-upgrade/unverifiable records, quarantines after, reports truthfully; CLI wiring intentionally deferred. - handshake replies are HMAC'd against a client nonce; constant-time secret compare; bounded accept failures and handshake concurrency; migration purge covers proxy.secret and quarantine files; spawn marker cleared via defer; unix-only constraints on sleep tests. Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * feat(dbproxy): wire bd dolt stop --force and prove the lifecycle lane end to end Stage 4 of mybd-psxg.5 (PR-A). bd dolt stop gains --force, routed through proxy.ForceStopUnverified only after the backend is confirmed stopped: it verifies executable identity before signaling, quarantines the record afterward, refuses foreign executables and verifiable v2 records, and reports text/JSON truthfully. The managed-local lane (BEADS_TEST_PROXIED_LOCAL=1) now exercises the whole security model against a real dolt sql-server: foreign-listener rejection, reused-PID no-kill, legacy fail-closed (free lock, held lock refusing a foreign executable, held lock force-stopping a genuine bd), orphan-backend reaping after a SIGKILLed supervisor, 5-process cold-start convergence, and stop/start interleaving. Windows/macOS lane equivalents are defined in the lane header, implementation staged later. Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): close cross-vendor review findings — handle lifetimes, pidfd races, force-stop scoping - B1: cleanupOrphanBackend closes the verified handle before waiting for exit, mirroring shutdown.go; an open Windows handle kept the dead PID allocated and made every orphan-backend recovery time out. - B2: Windows procid checks GetExitCodeProcess after OpenProcess, so a terminated process kept resolvable by a third-party handle no longer verifies; "Verify == true implies running" now holds on all platforms. - B3: Linux procid.Open opens the pidfd BEFORE token verification; the pidfd pins the PID number, closing the reuse window where a later Signal could kill an unrelated replacement process. - B4: PidfdSendSignal ESRCH on a fatal signal is treated as successful termination, matching the fallback path (same for Windows terminate). - B5: ForceStopUnverified now covers the backend (proxy-child) record with the same safety checks, and CanForceStopUnverified accepts refusals where the backend is also (or only) unverifiable — the paired-legacy pre-v2 topology was locked out of its own advertised recovery. bd dolt stop --force reports the backend record too. - B6a: an unverified live PID is only signaled when its command line ties it to this workspace (Linux /proc cmdline, Darwin ps args); Windows cannot read another process's command line and now refuses, which beats basename-only killing. Clear errors name the manual remedy. - B6b: one stable OS handle (Linux pidfd, Windows process handle) spans inspection and signaling, removing the inspect/kill TOCTOU. - S1: control-listener accept errors are non-fatal — log and retry with capped exponential backoff instead of tearing down a healthy proxy. - S2: requireUnverifiableRecord surfaces RootID resolution errors instead of failing open into the destructive branch. - S3: killSpawnedChild falls back to cmd.Process.Kill for our own un-reaped child when no verified handle exists, instead of leaking it. - S4: adoption handshake accepts reply schema v2 or newer, matching pidfile.ValidateV2 forward-compat policy. - S5: errors.Is for ERROR_INVALID_PARAMETER. - Tests: pidfd ESRCH-after-exit and Open-after-reap classification; paired-legacy force recovery end to end; foreign-workspace bd process refused; Windows Verify-after-terminate (compile-checked); schema-v3 handshake adoption; accept-loop survival under persistent errors. Agent-Signature: claude-fable-5-medium on behalf of maphew --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
846 lines
28 KiB
Go
846 lines
28 KiB
Go
//go:build !windows
|
|
|
|
package proxy
|
|
|
|
import (
|
|
"errors"
|
|
"fmt"
|
|
"net"
|
|
"os"
|
|
"os/exec"
|
|
"path/filepath"
|
|
"strconv"
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/steveyegge/beads/internal/configfile"
|
|
"github.com/steveyegge/beads/internal/procid"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/identity"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/pidfile"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/util"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
type helperProcess struct {
|
|
cmd *exec.Cmd
|
|
token procid.Token
|
|
done <-chan error
|
|
}
|
|
|
|
func startHelperProcess(t *testing.T) helperProcess {
|
|
t.Helper()
|
|
cmd := exec.Command("sleep", "30")
|
|
require.NoError(t, cmd.Start())
|
|
token, err := procid.Capture(cmd.Process.Pid)
|
|
require.NoError(t, err)
|
|
done := make(chan error, 1)
|
|
go func() {
|
|
done <- cmd.Wait()
|
|
close(done)
|
|
}()
|
|
t.Cleanup(func() {
|
|
matched, verifyErr := procid.Verify(cmd.Process.Pid, token)
|
|
if verifyErr == nil && matched {
|
|
handle, openErr := procid.Open(cmd.Process.Pid, token)
|
|
if openErr == nil {
|
|
_ = handle.Kill()
|
|
_ = handle.Close()
|
|
}
|
|
}
|
|
select {
|
|
case <-done:
|
|
case <-time.After(5 * time.Second):
|
|
t.Errorf("helper pid %d did not exit", cmd.Process.Pid)
|
|
}
|
|
})
|
|
return helperProcess{cmd: cmd, token: token, done: done}
|
|
}
|
|
|
|
func assertHelperAlive(t *testing.T, helper helperProcess) {
|
|
t.Helper()
|
|
matched, err := procid.Verify(helper.cmd.Process.Pid, helper.token)
|
|
require.NoError(t, err)
|
|
assert.True(t, matched, "helper pid %d should still be alive", helper.cmd.Process.Pid)
|
|
}
|
|
|
|
func assertHelperExited(t *testing.T, helper helperProcess) {
|
|
t.Helper()
|
|
select {
|
|
case <-helper.done:
|
|
case <-time.After(5 * time.Second):
|
|
t.Fatalf("helper pid %d did not exit", helper.cmd.Process.Pid)
|
|
}
|
|
matched, err := procid.Verify(helper.cmd.Process.Pid, helper.token)
|
|
require.NoError(t, err)
|
|
assert.False(t, matched)
|
|
}
|
|
|
|
func v2Record(t *testing.T, root, kind string, helper helperProcess) pidfile.PidFile {
|
|
t.Helper()
|
|
rootID, err := identity.RootID(root)
|
|
require.NoError(t, err)
|
|
return pidfile.PidFile{
|
|
Schema: pidfile.SchemaV2,
|
|
Kind: kind,
|
|
Pid: helper.cmd.Process.Pid,
|
|
Birth: string(helper.token),
|
|
Port: 3307,
|
|
RootID: rootID,
|
|
}
|
|
}
|
|
|
|
func TestReadAndDialClassifiesRecordsWithoutMutation(t *testing.T) {
|
|
t.Run("no record", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionNoRecord, got.status)
|
|
})
|
|
|
|
t.Run("malformed JSON", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
path := pidfile.Path(root, PIDFileName)
|
|
require.NoError(t, os.WriteFile(path, []byte(`{"pid":`), 0o600))
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionMalformed, got.status)
|
|
_, err := os.Stat(path)
|
|
assert.NoError(t, err, "discovery must not mutate the pidfile")
|
|
})
|
|
|
|
t.Run("invalid v2 fields", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pidfile.PidFile{
|
|
Schema: pidfile.SchemaV2,
|
|
Kind: "foreign-kind",
|
|
Pid: os.Getpid(),
|
|
Birth: "birth",
|
|
Port: 3307,
|
|
}))
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionMalformed, got.status)
|
|
assert.ErrorIs(t, got.err, pidfile.ErrKindMismatch)
|
|
})
|
|
|
|
t.Run("I/O error", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
require.NoError(t, os.Mkdir(pidfile.Path(root, PIDFileName), 0o700))
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionIOErr, got.status)
|
|
assert.Error(t, got.err)
|
|
})
|
|
|
|
t.Run("legacy", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pidfile.PidFile{Pid: os.Getpid(), Port: 3307}))
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionLegacy, got.status)
|
|
_, err := os.Stat(pidfile.Path(root, PIDFileName))
|
|
assert.NoError(t, err, "discovery must not mutate the pidfile")
|
|
})
|
|
|
|
t.Run("dead process", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
handle, err := procid.Open(pf.Pid, procid.Token(pf.Birth))
|
|
require.NoError(t, err)
|
|
require.NoError(t, handle.Kill())
|
|
require.NoError(t, handle.Close())
|
|
assertHelperExited(t, helper)
|
|
pf.ControlPort = 3308
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionStaleDead, got.status)
|
|
})
|
|
|
|
t.Run("wrong birth", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
pf.Birth += "-wrong"
|
|
pf.ControlPort = 3308
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionStaleDead, got.status)
|
|
assertHelperAlive(t, helper)
|
|
})
|
|
}
|
|
|
|
func TestReadAndDialProbeErrorsAreNonFatalDiscovery(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
pf.ControlPort = 3308
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
|
|
originalVerify := verifyProcessIdentity
|
|
originalRootID := resolveRootIdentity
|
|
originalSecret := readControlSecret
|
|
t.Cleanup(func() {
|
|
verifyProcessIdentity = originalVerify
|
|
resolveRootIdentity = originalRootID
|
|
readControlSecret = originalSecret
|
|
})
|
|
|
|
t.Run("process verification", func(t *testing.T) {
|
|
verifyProcessIdentity = func(int, procid.Token) (bool, error) {
|
|
return false, errors.New("access denied")
|
|
}
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionUnverifiable, got.status)
|
|
assert.NotEqual(t, adoptionIOErr, got.status)
|
|
verifyProcessIdentity = originalVerify
|
|
})
|
|
|
|
t.Run("root identity", func(t *testing.T) {
|
|
resolveRootIdentity = func(string) (string, error) {
|
|
return "", errors.New("root resolution failed")
|
|
}
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionUnverifiable, got.status)
|
|
assert.NotEqual(t, adoptionIOErr, got.status)
|
|
resolveRootIdentity = originalRootID
|
|
})
|
|
|
|
t.Run("secret read", func(t *testing.T) {
|
|
readControlSecret = func(string) (string, error) {
|
|
return "", errors.New("secret read failed")
|
|
}
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionUnverifiable, got.status)
|
|
assert.NotEqual(t, adoptionIOErr, got.status)
|
|
readControlSecret = originalSecret
|
|
})
|
|
assertHelperAlive(t, helper)
|
|
}
|
|
|
|
func TestReadAndDialForeignListenerWithUnrelatedLiveProcess(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { _ = ln.Close() })
|
|
go func() {
|
|
conn, acceptErr := ln.Accept()
|
|
if acceptErr == nil {
|
|
_ = conn.Close()
|
|
}
|
|
}()
|
|
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
pf.Port = ln.Addr().(*net.TCPAddr).Port
|
|
pf.ControlPort = pf.Port
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
_, err = identity.WriteSecret(root)
|
|
require.NoError(t, err)
|
|
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionIdentityMismatch, got.status)
|
|
assertHelperAlive(t, helper)
|
|
}
|
|
|
|
func TestReadAndDialForeignListenerWithDeadOrWrongProcessIdentity(t *testing.T) {
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { _ = ln.Close() })
|
|
port := ln.Addr().(*net.TCPAddr).Port
|
|
|
|
t.Run("dead pid", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
handle, openErr := procid.Open(pf.Pid, procid.Token(pf.Birth))
|
|
require.NoError(t, openErr)
|
|
require.NoError(t, handle.Kill())
|
|
require.NoError(t, handle.Close())
|
|
assertHelperExited(t, helper)
|
|
pf.Port = port
|
|
pf.ControlPort = port
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
assert.Equal(t, adoptionStaleDead, readAndDial(root).status)
|
|
})
|
|
|
|
t.Run("wrong birth", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
pf.Birth += "-wrong"
|
|
pf.Port = port
|
|
pf.ControlPort = port
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
assert.Equal(t, adoptionStaleDead, readAndDial(root).status)
|
|
assertHelperAlive(t, helper)
|
|
})
|
|
}
|
|
|
|
func TestReadAndDialRejectsAuthenticatedRootMismatch(t *testing.T) {
|
|
rootA := t.TempDir()
|
|
rootB := t.TempDir()
|
|
token, err := procid.Capture(os.Getpid())
|
|
require.NoError(t, err)
|
|
rootIDA, err := identity.RootID(rootA)
|
|
require.NoError(t, err)
|
|
|
|
dataListener, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { _ = dataListener.Close() })
|
|
dataPort := dataListener.Addr().(*net.TCPAddr).Port
|
|
|
|
secret, err := identity.WriteSecret(rootA)
|
|
require.NoError(t, err)
|
|
var reply identity.IdentReply
|
|
var replyMu sync.RWMutex
|
|
control, err := startControl(rootA, func() identity.IdentReply {
|
|
replyMu.RLock()
|
|
defer replyMu.RUnlock()
|
|
return reply
|
|
})
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { _ = control.Close() })
|
|
replyMu.Lock()
|
|
reply = identity.IdentReply{
|
|
Schema: pidfile.SchemaV2,
|
|
Role: pidfile.KindProxy,
|
|
RootID: rootIDA,
|
|
UpstreamID: "upstream",
|
|
PID: os.Getpid(),
|
|
Birth: string(token),
|
|
DataPort: dataPort,
|
|
ControlPort: control.Port(),
|
|
}
|
|
replyMu.Unlock()
|
|
pf := pidfile.PidFile{
|
|
Schema: pidfile.SchemaV2,
|
|
Kind: pidfile.KindProxy,
|
|
Pid: os.Getpid(),
|
|
Birth: string(token),
|
|
Port: dataPort,
|
|
ControlPort: control.Port(),
|
|
RootID: rootIDA,
|
|
UpstreamID: "upstream",
|
|
}
|
|
require.NoError(t, pidfile.Write(rootA, PIDFileName, pf))
|
|
require.NoError(t, pidfile.Write(rootB, PIDFileName, pf))
|
|
require.NoError(t, os.WriteFile(filepath.Join(rootB, identity.SecretFileName), []byte(secret+"\n"), 0o600))
|
|
|
|
got := readAndDial(rootB)
|
|
assert.Equal(t, adoptionIdentityMismatch, got.status)
|
|
}
|
|
|
|
// The handshake accepts schema v2 or newer, matching pidfile.ValidateV2's
|
|
// forward-compat policy for records: a newer proxy that still answers the v2
|
|
// handshake remains adoptable.
|
|
func TestReadAndDialAcceptsNewerSchemaHandshake(t *testing.T) {
|
|
root := t.TempDir()
|
|
token, err := procid.Capture(os.Getpid())
|
|
require.NoError(t, err)
|
|
rootID, err := identity.RootID(root)
|
|
require.NoError(t, err)
|
|
|
|
dataListener, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { _ = dataListener.Close() })
|
|
dataPort := dataListener.Addr().(*net.TCPAddr).Port
|
|
|
|
_, err = identity.WriteSecret(root)
|
|
require.NoError(t, err)
|
|
var reply identity.IdentReply
|
|
var replyMu sync.RWMutex
|
|
control, err := startControl(root, func() identity.IdentReply {
|
|
replyMu.RLock()
|
|
defer replyMu.RUnlock()
|
|
return reply
|
|
})
|
|
require.NoError(t, err)
|
|
t.Cleanup(func() { _ = control.Close() })
|
|
replyMu.Lock()
|
|
reply = identity.IdentReply{
|
|
Schema: pidfile.SchemaV2 + 1,
|
|
Role: pidfile.KindProxy,
|
|
RootID: rootID,
|
|
UpstreamID: "upstream",
|
|
PID: os.Getpid(),
|
|
Birth: string(token),
|
|
DataPort: dataPort,
|
|
ControlPort: control.Port(),
|
|
}
|
|
replyMu.Unlock()
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pidfile.PidFile{
|
|
Schema: pidfile.SchemaV2 + 1,
|
|
Kind: pidfile.KindProxy,
|
|
Pid: os.Getpid(),
|
|
Birth: string(token),
|
|
Port: dataPort,
|
|
ControlPort: control.Port(),
|
|
RootID: rootID,
|
|
UpstreamID: "upstream",
|
|
}))
|
|
|
|
got := readAndDial(root)
|
|
assert.Equal(t, adoptionAdopted, got.status)
|
|
assert.Equal(t, dataPort, got.endpoint.Port)
|
|
}
|
|
|
|
func TestLegacyRecordFailClosed(t *testing.T) {
|
|
t.Run("held lock returns actionable error without kill", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pidfile.PidFile{
|
|
Pid: helper.cmd.Process.Pid,
|
|
Port: 3307,
|
|
}))
|
|
lock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
defer lock.Unlock()
|
|
|
|
_, err = GetCreateDatabaseProxyServerEndpoint(root, externalOpenOpts(root))
|
|
require.Error(t, err)
|
|
assert.Contains(t, err.Error(), filepath.Join(root, LockFileName))
|
|
assert.Contains(t, err.Error(), "old bd binary")
|
|
assertHelperAlive(t, helper)
|
|
_, err = os.Stat(pidfile.Path(root, PIDFileName))
|
|
assert.NoError(t, err)
|
|
})
|
|
|
|
t.Run("free lock quarantines and enters spawn path without kill", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pidfile.PidFile{
|
|
Pid: helper.cmd.Process.Pid,
|
|
Port: 3307,
|
|
}))
|
|
discovery := readAndDial(root)
|
|
require.Equal(t, adoptionLegacy, discovery.status)
|
|
lock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
|
|
previous := ResolveExecutable
|
|
ResolveExecutable = func() (string, error) { return "", errors.New("spawn reached") }
|
|
t.Cleanup(func() { ResolveExecutable = previous })
|
|
_, err = spawnAndHandoff(root, externalOpenOpts(root), time.Now().Add(time.Second), "", lock, discovery)
|
|
require.ErrorContains(t, err, "spawn reached")
|
|
|
|
assertHelperAlive(t, helper)
|
|
assert.NoFileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.Len(t, quarantines(t, root, PIDFileName), 1)
|
|
})
|
|
}
|
|
|
|
func TestIdentityMismatchQuarantinesRecordWithoutKillingProcess(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
pf.ControlPort = 3308
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
lock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
|
|
previous := ResolveExecutable
|
|
ResolveExecutable = func() (string, error) { return "", errors.New("spawn reached") }
|
|
t.Cleanup(func() { ResolveExecutable = previous })
|
|
_, err = spawnAndHandoff(
|
|
root,
|
|
externalOpenOpts(root),
|
|
time.Now().Add(time.Second),
|
|
"",
|
|
lock,
|
|
adoptionResult{
|
|
status: adoptionIdentityMismatch,
|
|
pidfile: &pf,
|
|
err: errors.New("foreign control listener"),
|
|
},
|
|
)
|
|
require.ErrorContains(t, err, "spawn reached")
|
|
assertHelperAlive(t, helper)
|
|
assert.NoFileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.Len(t, quarantines(t, root, PIDFileName), 1)
|
|
}
|
|
|
|
func TestUnverifiableDiscoveryQuarantinesUnderLockWithoutKillingProcess(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
pf.ControlPort = 3308
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
lock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
|
|
previous := ResolveExecutable
|
|
ResolveExecutable = func() (string, error) { return "", errors.New("spawn reached") }
|
|
t.Cleanup(func() { ResolveExecutable = previous })
|
|
_, err = spawnAndHandoff(
|
|
root,
|
|
externalOpenOpts(root),
|
|
time.Now().Add(time.Second),
|
|
"",
|
|
lock,
|
|
adoptionResult{
|
|
status: adoptionUnverifiable,
|
|
pidfile: &pf,
|
|
err: errors.New("identity probe denied"),
|
|
},
|
|
)
|
|
require.ErrorContains(t, err, "spawn reached")
|
|
assertHelperAlive(t, helper)
|
|
assert.NoFileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.Len(t, quarantines(t, root, PIDFileName), 1)
|
|
}
|
|
|
|
func TestMalformedRecordQuarantinesBeforeSpawn(t *testing.T) {
|
|
root := t.TempDir()
|
|
require.NoError(t, os.WriteFile(pidfile.Path(root, PIDFileName), []byte(`{"pid":`), 0o600))
|
|
discovery := readAndDial(root)
|
|
require.Equal(t, adoptionMalformed, discovery.status)
|
|
lock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
|
|
previous := ResolveExecutable
|
|
ResolveExecutable = func() (string, error) { return "", errors.New("spawn reached") }
|
|
t.Cleanup(func() { ResolveExecutable = previous })
|
|
_, err = spawnAndHandoff(root, externalOpenOpts(root), time.Now().Add(time.Second), "", lock, discovery)
|
|
require.ErrorContains(t, err, "spawn reached")
|
|
assert.NoFileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.Len(t, quarantines(t, root, PIDFileName), 1)
|
|
}
|
|
|
|
func TestGetCreatePropagatesPIDFileIOErrorWithoutSpawning(t *testing.T) {
|
|
root := t.TempDir()
|
|
require.NoError(t, os.Mkdir(pidfile.Path(root, PIDFileName), 0o700))
|
|
_, err := GetCreateDatabaseProxyServerEndpoint(root, externalOpenOpts(root))
|
|
require.ErrorContains(t, err, "discover proxy")
|
|
assert.NoFileExists(t, filepath.Join(root, spawnMarkerFileName))
|
|
}
|
|
|
|
func TestCleanupOrphanBackendRefusesHeldChildLock(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
require.NoError(t, pidfile.Write(root, server.PIDFileName, v2Record(t, root, pidfile.KindDoltBackend, helper)))
|
|
|
|
proxyLock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
defer proxyLock.Unlock()
|
|
childLock, err := util.TryLock(filepath.Join(root, server.LockFileName))
|
|
require.NoError(t, err)
|
|
defer childLock.Unlock()
|
|
|
|
err = cleanupOrphanBackend(root)
|
|
require.ErrorContains(t, err, "refusing backend cleanup")
|
|
assertHelperAlive(t, helper)
|
|
assert.FileExists(t, pidfile.Path(root, server.PIDFileName))
|
|
assert.Empty(t, quarantines(t, root, server.PIDFileName))
|
|
}
|
|
|
|
func TestCleanupOrphanBackendRefusesRootMismatch(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindDoltBackend, helper)
|
|
pf.RootID = "different-workspace"
|
|
require.NoError(t, pidfile.Write(root, server.PIDFileName, pf))
|
|
|
|
proxyLock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
defer proxyLock.Unlock()
|
|
err = cleanupOrphanBackend(root)
|
|
require.ErrorContains(t, err, "root identity mismatch")
|
|
assertHelperAlive(t, helper)
|
|
assert.FileExists(t, pidfile.Path(root, server.PIDFileName))
|
|
assert.Empty(t, quarantines(t, root, server.PIDFileName))
|
|
}
|
|
|
|
func TestCleanupOrphanBackendLegacyRecords(t *testing.T) {
|
|
t.Run("dead pid quarantines", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
handle, err := procid.Open(helper.cmd.Process.Pid, helper.token)
|
|
require.NoError(t, err)
|
|
require.NoError(t, handle.Kill())
|
|
require.NoError(t, handle.Close())
|
|
assertHelperExited(t, helper)
|
|
require.NoError(t, pidfile.Write(root, server.PIDFileName, pidfile.PidFile{
|
|
Pid: helper.cmd.Process.Pid,
|
|
Port: 3306,
|
|
}))
|
|
|
|
proxyLock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
defer proxyLock.Unlock()
|
|
require.NoError(t, cleanupOrphanBackend(root))
|
|
assert.NoFileExists(t, pidfile.Path(root, server.PIDFileName))
|
|
assert.Len(t, quarantines(t, root, server.PIDFileName), 1)
|
|
})
|
|
|
|
t.Run("live pid refuses", func(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
require.NoError(t, pidfile.Write(root, server.PIDFileName, pidfile.PidFile{
|
|
Pid: helper.cmd.Process.Pid,
|
|
Port: 3306,
|
|
}))
|
|
|
|
proxyLock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
defer proxyLock.Unlock()
|
|
err = cleanupOrphanBackend(root)
|
|
require.ErrorContains(t, err, "unverifiable live process")
|
|
require.ErrorContains(t, err, pidfile.Path(root, server.PIDFileName)+".stale-<unix-timestamp>")
|
|
assertHelperAlive(t, helper)
|
|
assert.FileExists(t, pidfile.Path(root, server.PIDFileName))
|
|
assert.Empty(t, quarantines(t, root, server.PIDFileName))
|
|
})
|
|
}
|
|
|
|
func TestCleanupOrphanBackendBirthMismatchIsDeadRecord(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindDoltBackend, helper)
|
|
pf.Birth += "-recycled"
|
|
require.NoError(t, pidfile.Write(root, server.PIDFileName, pf))
|
|
|
|
proxyLock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
defer proxyLock.Unlock()
|
|
require.NoError(t, cleanupOrphanBackend(root))
|
|
assertHelperAlive(t, helper)
|
|
assert.NoFileExists(t, pidfile.Path(root, server.PIDFileName))
|
|
assert.Len(t, quarantines(t, root, server.PIDFileName), 1)
|
|
}
|
|
|
|
func TestCleanupOrphanBackendKillsVerifiedOrphanAndQuarantines(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
require.NoError(t, pidfile.Write(root, server.PIDFileName, v2Record(t, root, pidfile.KindDoltBackend, helper)))
|
|
|
|
proxyLock, err := util.TryLock(filepath.Join(root, LockFileName))
|
|
require.NoError(t, err)
|
|
previous := ResolveExecutable
|
|
ResolveExecutable = func() (string, error) { return "", errors.New("spawn reached after orphan cleanup") }
|
|
t.Cleanup(func() { ResolveExecutable = previous })
|
|
_, err = spawnAndHandoff(
|
|
root,
|
|
externalOpenOpts(root),
|
|
time.Now().Add(time.Second),
|
|
"",
|
|
proxyLock,
|
|
adoptionResult{status: adoptionNoRecord},
|
|
)
|
|
require.ErrorContains(t, err, "spawn reached after orphan cleanup")
|
|
|
|
assertHelperExited(t, helper)
|
|
assert.NoFileExists(t, pidfile.Path(root, server.PIDFileName))
|
|
assert.Len(t, quarantines(t, root, server.PIDFileName), 1)
|
|
}
|
|
|
|
func TestQuarantineNamingCollisionAndRetentionSweep(t *testing.T) {
|
|
root := t.TempDir()
|
|
now := time.Unix(2_000_000_000, 0)
|
|
currentPath := pidfile.Path(root, PIDFileName)
|
|
require.NoError(t, os.WriteFile(currentPath, []byte("first"), 0o600))
|
|
first, err := quarantineRecord(root, PIDFileName, now)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, currentPath+".stale-2000000000", first)
|
|
|
|
require.NoError(t, os.WriteFile(currentPath, []byte("second"), 0o600))
|
|
second, err := quarantineRecord(root, PIDFileName, now)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, currentPath+".stale-2000000001", second)
|
|
require.NoError(t, os.WriteFile(currentPath, []byte("current"), 0o600))
|
|
foreign := filepath.Join(root, "foreign.stale-1")
|
|
require.NoError(t, os.WriteFile(foreign, []byte("foreign"), 0o600))
|
|
recent := currentPath + ".stale-" + strconv.FormatInt(now.Add(2*24*time.Hour).Unix(), 10)
|
|
require.NoError(t, os.WriteFile(recent, []byte("recent"), 0o600))
|
|
|
|
require.NoError(t, sweepOldQuarantines(root, now.Add(31*24*time.Hour)))
|
|
assert.NoFileExists(t, first)
|
|
assert.NoFileExists(t, second)
|
|
assert.FileExists(t, currentPath, "current record must never be swept")
|
|
assert.FileExists(t, foreign, "foreign files must never be swept")
|
|
assert.FileExists(t, recent, "quarantines younger than 30 days must be retained")
|
|
}
|
|
|
|
func TestShutdownVerifiedRecordStopsProcess(t *testing.T) {
|
|
root := t.TempDir()
|
|
proxyHelper := startHelperProcess(t)
|
|
backendHelper := startHelperProcess(t)
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, v2Record(t, root, pidfile.KindProxy, proxyHelper)))
|
|
require.NoError(t, pidfile.Write(root, server.PIDFileName, v2Record(t, root, pidfile.KindDoltBackend, backendHelper)))
|
|
|
|
require.NoError(t, Shutdown(root))
|
|
assertHelperExited(t, proxyHelper)
|
|
assertHelperExited(t, backendHelper)
|
|
assert.NoFileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.NoFileExists(t, pidfile.Path(root, server.PIDFileName))
|
|
}
|
|
|
|
func TestShutdownReportsPartialOutcomeAndRefusesLegacyProcess(t *testing.T) {
|
|
root := t.TempDir()
|
|
proxyHelper := startHelperProcess(t)
|
|
backendHelper := startHelperProcess(t)
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pidfile.PidFile{
|
|
Pid: proxyHelper.cmd.Process.Pid,
|
|
Port: 3307,
|
|
}))
|
|
require.NoError(t, pidfile.Write(root, server.PIDFileName, v2Record(t, root, pidfile.KindDoltBackend, backendHelper)))
|
|
|
|
err := Shutdown(root)
|
|
require.Error(t, err)
|
|
assert.ErrorIs(t, err, ErrUnverifiableProcess)
|
|
assert.True(t, CanForceStopUnverified(err))
|
|
assert.Contains(t, err.Error(), "backend stopped; proxy left running")
|
|
assert.Contains(t, err.Error(), pidfile.Path(root, PIDFileName))
|
|
assert.Contains(t, err.Error(), "old bd binary")
|
|
assert.Contains(t, err.Error(), pidfile.Path(root, PIDFileName)+".stale-<unix-timestamp>")
|
|
assertHelperAlive(t, proxyHelper)
|
|
assertHelperExited(t, backendHelper)
|
|
assert.FileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.Empty(t, quarantines(t, root, PIDFileName))
|
|
}
|
|
|
|
func TestCanForceStopUnverifiedRequiresUnverifiableRefusals(t *testing.T) {
|
|
unverifiable := fmt.Errorf("%w: legacy record", ErrUnverifiableProcess)
|
|
other := errors.New("unrelated shutdown failure")
|
|
tests := []struct {
|
|
name string
|
|
err error
|
|
want bool
|
|
}{
|
|
{name: "proxy only", err: &shutdownError{proxyErr: unverifiable}, want: true},
|
|
{name: "backend only", err: &shutdownError{backendErr: unverifiable}, want: true},
|
|
{name: "paired legacy", err: &shutdownError{proxyErr: unverifiable, backendErr: unverifiable}, want: true},
|
|
{name: "unverifiable proxy with unrelated backend error", err: &shutdownError{proxyErr: unverifiable, backendErr: other}},
|
|
{name: "unrelated proxy error", err: &shutdownError{proxyErr: other}},
|
|
{name: "bare sentinel", err: unverifiable},
|
|
}
|
|
for _, tc := range tests {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
assert.Equal(t, tc.want, CanForceStopUnverified(tc.err))
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestShutdownQuarantinesBirthMismatchWithoutKillingProcess(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
pf.Birth += "-wrong"
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
|
|
require.NoError(t, Shutdown(root))
|
|
assertHelperAlive(t, helper)
|
|
assert.NoFileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.Len(t, quarantines(t, root, PIDFileName), 1)
|
|
}
|
|
|
|
func TestShutdownRefusesForeignRootProxyRecord(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
pf.RootID = "foreign-workspace-root"
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
|
|
err := Shutdown(root)
|
|
require.ErrorContains(t, err, "root identity mismatch")
|
|
require.ErrorContains(t, err, pidfile.Path(root, PIDFileName))
|
|
assertHelperAlive(t, helper)
|
|
assert.FileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.Empty(t, quarantines(t, root, PIDFileName))
|
|
}
|
|
|
|
func TestShutdownQuarantinesVerifiedDeadRecord(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
pf := v2Record(t, root, pidfile.KindProxy, helper)
|
|
handle, err := procid.Open(pf.Pid, procid.Token(pf.Birth))
|
|
require.NoError(t, err)
|
|
require.NoError(t, handle.Kill())
|
|
require.NoError(t, handle.Close())
|
|
assertHelperExited(t, helper)
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, pf))
|
|
|
|
require.NoError(t, Shutdown(root))
|
|
assert.NoFileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.Len(t, quarantines(t, root, PIDFileName), 1)
|
|
}
|
|
|
|
func TestShutdownQuarantinesDeadLegacyAndMalformedRecords(t *testing.T) {
|
|
tests := []struct {
|
|
name string
|
|
record func(helper helperProcess) pidfile.PidFile
|
|
}{
|
|
{
|
|
name: "legacy",
|
|
record: func(helper helperProcess) pidfile.PidFile {
|
|
return pidfile.PidFile{Pid: helper.cmd.Process.Pid, Port: 3307}
|
|
},
|
|
},
|
|
{
|
|
name: "malformed v2",
|
|
record: func(helper helperProcess) pidfile.PidFile {
|
|
return pidfile.PidFile{
|
|
Schema: pidfile.SchemaV2,
|
|
Kind: pidfile.KindProxy,
|
|
Pid: helper.cmd.Process.Pid,
|
|
Port: 3307,
|
|
}
|
|
},
|
|
},
|
|
}
|
|
for _, tc := range tests {
|
|
t.Run(tc.name, func(t *testing.T) {
|
|
root := t.TempDir()
|
|
helper := startHelperProcess(t)
|
|
record := tc.record(helper)
|
|
handle, err := procid.Open(helper.cmd.Process.Pid, helper.token)
|
|
require.NoError(t, err)
|
|
require.NoError(t, handle.Kill())
|
|
require.NoError(t, handle.Close())
|
|
assertHelperExited(t, helper)
|
|
require.NoError(t, pidfile.Write(root, PIDFileName, record))
|
|
|
|
require.NoError(t, Shutdown(root))
|
|
assert.NoFileExists(t, pidfile.Path(root, PIDFileName))
|
|
assert.Len(t, quarantines(t, root, PIDFileName), 1)
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestControlFilePathsIncludesSecretAndQuarantines(t *testing.T) {
|
|
root := t.TempDir()
|
|
proxyStale := pidfile.Path(root, PIDFileName) + ".stale-100"
|
|
backendStale := pidfile.Path(root, server.PIDFileName) + ".stale-200"
|
|
require.NoError(t, os.WriteFile(proxyStale, []byte("proxy"), 0o600))
|
|
require.NoError(t, os.WriteFile(backendStale, []byte("backend"), 0o600))
|
|
_, err := identity.WriteSecret(root)
|
|
require.NoError(t, err)
|
|
|
|
paths := ControlFilePaths(root)
|
|
assert.Contains(t, paths, filepath.Join(root, identity.SecretFileName))
|
|
assert.Contains(t, paths, proxyStale)
|
|
assert.Contains(t, paths, backendStale)
|
|
|
|
assert.Empty(t, PurgeControlFiles(root))
|
|
assert.NoFileExists(t, filepath.Join(root, identity.SecretFileName))
|
|
assert.NoFileExists(t, proxyStale)
|
|
assert.NoFileExists(t, backendStale)
|
|
}
|
|
|
|
// N-process cold-start convergence remains intentionally deferred to the
|
|
// lifecycle-lane stage; this security suite covers single-spawner invariants.
|
|
|
|
func externalOpenOpts(root string) OpenOpts {
|
|
return OpenOpts{
|
|
Backend: BackendExternal,
|
|
External: configfile.ExternalDoltConfig{Host: "127.0.0.1", Port: 3306},
|
|
LogFilePath: filepath.Join(root, LogFileName),
|
|
Port: 3307,
|
|
}
|
|
}
|
|
|
|
func quarantines(t *testing.T, root, name string) []string {
|
|
t.Helper()
|
|
matches, err := filepath.Glob(filepath.Join(root, name+".stale-*"))
|
|
require.NoError(t, err)
|
|
return matches
|
|
}
|