beads/internal/storage/dbproxy/proxy/force_stop.go
matt wilkie 21669eca60
fix(dbproxy): verified identity for the managed proxied-server lifecycle (adoption, cleanup, kills) (#5013)
* feat(dbproxy): add procid birth-token package and pidfile schema v2 fields

Foundation for verified proxy/backend adoption and safe stale-process
cleanup (mybd-psxg.5, PR-A stage 1): internal/procid provides versioned
process-birth tokens (Linux boot_id+starttime, Windows creation FILETIME,
Darwin p_starttime) with TOCTOU-safe signaling handles (pidfd on Linux,
re-verified process handle on Windows); pidfile gains additive schema-v2
fields (schema, kind, birth, root_id, control_port) and sentinel-error
validation. No behavior changes; existing v1 pidfiles still parse.

The procid Token/Verify shape intentionally matches the API requested in
gastownhall/beads#4513 so internal/doltserver can adopt it later.

Agent-Signature: codex-gpt-5.6-terra-medium on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* feat(dbproxy): publish verifiable proxy identity (secret, control handshake, v2 pidfiles)

Stage 2 of mybd-psxg.5 (PR-A): the managed proxy now proves who it is.
New leaf package internal/storage/dbproxy/identity provides the workspace
RootID (sha256 of the resolved root path), a rotated 0600 proxy.secret,
and the IDENT control protocol (client Identify + IdentReply). The proxy
opens a loopback control listener that answers only to the workspace
secret, and both pidfile writers publish schema-v2 records (kind, birth
token, root_id, control_port) captured via procid. A proxy or backend
that cannot prove its own identity fails startup instead of publishing.

Adoption, cleanup, and kill behavior are unchanged in this stage; the
consumers land next so each step stays reviewable.

Agent-Signature: codex-gpt-5.6-terra-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): verified adoption, fail-closed cleanup, and handle-safe kills

Stage 3 of mybd-psxg.5 (PR-A), the security core. Endpoint discovery now
classifies records with typed statuses instead of adopting any listener
that answers a TCP dial: adoption requires a schema-v2 pidfile, a live
birth-token match, the authenticated IDENT handshake, and root/pid/birth/
port cross-checks. Legacy and malformed records fail closed (quarantined
under proxy.lock, never adopted, never killed); quarantines are
timestamped renames with a 30-day sweep, so no forensic record is ever
unlinked.

Every kill site (spawn-path backend cleanup and bd dolt stop's Shutdown)
signals only through a procid.Handle opened against the recorded birth
token, refusing unverifiable live processes with an actionable error.
Orphaned Dolt backends left by a SIGKILLed proxy supervisor are now
detected regardless of child-lock state and reaped only after
verification. A proxy.spawn marker plus a persistent stop epoch closes
the window where bd dolt stop could report success while a fork already
in flight published a listener afterwards.

Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): harden identity fixes from cross-vendor adversarial review

Fixes every accepted finding from paired independent reviews of the
first three stages (reports in the coordination repo, 2026-07-24):

- darwin: classify wrapped ESRCH and the zero-length/EIO missing-pid
  sysctl result as process-gone; stale pidfiles self-heal again.
- fallback signaling (darwin, non-pidfd linux): a target that exits
  after a fatal signal is a successful kill, not an error.
- Shutdown verifies root_id on the proxy record too (copied-workspace
  kill hazard); positional bool pair replaced by killRecordChecks.
- birth-token mismatch now classifies as provably-dead everywhere;
  dead legacy/malformed records quarantine and self-heal; refusal
  messages name the record file and manual remedy.
- discovery-time verify/secret/root errors map to a non-adopting
  status routed through quarantine-under-lock instead of wedging
  before the lock probe.
- boot_id read failures are environmental errors, never process-gone
  (closes the one fail-open path found in review).
- ForceStopUnverified: executable-identity-checked operator escape for
  pre-upgrade/unverifiable records, quarantines after, reports
  truthfully; CLI wiring intentionally deferred.
- handshake replies are HMAC'd against a client nonce; constant-time
  secret compare; bounded accept failures and handshake concurrency;
  migration purge covers proxy.secret and quarantine files; spawn
  marker cleared via defer; unix-only constraints on sleep tests.

Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* feat(dbproxy): wire bd dolt stop --force and prove the lifecycle lane end to end

Stage 4 of mybd-psxg.5 (PR-A). bd dolt stop gains --force, routed through
proxy.ForceStopUnverified only after the backend is confirmed stopped:
it verifies executable identity before signaling, quarantines the record
afterward, refuses foreign executables and verifiable v2 records, and
reports text/JSON truthfully. The managed-local lane
(BEADS_TEST_PROXIED_LOCAL=1) now exercises the whole security model
against a real dolt sql-server: foreign-listener rejection, reused-PID
no-kill, legacy fail-closed (free lock, held lock refusing a foreign
executable, held lock force-stopping a genuine bd), orphan-backend
reaping after a SIGKILLed supervisor, 5-process cold-start convergence,
and stop/start interleaving. Windows/macOS lane equivalents are defined
in the lane header, implementation staged later.

Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): close cross-vendor review findings — handle lifetimes, pidfd races, force-stop scoping

- B1: cleanupOrphanBackend closes the verified handle before waiting for
  exit, mirroring shutdown.go; an open Windows handle kept the dead PID
  allocated and made every orphan-backend recovery time out.
- B2: Windows procid checks GetExitCodeProcess after OpenProcess, so a
  terminated process kept resolvable by a third-party handle no longer
  verifies; "Verify == true implies running" now holds on all platforms.
- B3: Linux procid.Open opens the pidfd BEFORE token verification; the
  pidfd pins the PID number, closing the reuse window where a later
  Signal could kill an unrelated replacement process.
- B4: PidfdSendSignal ESRCH on a fatal signal is treated as successful
  termination, matching the fallback path (same for Windows terminate).
- B5: ForceStopUnverified now covers the backend (proxy-child) record
  with the same safety checks, and CanForceStopUnverified accepts
  refusals where the backend is also (or only) unverifiable — the
  paired-legacy pre-v2 topology was locked out of its own advertised
  recovery. bd dolt stop --force reports the backend record too.
- B6a: an unverified live PID is only signaled when its command line
  ties it to this workspace (Linux /proc cmdline, Darwin ps args);
  Windows cannot read another process's command line and now refuses,
  which beats basename-only killing. Clear errors name the manual remedy.
- B6b: one stable OS handle (Linux pidfd, Windows process handle) spans
  inspection and signaling, removing the inspect/kill TOCTOU.
- S1: control-listener accept errors are non-fatal — log and retry with
  capped exponential backoff instead of tearing down a healthy proxy.
- S2: requireUnverifiableRecord surfaces RootID resolution errors
  instead of failing open into the destructive branch.
- S3: killSpawnedChild falls back to cmd.Process.Kill for our own
  un-reaped child when no verified handle exists, instead of leaking it.
- S4: adoption handshake accepts reply schema v2 or newer, matching
  pidfile.ValidateV2 forward-compat policy.
- S5: errors.Is for ERROR_INVALID_PARAMETER.
- Tests: pidfd ESRCH-after-exit and Open-after-reap classification;
  paired-legacy force recovery end to end; foreign-workspace bd process
  refused; Windows Verify-after-terminate (compile-checked); schema-v3
  handshake adoption; accept-loop survival under persistent errors.

Agent-Signature: claude-fable-5-medium on behalf of maphew

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 12:52:51 -07:00

329 lines
10 KiB
Go

package proxy
import (
"errors"
"fmt"
"path/filepath"
"strings"
"time"
"github.com/steveyegge/beads/internal/lockfile"
"github.com/steveyegge/beads/internal/storage/dbproxy/identity"
"github.com/steveyegge/beads/internal/storage/dbproxy/pidfile"
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
"github.com/steveyegge/beads/internal/storage/dbproxy/util"
)
// ForceStopOptions configures ForceStopUnverified.
type ForceStopOptions struct {
Timeout time.Duration
}
// ForceStopReport truthfully describes which irreversible force-stop actions
// completed, including partial progress returned alongside an error. The
// top-level fields describe the proxy record; Backend, when non-nil, carries
// the same fields for the backend (proxy-child) record.
type ForceStopReport struct {
RecordPath string
PID int
Executable string
RecordFound bool
LockWasHeld bool
ProcessWasGone bool
SignalSent bool
QuarantinedPath string
Backend *ForceStopReport
}
// ForceStopUnverified is the narrow recovery primitive used by
// `bd dolt stop --force` after an ordinary verified Shutdown refuses an
// unverifiable proxy or backend record. It applies the same procedure to the
// proxy record (proxy.pid) and the backend record (proxy-child.pid): a
// pre-v2 managed-local deployment leaves BOTH as legacy records, so covering
// only the proxy record would lock the advertised recovery out of the real
// upgrade topology.
//
// For each record: when its lock is free, the function holds it while
// inspecting, signaling, and quarantining the unchanged record. When the lock
// is held (the usual pre-upgrade-proxy case), it first inspects and signals
// the live PID, waits for the lock to become free, then quarantines only if
// the record is unchanged. Both flows accept an already-gone recorded
// process. An unverified live PID is never signaled unless its executable
// basename is exactly bd or dolt (with an optional .exe suffix) AND its
// command line scopes it to this workspace; where the platform cannot
// establish that scope, force-stop refuses rather than guessing.
func ForceStopUnverified(rootDir string, opts ...ForceStopOptions) (ForceStopReport, error) {
report := ForceStopReport{RecordPath: pidfile.Path(rootDir, PIDFileName)}
if len(opts) > 1 {
return report, errors.New("proxy.ForceStopUnverified: at most one options value is allowed")
}
timeout := shutdownConfirmDeadline
if len(opts) == 1 && opts[0].Timeout != 0 {
timeout = opts[0].Timeout
}
if timeout <= 0 {
return report, fmt.Errorf("proxy.ForceStopUnverified: timeout must be positive, got %s", timeout)
}
if err := advanceStopEpoch(rootDir); err != nil {
return report, fmt.Errorf("proxy.ForceStopUnverified: publish stop epoch: %w", err)
}
proxyErr := forceStopRecord(rootDir, LockFileName, PIDFileName, pidfile.KindProxy, timeout, &report)
backendReport := ForceStopReport{RecordPath: pidfile.Path(rootDir, server.PIDFileName)}
backendErr := forceStopRecord(
rootDir,
server.LockFileName,
server.PIDFileName,
pidfile.KindDoltBackend,
timeout,
&backendReport,
)
if backendReport.RecordFound || backendErr != nil {
report.Backend = &backendReport
}
return report, errors.Join(proxyErr, backendErr)
}
// forceStopRecord runs the inspect-signal-quarantine procedure for one
// process record, writing partial progress into report as it goes.
func forceStopRecord(
rootDir string,
lockName string,
pidName string,
wantKind string,
timeout time.Duration,
report *ForceStopReport,
) error {
deadline := time.Now().Add(timeout)
lockPath := filepath.Join(rootDir, lockName)
lock, err := util.TryLock(lockPath)
if err == nil {
defer lock.Unlock()
return forceStopRecordLocked(rootDir, pidName, wantKind, deadline, report)
}
if !lockfile.IsLocked(err) {
return fmt.Errorf("proxy.ForceStopUnverified: probe %s: %w", lockPath, err)
}
report.LockWasHeld = true
record, err := readForceStopRecord(rootDir, pidName, report)
if err != nil || record == nil {
return err
}
if err := requireUnverifiableRecord(rootDir, record, wantKind); err != nil {
return err
}
if err := inspectAndStopUnverifiedPID(rootDir, record.Pid, deadline, report); err != nil {
return err
}
lock, err = acquireForceStopLock(lockPath, deadline)
if err != nil {
return err
}
defer lock.Unlock()
return quarantineForceStopRecord(rootDir, pidName, record, report)
}
func forceStopRecordLocked(
rootDir string,
pidName string,
wantKind string,
deadline time.Time,
report *ForceStopReport,
) error {
record, err := readForceStopRecord(rootDir, pidName, report)
if err != nil || record == nil {
return err
}
if err := requireUnverifiableRecord(rootDir, record, wantKind); err != nil {
return err
}
if err := inspectAndStopUnverifiedPID(rootDir, record.Pid, deadline, report); err != nil {
return err
}
return quarantineForceStopRecord(rootDir, pidName, record, report)
}
func readForceStopRecord(rootDir, pidName string, report *ForceStopReport) (*pidfile.PidFile, error) {
record, err := pidfile.Read(rootDir, pidName)
if err != nil {
if isMalformedPIDFileError(err) {
return nil, unverifiableProcessError(
"force-stop",
report.RecordPath,
0,
err,
unverifiableProcessChecks{},
)
}
return nil, fmt.Errorf("proxy.ForceStopUnverified: read %s: %w", report.RecordPath, err)
}
if record == nil {
return nil, nil
}
report.RecordFound = true
report.PID = record.Pid
return record, nil
}
func requireUnverifiableRecord(rootDir string, record *pidfile.PidFile, wantKind string) error {
if err := record.ValidateV2(wantKind); err != nil {
return nil
}
rootID, err := identity.RootID(rootDir)
if err != nil {
// Failing open here would route a possibly-verifiable record into the
// destructive force path; surface the identity failure instead.
return fmt.Errorf("proxy.ForceStopUnverified: resolve workspace identity: %w", err)
}
if record.RootID == rootID {
return errors.New(
"proxy.ForceStopUnverified: record has a verifiable v2 workspace identity; use proxy.Shutdown",
)
}
return nil
}
func inspectAndStopUnverifiedPID(rootDir string, pid int, deadline time.Time, report *ForceStopReport) error {
if pid <= 0 {
return fmt.Errorf("proxy.ForceStopUnverified: record %s has invalid pid %d", report.RecordPath, pid)
}
// One stable handle covers inspection and signaling, so the PID cannot be
// recycled between the executable check and the kill on platforms with a
// pinning primitive (Linux pidfd, Windows process handle).
proc, gone, err := openUnverifiedProcess(pid)
if err != nil {
return fmt.Errorf("proxy.ForceStopUnverified: open pid %d: %w", pid, err)
}
if gone {
report.ProcessWasGone = true
return nil
}
defer proc.close()
executable, gone, err := proc.executableBasename()
if err != nil {
return fmt.Errorf("proxy.ForceStopUnverified: inspect executable for pid %d: %w", pid, err)
}
if gone {
report.ProcessWasGone = true
return nil
}
executable = normalizeForceStopExecutable(executable)
report.Executable = executable
if executable != "bd" && executable != "dolt" {
return fmt.Errorf(
"proxy.ForceStopUnverified: refusing to signal pid %d from %s: executable basename is %q, want bd or dolt",
pid,
report.RecordPath,
executable,
)
}
// Basename alone would let a recycled PID now running an unrelated bd or
// dolt be killed; require the command line to tie the process to THIS
// workspace, and refuse when that scope cannot be established.
scoped, gone, err := proc.commandLineContains(rootDir)
if err != nil {
return fmt.Errorf(
"proxy.ForceStopUnverified: refusing to signal pid %d from %s: workspace scope could not be established (%v); stop the process manually, then quarantine the record by renaming %s to %s.stale-<unix-timestamp> before retrying",
pid,
report.RecordPath,
err,
report.RecordPath,
report.RecordPath,
)
}
if gone {
report.ProcessWasGone = true
return nil
}
if !scoped {
return fmt.Errorf(
"proxy.ForceStopUnverified: refusing to signal pid %d from %s: its command line does not reference workspace %s, so it may be an unrelated %s process; stop it manually if it is yours, then quarantine the record by renaming %s to %s.stale-<unix-timestamp> before retrying",
pid,
report.RecordPath,
rootDir,
executable,
report.RecordPath,
report.RecordPath,
)
}
gone, err = proc.kill()
if err != nil {
return fmt.Errorf("proxy.ForceStopUnverified: signal pid %d: %w", pid, err)
}
if gone {
report.ProcessWasGone = true
return nil
}
report.SignalSent = true
for {
exited, err := proc.exited()
if err != nil {
return fmt.Errorf("proxy.ForceStopUnverified: confirm pid %d exit: %w", pid, err)
}
if exited {
return nil
}
if time.Now().After(deadline) {
return fmt.Errorf("proxy.ForceStopUnverified: timeout waiting for pid %d to exit", pid)
}
time.Sleep(shutdownConfirmPoll)
}
}
func normalizeForceStopExecutable(name string) string {
name = strings.TrimSpace(filepath.Base(name))
name = strings.TrimSuffix(name, " (deleted)")
name = strings.TrimSuffix(strings.ToLower(name), ".exe")
return name
}
func acquireForceStopLock(lockPath string, deadline time.Time) (*util.Lock, error) {
for {
lock, err := util.TryLock(lockPath)
if err == nil {
return lock, nil
}
if !lockfile.IsLocked(err) {
return nil, fmt.Errorf("proxy.ForceStopUnverified: acquire %s: %w", lockPath, err)
}
if time.Now().After(deadline) {
return nil, fmt.Errorf("proxy.ForceStopUnverified: timeout acquiring %s after signaling", lockPath)
}
time.Sleep(shutdownConfirmPoll)
}
}
func quarantineForceStopRecord(
rootDir string,
pidName string,
record *pidfile.PidFile,
report *ForceStopReport,
) error {
current, err := pidfile.Read(rootDir, pidName)
if err != nil {
return fmt.Errorf("proxy.ForceStopUnverified: re-read %s: %w", report.RecordPath, err)
}
if current == nil {
return nil
}
if *current != *record {
return fmt.Errorf(
"proxy.ForceStopUnverified: record %s changed after pid %d was stopped; refusing to quarantine the replacement",
report.RecordPath,
record.Pid,
)
}
target, err := quarantineRecord(rootDir, pidName, time.Now())
if err != nil {
return fmt.Errorf("proxy.ForceStopUnverified: quarantine %s: %w", report.RecordPath, err)
}
report.QuarantinedPath = target
return nil
}