* feat(dbproxy): add procid birth-token package and pidfile schema v2 fields Foundation for verified proxy/backend adoption and safe stale-process cleanup (mybd-psxg.5, PR-A stage 1): internal/procid provides versioned process-birth tokens (Linux boot_id+starttime, Windows creation FILETIME, Darwin p_starttime) with TOCTOU-safe signaling handles (pidfd on Linux, re-verified process handle on Windows); pidfile gains additive schema-v2 fields (schema, kind, birth, root_id, control_port) and sentinel-error validation. No behavior changes; existing v1 pidfiles still parse. The procid Token/Verify shape intentionally matches the API requested in gastownhall/beads#4513 so internal/doltserver can adopt it later. Agent-Signature: codex-gpt-5.6-terra-medium on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * feat(dbproxy): publish verifiable proxy identity (secret, control handshake, v2 pidfiles) Stage 2 of mybd-psxg.5 (PR-A): the managed proxy now proves who it is. New leaf package internal/storage/dbproxy/identity provides the workspace RootID (sha256 of the resolved root path), a rotated 0600 proxy.secret, and the IDENT control protocol (client Identify + IdentReply). The proxy opens a loopback control listener that answers only to the workspace secret, and both pidfile writers publish schema-v2 records (kind, birth token, root_id, control_port) captured via procid. A proxy or backend that cannot prove its own identity fails startup instead of publishing. Adoption, cleanup, and kill behavior are unchanged in this stage; the consumers land next so each step stays reviewable. Agent-Signature: codex-gpt-5.6-terra-high on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): verified adoption, fail-closed cleanup, and handle-safe kills Stage 3 of mybd-psxg.5 (PR-A), the security core. Endpoint discovery now classifies records with typed statuses instead of adopting any listener that answers a TCP dial: adoption requires a schema-v2 pidfile, a live birth-token match, the authenticated IDENT handshake, and root/pid/birth/ port cross-checks. Legacy and malformed records fail closed (quarantined under proxy.lock, never adopted, never killed); quarantines are timestamped renames with a 30-day sweep, so no forensic record is ever unlinked. Every kill site (spawn-path backend cleanup and bd dolt stop's Shutdown) signals only through a procid.Handle opened against the recorded birth token, refusing unverifiable live processes with an actionable error. Orphaned Dolt backends left by a SIGKILLed proxy supervisor are now detected regardless of child-lock state and reaped only after verification. A proxy.spawn marker plus a persistent stop epoch closes the window where bd dolt stop could report success while a fork already in flight published a listener afterwards. Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): harden identity fixes from cross-vendor adversarial review Fixes every accepted finding from paired independent reviews of the first three stages (reports in the coordination repo, 2026-07-24): - darwin: classify wrapped ESRCH and the zero-length/EIO missing-pid sysctl result as process-gone; stale pidfiles self-heal again. - fallback signaling (darwin, non-pidfd linux): a target that exits after a fatal signal is a successful kill, not an error. - Shutdown verifies root_id on the proxy record too (copied-workspace kill hazard); positional bool pair replaced by killRecordChecks. - birth-token mismatch now classifies as provably-dead everywhere; dead legacy/malformed records quarantine and self-heal; refusal messages name the record file and manual remedy. - discovery-time verify/secret/root errors map to a non-adopting status routed through quarantine-under-lock instead of wedging before the lock probe. - boot_id read failures are environmental errors, never process-gone (closes the one fail-open path found in review). - ForceStopUnverified: executable-identity-checked operator escape for pre-upgrade/unverifiable records, quarantines after, reports truthfully; CLI wiring intentionally deferred. - handshake replies are HMAC'd against a client nonce; constant-time secret compare; bounded accept failures and handshake concurrency; migration purge covers proxy.secret and quarantine files; spawn marker cleared via defer; unix-only constraints on sleep tests. Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * feat(dbproxy): wire bd dolt stop --force and prove the lifecycle lane end to end Stage 4 of mybd-psxg.5 (PR-A). bd dolt stop gains --force, routed through proxy.ForceStopUnverified only after the backend is confirmed stopped: it verifies executable identity before signaling, quarantines the record afterward, refuses foreign executables and verifiable v2 records, and reports text/JSON truthfully. The managed-local lane (BEADS_TEST_PROXIED_LOCAL=1) now exercises the whole security model against a real dolt sql-server: foreign-listener rejection, reused-PID no-kill, legacy fail-closed (free lock, held lock refusing a foreign executable, held lock force-stopping a genuine bd), orphan-backend reaping after a SIGKILLed supervisor, 5-process cold-start convergence, and stop/start interleaving. Windows/macOS lane equivalents are defined in the lane header, implementation staged later. Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): close cross-vendor review findings — handle lifetimes, pidfd races, force-stop scoping - B1: cleanupOrphanBackend closes the verified handle before waiting for exit, mirroring shutdown.go; an open Windows handle kept the dead PID allocated and made every orphan-backend recovery time out. - B2: Windows procid checks GetExitCodeProcess after OpenProcess, so a terminated process kept resolvable by a third-party handle no longer verifies; "Verify == true implies running" now holds on all platforms. - B3: Linux procid.Open opens the pidfd BEFORE token verification; the pidfd pins the PID number, closing the reuse window where a later Signal could kill an unrelated replacement process. - B4: PidfdSendSignal ESRCH on a fatal signal is treated as successful termination, matching the fallback path (same for Windows terminate). - B5: ForceStopUnverified now covers the backend (proxy-child) record with the same safety checks, and CanForceStopUnverified accepts refusals where the backend is also (or only) unverifiable — the paired-legacy pre-v2 topology was locked out of its own advertised recovery. bd dolt stop --force reports the backend record too. - B6a: an unverified live PID is only signaled when its command line ties it to this workspace (Linux /proc cmdline, Darwin ps args); Windows cannot read another process's command line and now refuses, which beats basename-only killing. Clear errors name the manual remedy. - B6b: one stable OS handle (Linux pidfd, Windows process handle) spans inspection and signaling, removing the inspect/kill TOCTOU. - S1: control-listener accept errors are non-fatal — log and retry with capped exponential backoff instead of tearing down a healthy proxy. - S2: requireUnverifiableRecord surfaces RootID resolution errors instead of failing open into the destructive branch. - S3: killSpawnedChild falls back to cmd.Process.Kill for our own un-reaped child when no verified handle exists, instead of leaking it. - S4: adoption handshake accepts reply schema v2 or newer, matching pidfile.ValidateV2 forward-compat policy. - S5: errors.Is for ERROR_INVALID_PARAMETER. - Tests: pidfd ESRCH-after-exit and Open-after-reap classification; paired-legacy force recovery end to end; foreign-workspace bd process refused; Windows Verify-after-terminate (compile-checked); schema-v3 handshake adoption; accept-loop survival under persistent errors. Agent-Signature: claude-fable-5-medium on behalf of maphew --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
329 lines
10 KiB
Go
329 lines
10 KiB
Go
package proxy
|
|
|
|
import (
|
|
"errors"
|
|
"fmt"
|
|
"path/filepath"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/steveyegge/beads/internal/lockfile"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/identity"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/pidfile"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/util"
|
|
)
|
|
|
|
// ForceStopOptions configures ForceStopUnverified.
|
|
type ForceStopOptions struct {
|
|
Timeout time.Duration
|
|
}
|
|
|
|
// ForceStopReport truthfully describes which irreversible force-stop actions
|
|
// completed, including partial progress returned alongside an error. The
|
|
// top-level fields describe the proxy record; Backend, when non-nil, carries
|
|
// the same fields for the backend (proxy-child) record.
|
|
type ForceStopReport struct {
|
|
RecordPath string
|
|
PID int
|
|
Executable string
|
|
RecordFound bool
|
|
LockWasHeld bool
|
|
ProcessWasGone bool
|
|
SignalSent bool
|
|
QuarantinedPath string
|
|
Backend *ForceStopReport
|
|
}
|
|
|
|
// ForceStopUnverified is the narrow recovery primitive used by
|
|
// `bd dolt stop --force` after an ordinary verified Shutdown refuses an
|
|
// unverifiable proxy or backend record. It applies the same procedure to the
|
|
// proxy record (proxy.pid) and the backend record (proxy-child.pid): a
|
|
// pre-v2 managed-local deployment leaves BOTH as legacy records, so covering
|
|
// only the proxy record would lock the advertised recovery out of the real
|
|
// upgrade topology.
|
|
//
|
|
// For each record: when its lock is free, the function holds it while
|
|
// inspecting, signaling, and quarantining the unchanged record. When the lock
|
|
// is held (the usual pre-upgrade-proxy case), it first inspects and signals
|
|
// the live PID, waits for the lock to become free, then quarantines only if
|
|
// the record is unchanged. Both flows accept an already-gone recorded
|
|
// process. An unverified live PID is never signaled unless its executable
|
|
// basename is exactly bd or dolt (with an optional .exe suffix) AND its
|
|
// command line scopes it to this workspace; where the platform cannot
|
|
// establish that scope, force-stop refuses rather than guessing.
|
|
func ForceStopUnverified(rootDir string, opts ...ForceStopOptions) (ForceStopReport, error) {
|
|
report := ForceStopReport{RecordPath: pidfile.Path(rootDir, PIDFileName)}
|
|
if len(opts) > 1 {
|
|
return report, errors.New("proxy.ForceStopUnverified: at most one options value is allowed")
|
|
}
|
|
timeout := shutdownConfirmDeadline
|
|
if len(opts) == 1 && opts[0].Timeout != 0 {
|
|
timeout = opts[0].Timeout
|
|
}
|
|
if timeout <= 0 {
|
|
return report, fmt.Errorf("proxy.ForceStopUnverified: timeout must be positive, got %s", timeout)
|
|
}
|
|
if err := advanceStopEpoch(rootDir); err != nil {
|
|
return report, fmt.Errorf("proxy.ForceStopUnverified: publish stop epoch: %w", err)
|
|
}
|
|
|
|
proxyErr := forceStopRecord(rootDir, LockFileName, PIDFileName, pidfile.KindProxy, timeout, &report)
|
|
|
|
backendReport := ForceStopReport{RecordPath: pidfile.Path(rootDir, server.PIDFileName)}
|
|
backendErr := forceStopRecord(
|
|
rootDir,
|
|
server.LockFileName,
|
|
server.PIDFileName,
|
|
pidfile.KindDoltBackend,
|
|
timeout,
|
|
&backendReport,
|
|
)
|
|
if backendReport.RecordFound || backendErr != nil {
|
|
report.Backend = &backendReport
|
|
}
|
|
return report, errors.Join(proxyErr, backendErr)
|
|
}
|
|
|
|
// forceStopRecord runs the inspect-signal-quarantine procedure for one
|
|
// process record, writing partial progress into report as it goes.
|
|
func forceStopRecord(
|
|
rootDir string,
|
|
lockName string,
|
|
pidName string,
|
|
wantKind string,
|
|
timeout time.Duration,
|
|
report *ForceStopReport,
|
|
) error {
|
|
deadline := time.Now().Add(timeout)
|
|
lockPath := filepath.Join(rootDir, lockName)
|
|
lock, err := util.TryLock(lockPath)
|
|
if err == nil {
|
|
defer lock.Unlock()
|
|
return forceStopRecordLocked(rootDir, pidName, wantKind, deadline, report)
|
|
}
|
|
if !lockfile.IsLocked(err) {
|
|
return fmt.Errorf("proxy.ForceStopUnverified: probe %s: %w", lockPath, err)
|
|
}
|
|
report.LockWasHeld = true
|
|
|
|
record, err := readForceStopRecord(rootDir, pidName, report)
|
|
if err != nil || record == nil {
|
|
return err
|
|
}
|
|
if err := requireUnverifiableRecord(rootDir, record, wantKind); err != nil {
|
|
return err
|
|
}
|
|
if err := inspectAndStopUnverifiedPID(rootDir, record.Pid, deadline, report); err != nil {
|
|
return err
|
|
}
|
|
|
|
lock, err = acquireForceStopLock(lockPath, deadline)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
defer lock.Unlock()
|
|
return quarantineForceStopRecord(rootDir, pidName, record, report)
|
|
}
|
|
|
|
func forceStopRecordLocked(
|
|
rootDir string,
|
|
pidName string,
|
|
wantKind string,
|
|
deadline time.Time,
|
|
report *ForceStopReport,
|
|
) error {
|
|
record, err := readForceStopRecord(rootDir, pidName, report)
|
|
if err != nil || record == nil {
|
|
return err
|
|
}
|
|
if err := requireUnverifiableRecord(rootDir, record, wantKind); err != nil {
|
|
return err
|
|
}
|
|
if err := inspectAndStopUnverifiedPID(rootDir, record.Pid, deadline, report); err != nil {
|
|
return err
|
|
}
|
|
return quarantineForceStopRecord(rootDir, pidName, record, report)
|
|
}
|
|
|
|
func readForceStopRecord(rootDir, pidName string, report *ForceStopReport) (*pidfile.PidFile, error) {
|
|
record, err := pidfile.Read(rootDir, pidName)
|
|
if err != nil {
|
|
if isMalformedPIDFileError(err) {
|
|
return nil, unverifiableProcessError(
|
|
"force-stop",
|
|
report.RecordPath,
|
|
0,
|
|
err,
|
|
unverifiableProcessChecks{},
|
|
)
|
|
}
|
|
return nil, fmt.Errorf("proxy.ForceStopUnverified: read %s: %w", report.RecordPath, err)
|
|
}
|
|
if record == nil {
|
|
return nil, nil
|
|
}
|
|
report.RecordFound = true
|
|
report.PID = record.Pid
|
|
return record, nil
|
|
}
|
|
|
|
func requireUnverifiableRecord(rootDir string, record *pidfile.PidFile, wantKind string) error {
|
|
if err := record.ValidateV2(wantKind); err != nil {
|
|
return nil
|
|
}
|
|
rootID, err := identity.RootID(rootDir)
|
|
if err != nil {
|
|
// Failing open here would route a possibly-verifiable record into the
|
|
// destructive force path; surface the identity failure instead.
|
|
return fmt.Errorf("proxy.ForceStopUnverified: resolve workspace identity: %w", err)
|
|
}
|
|
if record.RootID == rootID {
|
|
return errors.New(
|
|
"proxy.ForceStopUnverified: record has a verifiable v2 workspace identity; use proxy.Shutdown",
|
|
)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
func inspectAndStopUnverifiedPID(rootDir string, pid int, deadline time.Time, report *ForceStopReport) error {
|
|
if pid <= 0 {
|
|
return fmt.Errorf("proxy.ForceStopUnverified: record %s has invalid pid %d", report.RecordPath, pid)
|
|
}
|
|
// One stable handle covers inspection and signaling, so the PID cannot be
|
|
// recycled between the executable check and the kill on platforms with a
|
|
// pinning primitive (Linux pidfd, Windows process handle).
|
|
proc, gone, err := openUnverifiedProcess(pid)
|
|
if err != nil {
|
|
return fmt.Errorf("proxy.ForceStopUnverified: open pid %d: %w", pid, err)
|
|
}
|
|
if gone {
|
|
report.ProcessWasGone = true
|
|
return nil
|
|
}
|
|
defer proc.close()
|
|
|
|
executable, gone, err := proc.executableBasename()
|
|
if err != nil {
|
|
return fmt.Errorf("proxy.ForceStopUnverified: inspect executable for pid %d: %w", pid, err)
|
|
}
|
|
if gone {
|
|
report.ProcessWasGone = true
|
|
return nil
|
|
}
|
|
executable = normalizeForceStopExecutable(executable)
|
|
report.Executable = executable
|
|
if executable != "bd" && executable != "dolt" {
|
|
return fmt.Errorf(
|
|
"proxy.ForceStopUnverified: refusing to signal pid %d from %s: executable basename is %q, want bd or dolt",
|
|
pid,
|
|
report.RecordPath,
|
|
executable,
|
|
)
|
|
}
|
|
|
|
// Basename alone would let a recycled PID now running an unrelated bd or
|
|
// dolt be killed; require the command line to tie the process to THIS
|
|
// workspace, and refuse when that scope cannot be established.
|
|
scoped, gone, err := proc.commandLineContains(rootDir)
|
|
if err != nil {
|
|
return fmt.Errorf(
|
|
"proxy.ForceStopUnverified: refusing to signal pid %d from %s: workspace scope could not be established (%v); stop the process manually, then quarantine the record by renaming %s to %s.stale-<unix-timestamp> before retrying",
|
|
pid,
|
|
report.RecordPath,
|
|
err,
|
|
report.RecordPath,
|
|
report.RecordPath,
|
|
)
|
|
}
|
|
if gone {
|
|
report.ProcessWasGone = true
|
|
return nil
|
|
}
|
|
if !scoped {
|
|
return fmt.Errorf(
|
|
"proxy.ForceStopUnverified: refusing to signal pid %d from %s: its command line does not reference workspace %s, so it may be an unrelated %s process; stop it manually if it is yours, then quarantine the record by renaming %s to %s.stale-<unix-timestamp> before retrying",
|
|
pid,
|
|
report.RecordPath,
|
|
rootDir,
|
|
executable,
|
|
report.RecordPath,
|
|
report.RecordPath,
|
|
)
|
|
}
|
|
|
|
gone, err = proc.kill()
|
|
if err != nil {
|
|
return fmt.Errorf("proxy.ForceStopUnverified: signal pid %d: %w", pid, err)
|
|
}
|
|
if gone {
|
|
report.ProcessWasGone = true
|
|
return nil
|
|
}
|
|
report.SignalSent = true
|
|
|
|
for {
|
|
exited, err := proc.exited()
|
|
if err != nil {
|
|
return fmt.Errorf("proxy.ForceStopUnverified: confirm pid %d exit: %w", pid, err)
|
|
}
|
|
if exited {
|
|
return nil
|
|
}
|
|
if time.Now().After(deadline) {
|
|
return fmt.Errorf("proxy.ForceStopUnverified: timeout waiting for pid %d to exit", pid)
|
|
}
|
|
time.Sleep(shutdownConfirmPoll)
|
|
}
|
|
}
|
|
|
|
func normalizeForceStopExecutable(name string) string {
|
|
name = strings.TrimSpace(filepath.Base(name))
|
|
name = strings.TrimSuffix(name, " (deleted)")
|
|
name = strings.TrimSuffix(strings.ToLower(name), ".exe")
|
|
return name
|
|
}
|
|
|
|
func acquireForceStopLock(lockPath string, deadline time.Time) (*util.Lock, error) {
|
|
for {
|
|
lock, err := util.TryLock(lockPath)
|
|
if err == nil {
|
|
return lock, nil
|
|
}
|
|
if !lockfile.IsLocked(err) {
|
|
return nil, fmt.Errorf("proxy.ForceStopUnverified: acquire %s: %w", lockPath, err)
|
|
}
|
|
if time.Now().After(deadline) {
|
|
return nil, fmt.Errorf("proxy.ForceStopUnverified: timeout acquiring %s after signaling", lockPath)
|
|
}
|
|
time.Sleep(shutdownConfirmPoll)
|
|
}
|
|
}
|
|
|
|
func quarantineForceStopRecord(
|
|
rootDir string,
|
|
pidName string,
|
|
record *pidfile.PidFile,
|
|
report *ForceStopReport,
|
|
) error {
|
|
current, err := pidfile.Read(rootDir, pidName)
|
|
if err != nil {
|
|
return fmt.Errorf("proxy.ForceStopUnverified: re-read %s: %w", report.RecordPath, err)
|
|
}
|
|
if current == nil {
|
|
return nil
|
|
}
|
|
if *current != *record {
|
|
return fmt.Errorf(
|
|
"proxy.ForceStopUnverified: record %s changed after pid %d was stopped; refusing to quarantine the replacement",
|
|
report.RecordPath,
|
|
record.Pid,
|
|
)
|
|
}
|
|
target, err := quarantineRecord(rootDir, pidName, time.Now())
|
|
if err != nil {
|
|
return fmt.Errorf("proxy.ForceStopUnverified: quarantine %s: %w", report.RecordPath, err)
|
|
}
|
|
report.QuarantinedPath = target
|
|
return nil
|
|
}
|