beads/internal/procid/procid_linux.go
matt wilkie 21669eca60
fix(dbproxy): verified identity for the managed proxied-server lifecycle (adoption, cleanup, kills) (#5013)
* feat(dbproxy): add procid birth-token package and pidfile schema v2 fields

Foundation for verified proxy/backend adoption and safe stale-process
cleanup (mybd-psxg.5, PR-A stage 1): internal/procid provides versioned
process-birth tokens (Linux boot_id+starttime, Windows creation FILETIME,
Darwin p_starttime) with TOCTOU-safe signaling handles (pidfd on Linux,
re-verified process handle on Windows); pidfile gains additive schema-v2
fields (schema, kind, birth, root_id, control_port) and sentinel-error
validation. No behavior changes; existing v1 pidfiles still parse.

The procid Token/Verify shape intentionally matches the API requested in
gastownhall/beads#4513 so internal/doltserver can adopt it later.

Agent-Signature: codex-gpt-5.6-terra-medium on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* feat(dbproxy): publish verifiable proxy identity (secret, control handshake, v2 pidfiles)

Stage 2 of mybd-psxg.5 (PR-A): the managed proxy now proves who it is.
New leaf package internal/storage/dbproxy/identity provides the workspace
RootID (sha256 of the resolved root path), a rotated 0600 proxy.secret,
and the IDENT control protocol (client Identify + IdentReply). The proxy
opens a loopback control listener that answers only to the workspace
secret, and both pidfile writers publish schema-v2 records (kind, birth
token, root_id, control_port) captured via procid. A proxy or backend
that cannot prove its own identity fails startup instead of publishing.

Adoption, cleanup, and kill behavior are unchanged in this stage; the
consumers land next so each step stays reviewable.

Agent-Signature: codex-gpt-5.6-terra-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): verified adoption, fail-closed cleanup, and handle-safe kills

Stage 3 of mybd-psxg.5 (PR-A), the security core. Endpoint discovery now
classifies records with typed statuses instead of adopting any listener
that answers a TCP dial: adoption requires a schema-v2 pidfile, a live
birth-token match, the authenticated IDENT handshake, and root/pid/birth/
port cross-checks. Legacy and malformed records fail closed (quarantined
under proxy.lock, never adopted, never killed); quarantines are
timestamped renames with a 30-day sweep, so no forensic record is ever
unlinked.

Every kill site (spawn-path backend cleanup and bd dolt stop's Shutdown)
signals only through a procid.Handle opened against the recorded birth
token, refusing unverifiable live processes with an actionable error.
Orphaned Dolt backends left by a SIGKILLed proxy supervisor are now
detected regardless of child-lock state and reaped only after
verification. A proxy.spawn marker plus a persistent stop epoch closes
the window where bd dolt stop could report success while a fork already
in flight published a listener afterwards.

Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): harden identity fixes from cross-vendor adversarial review

Fixes every accepted finding from paired independent reviews of the
first three stages (reports in the coordination repo, 2026-07-24):

- darwin: classify wrapped ESRCH and the zero-length/EIO missing-pid
  sysctl result as process-gone; stale pidfiles self-heal again.
- fallback signaling (darwin, non-pidfd linux): a target that exits
  after a fatal signal is a successful kill, not an error.
- Shutdown verifies root_id on the proxy record too (copied-workspace
  kill hazard); positional bool pair replaced by killRecordChecks.
- birth-token mismatch now classifies as provably-dead everywhere;
  dead legacy/malformed records quarantine and self-heal; refusal
  messages name the record file and manual remedy.
- discovery-time verify/secret/root errors map to a non-adopting
  status routed through quarantine-under-lock instead of wedging
  before the lock probe.
- boot_id read failures are environmental errors, never process-gone
  (closes the one fail-open path found in review).
- ForceStopUnverified: executable-identity-checked operator escape for
  pre-upgrade/unverifiable records, quarantines after, reports
  truthfully; CLI wiring intentionally deferred.
- handshake replies are HMAC'd against a client nonce; constant-time
  secret compare; bounded accept failures and handshake concurrency;
  migration purge covers proxy.secret and quarantine files; spawn
  marker cleared via defer; unix-only constraints on sleep tests.

Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* feat(dbproxy): wire bd dolt stop --force and prove the lifecycle lane end to end

Stage 4 of mybd-psxg.5 (PR-A). bd dolt stop gains --force, routed through
proxy.ForceStopUnverified only after the backend is confirmed stopped:
it verifies executable identity before signaling, quarantines the record
afterward, refuses foreign executables and verifiable v2 records, and
reports text/JSON truthfully. The managed-local lane
(BEADS_TEST_PROXIED_LOCAL=1) now exercises the whole security model
against a real dolt sql-server: foreign-listener rejection, reused-PID
no-kill, legacy fail-closed (free lock, held lock refusing a foreign
executable, held lock force-stopping a genuine bd), orphan-backend
reaping after a SIGKILLed supervisor, 5-process cold-start convergence,
and stop/start interleaving. Windows/macOS lane equivalents are defined
in the lane header, implementation staged later.

Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): close cross-vendor review findings — handle lifetimes, pidfd races, force-stop scoping

- B1: cleanupOrphanBackend closes the verified handle before waiting for
  exit, mirroring shutdown.go; an open Windows handle kept the dead PID
  allocated and made every orphan-backend recovery time out.
- B2: Windows procid checks GetExitCodeProcess after OpenProcess, so a
  terminated process kept resolvable by a third-party handle no longer
  verifies; "Verify == true implies running" now holds on all platforms.
- B3: Linux procid.Open opens the pidfd BEFORE token verification; the
  pidfd pins the PID number, closing the reuse window where a later
  Signal could kill an unrelated replacement process.
- B4: PidfdSendSignal ESRCH on a fatal signal is treated as successful
  termination, matching the fallback path (same for Windows terminate).
- B5: ForceStopUnverified now covers the backend (proxy-child) record
  with the same safety checks, and CanForceStopUnverified accepts
  refusals where the backend is also (or only) unverifiable — the
  paired-legacy pre-v2 topology was locked out of its own advertised
  recovery. bd dolt stop --force reports the backend record too.
- B6a: an unverified live PID is only signaled when its command line
  ties it to this workspace (Linux /proc cmdline, Darwin ps args);
  Windows cannot read another process's command line and now refuses,
  which beats basename-only killing. Clear errors name the manual remedy.
- B6b: one stable OS handle (Linux pidfd, Windows process handle) spans
  inspection and signaling, removing the inspect/kill TOCTOU.
- S1: control-listener accept errors are non-fatal — log and retry with
  capped exponential backoff instead of tearing down a healthy proxy.
- S2: requireUnverifiableRecord surfaces RootID resolution errors
  instead of failing open into the destructive branch.
- S3: killSpawnedChild falls back to cmd.Process.Kill for our own
  un-reaped child when no verified handle exists, instead of leaking it.
- S4: adoption handshake accepts reply schema v2 or newer, matching
  pidfile.ValidateV2 forward-compat policy.
- S5: errors.Is for ERROR_INVALID_PARAMETER.
- Tests: pidfd ESRCH-after-exit and Open-after-reap classification;
  paired-legacy force recovery end to end; foreign-workspace bd process
  refused; Windows Verify-after-terminate (compile-checked); schema-v3
  handshake adoption; accept-loop survival under persistent errors.

Agent-Signature: claude-fable-5-medium on behalf of maphew

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 12:52:51 -07:00

253 lines
6.6 KiB
Go

//go:build linux
package procid
import (
"errors"
"fmt"
"os"
"strconv"
"strings"
"syscall"
"time"
"golang.org/x/sys/unix"
)
const fallbackSignalConfirmTimeout = 2 * time.Second
var (
bootIDPath = "/proc/sys/kernel/random/boot_id"
pidfdOpen = unix.PidfdOpen
)
// Handle identifies a process through a pidfd when the running kernel supports
// it. On older kernels without pidfds, it falls back to verify, signal, verify;
// that fallback retains a small PID-reuse race.
type Handle struct {
pid int
token Token
pidfd int
}
// Capture resolves pid's current process-birth token.
func Capture(pid int) (Token, error) {
bootID, err := os.ReadFile(bootIDPath)
if err != nil {
// Do not unwrap this environmental failure: a missing or inaccessible
// boot_id is not evidence that the requested process is gone.
return "", &bootIDReadError{path: bootIDPath, err: err}
}
startTime, err := processStartTime(pid)
if err != nil {
return "", err
}
return Token("linux-v1:" + strings.TrimSpace(string(bootID)) + ":" + startTime), nil
}
// Verify reports whether pid still refers to tok. A process which has exited
// returns false, nil.
func Verify(pid int, tok Token) (bool, error) {
current, err := Capture(pid)
if err != nil {
if isGone(err) {
return false, nil
}
return false, err
}
return current == tok, nil
}
// Open verifies pid's token and opens a handle suitable for safe signaling.
//
// The pidfd is opened before the token check: a pidfd pins the PID number
// against reuse, so verifying afterwards proves the fd refers to the process
// the token describes. Verifying first would leave a window where the
// verified process exits, the PID is recycled, and the pidfd targets the
// unrelated replacement.
func Open(pid int, tok Token) (*Handle, error) {
fd, err := pidfdOpen(pid, 0)
if err != nil {
if !errors.Is(err, unix.ENOSYS) {
return nil, fmt.Errorf("procid: pidfd open %d: %w", pid, err)
}
// Kernel without pidfds: fall back to verify-then-signal, which
// retains the documented small PID-reuse race.
match, verifyErr := Verify(pid, tok)
if verifyErr != nil {
return nil, verifyErr
}
if !match {
return nil, fmt.Errorf("procid: process %d does not match token", pid)
}
return &Handle{pid: pid, token: tok, pidfd: -1}, nil
}
match, verifyErr := Verify(pid, tok)
if verifyErr != nil {
_ = unix.Close(fd)
return nil, verifyErr
}
if !match {
_ = unix.Close(fd)
return nil, fmt.Errorf("procid: process %d does not match token", pid)
}
return &Handle{pid: pid, token: tok, pidfd: fd}, nil
}
// Signal sends sig to the verified process.
func (h *Handle) Signal(sig os.Signal) error {
if h.pidfd >= 0 {
unixSig, ok := sig.(syscall.Signal)
if !ok {
return fmt.Errorf("procid: unsupported signal %v", sig)
}
if err := unix.PidfdSendSignal(h.pidfd, unixSig, nil, 0); err != nil {
if isFatalSignal(unixSig) && errors.Is(err, unix.ESRCH) {
// The target exited on its own after Open; a fatal signal's
// goal is already met, matching the fallback path.
return nil
}
return fmt.Errorf("procid: pidfd signal %d: %w", h.pid, err)
}
return nil
}
if err := h.verifyThenSignal(sig); err != nil {
return err
}
return nil
}
// Kill sends SIGKILL to the verified process.
func (h *Handle) Kill() error { return h.Signal(syscall.SIGKILL) }
// Close releases the underlying pidfd, if one was opened.
func (h *Handle) Close() error {
if h.pidfd < 0 {
return nil
}
err := unix.Close(h.pidfd)
h.pidfd = -1
if err != nil {
return fmt.Errorf("procid: close pidfd: %w", err)
}
return nil
}
func (h *Handle) verifyThenSignal(sig os.Signal) error {
match, err := Verify(h.pid, h.token)
if err != nil {
return err
}
if !match {
return fmt.Errorf("procid: process %d no longer matches token", h.pid)
}
unixSig, ok := sig.(syscall.Signal)
if !ok {
return fmt.Errorf("procid: unsupported signal %v", sig)
}
if err := syscall.Kill(h.pid, unixSig); err != nil {
if isFatalSignal(unixSig) && errors.Is(err, unix.ESRCH) {
return nil
}
return fmt.Errorf("procid: signal %d: %w", h.pid, err)
}
if isFatalSignal(unixSig) {
return h.confirmFatalSignal()
}
match, err = Verify(h.pid, h.token)
if err != nil {
return err
}
if !match {
return fmt.Errorf("procid: process %d no longer matches token", h.pid)
}
return nil
}
func (h *Handle) confirmFatalSignal() error {
deadline := time.Now().Add(fallbackSignalConfirmTimeout)
for {
match, err := Verify(h.pid, h.token)
if err != nil {
return err
}
if !match {
return nil
}
if time.Now().After(deadline) {
return fmt.Errorf(
"procid: process %d still matches token after fatal signal and %s re-check",
h.pid,
fallbackSignalConfirmTimeout,
)
}
time.Sleep(10 * time.Millisecond)
}
}
func isFatalSignal(sig syscall.Signal) bool {
return sig == syscall.SIGKILL || sig == syscall.SIGTERM
}
func processStartTime(pid int) (string, error) {
data, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
if err != nil {
return "", &processStatReadError{pid: pid, err: err}
}
return parseStartTime(string(data))
}
func parseStartTime(stat string) (string, error) {
endComm := strings.LastIndex(stat, ")")
if endComm == -1 {
return "", errors.New("procid: malformed proc stat: missing comm terminator")
}
fields := strings.Fields(stat[endComm+1:])
// The remainder starts with state (field 3), so starttime (field 22) is
// its twentieth field.
if len(fields) < 20 {
return "", errors.New("procid: malformed proc stat: missing starttime")
}
if fields[0] == "Z" || fields[0] == "X" || fields[0] == "x" {
return "", fmt.Errorf("procid: process is no longer running: %w", unix.ESRCH)
}
if _, err := strconv.ParseUint(fields[19], 10, 64); err != nil {
return "", fmt.Errorf("procid: malformed proc stat starttime: %w", err)
}
return fields[19], nil
}
type bootIDReadError struct {
path string
err error
}
func (e *bootIDReadError) Error() string {
return fmt.Sprintf("procid: read boot ID %s: %v", e.path, e.err)
}
type processStatReadError struct {
pid int
err error
}
func (e *processStatReadError) Error() string {
return fmt.Sprintf("procid: read stat for %d: %v", e.pid, e.err)
}
func (e *processStatReadError) Unwrap() error {
return e.err
}
func isGone(err error) bool {
var statErr *processStatReadError
return (errors.As(err, &statErr) && errors.Is(statErr.err, os.ErrNotExist)) ||
errors.Is(err, unix.ESRCH)
}
// IsProcessGone reports whether err means the referenced process no longer
// exists. Callers use it to distinguish a dead recorded process from a live
// process whose birth token does not match.
func IsProcessGone(err error) bool {
return isGone(err)
}