beads/internal/storage/dbproxy/proxy/server_test.go
matt wilkie 4c63e15107
fix(dbproxy): race-free proxy port allocation and managed loopback listener policy (#5024)
* feat(dbproxy): add procid birth-token package and pidfile schema v2 fields

Foundation for verified proxy/backend adoption and safe stale-process
cleanup (mybd-psxg.5, PR-A stage 1): internal/procid provides versioned
process-birth tokens (Linux boot_id+starttime, Windows creation FILETIME,
Darwin p_starttime) with TOCTOU-safe signaling handles (pidfd on Linux,
re-verified process handle on Windows); pidfile gains additive schema-v2
fields (schema, kind, birth, root_id, control_port) and sentinel-error
validation. No behavior changes; existing v1 pidfiles still parse.

The procid Token/Verify shape intentionally matches the API requested in
gastownhall/beads#4513 so internal/doltserver can adopt it later.

Agent-Signature: codex-gpt-5.6-terra-medium on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* feat(dbproxy): publish verifiable proxy identity (secret, control handshake, v2 pidfiles)

Stage 2 of mybd-psxg.5 (PR-A): the managed proxy now proves who it is.
New leaf package internal/storage/dbproxy/identity provides the workspace
RootID (sha256 of the resolved root path), a rotated 0600 proxy.secret,
and the IDENT control protocol (client Identify + IdentReply). The proxy
opens a loopback control listener that answers only to the workspace
secret, and both pidfile writers publish schema-v2 records (kind, birth
token, root_id, control_port) captured via procid. A proxy or backend
that cannot prove its own identity fails startup instead of publishing.

Adoption, cleanup, and kill behavior are unchanged in this stage; the
consumers land next so each step stays reviewable.

Agent-Signature: codex-gpt-5.6-terra-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): verified adoption, fail-closed cleanup, and handle-safe kills

Stage 3 of mybd-psxg.5 (PR-A), the security core. Endpoint discovery now
classifies records with typed statuses instead of adopting any listener
that answers a TCP dial: adoption requires a schema-v2 pidfile, a live
birth-token match, the authenticated IDENT handshake, and root/pid/birth/
port cross-checks. Legacy and malformed records fail closed (quarantined
under proxy.lock, never adopted, never killed); quarantines are
timestamped renames with a 30-day sweep, so no forensic record is ever
unlinked.

Every kill site (spawn-path backend cleanup and bd dolt stop's Shutdown)
signals only through a procid.Handle opened against the recorded birth
token, refusing unverifiable live processes with an actionable error.
Orphaned Dolt backends left by a SIGKILLed proxy supervisor are now
detected regardless of child-lock state and reaped only after
verification. A proxy.spawn marker plus a persistent stop epoch closes
the window where bd dolt stop could report success while a fork already
in flight published a listener afterwards.

Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): harden identity fixes from cross-vendor adversarial review

Fixes every accepted finding from paired independent reviews of the
first three stages (reports in the coordination repo, 2026-07-24):

- darwin: classify wrapped ESRCH and the zero-length/EIO missing-pid
  sysctl result as process-gone; stale pidfiles self-heal again.
- fallback signaling (darwin, non-pidfd linux): a target that exits
  after a fatal signal is a successful kill, not an error.
- Shutdown verifies root_id on the proxy record too (copied-workspace
  kill hazard); positional bool pair replaced by killRecordChecks.
- birth-token mismatch now classifies as provably-dead everywhere;
  dead legacy/malformed records quarantine and self-heal; refusal
  messages name the record file and manual remedy.
- discovery-time verify/secret/root errors map to a non-adopting
  status routed through quarantine-under-lock instead of wedging
  before the lock probe.
- boot_id read failures are environmental errors, never process-gone
  (closes the one fail-open path found in review).
- ForceStopUnverified: executable-identity-checked operator escape for
  pre-upgrade/unverifiable records, quarantines after, reports
  truthfully; CLI wiring intentionally deferred.
- handshake replies are HMAC'd against a client nonce; constant-time
  secret compare; bounded accept failures and handshake concurrency;
  migration purge covers proxy.secret and quarantine files; spawn
  marker cleared via defer; unix-only constraints on sleep tests.

Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* feat(dbproxy): wire bd dolt stop --force and prove the lifecycle lane end to end

Stage 4 of mybd-psxg.5 (PR-A). bd dolt stop gains --force, routed through
proxy.ForceStopUnverified only after the backend is confirmed stopped:
it verifies executable identity before signaling, quarantines the record
afterward, refuses foreign executables and verifiable v2 records, and
reports text/JSON truthfully. The managed-local lane
(BEADS_TEST_PROXIED_LOCAL=1) now exercises the whole security model
against a real dolt sql-server: foreign-listener rejection, reused-PID
no-kill, legacy fail-closed (free lock, held lock refusing a foreign
executable, held lock force-stopping a genuine bd), orphan-backend
reaping after a SIGKILLed supervisor, 5-process cold-start convergence,
and stop/start interleaving. Windows/macOS lane equivalents are defined
in the lane header, implementation staged later.

Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): close cross-vendor review findings — handle lifetimes, pidfd races, force-stop scoping

- B1: cleanupOrphanBackend closes the verified handle before waiting for
  exit, mirroring shutdown.go; an open Windows handle kept the dead PID
  allocated and made every orphan-backend recovery time out.
- B2: Windows procid checks GetExitCodeProcess after OpenProcess, so a
  terminated process kept resolvable by a third-party handle no longer
  verifies; "Verify == true implies running" now holds on all platforms.
- B3: Linux procid.Open opens the pidfd BEFORE token verification; the
  pidfd pins the PID number, closing the reuse window where a later
  Signal could kill an unrelated replacement process.
- B4: PidfdSendSignal ESRCH on a fatal signal is treated as successful
  termination, matching the fallback path (same for Windows terminate).
- B5: ForceStopUnverified now covers the backend (proxy-child) record
  with the same safety checks, and CanForceStopUnverified accepts
  refusals where the backend is also (or only) unverifiable — the
  paired-legacy pre-v2 topology was locked out of its own advertised
  recovery. bd dolt stop --force reports the backend record too.
- B6a: an unverified live PID is only signaled when its command line
  ties it to this workspace (Linux /proc cmdline, Darwin ps args);
  Windows cannot read another process's command line and now refuses,
  which beats basename-only killing. Clear errors name the manual remedy.
- B6b: one stable OS handle (Linux pidfd, Windows process handle) spans
  inspection and signaling, removing the inspect/kill TOCTOU.
- S1: control-listener accept errors are non-fatal — log and retry with
  capped exponential backoff instead of tearing down a healthy proxy.
- S2: requireUnverifiableRecord surfaces RootID resolution errors
  instead of failing open into the destructive branch.
- S3: killSpawnedChild falls back to cmd.Process.Kill for our own
  un-reaped child when no verified handle exists, instead of leaking it.
- S4: adoption handshake accepts reply schema v2 or newer, matching
  pidfile.ValidateV2 forward-compat policy.
- S5: errors.Is for ERROR_INVALID_PARAMETER.
- Tests: pidfd ESRCH-after-exit and Open-after-reap classification;
  paired-legacy force recovery end to end; foreign-workspace bd process
  refused; Windows Verify-after-terminate (compile-checked); schema-v3
  handshake adoption; accept-loop survival under persistent errors.

Agent-Signature: claude-fable-5-medium on behalf of maphew

* fix(dbproxy): race-free proxy port allocation and managed loopback listener policy

Companion to the identity PR (stacks on it). The parent no longer
pre-picks the proxy data port: port 0 flows to the child, which binds
127.0.0.1:0 and publishes the OS-assigned port through its verified
pidfile, eliminating the bind-close-launch race; explicitly configured
ports keep fixed-port failure semantics. PickFreePort survives only for
the dolt-config path, annotated for the follow-up RFC on backend port
collisions.

Managed dolt configs are now validated by a single numeric-loopback
listener policy at init and at runtime resolution, covering all config
paths (generated, BEADS_PROXIED_SERVER_CONFIG, sidecar, pre-marker,
hand-edited marker, interpolation fallback). Hostnames including
localhost are rejected with remediation text naming the
--proxied-server-external-* flags; an omitted host is accepted because
dolt servercfg defaults it. There is deliberately no bypass env:
managed mode assumes loopback for its root/empty-password account.
External proxied topology is exempt by construction.

Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi

* fix(dbproxy): whole-config managed listener policy, epoch-fenced startup, diagnosis fixes

Review findings from two independent passes over the listener-policy
commit (gastownhall/beads#5024), rebased onto the rewritten #5013 head:

- F1: the managed policy now covers the whole sql-server YAML, not just
  listener.host. validateManagedListenerHost is renamed
  validateManagedServerConfigPolicy and additionally rejects
  listener.socket (incl. the empty-string form, which maps to Dolt's
  default socket path), remotesapi.port, and any cluster block — all of
  which open non-loopback listeners for the root/empty-password child.
- F2: managed mode requires an explicit numeric loopback listener.host.
  An omitted host is no longer trusted: servercfg resolves it to
  "localhost", which CheckForUnixSocket also treats as license to open
  the shared-namespace default unix socket. This removes the dead
  net.ParseIP("localhost") branch and the omitted/explicit asymmetry.
- F3: one interpolation-aware policy check instead of five scattered
  raw-bytes ones. ensureProxiedServerConfig now delegates resolution to
  resolveOrCreateProxiedServerConfig and runs a single
  validateProxiedServerConfig (YamlConfigFromFile, ${VAR}-expanding) on
  the resolved path, so a marker config with an interpolated host is
  judged on what it expands to.
- F4: the proxy child re-checks the stop epoch immediately before
  publishing proxy.pid and aborts if it advanced (the spawn marker is
  cleared at lock acquisition, so a `bd dolt stop` racing a slow backend
  start otherwise has no record to wait on). The parent's observed epoch
  is threaded through a new db-proxy-child --stop-epoch flag into
  ProxyOpts.StopEpoch.
- F5a/b: child-exit diagnosis distinguishes a LockHeldExitCode spawn
  race from real failures, points at the child's log path otherwise, and
  timeout messages say "its OS-assigned port" instead of "port 0".
- F5c/d: validateManagedProxiedServerConfigAtInit now honors
  --proxied-server-root-path (the sidecar is not persisted yet at
  validation time) and drops the redundant explicit-config re-validation
  already done at flag parse.
- F5e: --proxied-server-config-path flag help states the hard managed
  constraint (docs/cli-reference regenerates from the release pin, so
  only the source help string changes here).
- F5f: the listener-policy lifecycle test documents its accepted
  PickFreePort EADDRINUSE flake window.

Tests: whole-config policy table (socket/remotesapi/cluster/omitted/
localhost/loopback forms), marker ${VAR} host interpolation accept and
reject, root-path-flag validation at init (unit + gated integration),
and a deterministic stop-epoch-advance-during-startup abort test.

Agent-Signature: claude-fable-5-medium on behalf of maphew

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 22:29:11 -07:00

806 lines
21 KiB
Go

package proxy_test
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"net"
"os"
"path/filepath"
"strconv"
"sync"
"testing"
"time"
"github.com/steveyegge/beads/internal/procid"
"github.com/steveyegge/beads/internal/storage/dbproxy/identity"
"github.com/steveyegge/beads/internal/storage/dbproxy/pidfile"
"github.com/steveyegge/beads/internal/storage/dbproxy/proxy"
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
"github.com/steveyegge/beads/internal/storage/dbproxy/util"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const (
listenWait = 10 * time.Second
shutdownWait = 5 * time.Second
ioTimeout = 2 * time.Second
)
func freeTCPPort(t *testing.T) int {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
port := ln.Addr().(*net.TCPAddr).Port
require.NoError(t, ln.Close())
return port
}
func proxyAddr(port int) string {
return net.JoinHostPort("127.0.0.1", strconv.Itoa(port))
}
type proxyHandle struct {
cancel context.CancelFunc
done chan error // buffered=1, holds Start's return value
exited chan struct{} // closed when Start returns
}
func (h *proxyHandle) Cancel() { h.cancel() }
func (h *proxyHandle) waitErr(t *testing.T, timeout time.Duration) error {
t.Helper()
timer := time.NewTimer(timeout)
defer timer.Stop()
select {
case err := <-h.done:
return err
case <-timer.C:
t.Fatalf("proxy.ListenAndServe did not return within %s", timeout)
return nil
}
}
func runProxy(t *testing.T, opts proxy.ProxyOpts) *proxyHandle {
t.Helper()
ctx, cancel := context.WithCancel(context.Background())
h := &proxyHandle{
cancel: cancel,
done: make(chan error, 1),
exited: make(chan struct{}),
}
p := proxy.NewProxyServer(opts)
go func() {
defer close(h.exited)
h.done <- p.ListenAndServe(ctx)
}()
t.Cleanup(func() {
cancel()
timer := time.NewTimer(shutdownWait)
defer timer.Stop()
select {
case <-h.exited:
case <-timer.C:
t.Errorf("proxy did not exit within %s of cancel", shutdownWait)
}
})
return h
}
func waitListening(t *testing.T, root string, timeout time.Duration) {
t.Helper()
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
pf, err := pidfile.Read(root, proxy.PIDFileName)
if err == nil && pf != nil {
return
}
time.Sleep(10 * time.Millisecond)
}
t.Fatalf("proxy pidfile not present within %s", timeout)
}
func assertNoPidFile(t *testing.T, root string) {
t.Helper()
pf, err := pidfile.Read(root, proxy.PIDFileName)
require.NoError(t, err)
assert.Nil(t, pf)
}
func dialProxy(t *testing.T, port int) net.Conn {
t.Helper()
c, err := net.DialTimeout("tcp", proxyAddr(port), ioTimeout)
require.NoError(t, err)
require.NoError(t, c.SetDeadline(time.Now().Add(ioTimeout)))
return c
}
func TestProxy_HappyPath_Echo(t *testing.T) {
t.Parallel()
ts := server.New()
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port,
IdleTimeout: 0,
Server: ts,
Stats: stats,
})
waitListening(t, root, listenWait)
conn := dialProxy(t, port)
_, err := conn.Write([]byte("hello"))
require.NoError(t, err)
buf := make([]byte, 5)
_, err = io.ReadFull(conn, buf)
require.NoError(t, err)
assert.Equal(t, "hello", string(buf))
pf, err := pidfile.Read(root, proxy.PIDFileName)
require.NoError(t, err)
require.NotNil(t, pf)
assert.Equal(t, os.Getpid(), pf.Pid)
assert.Equal(t, port, pf.Port)
require.NoError(t, conn.Close())
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
s := stats.Snapshot()
assert.Equal(t, int64(1), s.ListenAndServeCalls)
assert.Equal(t, int64(1), s.BackendStartCalls)
assert.Equal(t, int64(1), s.BackendStopCalls)
assert.Equal(t, int64(1), s.AcceptCalls)
assert.Equal(t, int64(1), s.BackendDialAttempts)
assert.Equal(t, int64(1), s.BackendDialSuccess)
assert.Equal(t, int64(0), s.BackendDialErrors)
assert.Equal(t, int64(1), s.HandledConns)
assert.Equal(t, int64(5), s.BytesClientToBackend)
assert.Equal(t, int64(5), s.BytesBackendToClient)
bs := ts.Snapshot()
assert.Equal(t, int64(1), bs.StartCalls)
// readiness Dial + client Dial both go through the backend
assert.Equal(t, int64(2), bs.DialCalls)
assert.Equal(t, int64(2), bs.AcceptedConns)
assert.Equal(t, int64(5), bs.BytesIn)
assert.Equal(t, int64(5), bs.BytesOut)
assert.Equal(t, int64(1), bs.StopCalls)
assertNoPidFile(t, root)
}
func TestProxy_PortZeroPublishesActualDialablePort(t *testing.T) {
t.Parallel()
root := t.TempDir()
ts := server.New()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root,
Port: 0,
Server: ts,
})
waitListening(t, root, listenWait)
pf, err := pidfile.Read(root, proxy.PIDFileName)
require.NoError(t, err)
require.NotNil(t, pf)
require.NotZero(t, pf.Port, "pidfile must publish the kernel-assigned port")
conn := dialProxy(t, pf.Port)
_, err = conn.Write([]byte("port-zero"))
require.NoError(t, err)
got := make([]byte, len("port-zero"))
_, err = io.ReadFull(conn, got)
require.NoError(t, err)
assert.Equal(t, "port-zero", string(got))
require.NoError(t, conn.Close())
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
assertNoPidFile(t, root)
}
func TestProxy_PidFile_WrittenAndRemoved(t *testing.T) {
t.Parallel()
ts := server.New()
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port, Server: ts,
})
waitListening(t, root, listenWait)
pf, err := pidfile.Read(root, proxy.PIDFileName)
require.NoError(t, err)
require.NotNil(t, pf)
assert.Equal(t, os.Getpid(), pf.Pid)
assert.Equal(t, port, pf.Port)
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
assertNoPidFile(t, root)
}
func TestProxy_PublishesVerifiableIdentity(t *testing.T) {
t.Parallel()
root := t.TempDir()
ts := server.New()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root,
Port: freeTCPPort(t),
Server: ts,
})
waitListening(t, root, listenWait)
pf, err := pidfile.Read(root, proxy.PIDFileName)
require.NoError(t, err)
require.NotNil(t, pf)
require.NoError(t, pf.ValidateV2(pidfile.KindProxy))
match, err := procid.Verify(pf.Pid, procid.Token(pf.Birth))
require.NoError(t, err)
assert.True(t, match)
secret, err := identity.ReadSecret(root)
require.NoError(t, err)
got, err := identity.Identify("127.0.0.1", pf.ControlPort, secret, ioTimeout)
require.NoError(t, err)
rootID, err := identity.RootID(root)
require.NoError(t, err)
assert.Equal(t, pidfile.SchemaV2, got.Schema)
assert.Equal(t, pidfile.KindProxy, got.Role)
assert.Equal(t, pf.Pid, got.PID)
assert.Equal(t, pf.Birth, got.Birth)
assert.Equal(t, pf.Port, got.DataPort)
assert.Equal(t, pf.RootID, got.RootID)
assert.Equal(t, rootID, got.RootID)
assert.Equal(t, pf.ControlPort, got.ControlPort)
assert.Equal(t, int64(1), ts.Snapshot().IDCalls, "pidfile and control reply must share one captured upstream ID")
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
_, err = net.DialTimeout("tcp", proxyAddr(pf.ControlPort), ioTimeout)
assert.Error(t, err)
}
func TestProxy_ExplicitPortInUseFailsWithoutRepick(t *testing.T) {
t.Parallel()
hold, err := net.Listen("tcp", "127.0.0.1:0")
require.NoError(t, err)
defer hold.Close()
port := hold.Addr().(*net.TCPAddr).Port
ts := server.New()
stats := &proxy.Stats{}
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port, Server: ts, Stats: stats,
})
err = h.waitErr(t, shutdownWait)
require.Error(t, err)
assert.Contains(t, err.Error(), "listen on")
assert.Contains(t, err.Error(), strconv.Itoa(port))
s := stats.Snapshot()
assert.Equal(t, int64(0), s.ListenAndServeCalls)
assert.Equal(t, int64(0), s.BackendStartCalls)
assert.Equal(t, int64(0), ts.Snapshot().StartCalls)
assertNoPidFile(t, root)
}
func TestProxy_BackendStartError(t *testing.T) {
t.Parallel()
ts := server.New()
ts.StartErr = errors.New("boom")
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port, Server: ts, Stats: stats,
})
err := h.waitErr(t, shutdownWait)
require.Error(t, err)
assert.Contains(t, err.Error(), "start database server")
assert.Contains(t, err.Error(), "boom")
s := stats.Snapshot()
assert.Equal(t, int64(1), s.ListenAndServeCalls)
assert.Equal(t, int64(1), s.BackendStartCalls)
assert.Equal(t, int64(0), s.BackendStopCalls)
assertNoPidFile(t, root)
}
func TestProxy_BackendNotReady_CtxCancel(t *testing.T) {
t.Parallel()
ts := server.New()
ts.DialErr = errors.New("not ready")
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port, Server: ts, Stats: stats,
})
// The pidfile is written only after readiness succeeds, but DialErr
// keeps readiness failing — so waitListening would hang. Wait until
// the proxy is in the readiness loop (>=1 Dial attempt observed).
require.Eventually(t, func() bool {
return ts.Snapshot().DialCalls >= 1
}, listenWait, 10*time.Millisecond)
h.Cancel()
err := h.waitErr(t, shutdownWait)
require.Error(t, err)
assert.Contains(t, err.Error(), "database server not ready")
s := stats.Snapshot()
assert.Equal(t, int64(1), s.ListenAndServeCalls)
assert.Equal(t, int64(1), s.BackendStartCalls)
assert.Equal(t, int64(1), s.BackendStopCalls)
bs := ts.Snapshot()
assert.Equal(t, int64(1), bs.StartCalls)
assert.GreaterOrEqual(t, bs.DialCalls, int64(1))
assert.Equal(t, int64(1), bs.StopCalls)
assertNoPidFile(t, root)
}
func TestProxy_BackendDialError(t *testing.T) {
t.Parallel()
ts := server.New()
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port, Server: ts, Stats: stats,
})
// Wait for readiness Dial to succeed, then flip DialErr so subsequent
// proxied connections fail.
waitListening(t, root, listenWait)
ts.SetDialErr(errors.New("refused"))
for i := 0; i < 2; i++ {
c := dialProxy(t, port)
// Proxy accepts then closes us out after the dial error.
_, err := c.Read(make([]byte, 1))
assert.Error(t, err, "iteration %d: expected EOF/error from proxy", i)
require.NoError(t, c.Close())
}
require.Eventually(t, func() bool {
s := stats.Snapshot()
return s.AcceptCalls >= 2 &&
s.BackendDialAttempts >= 2 &&
s.BackendDialErrors >= 2
}, ioTimeout, 10*time.Millisecond, "stats did not reach 2 dial errors")
s := stats.Snapshot()
assert.Equal(t, int64(2), s.AcceptCalls)
assert.Equal(t, int64(2), s.BackendDialAttempts)
assert.Equal(t, int64(2), s.BackendDialErrors)
assert.Equal(t, int64(0), s.BackendDialSuccess)
assert.Equal(t, int64(0), s.HandledConns)
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
assertNoPidFile(t, root)
}
func TestProxy_IdleTimeout_Fires(t *testing.T) {
t.Parallel()
ts := server.New()
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port,
IdleTimeout: 2 * time.Second,
Server: ts, Stats: stats,
})
waitListening(t, root, listenWait)
// No client traffic — proxy should self-terminate on idle.
require.NoError(t, h.waitErr(t, 6*time.Second))
s := stats.Snapshot()
assert.Equal(t, int64(1), s.IdleTimeouts)
assert.Equal(t, int64(1), s.BackendStopCalls)
assert.Equal(t, int64(1), ts.Snapshot().StopCalls)
assertNoPidFile(t, root)
}
func TestProxy_IdleTimeout_BlockedByActiveConn(t *testing.T) {
t.Parallel()
ts := server.New()
ts.Handler = server.DiscardHandler
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port,
IdleTimeout: 2 * time.Second,
Server: ts, Stats: stats,
})
waitListening(t, root, listenWait)
conn := dialProxy(t, port)
defer conn.Close()
// Wait past two full idle windows to confirm activeConns gates the watcher.
time.Sleep(4 * time.Second)
assert.Equal(t, int64(0), stats.Snapshot().IdleTimeouts)
// Tear down cleanly.
require.NoError(t, conn.Close())
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
assertNoPidFile(t, root)
}
func TestProxy_IdleTimeout_Never(t *testing.T) {
t.Parallel()
ts := server.New()
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port,
IdleTimeout: proxy.IdleTimeoutNever,
Server: ts, Stats: stats,
})
waitListening(t, root, listenWait)
// No client traffic and no timeout — proxy must stay up.
time.Sleep(3 * time.Second)
assert.Equal(t, int64(0), stats.Snapshot().IdleTimeouts)
pf, err := pidfile.Read(root, proxy.PIDFileName)
require.NoError(t, err)
require.NotNil(t, pf, "proxy should still be running")
_ = dialProxy(t, port).Close()
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
assertNoPidFile(t, root)
}
func TestProxy_ConcurrentConnections(t *testing.T) {
t.Parallel()
ts := server.New() // default echo handler
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port, Server: ts, Stats: stats,
})
waitListening(t, root, listenWait)
const N = 5
const payloadLen = 4
var wg sync.WaitGroup
errs := make(chan error, N)
for i := 0; i < N; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
c, err := net.DialTimeout("tcp", proxyAddr(port), ioTimeout)
if err != nil {
errs <- fmt.Errorf("dial: %w", err)
return
}
defer c.Close()
_ = c.SetDeadline(time.Now().Add(ioTimeout))
payload := []byte(fmt.Sprintf("p%03d", i))
if _, err := c.Write(payload); err != nil {
errs <- fmt.Errorf("write: %w", err)
return
}
buf := make([]byte, payloadLen)
if _, err := io.ReadFull(c, buf); err != nil {
errs <- fmt.Errorf("read: %w", err)
return
}
if !bytes.Equal(payload, buf) {
errs <- fmt.Errorf("echo mismatch: got %q want %q", buf, payload)
}
}(i)
}
wg.Wait()
close(errs)
for e := range errs {
t.Error(e)
}
require.Eventually(t, func() bool {
s := stats.Snapshot()
return s.HandledConns == N &&
s.BytesClientToBackend == int64(N*payloadLen) &&
s.BytesBackendToClient == int64(N*payloadLen)
}, ioTimeout, 10*time.Millisecond, "stats did not converge for N=%d clients", N)
s := stats.Snapshot()
assert.Equal(t, int64(N), s.AcceptCalls)
assert.Equal(t, int64(N), s.BackendDialSuccess)
assert.Equal(t, int64(0), s.BackendDialErrors)
assert.Equal(t, int64(N), s.HandledConns)
assert.Equal(t, int64(N*payloadLen), s.BytesClientToBackend)
assert.Equal(t, int64(N*payloadLen), s.BytesBackendToClient)
// readiness Dial counts as one extra AcceptedConn against the backend
assert.Equal(t, int64(N+1), ts.Snapshot().AcceptedConns)
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
}
func TestProxy_BidirectionalCopy_AsymmetricBytes(t *testing.T) {
t.Parallel()
ts := server.New()
const clientMsgLen = 3
payload := []byte("backend-says-hi") // 15 bytes
// Read the client's send first so direction-1's Write completes before
// the backend side closes; then write the asymmetric payload back.
ts.Handler = func(c net.Conn) {
defer c.Close()
buf := make([]byte, clientMsgLen)
_, _ = io.ReadFull(c, buf)
_, _ = c.Write(payload)
}
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port, Server: ts, Stats: stats,
})
waitListening(t, root, listenWait)
conn := dialProxy(t, port)
_, err := conn.Write([]byte("abc"))
require.NoError(t, err)
got, err := io.ReadAll(conn)
require.NoError(t, err)
assert.Equal(t, payload, got)
require.NoError(t, conn.Close())
require.Eventually(t, func() bool {
s := stats.Snapshot()
return s.HandledConns == 1 &&
s.BytesClientToBackend == clientMsgLen &&
s.BytesBackendToClient == int64(len(payload))
}, ioTimeout, 10*time.Millisecond, "asymmetric byte counters did not converge")
bs := ts.Snapshot()
assert.Equal(t, int64(clientMsgLen), bs.BytesIn)
assert.Equal(t, int64(len(payload)), bs.BytesOut)
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
}
func TestProxy_Cancel_DrainsInFlightConn(t *testing.T) {
t.Parallel()
ts := server.New()
ts.Handler = server.DiscardHandler // backend reads, never writes
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port, Server: ts, Stats: stats,
})
waitListening(t, root, listenWait)
conn := dialProxy(t, port)
defer conn.Close()
_, err := conn.Write([]byte("x"))
require.NoError(t, err)
h.Cancel()
require.NoError(t, h.waitErr(t, shutdownWait))
// Proxy tore down our conn during shutdown.
_, err = conn.Read(make([]byte, 1))
assert.Error(t, err)
assert.Equal(t, int64(1), stats.Snapshot().BackendStopCalls)
assert.Equal(t, int64(1), ts.Snapshot().StopCalls)
assertNoPidFile(t, root)
}
// TestProxy_ConcurrentInstantiation_OnlyOneWinsLock launches N proxy servers
// in parallel against the same rootdir. Exactly one wins the proxy.lock
// flock race, runs to completion, and accounts for a full ListenAndServe /
// BackendStart / BackendStop lifecycle. The losers return ErrLockHeld before
// reaching any side-effecting step.
//
// This documents the proxy package's contract: ListenAndServe serializes
// instantiation per-rootDir via proxy.lock. Same-rootDir contention always
// produces ErrLockHeld for losers — independent of whether they would have
// raced on Listen, pidfile, or backend start.
func TestProxy_ConcurrentInstantiation_OnlyOneWinsLock(t *testing.T) {
t.Parallel()
root := t.TempDir()
port := freeTCPPort(t)
const N = 5
const settle = 500 * time.Millisecond
type result struct {
err error
stats *proxy.Stats
}
barrier := make(chan struct{})
results := make(chan result, N)
for i := 0; i < N; i++ {
go func() {
ts := server.New()
stats := &proxy.Stats{}
p := proxy.NewProxyServer(proxy.ProxyOpts{
RootDir: root, Port: port,
Server: ts, Stats: stats,
})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
<-barrier
go func() { done <- p.ListenAndServe(ctx) }()
// After settle: losers have already returned (lock probe fails in
// microseconds); the winner is blocked in the accept loop. Cancel
// everyone uniformly and collect their final return value.
time.Sleep(settle)
cancel()
results <- result{err: <-done, stats: stats}
}()
}
close(barrier)
var lockHeld, winners int
for i := 0; i < N; i++ {
r := <-results
if r.err != nil {
require.ErrorIs(t, r.err, proxy.ErrLockHeld)
lockHeld++
s := r.stats.Snapshot()
// Lock acquisition fires before any IncListenAndServe / IncBackendStart.
assert.Equal(t, int64(0), s.ListenAndServeCalls)
assert.Equal(t, int64(0), s.BackendStartCalls)
assert.Equal(t, int64(0), s.BackendStopCalls)
} else {
winners++
s := r.stats.Snapshot()
assert.Equal(t, int64(1), s.ListenAndServeCalls)
assert.Equal(t, int64(1), s.BackendStartCalls)
assert.Equal(t, int64(1), s.BackendStopCalls)
}
}
assert.Equal(t, 1, winners, "expected exactly 1 lock winner")
assert.Equal(t, N-1, lockHeld)
// Winner's pidfile was cleaned up by its deferred Remove.
assertNoPidFile(t, root)
}
func TestProxy_LockHeld_ReturnsSentinel(t *testing.T) {
t.Parallel()
root := t.TempDir()
port := freeTCPPort(t)
held, err := util.TryLock(filepath.Join(root, proxy.LockFileName))
require.NoError(t, err)
defer held.Unlock()
ts := server.New()
stats := &proxy.Stats{}
p := proxy.NewProxyServer(proxy.ProxyOpts{
RootDir: root, Port: port,
Server: ts, Stats: stats,
})
err = p.ListenAndServe(context.Background())
require.Error(t, err)
assert.ErrorIs(t, err, proxy.ErrLockHeld)
s := stats.Snapshot()
assert.Equal(t, int64(0), s.ListenAndServeCalls)
assert.Equal(t, int64(0), s.BackendStartCalls)
assert.Equal(t, int64(0), s.BackendStopCalls)
assert.Equal(t, int64(0), ts.Snapshot().StartCalls)
assertNoPidFile(t, root)
}
func TestProxy_TraceLog_LifecycleEvents(t *testing.T) {
t.Parallel()
ts := server.New()
stats := &proxy.Stats{}
port := freeTCPPort(t)
root := t.TempDir()
const idle = 2 * time.Second
const tickSlack = 1200 * time.Millisecond
h := runProxy(t, proxy.ProxyOpts{
RootDir: root, Port: port,
IdleTimeout: idle,
Server: ts, Stats: stats,
})
waitListening(t, root, listenWait)
time.Sleep(tickSlack)
conn := dialProxy(t, port)
_, err := conn.Write([]byte("ping"))
require.NoError(t, err)
buf := make([]byte, 4)
_, err = io.ReadFull(conn, buf)
require.NoError(t, err)
require.Equal(t, "ping", string(buf))
time.Sleep(tickSlack)
require.NoError(t, conn.Close())
require.NoError(t, h.waitErr(t, 3*idle))
body, err := os.ReadFile(filepath.Join(root, proxy.LogFileName))
require.NoError(t, err)
text := string(body)
for _, want := range []string{
"acceptLoop start",
"acceptLoop accepted",
"idleWatcher start",
"idleWatcher armed",
"idleWatcher cleared",
"idleWatcher expired",
"handleConn(",
"backend dial ok",
"client→backend done",
"backend→client done",
"acceptLoop exit",
} {
assert.Contains(t, text, want, "log missing %q", want)
}
}