* feat(dbproxy): add procid birth-token package and pidfile schema v2 fields Foundation for verified proxy/backend adoption and safe stale-process cleanup (mybd-psxg.5, PR-A stage 1): internal/procid provides versioned process-birth tokens (Linux boot_id+starttime, Windows creation FILETIME, Darwin p_starttime) with TOCTOU-safe signaling handles (pidfd on Linux, re-verified process handle on Windows); pidfile gains additive schema-v2 fields (schema, kind, birth, root_id, control_port) and sentinel-error validation. No behavior changes; existing v1 pidfiles still parse. The procid Token/Verify shape intentionally matches the API requested in gastownhall/beads#4513 so internal/doltserver can adopt it later. Agent-Signature: codex-gpt-5.6-terra-medium on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * feat(dbproxy): publish verifiable proxy identity (secret, control handshake, v2 pidfiles) Stage 2 of mybd-psxg.5 (PR-A): the managed proxy now proves who it is. New leaf package internal/storage/dbproxy/identity provides the workspace RootID (sha256 of the resolved root path), a rotated 0600 proxy.secret, and the IDENT control protocol (client Identify + IdentReply). The proxy opens a loopback control listener that answers only to the workspace secret, and both pidfile writers publish schema-v2 records (kind, birth token, root_id, control_port) captured via procid. A proxy or backend that cannot prove its own identity fails startup instead of publishing. Adoption, cleanup, and kill behavior are unchanged in this stage; the consumers land next so each step stays reviewable. Agent-Signature: codex-gpt-5.6-terra-high on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): verified adoption, fail-closed cleanup, and handle-safe kills Stage 3 of mybd-psxg.5 (PR-A), the security core. Endpoint discovery now classifies records with typed statuses instead of adopting any listener that answers a TCP dial: adoption requires a schema-v2 pidfile, a live birth-token match, the authenticated IDENT handshake, and root/pid/birth/ port cross-checks. Legacy and malformed records fail closed (quarantined under proxy.lock, never adopted, never killed); quarantines are timestamped renames with a 30-day sweep, so no forensic record is ever unlinked. Every kill site (spawn-path backend cleanup and bd dolt stop's Shutdown) signals only through a procid.Handle opened against the recorded birth token, refusing unverifiable live processes with an actionable error. Orphaned Dolt backends left by a SIGKILLed proxy supervisor are now detected regardless of child-lock state and reaped only after verification. A proxy.spawn marker plus a persistent stop epoch closes the window where bd dolt stop could report success while a fork already in flight published a listener afterwards. Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): harden identity fixes from cross-vendor adversarial review Fixes every accepted finding from paired independent reviews of the first three stages (reports in the coordination repo, 2026-07-24): - darwin: classify wrapped ESRCH and the zero-length/EIO missing-pid sysctl result as process-gone; stale pidfiles self-heal again. - fallback signaling (darwin, non-pidfd linux): a target that exits after a fatal signal is a successful kill, not an error. - Shutdown verifies root_id on the proxy record too (copied-workspace kill hazard); positional bool pair replaced by killRecordChecks. - birth-token mismatch now classifies as provably-dead everywhere; dead legacy/malformed records quarantine and self-heal; refusal messages name the record file and manual remedy. - discovery-time verify/secret/root errors map to a non-adopting status routed through quarantine-under-lock instead of wedging before the lock probe. - boot_id read failures are environmental errors, never process-gone (closes the one fail-open path found in review). - ForceStopUnverified: executable-identity-checked operator escape for pre-upgrade/unverifiable records, quarantines after, reports truthfully; CLI wiring intentionally deferred. - handshake replies are HMAC'd against a client nonce; constant-time secret compare; bounded accept failures and handshake concurrency; migration purge covers proxy.secret and quarantine files; spawn marker cleared via defer; unix-only constraints on sleep tests. Agent-Signature: codex-gpt-5.6-sol-xhigh on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * feat(dbproxy): wire bd dolt stop --force and prove the lifecycle lane end to end Stage 4 of mybd-psxg.5 (PR-A). bd dolt stop gains --force, routed through proxy.ForceStopUnverified only after the backend is confirmed stopped: it verifies executable identity before signaling, quarantines the record afterward, refuses foreign executables and verifiable v2 records, and reports text/JSON truthfully. The managed-local lane (BEADS_TEST_PROXIED_LOCAL=1) now exercises the whole security model against a real dolt sql-server: foreign-listener rejection, reused-PID no-kill, legacy fail-closed (free lock, held lock refusing a foreign executable, held lock force-stopping a genuine bd), orphan-backend reaping after a SIGKILLed supervisor, 5-process cold-start convergence, and stop/start interleaving. Windows/macOS lane equivalents are defined in the lane header, implementation staged later. Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): close cross-vendor review findings — handle lifetimes, pidfd races, force-stop scoping - B1: cleanupOrphanBackend closes the verified handle before waiting for exit, mirroring shutdown.go; an open Windows handle kept the dead PID allocated and made every orphan-backend recovery time out. - B2: Windows procid checks GetExitCodeProcess after OpenProcess, so a terminated process kept resolvable by a third-party handle no longer verifies; "Verify == true implies running" now holds on all platforms. - B3: Linux procid.Open opens the pidfd BEFORE token verification; the pidfd pins the PID number, closing the reuse window where a later Signal could kill an unrelated replacement process. - B4: PidfdSendSignal ESRCH on a fatal signal is treated as successful termination, matching the fallback path (same for Windows terminate). - B5: ForceStopUnverified now covers the backend (proxy-child) record with the same safety checks, and CanForceStopUnverified accepts refusals where the backend is also (or only) unverifiable — the paired-legacy pre-v2 topology was locked out of its own advertised recovery. bd dolt stop --force reports the backend record too. - B6a: an unverified live PID is only signaled when its command line ties it to this workspace (Linux /proc cmdline, Darwin ps args); Windows cannot read another process's command line and now refuses, which beats basename-only killing. Clear errors name the manual remedy. - B6b: one stable OS handle (Linux pidfd, Windows process handle) spans inspection and signaling, removing the inspect/kill TOCTOU. - S1: control-listener accept errors are non-fatal — log and retry with capped exponential backoff instead of tearing down a healthy proxy. - S2: requireUnverifiableRecord surfaces RootID resolution errors instead of failing open into the destructive branch. - S3: killSpawnedChild falls back to cmd.Process.Kill for our own un-reaped child when no verified handle exists, instead of leaking it. - S4: adoption handshake accepts reply schema v2 or newer, matching pidfile.ValidateV2 forward-compat policy. - S5: errors.Is for ERROR_INVALID_PARAMETER. - Tests: pidfd ESRCH-after-exit and Open-after-reap classification; paired-legacy force recovery end to end; foreign-workspace bd process refused; Windows Verify-after-terminate (compile-checked); schema-v3 handshake adoption; accept-loop survival under persistent errors. Agent-Signature: claude-fable-5-medium on behalf of maphew * fix(dbproxy): race-free proxy port allocation and managed loopback listener policy Companion to the identity PR (stacks on it). The parent no longer pre-picks the proxy data port: port 0 flows to the child, which binds 127.0.0.1:0 and publishes the OS-assigned port through its verified pidfile, eliminating the bind-close-launch race; explicitly configured ports keep fixed-port failure semantics. PickFreePort survives only for the dolt-config path, annotated for the follow-up RFC on backend port collisions. Managed dolt configs are now validated by a single numeric-loopback listener policy at init and at runtime resolution, covering all config paths (generated, BEADS_PROXIED_SERVER_CONFIG, sidecar, pre-marker, hand-edited marker, interpolation fallback). Hostnames including localhost are rejected with remediation text naming the --proxied-server-external-* flags; an omitted host is accepted because dolt servercfg defaults it. There is deliberately no bypass env: managed mode assumes loopback for its root/empty-password account. External proxied topology is exempt by construction. Agent-Signature: codex-gpt-5.6-sol-high on behalf of matt wilkie Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WZHiYc3wjcdy1wrUP2JiQi * fix(dbproxy): whole-config managed listener policy, epoch-fenced startup, diagnosis fixes Review findings from two independent passes over the listener-policy commit (gastownhall/beads#5024), rebased onto the rewritten #5013 head: - F1: the managed policy now covers the whole sql-server YAML, not just listener.host. validateManagedListenerHost is renamed validateManagedServerConfigPolicy and additionally rejects listener.socket (incl. the empty-string form, which maps to Dolt's default socket path), remotesapi.port, and any cluster block — all of which open non-loopback listeners for the root/empty-password child. - F2: managed mode requires an explicit numeric loopback listener.host. An omitted host is no longer trusted: servercfg resolves it to "localhost", which CheckForUnixSocket also treats as license to open the shared-namespace default unix socket. This removes the dead net.ParseIP("localhost") branch and the omitted/explicit asymmetry. - F3: one interpolation-aware policy check instead of five scattered raw-bytes ones. ensureProxiedServerConfig now delegates resolution to resolveOrCreateProxiedServerConfig and runs a single validateProxiedServerConfig (YamlConfigFromFile, ${VAR}-expanding) on the resolved path, so a marker config with an interpolated host is judged on what it expands to. - F4: the proxy child re-checks the stop epoch immediately before publishing proxy.pid and aborts if it advanced (the spawn marker is cleared at lock acquisition, so a `bd dolt stop` racing a slow backend start otherwise has no record to wait on). The parent's observed epoch is threaded through a new db-proxy-child --stop-epoch flag into ProxyOpts.StopEpoch. - F5a/b: child-exit diagnosis distinguishes a LockHeldExitCode spawn race from real failures, points at the child's log path otherwise, and timeout messages say "its OS-assigned port" instead of "port 0". - F5c/d: validateManagedProxiedServerConfigAtInit now honors --proxied-server-root-path (the sidecar is not persisted yet at validation time) and drops the redundant explicit-config re-validation already done at flag parse. - F5e: --proxied-server-config-path flag help states the hard managed constraint (docs/cli-reference regenerates from the release pin, so only the source help string changes here). - F5f: the listener-policy lifecycle test documents its accepted PickFreePort EADDRINUSE flake window. Tests: whole-config policy table (socket/remotesapi/cluster/omitted/ localhost/loopback forms), marker ${VAR} host interpolation accept and reject, root-path-flag validation at init (unit + gated integration), and a deterministic stop-epoch-advance-during-startup abort test. Agent-Signature: claude-fable-5-medium on behalf of maphew --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
806 lines
21 KiB
Go
806 lines
21 KiB
Go
package proxy_test
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
"net"
|
|
"os"
|
|
"path/filepath"
|
|
"strconv"
|
|
"sync"
|
|
"testing"
|
|
"time"
|
|
|
|
"github.com/steveyegge/beads/internal/procid"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/identity"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/pidfile"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/proxy"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
|
|
"github.com/steveyegge/beads/internal/storage/dbproxy/util"
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
)
|
|
|
|
const (
|
|
listenWait = 10 * time.Second
|
|
shutdownWait = 5 * time.Second
|
|
ioTimeout = 2 * time.Second
|
|
)
|
|
|
|
func freeTCPPort(t *testing.T) int {
|
|
t.Helper()
|
|
ln, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
port := ln.Addr().(*net.TCPAddr).Port
|
|
require.NoError(t, ln.Close())
|
|
return port
|
|
}
|
|
|
|
func proxyAddr(port int) string {
|
|
return net.JoinHostPort("127.0.0.1", strconv.Itoa(port))
|
|
}
|
|
|
|
type proxyHandle struct {
|
|
cancel context.CancelFunc
|
|
done chan error // buffered=1, holds Start's return value
|
|
exited chan struct{} // closed when Start returns
|
|
}
|
|
|
|
func (h *proxyHandle) Cancel() { h.cancel() }
|
|
|
|
func (h *proxyHandle) waitErr(t *testing.T, timeout time.Duration) error {
|
|
t.Helper()
|
|
timer := time.NewTimer(timeout)
|
|
defer timer.Stop()
|
|
select {
|
|
case err := <-h.done:
|
|
return err
|
|
case <-timer.C:
|
|
t.Fatalf("proxy.ListenAndServe did not return within %s", timeout)
|
|
return nil
|
|
}
|
|
}
|
|
|
|
func runProxy(t *testing.T, opts proxy.ProxyOpts) *proxyHandle {
|
|
t.Helper()
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
h := &proxyHandle{
|
|
cancel: cancel,
|
|
done: make(chan error, 1),
|
|
exited: make(chan struct{}),
|
|
}
|
|
p := proxy.NewProxyServer(opts)
|
|
go func() {
|
|
defer close(h.exited)
|
|
h.done <- p.ListenAndServe(ctx)
|
|
}()
|
|
t.Cleanup(func() {
|
|
cancel()
|
|
timer := time.NewTimer(shutdownWait)
|
|
defer timer.Stop()
|
|
select {
|
|
case <-h.exited:
|
|
case <-timer.C:
|
|
t.Errorf("proxy did not exit within %s of cancel", shutdownWait)
|
|
}
|
|
})
|
|
return h
|
|
}
|
|
|
|
func waitListening(t *testing.T, root string, timeout time.Duration) {
|
|
t.Helper()
|
|
deadline := time.Now().Add(timeout)
|
|
for time.Now().Before(deadline) {
|
|
pf, err := pidfile.Read(root, proxy.PIDFileName)
|
|
if err == nil && pf != nil {
|
|
return
|
|
}
|
|
time.Sleep(10 * time.Millisecond)
|
|
}
|
|
t.Fatalf("proxy pidfile not present within %s", timeout)
|
|
}
|
|
|
|
func assertNoPidFile(t *testing.T, root string) {
|
|
t.Helper()
|
|
pf, err := pidfile.Read(root, proxy.PIDFileName)
|
|
require.NoError(t, err)
|
|
assert.Nil(t, pf)
|
|
}
|
|
|
|
func dialProxy(t *testing.T, port int) net.Conn {
|
|
t.Helper()
|
|
c, err := net.DialTimeout("tcp", proxyAddr(port), ioTimeout)
|
|
require.NoError(t, err)
|
|
require.NoError(t, c.SetDeadline(time.Now().Add(ioTimeout)))
|
|
return c
|
|
}
|
|
|
|
func TestProxy_HappyPath_Echo(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port,
|
|
IdleTimeout: 0,
|
|
Server: ts,
|
|
Stats: stats,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
conn := dialProxy(t, port)
|
|
_, err := conn.Write([]byte("hello"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 5)
|
|
_, err = io.ReadFull(conn, buf)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "hello", string(buf))
|
|
|
|
pf, err := pidfile.Read(root, proxy.PIDFileName)
|
|
require.NoError(t, err)
|
|
require.NotNil(t, pf)
|
|
assert.Equal(t, os.Getpid(), pf.Pid)
|
|
assert.Equal(t, port, pf.Port)
|
|
|
|
require.NoError(t, conn.Close())
|
|
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
|
|
s := stats.Snapshot()
|
|
assert.Equal(t, int64(1), s.ListenAndServeCalls)
|
|
assert.Equal(t, int64(1), s.BackendStartCalls)
|
|
assert.Equal(t, int64(1), s.BackendStopCalls)
|
|
assert.Equal(t, int64(1), s.AcceptCalls)
|
|
assert.Equal(t, int64(1), s.BackendDialAttempts)
|
|
assert.Equal(t, int64(1), s.BackendDialSuccess)
|
|
assert.Equal(t, int64(0), s.BackendDialErrors)
|
|
assert.Equal(t, int64(1), s.HandledConns)
|
|
assert.Equal(t, int64(5), s.BytesClientToBackend)
|
|
assert.Equal(t, int64(5), s.BytesBackendToClient)
|
|
|
|
bs := ts.Snapshot()
|
|
assert.Equal(t, int64(1), bs.StartCalls)
|
|
// readiness Dial + client Dial both go through the backend
|
|
assert.Equal(t, int64(2), bs.DialCalls)
|
|
assert.Equal(t, int64(2), bs.AcceptedConns)
|
|
assert.Equal(t, int64(5), bs.BytesIn)
|
|
assert.Equal(t, int64(5), bs.BytesOut)
|
|
assert.Equal(t, int64(1), bs.StopCalls)
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_PortZeroPublishesActualDialablePort(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
root := t.TempDir()
|
|
ts := server.New()
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root,
|
|
Port: 0,
|
|
Server: ts,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
pf, err := pidfile.Read(root, proxy.PIDFileName)
|
|
require.NoError(t, err)
|
|
require.NotNil(t, pf)
|
|
require.NotZero(t, pf.Port, "pidfile must publish the kernel-assigned port")
|
|
|
|
conn := dialProxy(t, pf.Port)
|
|
_, err = conn.Write([]byte("port-zero"))
|
|
require.NoError(t, err)
|
|
got := make([]byte, len("port-zero"))
|
|
_, err = io.ReadFull(conn, got)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, "port-zero", string(got))
|
|
require.NoError(t, conn.Close())
|
|
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_PidFile_WrittenAndRemoved(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port, Server: ts,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
pf, err := pidfile.Read(root, proxy.PIDFileName)
|
|
require.NoError(t, err)
|
|
require.NotNil(t, pf)
|
|
assert.Equal(t, os.Getpid(), pf.Pid)
|
|
assert.Equal(t, port, pf.Port)
|
|
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_PublishesVerifiableIdentity(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
root := t.TempDir()
|
|
ts := server.New()
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root,
|
|
Port: freeTCPPort(t),
|
|
Server: ts,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
pf, err := pidfile.Read(root, proxy.PIDFileName)
|
|
require.NoError(t, err)
|
|
require.NotNil(t, pf)
|
|
require.NoError(t, pf.ValidateV2(pidfile.KindProxy))
|
|
match, err := procid.Verify(pf.Pid, procid.Token(pf.Birth))
|
|
require.NoError(t, err)
|
|
assert.True(t, match)
|
|
|
|
secret, err := identity.ReadSecret(root)
|
|
require.NoError(t, err)
|
|
got, err := identity.Identify("127.0.0.1", pf.ControlPort, secret, ioTimeout)
|
|
require.NoError(t, err)
|
|
rootID, err := identity.RootID(root)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, pidfile.SchemaV2, got.Schema)
|
|
assert.Equal(t, pidfile.KindProxy, got.Role)
|
|
assert.Equal(t, pf.Pid, got.PID)
|
|
assert.Equal(t, pf.Birth, got.Birth)
|
|
assert.Equal(t, pf.Port, got.DataPort)
|
|
assert.Equal(t, pf.RootID, got.RootID)
|
|
assert.Equal(t, rootID, got.RootID)
|
|
assert.Equal(t, pf.ControlPort, got.ControlPort)
|
|
assert.Equal(t, int64(1), ts.Snapshot().IDCalls, "pidfile and control reply must share one captured upstream ID")
|
|
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
_, err = net.DialTimeout("tcp", proxyAddr(pf.ControlPort), ioTimeout)
|
|
assert.Error(t, err)
|
|
}
|
|
|
|
func TestProxy_ExplicitPortInUseFailsWithoutRepick(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
hold, err := net.Listen("tcp", "127.0.0.1:0")
|
|
require.NoError(t, err)
|
|
defer hold.Close()
|
|
port := hold.Addr().(*net.TCPAddr).Port
|
|
|
|
ts := server.New()
|
|
stats := &proxy.Stats{}
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port, Server: ts, Stats: stats,
|
|
})
|
|
err = h.waitErr(t, shutdownWait)
|
|
require.Error(t, err)
|
|
assert.Contains(t, err.Error(), "listen on")
|
|
assert.Contains(t, err.Error(), strconv.Itoa(port))
|
|
|
|
s := stats.Snapshot()
|
|
assert.Equal(t, int64(0), s.ListenAndServeCalls)
|
|
assert.Equal(t, int64(0), s.BackendStartCalls)
|
|
assert.Equal(t, int64(0), ts.Snapshot().StartCalls)
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_BackendStartError(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
ts.StartErr = errors.New("boom")
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port, Server: ts, Stats: stats,
|
|
})
|
|
err := h.waitErr(t, shutdownWait)
|
|
require.Error(t, err)
|
|
assert.Contains(t, err.Error(), "start database server")
|
|
assert.Contains(t, err.Error(), "boom")
|
|
|
|
s := stats.Snapshot()
|
|
assert.Equal(t, int64(1), s.ListenAndServeCalls)
|
|
assert.Equal(t, int64(1), s.BackendStartCalls)
|
|
assert.Equal(t, int64(0), s.BackendStopCalls)
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_BackendNotReady_CtxCancel(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
ts.DialErr = errors.New("not ready")
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port, Server: ts, Stats: stats,
|
|
})
|
|
// The pidfile is written only after readiness succeeds, but DialErr
|
|
// keeps readiness failing — so waitListening would hang. Wait until
|
|
// the proxy is in the readiness loop (>=1 Dial attempt observed).
|
|
require.Eventually(t, func() bool {
|
|
return ts.Snapshot().DialCalls >= 1
|
|
}, listenWait, 10*time.Millisecond)
|
|
h.Cancel()
|
|
err := h.waitErr(t, shutdownWait)
|
|
require.Error(t, err)
|
|
assert.Contains(t, err.Error(), "database server not ready")
|
|
|
|
s := stats.Snapshot()
|
|
assert.Equal(t, int64(1), s.ListenAndServeCalls)
|
|
assert.Equal(t, int64(1), s.BackendStartCalls)
|
|
assert.Equal(t, int64(1), s.BackendStopCalls)
|
|
|
|
bs := ts.Snapshot()
|
|
assert.Equal(t, int64(1), bs.StartCalls)
|
|
assert.GreaterOrEqual(t, bs.DialCalls, int64(1))
|
|
assert.Equal(t, int64(1), bs.StopCalls)
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_BackendDialError(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port, Server: ts, Stats: stats,
|
|
})
|
|
// Wait for readiness Dial to succeed, then flip DialErr so subsequent
|
|
// proxied connections fail.
|
|
waitListening(t, root, listenWait)
|
|
ts.SetDialErr(errors.New("refused"))
|
|
|
|
for i := 0; i < 2; i++ {
|
|
c := dialProxy(t, port)
|
|
// Proxy accepts then closes us out after the dial error.
|
|
_, err := c.Read(make([]byte, 1))
|
|
assert.Error(t, err, "iteration %d: expected EOF/error from proxy", i)
|
|
require.NoError(t, c.Close())
|
|
}
|
|
|
|
require.Eventually(t, func() bool {
|
|
s := stats.Snapshot()
|
|
return s.AcceptCalls >= 2 &&
|
|
s.BackendDialAttempts >= 2 &&
|
|
s.BackendDialErrors >= 2
|
|
}, ioTimeout, 10*time.Millisecond, "stats did not reach 2 dial errors")
|
|
|
|
s := stats.Snapshot()
|
|
assert.Equal(t, int64(2), s.AcceptCalls)
|
|
assert.Equal(t, int64(2), s.BackendDialAttempts)
|
|
assert.Equal(t, int64(2), s.BackendDialErrors)
|
|
assert.Equal(t, int64(0), s.BackendDialSuccess)
|
|
assert.Equal(t, int64(0), s.HandledConns)
|
|
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_IdleTimeout_Fires(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port,
|
|
IdleTimeout: 2 * time.Second,
|
|
Server: ts, Stats: stats,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
// No client traffic — proxy should self-terminate on idle.
|
|
require.NoError(t, h.waitErr(t, 6*time.Second))
|
|
|
|
s := stats.Snapshot()
|
|
assert.Equal(t, int64(1), s.IdleTimeouts)
|
|
assert.Equal(t, int64(1), s.BackendStopCalls)
|
|
assert.Equal(t, int64(1), ts.Snapshot().StopCalls)
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_IdleTimeout_BlockedByActiveConn(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
ts.Handler = server.DiscardHandler
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port,
|
|
IdleTimeout: 2 * time.Second,
|
|
Server: ts, Stats: stats,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
conn := dialProxy(t, port)
|
|
defer conn.Close()
|
|
|
|
// Wait past two full idle windows to confirm activeConns gates the watcher.
|
|
time.Sleep(4 * time.Second)
|
|
assert.Equal(t, int64(0), stats.Snapshot().IdleTimeouts)
|
|
|
|
// Tear down cleanly.
|
|
require.NoError(t, conn.Close())
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_IdleTimeout_Never(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port,
|
|
IdleTimeout: proxy.IdleTimeoutNever,
|
|
Server: ts, Stats: stats,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
// No client traffic and no timeout — proxy must stay up.
|
|
time.Sleep(3 * time.Second)
|
|
assert.Equal(t, int64(0), stats.Snapshot().IdleTimeouts)
|
|
pf, err := pidfile.Read(root, proxy.PIDFileName)
|
|
require.NoError(t, err)
|
|
require.NotNil(t, pf, "proxy should still be running")
|
|
_ = dialProxy(t, port).Close()
|
|
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_ConcurrentConnections(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New() // default echo handler
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port, Server: ts, Stats: stats,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
const N = 5
|
|
const payloadLen = 4
|
|
|
|
var wg sync.WaitGroup
|
|
errs := make(chan error, N)
|
|
for i := 0; i < N; i++ {
|
|
wg.Add(1)
|
|
go func(i int) {
|
|
defer wg.Done()
|
|
c, err := net.DialTimeout("tcp", proxyAddr(port), ioTimeout)
|
|
if err != nil {
|
|
errs <- fmt.Errorf("dial: %w", err)
|
|
return
|
|
}
|
|
defer c.Close()
|
|
_ = c.SetDeadline(time.Now().Add(ioTimeout))
|
|
|
|
payload := []byte(fmt.Sprintf("p%03d", i))
|
|
if _, err := c.Write(payload); err != nil {
|
|
errs <- fmt.Errorf("write: %w", err)
|
|
return
|
|
}
|
|
buf := make([]byte, payloadLen)
|
|
if _, err := io.ReadFull(c, buf); err != nil {
|
|
errs <- fmt.Errorf("read: %w", err)
|
|
return
|
|
}
|
|
if !bytes.Equal(payload, buf) {
|
|
errs <- fmt.Errorf("echo mismatch: got %q want %q", buf, payload)
|
|
}
|
|
}(i)
|
|
}
|
|
wg.Wait()
|
|
close(errs)
|
|
for e := range errs {
|
|
t.Error(e)
|
|
}
|
|
|
|
require.Eventually(t, func() bool {
|
|
s := stats.Snapshot()
|
|
return s.HandledConns == N &&
|
|
s.BytesClientToBackend == int64(N*payloadLen) &&
|
|
s.BytesBackendToClient == int64(N*payloadLen)
|
|
}, ioTimeout, 10*time.Millisecond, "stats did not converge for N=%d clients", N)
|
|
|
|
s := stats.Snapshot()
|
|
assert.Equal(t, int64(N), s.AcceptCalls)
|
|
assert.Equal(t, int64(N), s.BackendDialSuccess)
|
|
assert.Equal(t, int64(0), s.BackendDialErrors)
|
|
assert.Equal(t, int64(N), s.HandledConns)
|
|
assert.Equal(t, int64(N*payloadLen), s.BytesClientToBackend)
|
|
assert.Equal(t, int64(N*payloadLen), s.BytesBackendToClient)
|
|
|
|
// readiness Dial counts as one extra AcceptedConn against the backend
|
|
assert.Equal(t, int64(N+1), ts.Snapshot().AcceptedConns)
|
|
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
}
|
|
|
|
func TestProxy_BidirectionalCopy_AsymmetricBytes(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
const clientMsgLen = 3
|
|
payload := []byte("backend-says-hi") // 15 bytes
|
|
// Read the client's send first so direction-1's Write completes before
|
|
// the backend side closes; then write the asymmetric payload back.
|
|
ts.Handler = func(c net.Conn) {
|
|
defer c.Close()
|
|
buf := make([]byte, clientMsgLen)
|
|
_, _ = io.ReadFull(c, buf)
|
|
_, _ = c.Write(payload)
|
|
}
|
|
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port, Server: ts, Stats: stats,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
conn := dialProxy(t, port)
|
|
_, err := conn.Write([]byte("abc"))
|
|
require.NoError(t, err)
|
|
got, err := io.ReadAll(conn)
|
|
require.NoError(t, err)
|
|
assert.Equal(t, payload, got)
|
|
require.NoError(t, conn.Close())
|
|
|
|
require.Eventually(t, func() bool {
|
|
s := stats.Snapshot()
|
|
return s.HandledConns == 1 &&
|
|
s.BytesClientToBackend == clientMsgLen &&
|
|
s.BytesBackendToClient == int64(len(payload))
|
|
}, ioTimeout, 10*time.Millisecond, "asymmetric byte counters did not converge")
|
|
|
|
bs := ts.Snapshot()
|
|
assert.Equal(t, int64(clientMsgLen), bs.BytesIn)
|
|
assert.Equal(t, int64(len(payload)), bs.BytesOut)
|
|
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
}
|
|
|
|
func TestProxy_Cancel_DrainsInFlightConn(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
ts.Handler = server.DiscardHandler // backend reads, never writes
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port, Server: ts, Stats: stats,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
conn := dialProxy(t, port)
|
|
defer conn.Close()
|
|
_, err := conn.Write([]byte("x"))
|
|
require.NoError(t, err)
|
|
|
|
h.Cancel()
|
|
require.NoError(t, h.waitErr(t, shutdownWait))
|
|
|
|
// Proxy tore down our conn during shutdown.
|
|
_, err = conn.Read(make([]byte, 1))
|
|
assert.Error(t, err)
|
|
|
|
assert.Equal(t, int64(1), stats.Snapshot().BackendStopCalls)
|
|
assert.Equal(t, int64(1), ts.Snapshot().StopCalls)
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
// TestProxy_ConcurrentInstantiation_OnlyOneWinsLock launches N proxy servers
|
|
// in parallel against the same rootdir. Exactly one wins the proxy.lock
|
|
// flock race, runs to completion, and accounts for a full ListenAndServe /
|
|
// BackendStart / BackendStop lifecycle. The losers return ErrLockHeld before
|
|
// reaching any side-effecting step.
|
|
//
|
|
// This documents the proxy package's contract: ListenAndServe serializes
|
|
// instantiation per-rootDir via proxy.lock. Same-rootDir contention always
|
|
// produces ErrLockHeld for losers — independent of whether they would have
|
|
// raced on Listen, pidfile, or backend start.
|
|
func TestProxy_ConcurrentInstantiation_OnlyOneWinsLock(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
root := t.TempDir()
|
|
port := freeTCPPort(t)
|
|
|
|
const N = 5
|
|
const settle = 500 * time.Millisecond
|
|
|
|
type result struct {
|
|
err error
|
|
stats *proxy.Stats
|
|
}
|
|
|
|
barrier := make(chan struct{})
|
|
results := make(chan result, N)
|
|
|
|
for i := 0; i < N; i++ {
|
|
go func() {
|
|
ts := server.New()
|
|
stats := &proxy.Stats{}
|
|
p := proxy.NewProxyServer(proxy.ProxyOpts{
|
|
RootDir: root, Port: port,
|
|
Server: ts, Stats: stats,
|
|
})
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
done := make(chan error, 1)
|
|
<-barrier
|
|
go func() { done <- p.ListenAndServe(ctx) }()
|
|
|
|
// After settle: losers have already returned (lock probe fails in
|
|
// microseconds); the winner is blocked in the accept loop. Cancel
|
|
// everyone uniformly and collect their final return value.
|
|
time.Sleep(settle)
|
|
cancel()
|
|
results <- result{err: <-done, stats: stats}
|
|
}()
|
|
}
|
|
|
|
close(barrier)
|
|
|
|
var lockHeld, winners int
|
|
for i := 0; i < N; i++ {
|
|
r := <-results
|
|
if r.err != nil {
|
|
require.ErrorIs(t, r.err, proxy.ErrLockHeld)
|
|
lockHeld++
|
|
s := r.stats.Snapshot()
|
|
// Lock acquisition fires before any IncListenAndServe / IncBackendStart.
|
|
assert.Equal(t, int64(0), s.ListenAndServeCalls)
|
|
assert.Equal(t, int64(0), s.BackendStartCalls)
|
|
assert.Equal(t, int64(0), s.BackendStopCalls)
|
|
} else {
|
|
winners++
|
|
s := r.stats.Snapshot()
|
|
assert.Equal(t, int64(1), s.ListenAndServeCalls)
|
|
assert.Equal(t, int64(1), s.BackendStartCalls)
|
|
assert.Equal(t, int64(1), s.BackendStopCalls)
|
|
}
|
|
}
|
|
|
|
assert.Equal(t, 1, winners, "expected exactly 1 lock winner")
|
|
assert.Equal(t, N-1, lockHeld)
|
|
|
|
// Winner's pidfile was cleaned up by its deferred Remove.
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_LockHeld_ReturnsSentinel(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
root := t.TempDir()
|
|
port := freeTCPPort(t)
|
|
|
|
held, err := util.TryLock(filepath.Join(root, proxy.LockFileName))
|
|
require.NoError(t, err)
|
|
defer held.Unlock()
|
|
|
|
ts := server.New()
|
|
stats := &proxy.Stats{}
|
|
p := proxy.NewProxyServer(proxy.ProxyOpts{
|
|
RootDir: root, Port: port,
|
|
Server: ts, Stats: stats,
|
|
})
|
|
err = p.ListenAndServe(context.Background())
|
|
require.Error(t, err)
|
|
assert.ErrorIs(t, err, proxy.ErrLockHeld)
|
|
|
|
s := stats.Snapshot()
|
|
assert.Equal(t, int64(0), s.ListenAndServeCalls)
|
|
assert.Equal(t, int64(0), s.BackendStartCalls)
|
|
assert.Equal(t, int64(0), s.BackendStopCalls)
|
|
assert.Equal(t, int64(0), ts.Snapshot().StartCalls)
|
|
|
|
assertNoPidFile(t, root)
|
|
}
|
|
|
|
func TestProxy_TraceLog_LifecycleEvents(t *testing.T) {
|
|
t.Parallel()
|
|
|
|
ts := server.New()
|
|
stats := &proxy.Stats{}
|
|
port := freeTCPPort(t)
|
|
root := t.TempDir()
|
|
|
|
const idle = 2 * time.Second
|
|
const tickSlack = 1200 * time.Millisecond
|
|
h := runProxy(t, proxy.ProxyOpts{
|
|
RootDir: root, Port: port,
|
|
IdleTimeout: idle,
|
|
Server: ts, Stats: stats,
|
|
})
|
|
waitListening(t, root, listenWait)
|
|
|
|
time.Sleep(tickSlack)
|
|
|
|
conn := dialProxy(t, port)
|
|
_, err := conn.Write([]byte("ping"))
|
|
require.NoError(t, err)
|
|
buf := make([]byte, 4)
|
|
_, err = io.ReadFull(conn, buf)
|
|
require.NoError(t, err)
|
|
require.Equal(t, "ping", string(buf))
|
|
|
|
time.Sleep(tickSlack)
|
|
|
|
require.NoError(t, conn.Close())
|
|
|
|
require.NoError(t, h.waitErr(t, 3*idle))
|
|
|
|
body, err := os.ReadFile(filepath.Join(root, proxy.LogFileName))
|
|
require.NoError(t, err)
|
|
text := string(body)
|
|
|
|
for _, want := range []string{
|
|
"acceptLoop start",
|
|
"acceptLoop accepted",
|
|
"idleWatcher start",
|
|
"idleWatcher armed",
|
|
"idleWatcher cleared",
|
|
"idleWatcher expired",
|
|
"handleConn(",
|
|
"backend dial ok",
|
|
"client→backend done",
|
|
"backend→client done",
|
|
"acceptLoop exit",
|
|
} {
|
|
assert.Contains(t, text, want, "log missing %q", want)
|
|
}
|
|
}
|