beads/cmd/bd/proxied_local_backend_exit_test.go
Julian Knutsen d803ec9126
fix(dbproxy): retire the proxy when its Dolt backend exits cleanly (#6937)
* fix(dbproxy): retire the proxy when its Dolt backend exits cleanly

DoltServer supervised `dolt sql-server` with cmd.Wait inside an
errgroup, and Running read the errgroup context. errgroup cancels that
context only for a non-nil result, so a backend that shut down
gracefully (dolt exits 0 on SIGTERM) was reported as running forever.
The proxy's backend health watcher never fired: the proxy stayed up and
adoptable in front of a dead store. With --proxied-server-idle-timeout 0
(the setting Gas City uses) it never went away; `bd dolt status` said
"running" and every command, `bd ping` included, failed until
`bd dolt stop`.

The supervising goroutine now reports a clean exit as errBackendExited,
so Running flips on any backend exit and Stop treats it as benign. This
follows the existing proxied lifecycle contract (child death retires
the owned proxy; the next bd invocation starts a fresh proxy/backend
pair) rather than adding in-proxy restarts: spawn, adoption, stop
epochs and identity already live in the on-demand start path, and a
second restart loop inside the proxy would duplicate that and could
mask a crash-looping backend.

`bd dolt status` no longer headlines a proxy whose managed backend is
gone as "running"; it says the proxy is up but not serving and points
at `bd dolt stop` (for proxies from an older bd). The JSON payload is
unchanged. `bd ping` already verifies by running SELECT 1 through the
proxy, so after this fix a successful ping reflects a live pair.

Tests: DoltServer Running goes false after SIGTERM (clean exit) and
SIGKILL, and Stop succeeds; managed-local lane test that SIGTERMs the
backend with idle timeout 0 and requires the proxy to retire and
`bd ping --json` to succeed on a fresh pair; status render case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dolt status): report running=false for a proxy over a dead managed backend

Review follow-up: the JSON running field now means "serving" (a live proxy
and, on a managed topology, a live dolt behind it), so scripts that key off
running alone are not fooled by a stranded proxy left by an older bd.
proxy_pid/proxy_port and backend_* are unchanged. Adds the PR link to the
CHANGELOG entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): resolve merge markers left by the origin/main merge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 01:48:15 -07:00

71 lines
2.9 KiB
Go

//go:build cgo && unix
package main
import (
"encoding/json"
"strings"
"syscall"
"testing"
"time"
)
// backendCleanExitRetireTimeout bounds how long the proxy may outlive a dolt
// backend that shut down gracefully. dolt's own SIGTERM shutdown dominates it;
// the proxy's health watcher polls every 100ms.
const backendCleanExitRetireTimeout = 30 * time.Second
// TestManagedLocalProxiedBackendCleanExitRetiresProxy extends the child-death
// contract (see TestManagedLocalProxiedOutageReconnectContract, which kills
// the backend) to a backend that exits CLEANLY: SIGTERM makes dolt shut down
// gracefully with status 0. The proxy must retire just as it does for a
// crash, so the next bd command starts a fresh proxy/backend pair and `bd
// ping` answers from a live store.
//
// Before the fix the proxy only noticed non-zero exits. With idle timeout 0
// (never idle out) it stayed up and adoptable in front of a dead backend
// indefinitely: `bd dolt status` said "running", and every command, `bd ping`
// included, failed until an operator ran `bd dolt stop`.
func TestManagedLocalProxiedBackendCleanExitRetiresProxy(t *testing.T) {
requireManagedLocalProxiedEnv(t)
bd := buildEmbeddedBD(t)
p := bdProxiedInit(t, bd, "clean_exit", "--proxied-server-idle-timeout", "0")
sentinel := bdProxiedCreate(t, bd, p.dir, "clean exit sentinel")
proxyBefore := readManagedProxyPidFile(t, p)
if proxyBefore == nil || !processAlive(proxyBefore.Pid) {
t.Fatalf("managed-local proxy is not running: %+v", proxyBefore)
}
backendBefore := readManagedBackendPidFile(t, p)
if backendBefore == nil || !processAlive(backendBefore.Pid) {
t.Fatalf("managed-local backend is not running: %+v", backendBefore)
}
if err := syscall.Kill(backendBefore.Pid, syscall.SIGTERM); err != nil {
t.Fatalf("SIGTERM dolt backend pid %d: %v", backendBefore.Pid, err)
}
waitForManagedProxiedShutdown(t, p, proxyBefore.Pid, backendBefore.Pid, backendCleanExitRetireTimeout)
out, err := bdProxiedRun(t, bd, p.dir, "ping", "--json")
if err != nil {
t.Fatalf("bd ping after the backend exited: %v\n%s", err, out)
}
var ping struct {
Status string `json:"status"`
}
if err := json.Unmarshal([]byte(strings.TrimSpace(string(out))), &ping); err != nil || ping.Status != "ok" {
t.Fatalf("bd ping --json: status=%q err=%v\n%s", ping.Status, err, out)
}
if got := bdProxiedShow(t, bd, p.dir, sentinel.ID); got.Title != "clean exit sentinel" {
t.Fatalf("sentinel after restart: %+v", got)
}
proxyAfter := readManagedProxyPidFile(t, p)
backendAfter := readManagedBackendPidFile(t, p)
if proxyAfter == nil || proxyAfter.Pid == proxyBefore.Pid || !processAlive(proxyAfter.Pid) {
t.Fatalf("expected a fresh live proxy: before=%+v after=%+v", proxyBefore, proxyAfter)
}
if backendAfter == nil || backendAfter.Pid == backendBefore.Pid || !processAlive(backendAfter.Pid) {
t.Fatalf("expected a fresh live backend: before=%+v after=%+v", backendBefore, backendAfter)
}
}