fix(dbproxy): retire the proxy when its Dolt backend exits cleanly (#6937)

* fix(dbproxy): retire the proxy when its Dolt backend exits cleanly

DoltServer supervised `dolt sql-server` with cmd.Wait inside an
errgroup, and Running read the errgroup context. errgroup cancels that
context only for a non-nil result, so a backend that shut down
gracefully (dolt exits 0 on SIGTERM) was reported as running forever.
The proxy's backend health watcher never fired: the proxy stayed up and
adoptable in front of a dead store. With --proxied-server-idle-timeout 0
(the setting Gas City uses) it never went away; `bd dolt status` said
"running" and every command, `bd ping` included, failed until
`bd dolt stop`.

The supervising goroutine now reports a clean exit as errBackendExited,
so Running flips on any backend exit and Stop treats it as benign. This
follows the existing proxied lifecycle contract (child death retires
the owned proxy; the next bd invocation starts a fresh proxy/backend
pair) rather than adding in-proxy restarts: spawn, adoption, stop
epochs and identity already live in the on-demand start path, and a
second restart loop inside the proxy would duplicate that and could
mask a crash-looping backend.

`bd dolt status` no longer headlines a proxy whose managed backend is
gone as "running"; it says the proxy is up but not serving and points
at `bd dolt stop` (for proxies from an older bd). The JSON payload is
unchanged. `bd ping` already verifies by running SELECT 1 through the
proxy, so after this fix a successful ping reflects a live pair.

Tests: DoltServer Running goes false after SIGTERM (clean exit) and
SIGKILL, and Stop succeeds; managed-local lane test that SIGTERMs the
backend with idle timeout 0 and requires the proxy to retire and
`bd ping --json` to succeed on a fresh pair; status render case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(dolt status): report running=false for a proxy over a dead managed backend

Review follow-up: the JSON running field now means "serving" (a live proxy
and, on a managed topology, a live dolt behind it), so scripts that key off
running alone are not fooled by a stranded proxy left by an older bd.
proxy_pid/proxy_port and backend_* are unchanged. Adds the PR link to the
CHANGELOG entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changelog): resolve merge markers left by the origin/main merge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Julian Knutsen 2026-09-29 01:48:15 -07:00 • committed by GitHub
commit d803ec9126
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 233 additions and 9 deletions

View file

@ -133,7 +133,8 @@ jobs:
TestManagedLocalProxiedBackupRoundTrip \
TestManagedLocalProxiedAutoBackup \
TestManagedLocalProxiedBackupRestoreRefusedWhileAttached \
TestManagedLocalProxiedPurgeWispsPlaneRetention
TestManagedLocalProxiedPurgeWispsPlaneRetention \
TestManagedLocalProxiedBackendCleanExitRetiresProxy
do
if ! grep -qx "$test_name" <<<"$listed"; then
echo "managed-local lifecycle test $test_name is missing; got: '$listed'" >&2

View file

@ -9,6 +9,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- **A proxied workspace's proxy retires when its Dolt backend exits
cleanly.** The proxy noticed its `dolt sql-server` child exiting only when
the exit status was non-zero. A backend that shut down gracefully (for
example on SIGTERM) left the proxy up and adoptable in front of a dead store:
with `--proxied-server-idle-timeout 0` it never went away, `bd dolt status`
called it running, and every command failed until `bd dolt stop`. The proxy
now retires on any backend exit, as it already did on a crash, so the next bd
command (including `bd ping`) starts a fresh proxy and backend. `bd dolt
status` no longer reports a proxy whose managed backend is gone as running:
the text says "not serving" and the JSON reports `running: false` (with
`proxy_pid` still set), including for a stranded proxy started by an older
bd. ([#6937](https://github.com/gastownhall/beads/pull/6937))
- **`bd sql` no longer drops the rows of CTE queries or CALL result sets**
([#6932](https://github.com/gastownhall/beads/pull/6932)). In proxied-server
mode, `WITH name(cols) AS (...) SELECT ...` and `WITH RECURSIVE ...` queries

View file

@ -929,6 +929,7 @@ go_test(
"provenance_test.go",
"proxied_direct_only_refusal_frontdoor_test.go",
"proxied_integration_helpers_test.go",
"proxied_local_backend_exit_test.go",
"proxied_local_dolt_lifecycle_test.go",
"proxied_local_helpers_test.go",
"proxied_local_identity_lifecycle_linux_test.go",

View file

@ -43,9 +43,12 @@ func proxiedDoltStartRefusal() *ProxyCapabilityError {
// proxiedDoltStatus is the `bd dolt status --json` payload for a proxied
// workspace. It reports the two processes separately because they fail
// separately and an operator's next move differs: Running describes the proxy,
// which is the endpoint every bd command connects through, while the Backend*
// fields describe the dolt server behind it.
// separately and an operator's next move differs: Running says whether the
// workspace is serving (a live proxy, and on a managed topology a live dolt
// behind it), Proxy* describe the proxy every bd command connects through, and
// the Backend* fields describe the dolt server behind it. A proxy left in front
// of a dead managed backend (possible with proxies started by an older bd)
// therefore reports running=false with proxy_pid still set.
type proxiedDoltStatus struct {
Mode string `json:"mode"`
Root string `json:"root"`
@ -98,9 +101,17 @@ func collectProxiedDoltStatus(beadsDir string) (proxiedDoltStatus, error) {
status.BackendEndpoint = externalDoltEndpoint(*info.External)
}
}
status.Running = proxiedStatusServing(live.ProxyRunning, status.BackendManaged, status.BackendRunning)
return status, nil
}
// proxiedStatusServing reports whether a proxied workspace is serving: its
// proxy is live and, when bd manages the backend, that dolt server is live
// too. A proxy in front of a dead managed backend serves nothing.
func proxiedStatusServing(proxyRunning, backendManaged, backendRunning bool) bool {
return proxyRunning && (!backendManaged || backendRunning)
}
func externalDoltEndpoint(cfg configfile.ExternalDoltConfig) string {
if cfg.Socket != "" {
return "unix:" + cfg.Socket
@ -116,11 +127,20 @@ func renderProxiedDoltStatus(status proxiedDoltStatus) {
return
}
if status.Running {
// A live proxy in front of a managed backend that is gone serves nothing,
// so do not call it running. The proxy retires itself once it notices
// the backend exit; a proxy from an older bd may not.
backendDown := !status.Running && status.ProxyPID != 0 && status.BackendManaged && !status.BackendRunning
switch {
case backendDown:
fmt.Printf("Dolt server: not serving (%s): proxy is up but its dolt backend is not running\n", status.Mode)
fmt.Printf(" Proxy PID: %d\n", status.ProxyPID)
fmt.Printf(" Proxy port: %d\n", status.ProxyPort)
case status.Running:
fmt.Printf("Dolt server: running (%s)\n", status.Mode)
fmt.Printf(" Proxy PID: %d\n", status.ProxyPID)
fmt.Printf(" Proxy port: %d\n", status.ProxyPort)
} else {
default:
fmt.Printf("Dolt server: not running (%s)\n", status.Mode)
}
fmt.Printf(" Root: %s\n", status.Root)
@ -135,9 +155,12 @@ func renderProxiedDoltStatus(status proxiedDoltStatus) {
if status.IdleTimeout != "" {
fmt.Printf(" Idle timeout: %s\n", status.IdleTimeout)
}
if !status.Running {
if !status.Running && !backendDown {
fmt.Println(" The proxy starts on demand; the next bd command launches it.")
}
if backendDown {
fmt.Println(" Run 'bd dolt stop' to retire the proxy; the next bd command starts a fresh proxy and backend.")
}
if isDoltLocalOnly() {
fmt.Println(" Remote sync: disabled (dolt.local-only=true)")
}

View file

@ -151,6 +151,28 @@ func TestRenderProxiedDoltStatus(t *testing.T) {
}
})
t.Run("text does not call a proxy over a dead managed backend running", func(t *testing.T) {
defer restoreJSONOutput(t)()
jsonOutput = false
stranded := running
stranded.Running = false
stranded.BackendRunning, stranded.BackendPID, stranded.BackendPort = false, 0, 0
out := captureStdout(t, func() error { renderProxiedDoltStatus(stranded); return nil })
if strings.Contains(out, "Dolt server: running") {
t.Errorf("proxy with a dead backend reported as running:\n%s", out)
}
for _, want := range []string{
"Dolt server: not serving (proxied-server)",
"Proxy PID: 111",
"Backend: not running",
"bd dolt stop",
} {
if !strings.Contains(out, want) {
t.Errorf("missing %q in:\n%s", want, out)
}
}
})
t.Run("text marks an external backend as somebody else's", func(t *testing.T) {
defer restoreJSONOutput(t)()
jsonOutput = false
@ -180,6 +202,24 @@ func TestRenderProxiedDoltStatus(t *testing.T) {
})
}
func TestProxiedStatusServing(t *testing.T) {
for _, tc := range []struct {
name string
proxyRunning, managed, backendRunning bool
want bool
}{
{"managed pair live", true, true, true, true},
{"proxy over dead managed backend", true, true, false, false},
{"external backend", true, false, false, true},
{"proxy down", false, true, true, false},
{"nothing running", false, true, false, false},
} {
if got := proxiedStatusServing(tc.proxyRunning, tc.managed, tc.backendRunning); got != tc.want {
t.Errorf("%s: proxiedStatusServing(%v, %v, %v) = %v, want %v", tc.name, tc.proxyRunning, tc.managed, tc.backendRunning, got, tc.want)
}
}
}
func restoreJSONOutput(t *testing.T) func() {
t.Helper()
orig := jsonOutput

View file

@ -0,0 +1,71 @@
//go:build cgo && unix
package main
import (
"encoding/json"
"strings"
"syscall"
"testing"
"time"
)
// backendCleanExitRetireTimeout bounds how long the proxy may outlive a dolt
// backend that shut down gracefully. dolt's own SIGTERM shutdown dominates it;
// the proxy's health watcher polls every 100ms.
const backendCleanExitRetireTimeout = 30 * time.Second
// TestManagedLocalProxiedBackendCleanExitRetiresProxy extends the child-death
// contract (see TestManagedLocalProxiedOutageReconnectContract, which kills
// the backend) to a backend that exits CLEANLY: SIGTERM makes dolt shut down
// gracefully with status 0. The proxy must retire just as it does for a
// crash, so the next bd command starts a fresh proxy/backend pair and `bd
// ping` answers from a live store.
//
// Before the fix the proxy only noticed non-zero exits. With idle timeout 0
// (never idle out) it stayed up and adoptable in front of a dead backend
// indefinitely: `bd dolt status` said "running", and every command, `bd ping`
// included, failed until an operator ran `bd dolt stop`.
func TestManagedLocalProxiedBackendCleanExitRetiresProxy(t *testing.T) {
requireManagedLocalProxiedEnv(t)
bd := buildEmbeddedBD(t)
p := bdProxiedInit(t, bd, "clean_exit", "--proxied-server-idle-timeout", "0")
sentinel := bdProxiedCreate(t, bd, p.dir, "clean exit sentinel")
proxyBefore := readManagedProxyPidFile(t, p)
if proxyBefore == nil || !processAlive(proxyBefore.Pid) {
t.Fatalf("managed-local proxy is not running: %+v", proxyBefore)
}
backendBefore := readManagedBackendPidFile(t, p)
if backendBefore == nil || !processAlive(backendBefore.Pid) {
t.Fatalf("managed-local backend is not running: %+v", backendBefore)
}
if err := syscall.Kill(backendBefore.Pid, syscall.SIGTERM); err != nil {
t.Fatalf("SIGTERM dolt backend pid %d: %v", backendBefore.Pid, err)
}
waitForManagedProxiedShutdown(t, p, proxyBefore.Pid, backendBefore.Pid, backendCleanExitRetireTimeout)
out, err := bdProxiedRun(t, bd, p.dir, "ping", "--json")
if err != nil {
t.Fatalf("bd ping after the backend exited: %v\n%s", err, out)
}
var ping struct {
Status string `json:"status"`
}
if err := json.Unmarshal([]byte(strings.TrimSpace(string(out))), &ping); err != nil || ping.Status != "ok" {
t.Fatalf("bd ping --json: status=%q err=%v\n%s", ping.Status, err, out)
}
if got := bdProxiedShow(t, bd, p.dir, sentinel.ID); got.Title != "clean exit sentinel" {
t.Fatalf("sentinel after restart: %+v", got)
}
proxyAfter := readManagedProxyPidFile(t, p)
backendAfter := readManagedBackendPidFile(t, p)
if proxyAfter == nil || proxyAfter.Pid == proxyBefore.Pid || !processAlive(proxyAfter.Pid) {
t.Fatalf("expected a fresh live proxy: before=%+v after=%+v", proxyBefore, proxyAfter)
}
if backendAfter == nil || backendAfter.Pid == backendBefore.Pid || !processAlive(backendAfter.Pid) {
t.Fatalf("expected a fresh live backend: before=%+v after=%+v", backendBefore, backendAfter)
}
}

View file

@ -29,6 +29,7 @@ go_library(
go_test(
name = "server_test",
srcs = [
"doltserver_exit_unix_test.go",
"doltserver_test.go",
"external_dolt_server_test.go",
"testmain_test.go",

View file

@ -36,6 +36,13 @@ const (
LockFileName = "proxy-child.lock"
)
// errBackendExited is the result of the supervising goroutine when the dolt
// sql-server exits on its own with status 0 (for example dolt's graceful
// shutdown on SIGTERM). errgroup cancels egCtx only for a non-nil result, and
// Running reads egCtx, so without it a cleanly exited backend would be
// reported as running forever.
var errBackendExited = errors.New("dolt sql-server exited")
const (
startReadyTimeout = 30 * time.Second
startReadyPollInterval = 50 * time.Millisecond
@ -303,7 +310,10 @@ func (s *DoltServer) Start(ctx context.Context) error {
eg.Go(func() error {
defer lock.Unlock()
return cmd.Wait()
if err := cmd.Wait(); err != nil {
return err
}
return errBackendExited
})
if err := s.waitReady(ctx); err != nil {
@ -358,7 +368,7 @@ func (s *DoltServer) Stop(ctx context.Context) error {
if s.eg != nil {
waitErr = s.eg.Wait()
var exitErr *exec.ExitError
if errors.As(waitErr, &exitErr) || errors.Is(waitErr, context.Canceled) {
if errors.As(waitErr, &exitErr) || errors.Is(waitErr, context.Canceled) || errors.Is(waitErr, errBackendExited) {
waitErr = nil
}
}

View file

@ -0,0 +1,64 @@
//go:build unix
package server_test
import (
"context"
"syscall"
"testing"
"time"
"github.com/steveyegge/beads/internal/storage/dbproxy/pidfile"
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// backendExitNoticeTimeout bounds how long Running may keep reporting a
// backend that has left the process table. dolt's own graceful shutdown on
// SIGTERM dominates this; the detection itself is immediate.
const (
backendExitNoticeTimeout = 30 * time.Second
backendExitNoticePoll = 50 * time.Millisecond
)
// TestDoltServer_RunningFalseAfterBackendExits verifies Running observes the
// dolt sql-server leaving on its own, whatever its exit status. A SIGTERM'd
// dolt shuts down gracefully and exits 0; before the fix only a non-zero exit
// flipped Running, so the proxy's backend health watcher never fired and the
// proxy stayed up (and adoptable) in front of a dead backend.
func TestDoltServer_RunningFalseAfterBackendExits(t *testing.T) {
for _, tc := range []struct {
name string
sig syscall.Signal
}{
{name: "clean exit on SIGTERM", sig: syscall.SIGTERM},
{name: "killed", sig: syscall.SIGKILL},
} {
t.Run(tc.name, func(t *testing.T) {
s, rootDir := newDoltServer(t)
ctx := context.Background()
require.NoError(t, s.Start(ctx))
require.True(t, s.Running(ctx))
pf, err := pidfile.Read(rootDir, server.PIDFileName)
require.NoError(t, err)
require.NotNil(t, pf)
require.NoError(t, syscall.Kill(pf.Pid, tc.sig))
deadline := time.Now().Add(backendExitNoticeTimeout)
for s.Running(ctx) && time.Now().Before(deadline) {
time.Sleep(backendExitNoticePoll)
}
require.False(t, s.Running(ctx), "Running still true %s after the dolt sql-server (pid %d) got %s", backendExitNoticeTimeout, pf.Pid, tc.sig)
// A backend that exited on its own is not a Stop failure.
stopCtx, cancel := context.WithTimeout(ctx, stopTimeout)
defer cancel()
require.NoError(t, s.Stop(stopCtx))
left, err := pidfile.Read(rootDir, server.PIDFileName)
require.NoError(t, err)
assert.Nil(t, left, "Stop must remove the backend pid record")
})
}
}