beads/internal/doltserver/doltserver.go
matt wilkie 6d81e1c73a
feat(workspacegate): shared physical-root resolver, beads.OpenGated, and CLI gate wiring (part 2) (#5093)
* feat(workspacegate): wire the gate through a shared physical-root resolver, OpenGated, and the CLI (part 2)

Part 2 of the workspace-gate work started in #5046, which landed the
internal/workspacegate primitive and deliberately held back OpenGated
because its private physical-root derivation was wrong in exactly the
ways review found: it skipped the store's real mode resolution and
silently skipped the physical gate on metadata-less workspaces.

internal/doltserver.ResolvePhysicalRoots is now the one shared answer
to 'which local directory will this workspace's store actually open':
side-effect free (never triggers configfile.Load's legacy-migration
write, never mkdirs the shared-server tree), central-config aware
(the dolt_mode inheritance that previously existed only inside
storage/dolt/open.go), correct for embedded (embeddeddolt, where
ResolveDoltDir answers dolt/), proxied (env > client-info sidecar >
default, hoisted out of package main so it cannot diverge), shared
(no-mkdir path), remote backends (workspace gate only), and the
supported no-metadata layouts (embeddeddolt-then-dolt stat probe).

beads.OpenGated gives library consumers a store that holds SHARED
workspace+physical gates for its lifetime. The CLI acquires at one
chokepoint after workspace resolution - SHARED for store-opening
commands, EXCLUSIVE for backup restore - and in-command for the
noDbCommands that replace state: init, migrate from-*-to-* (both
source and destination roots, ahead of migrate.lock per the normative
lock order), and bootstrap.

Posture is deliberate and asymmetric: SHARED acquisition failures that
are not contention warn once and continue ungated (the gate is
cooperative; normal commands must not brick existing deployments),
while contention aborts with the holder's identity, and EXCLUSIVE
failures of any kind refuse - maintenance must not pretend to a fence
it does not hold. Known residual: discovery-path config loads before
the chokepoint can still fire the legacy-migration write; deferring
that is follow-up scope alongside doctor, secondary-target opens, and
bd dolt set.

Agent-Signature: claude-fable-5-medium on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AQHiBWVPSXDCcUSG9EGw9d

* fix(workspacegate): nil store on failed factory open before gate-release cleanup

newDoltStore's concrete factory branches (e.g. server-mode dolt.New) return
a typed-nil pointer on error; assigning it to the storage.DoltStorage
interface makes 'store != nil' true in closeStoreBeforeGateRelease, which
then calls Close on a nil receiver and panics (SIGSEGV observed in
TestPersistentPreRunHonorsSkipStoreAnnotation, verify run 20260727T000727Z).
No store was opened on that path, so drop the value before the error return.

Agent-Signature: claude-fable-5-high on behalf of maphew
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P3JtQ3ukoVmfg6LCKcKBBh

* fix(workspacegate): address review — unparam, gate-busy lock classification, bootstrap dolt-dir gating

Three fixes from the adversarial review of gastownhall/beads#5093
(see reports/2026-07-26-reviewer-dispatch-sweep.md, "#5093" section):

- unparam (CI lint fail): drop the dead PhysicalRoots return value from
  buildWorkspaceGateSet; both call sites already discarded it, and the
  code comments already document that provenance is deliberately not
  surfaced in user-facing errors. Updated both callers.

- TestEmbeddedInitConcurrent regression (CI test fail): the new EXCLUSIVE
  workspace gate in cmd/bd/init.go short-circuits before the embedded
  Dolt flock is ever attempted, so a losing concurrent `bd init` now
  reports gate-busy output that isEmbeddedLockOutput did not recognize.
  Taught isEmbeddedLockOutput to also match workspacegate.ErrBusy's
  "workspace gate busy" text, and added a deterministic
  TestInitGateBusyClassifiedAsLockContention that holds the gate
  in-process and exercises the exact classification path, independent of
  subprocess race timing.

- Bootstrap dolt-dir gating gap: executeBootstrapPlan gated
  plan.BeadsDir's resolved physical roots but not
  doltserver.ResolveDoltDir(plan.BeadsDir), the directory
  executeInitAction/executeRestoreAction/executeJSONLImportAction
  actually open. Pass it as an extraRoot, mirroring how bd init passes
  its own resolved db path.

Also fixes a locally-discovered flake in
TestChokepointSharedExcludesMigrateExclusive: acquireMigrateGates now
threads the package-global rootCtx into acquireExclusiveWorkspaceGates,
so a prior in-process test that ran the full command path (which sets
rootCtx via setupGracefulShutdown and cancels it via rootCancel() without
resetting the var) can leak an already-canceled context into this test.
Pin rootCtx to nil for the test's duration, matching the isolation
pattern already used elsewhere in this file. Reproduced deterministically
by injecting a canceled context before the fix, confirmed neutralized
after.

TestEmbeddedInitConcurrent itself still does not reproduce the CI
gate-busy symptom on this dev machine (10/10 processes succeed within
the 5s exclusive-gate wait, so the test's own "at least 1 lock error"
assertion still fails locally) — that is a machine-speed-dependent
property of the test's contention window, not the classification bug;
the classification fix is independently verified by the new deterministic
test and by code inspection of workspacegate.ErrBusy's text.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 01:41:20 -07:00

1843 lines
67 KiB
Go

// Package doltserver manages the lifecycle of a local dolt sql-server process.
// It provides transparent auto-start so that `bd init` and `bd <command>` work
// without manual server management.
//
// Port assignment uses OS-assigned ephemeral ports by default. When no explicit
// port is configured (env var, config.yaml, metadata.json), Start() asks the OS
// for a free port via net.Listen(":0"), passes it to dolt sql-server, and writes
// the actual port to dolt-server.port. This eliminates the birthday-problem
// collisions that plagued the old hash-derived port scheme (GH#2098, GH#2372).
//
// Users with explicit port config via BEADS_DOLT_SERVER_PORT env var or
// config.yaml always use that port instead, with conflict detection via
// reclaimPort.
//
// Server state files (PID, port, log, lock) live in the .beads/ directory.
package doltserver
import (
"context"
"database/sql"
"errors"
"fmt"
"net"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/dolthub/dolt/go/libraries/doltcore/servercfg"
"github.com/dolthub/dolt/go/libraries/utils/filesys"
_ "github.com/go-sql-driver/mysql"
"gopkg.in/yaml.v3"
"github.com/steveyegge/beads/internal/config"
"github.com/steveyegge/beads/internal/configfile"
"github.com/steveyegge/beads/internal/debug"
"github.com/steveyegge/beads/internal/lockfile"
"github.com/steveyegge/beads/internal/storage/doltutil"
)
// ErrServerNotRunning is returned by Stop when the Dolt server is not running.
// Callers can use errors.Is to distinguish this expected condition from real
// failures (GH#2670).
var ErrServerNotRunning = errors.New("dolt server is not running")
// IgnoreNotRunning strips ErrServerNotRunning from err and returns any
// remaining errors (typically cleanup failures). If the only error was the
// sentinel, it returns nil. Handles both errors.Join (multi-unwrap) and
// standard fmt.Errorf wrapping (single-unwrap).
//
// IMPORTANT: call directly on Stop()/StopWithForce() return values only.
// Do not wrap the error before passing it here — wrapping may hide joined
// cleanup errors from the multi-unwrap path.
func IgnoreNotRunning(err error) error {
if err == nil {
return nil
}
if !errors.Is(err, ErrServerNotRunning) {
return err // unrelated error, pass through
}
// Multi-error from errors.Join: filter out the sentinel, keep the rest.
if joined, ok := err.(interface{ Unwrap() []error }); ok {
var remaining []error
for _, e := range joined.Unwrap() {
if e != nil && !errors.Is(e, ErrServerNotRunning) {
remaining = append(remaining, e)
}
}
return errors.Join(remaining...)
}
// Single-wrapped error (e.g., fmt.Errorf("%w", ErrServerNotRunning)):
// the sentinel is the only meaningful content, so treat as pure sentinel.
return nil
}
// PIDFileName and PortFileName are the canonical state file names used by the
// Dolt server lifecycle. They are exported so cross-package tests can reference
// the same names as the production code.
const (
PIDFileName = "dolt-server.pid"
PortFileName = "dolt-server.port"
)
// maxEphemeralPortAttempts is the number of times Start() retries ephemeral
// port allocation when the TOCTOU race causes a bind failure.
const maxEphemeralPortAttempts = 10
// DefaultSharedServerPort is the default port for shared server mode.
// Uses 3308 to avoid conflict with the orchestrator which uses 3307.
const DefaultSharedServerPort = 3308
// GlobalDatabaseName is the SQL database name for the project-agnostic
// global issue database in shared-server mode.
const GlobalDatabaseName = "beads_global"
// GlobalIssuePrefix is the issue prefix used in the global database.
const GlobalIssuePrefix = "global"
// GlobalProjectID is the well-known sentinel UUID for the global database.
// Used for project identity verification — the global DB doesn't belong to
// any single project, so it uses this fixed value instead of a random UUID.
const GlobalProjectID = "00000000-0000-0000-0000-000000000000"
// IsSharedServerMode returns true if shared server mode is enabled.
// Checks (in priority order):
// 1. BEADS_DOLT_SHARED_SERVER env var ("1" or "true")
// 2. dolt.shared-server in config.yaml
//
// Shared server mode means all projects on this machine share a single
// dolt sql-server process at SharedServerDir(), each using its own
// database (already unique via prefix-based naming in bd init).
func IsSharedServerMode() bool {
if v := os.Getenv("BEADS_DOLT_SHARED_SERVER"); v == "1" || strings.EqualFold(v, "true") {
return true
}
return config.GetBool("dolt.shared-server")
}
func IsDebugMode() bool {
if v := os.Getenv("BEADS_DOLT_DEBUG"); v == "1" || strings.EqualFold(v, "true") {
return true
}
return config.GetBool("dolt.debug")
}
func DebugProfileDir(beadsDir string) string {
p := filepath.Join(resolveServerDir(beadsDir), "dolt-pprof")
if abs, err := filepath.Abs(p); err == nil {
return abs
}
return p
}
const debugProfileFilename = "cpu.pprof"
func rotateDebugProfile(beadsDir string) {
profDir := DebugProfileDir(beadsDir)
src := filepath.Join(profDir, debugProfileFilename)
info, err := os.Stat(src)
if err != nil || info.Size() == 0 {
// No profile to rotate (server killed before flush, or never started in debug).
return
}
ts := time.Now().UTC().Format("20060102T150405Z")
dst := filepath.Join(profDir, fmt.Sprintf("cpu-%s.pprof", ts))
if err := os.Rename(src, dst); err != nil {
fmt.Fprintf(os.Stderr, "Warning: could not rotate %s → %s: %v\n", src, dst, err)
return
}
fmt.Fprintf(os.Stderr, "Debug: cpu profile rotated to %s\n", dst)
}
// IsAutoStartDisabled returns true if the dolt server should NOT be
// auto-started or managed by bd. When true, KillStaleServers and
// auto-start are suppressed — the server is externally managed (e.g.,
// by systemd).
//
// Either source can disable auto-start independently — there is no way
// to force-enable via env when the config file says disabled. Accepted
// disable values: any value strconv.ParseBool recognizes as false
// ("0", "f", "F", "false", "FALSE", "False") plus "off" (case-insensitive)
// for backward compatibility.
//
// This is used by KillStaleServers and Start to avoid killing or
// interfering with externally-managed dolt processes (GH#2641).
func IsAutoStartDisabled() bool {
if isFalsyBool(os.Getenv("BEADS_DOLT_AUTO_START")) {
return true
}
return isFalsyBool(config.GetString("dolt.auto-start"))
}
// isFalsyBool returns true when s is a recognized "false" value:
// anything strconv.ParseBool accepts as false, or "off" (case-insensitive).
// Leading/trailing whitespace is trimmed before parsing.
func isFalsyBool(s string) bool {
s = strings.TrimSpace(s)
if strings.EqualFold(s, "off") {
return true
}
b, err := strconv.ParseBool(s)
return err == nil && !b
}
// readyTimeout returns the timeout used by waitForReady when starting the
// dolt sql-server. Defaults to 10 seconds, but can be overridden via the
// BEADS_DOLT_READY_TIMEOUT environment variable (positive integer seconds).
// First-run Dolt SQL engine initialization can take ~60s on slower hardware
// where the privileges.db, stats subrepo, and other bootstrap work must
// happen before the MySQL listener accepts TCP connections. See GH#3142.
func readyTimeout() time.Duration {
const defaultTimeout = 10 * time.Second
v := strings.TrimSpace(os.Getenv("BEADS_DOLT_READY_TIMEOUT"))
if v == "" {
return defaultTimeout
}
secs, err := strconv.Atoi(v)
if err != nil || secs < 1 {
fmt.Fprintf(os.Stderr,
"Warning: BEADS_DOLT_READY_TIMEOUT=%q is not a positive integer; using default %s\n",
v, defaultTimeout)
return defaultTimeout
}
return time.Duration(secs) * time.Second
}
// SharedServerPath returns the directory for shared server state files without
// creating it. Override with BEADS_SHARED_SERVER_DIR for testing or custom
// layouts; otherwise it resolves to ~/.beads/shared-server/.
func SharedServerPath() (string, error) {
if d := os.Getenv("BEADS_SHARED_SERVER_DIR"); d != "" {
return d, nil
}
home, err := os.UserHomeDir()
if err != nil {
return "", fmt.Errorf("cannot determine home directory: %w", err)
}
return filepath.Join(home, ".beads", "shared-server"), nil
}
// SharedServerDir returns the directory for shared server state files.
// Returns ~/.beads/shared-server/ (created on first use).
// Override with BEADS_SHARED_SERVER_DIR env var for testing or custom layouts.
func SharedServerDir() (string, error) {
dir, err := SharedServerPath()
if err != nil {
return "", err
}
if err := os.MkdirAll(dir, config.BeadsDirPerm); err != nil {
return "", fmt.Errorf("cannot create shared server directory %s: %w", dir, err)
}
return dir, nil
}
// SharedDoltDir returns the dolt data directory for the shared server.
// Returns ~/.beads/shared-server/dolt/ (created on first use).
func SharedDoltDir() (string, error) {
serverDir, err := SharedServerDir()
if err != nil {
return "", err
}
dir := filepath.Join(serverDir, "dolt")
if err := os.MkdirAll(dir, config.BeadsDirPerm); err != nil {
return "", fmt.Errorf("cannot create shared dolt directory %s: %w", dir, err)
}
return dir, nil
}
// resolveServerDir returns the canonical server directory for dolt state files.
// In shared server mode, returns ~/.beads/shared-server/ instead of the
// project's .beads/ directory.
func resolveServerDir(beadsDir string) string {
if IsSharedServerMode() {
dir, err := SharedServerDir()
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: shared server directory unavailable, using per-project mode: %v\n", err)
return beadsDir
}
return dir
}
return beadsDir
}
// ResolveServerDir is the exported version of resolveServerDir.
// CLI commands use this to resolve the server directory before calling
// Start, Stop, or IsRunning.
func ResolveServerDir(beadsDir string) string {
return resolveServerDir(beadsDir)
}
// ResolveDoltDir returns the dolt data directory for the given beadsDir.
// It checks the BEADS_DOLT_DATA_DIR env var and metadata.json for a custom
// dolt_data_dir, falling back to the default .beads/dolt/ path.
//
// Note: we check for metadata.json existence before calling configfile.Load
// to avoid triggering the config.json → metadata.json migration side effect,
// which would create files in the .beads/ directory unexpectedly.
func ResolveDoltDir(beadsDir string) string {
// Shared server mode: use centralized dolt data directory
if IsSharedServerMode() {
dir, err := SharedDoltDir()
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: shared dolt directory unavailable, using per-project mode: %v\n", err)
} else {
return dir
}
}
// Env var, metadata.json dolt_data_dir (stat-guarded), then default —
// shared with the side-effect-free DoltDirPath so the two resolvers
// cannot drift.
return projectDoltDirPath(beadsDir)
}
// Config holds the server configuration.
type Config struct {
BeadsDir string // Path to .beads/ directory
Port int // MySQL protocol port (0 = allocate ephemeral port on Start)
Host string // Bind address (default: 127.0.0.1)
Mode ServerMode // Server ownership mode (Owned, External, Embedded)
// PortSource records which step of the precedence chain (see
// portSources) resolved Port. PortSourceUnset when Port == 0. Callers
// that auto-start a replacement server use this to decide whether
// silently retargeting to a different port is safe (GH#4052): safe for
// bd's own port-file bookkeeping, not safe for a source where the user
// (or config on the user's behalf) explicitly asserted the port.
PortSource PortSource
// PortSharedServer records whether Port was resolved in shared-server
// mode (BEADS_DOLT_SHARED_SERVER=1): either because port resolution
// consulted the shared server directory's sources, or because no source
// resolved a port and DefaultConfig fell back to DefaultSharedServerPort.
// Orthogonal to PortSource — a shared-mode port can come from any source
// (env, port file, config.yaml, ...). Callers use this together with
// PortSource.IsAuthoritative() to decide whether auto-starting a
// repo-local server on a different port is safe: it never is here,
// because the shared server is a different database than whatever
// auto-start would create locally (GH#4052).
PortSharedServer bool
}
// State holds runtime information about a managed server.
type State struct {
Running bool `json:"running"`
PID int `json:"pid"`
Port int `json:"port"`
DataDir string `json:"data_dir"`
}
// file paths within .beads/
func pidPath(beadsDir string) string { return filepath.Join(beadsDir, PIDFileName) }
func logPath(beadsDir string) string { return filepath.Join(beadsDir, "dolt-server.log") }
func lockPath(beadsDir string) string { return filepath.Join(beadsDir, "dolt-server.lock") }
func portPath(beadsDir string) string { return filepath.Join(beadsDir, PortFileName) }
// MaxDoltServers is the hard ceiling on concurrent dolt sql-server processes.
// Allows up to 3 (e.g., multiple projects).
func maxDoltServers() int {
return 3
}
// allocateEphemeralPort asks the OS for a free TCP port on host.
// It binds to port 0, reads the assigned port, and closes the listener.
// The caller should pass the returned port to dolt sql-server promptly
// to minimize the TOCTOU window.
func allocateEphemeralPort(host string) (int, error) {
ln, err := net.Listen("tcp", net.JoinHostPort(host, "0"))
if err != nil {
return 0, fmt.Errorf("allocating ephemeral port: %w", err)
}
port := ln.Addr().(*net.TCPAddr).Port
_ = ln.Close()
return port, nil
}
// isPortAvailable checks if a TCP port is available for binding.
func isPortAvailable(host string, port int) bool {
addr := net.JoinHostPort(host, strconv.Itoa(port))
ln, err := net.Listen("tcp", addr)
if err != nil {
return false
}
_ = ln.Close()
return true
}
// reclaimPort ensures an explicit (user-configured) port is available for use.
// Only called for explicit ports (env var, config.yaml, metadata.json).
// If the port is busy:
// - If our dolt server (same data dir) → return its PID for adoption
// - If a stale/orphan dolt sql-server holds it → kill it and reclaim
// - If another project's dolt or a non-dolt process → return error
//
// Returns (adoptPID, nil) when an existing server should be adopted.
// Returns (0, nil) when the port is free for a new server.
// Returns (0, err) when the port can't be used.
func reclaimPort(host string, port int, beadsDir string) (adoptPID int, err error) {
if isPortAvailable(host, port) {
return 0, nil // port is free
}
// Port is busy — find out what's using it
pid := findPIDOnPort(port)
if pid == 0 {
// Can't identify the process; port may be in TIME_WAIT or transient use.
// Wait briefly and retry.
time.Sleep(2 * time.Second)
if isPortAvailable(host, port) {
return 0, nil
}
return 0, fmt.Errorf("port %d is busy but cannot identify the process.\n\nCheck with: %s", port, fmt.Sprintf(portConflictHint, port))
}
// Check if it's a dolt sql-server process
if !isDoltProcess(pid) {
return 0, fmt.Errorf("port %d is in use by a non-dolt process (PID %d).\n\n%s\n\nFree the port or configure a different one with: bd dolt set port <port>", port, pid, portConflictDiagnostics(port))
}
// It's a dolt process. Check if it's one we should adopt.
// Check if the process is using our data directory (CWD matches our dolt dir).
// dolt sql-server is started with cmd.Dir = doltDir, so CWD is the data dir.
doltDir := ResolveDoltDir(beadsDir)
if isProcessInDir(pid, doltDir) {
return pid, nil // our server — adopt it
}
// Another beads project's Dolt server is on this port.
return 0, fmt.Errorf("port %d is in use by another project's dolt server (PID %d).\n\n%s\n\nFree the port or use a different one with: bd dolt set port <port>", port, pid, portConflictDiagnostics(port))
}
// portConflictDiagnostics returns a multi-line block of operator-actionable
// hints for diagnosing what's holding a port. Combines the platform-specific
// listener-discovery command with a docker-in-the-loop hint that frequently
// applies in practice — operators running their own dolt sql-server in a
// container don't realize bd would otherwise try to start a competing
// instance and lose the race (GH#3516).
func portConflictDiagnostics(port int) string {
return fmt.Sprintf("Identify the listener:\n %s\n\n"+
"If the listener is YOUR own Dolt instance (e.g., a docker container "+
"or systemd unit you manage), bd does not need to start a new server. "+
"Configure bd to talk to the existing server instead:\n"+
" export BEADS_DOLT_SERVER_HOST=<host> # 127.0.0.1 for local container\n"+
" export BEADS_DOLT_SERVER_PORT=%d\n"+
" bd dolt status # verify reachable",
fmt.Sprintf(portConflictHint, port), port)
}
// countDoltProcesses returns the number of running dolt sql-server processes.
func countDoltProcesses() int { return len(listDoltProcessPIDs()) }
// isDoltProcess checks if a PID belongs to a running dolt sql-server.
func isDoltProcess(pid int) bool {
for _, p := range listDoltProcessPIDs() {
if p == pid {
return true
}
}
return false
}
// readPortFile reads the actual port from the port file, if it exists.
// Returns 0 if the file doesn't exist or is unreadable.
func readPortFile(beadsDir string) int {
data, err := os.ReadFile(portPath(beadsDir))
if err != nil {
return 0
}
port, err := strconv.Atoi(strings.TrimSpace(string(data)))
if err != nil {
return 0
}
return port
}
// writePortFile records the actual port the server is listening on.
// Write-temp-then-rename: a plain os.WriteFile truncates in place, so a
// concurrent readPortFile can observe an empty or partial file and resolve
// port 0 (or a truncated port). Rename within .beads is atomic.
func writePortFile(beadsDir string, port int) error {
path := portPath(beadsDir)
tmp, err := os.CreateTemp(beadsDir, PortFileName+".tmp-*")
if err != nil {
return err
}
tmpName := tmp.Name()
defer os.Remove(tmpName) //nolint:errcheck // no-op after successful rename
if _, err := tmp.WriteString(strconv.Itoa(port)); err != nil {
_ = tmp.Close()
return err
}
if err := tmp.Chmod(0o600); err != nil {
_ = tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
return os.Rename(tmpName, path)
}
// EnsurePortFile makes the repo-local port file match the connected server port.
// This is a best-effort repair path for upgraded repos that are missing
// .beads/dolt-server.port even though commands can still connect.
func EnsurePortFile(beadsDir string, port int) error {
if beadsDir == "" || port <= 0 {
return nil
}
existing := readPortFile(beadsDir)
if existing == port {
return nil
}
if existing > 0 {
fmt.Fprintf(os.Stderr, "Info: updating port file %d → %d in %s\n", existing, port, beadsDir)
}
return writePortFile(beadsDir, port)
}
// ReadPortFile returns the port from the project's dolt-server.port file,
// or 0 if the file doesn't exist or is invalid. Exported for use by bd init
// to detect whether this project has its own running server (GH#2336).
func ReadPortFile(beadsDir string) int {
return readPortFile(beadsDir)
}
// PortFileSnapshot captures the exact prior contents of a project's port
// file, so a caller that speculatively lets EnsureRunningDetailed write a
// new one can restore the pre-call state exactly if it later decides not to
// use the new server (GH#4052 fail-closed paths). Existed is false when the
// file did not exist before the snapshot; in that case Data is nil and
// RestorePortFile removes the file rather than rewriting it.
type PortFileSnapshot struct {
Data []byte
Existed bool
}
// SnapshotPortFile captures the current on-disk state of beadsDir's port
// file (see PortFileSnapshot). Read-only; does not mutate anything.
func SnapshotPortFile(beadsDir string) (PortFileSnapshot, error) {
data, err := os.ReadFile(portPath(beadsDir))
if err != nil {
if os.IsNotExist(err) {
return PortFileSnapshot{}, nil
}
return PortFileSnapshot{}, err
}
// Copy: os.ReadFile's backing array should not be retained/mutated by
// later callers of the same path.
cp := make([]byte, len(data))
copy(cp, data)
return PortFileSnapshot{Data: cp, Existed: true}, nil
}
// RestorePortFile restores beadsDir's port file to the state captured by
// snap: removes the file if it did not exist before, or rewrites it with the
// exact prior bytes (write-temp-then-rename, matching writePortFile, so a
// concurrent reader never observes a partially written file). Used on
// fail-closed auto-start paths (GH#4052) that must leave no port-file trace
// of a server they decided not to use.
func RestorePortFile(beadsDir string, snap PortFileSnapshot) error {
path := portPath(beadsDir)
if !snap.Existed {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
return err
}
return nil
}
tmp, err := os.CreateTemp(beadsDir, PortFileName+".tmp-*")
if err != nil {
return err
}
tmpName := tmp.Name()
defer os.Remove(tmpName) //nolint:errcheck // no-op after successful rename
if _, err := tmp.Write(snap.Data); err != nil {
_ = tmp.Close()
return err
}
if err := tmp.Chmod(0o600); err != nil {
_ = tmp.Close()
return err
}
if err := tmp.Close(); err != nil {
return err
}
return os.Rename(tmpName, path)
}
func configYamlPort(beadsDir string) int {
path := filepath.Join(ResolveDoltDir(beadsDir), "config.yaml")
if _, err := os.Stat(path); err != nil {
return 0
}
cfg, err := servercfg.YamlConfigFromFile(filesys.LocalFS, path)
if err != nil {
return 0
}
return cfg.Port()
}
// PortSource identifies which step of the port-resolution precedence chain
// (see portSources) produced a Config's Port. Callers use it to distinguish
// a port the user explicitly asserted (authoritative) from bd's own
// bookkeeping (the gitignored port file), which auto-start is free to
// replace without confirmation (GH#4052).
type PortSource string
const (
// PortSourceUnset means no source resolved a port (Port == 0); Start()
// will allocate an ephemeral port.
PortSourceUnset PortSource = ""
// PortSourceEnv is the BEADS_DOLT_SERVER_PORT environment variable.
PortSourceEnv PortSource = "env"
// PortSourcePortFile is the gitignored .beads/dolt-server.port file that
// bd itself writes and rewrites as part of normal server bookkeeping.
PortSourcePortFile PortSource = "port_file"
// PortSourceDoltConfigYaml is the Dolt server's own config.yaml
// (listener.port) in the dolt data directory.
PortSourceDoltConfigYaml PortSource = "dolt_config_yaml"
// PortSourceConfigYaml is Beads' project/global config.yaml (dolt.port).
PortSourceConfigYaml PortSource = "config_yaml"
// PortSourceGlobalConfig is retained as an alias of PortSourceConfigYaml
// for callers that think of it as "global config" (~/.config/bd/config.yaml
// resolves through the same config.GetYamlConfig("dolt.port") lookup).
PortSourceGlobalConfig = PortSourceConfigYaml
// PortSourceMetadataJSON is the deprecated metadata.json dolt_server_port
// fallback. Still an explicit user/tooling assertion, not bd bookkeeping.
PortSourceMetadataJSON PortSource = "metadata_json"
)
// IsAuthoritative reports whether this source represents a user (or
// tooling-on-the-user's-behalf) assertion of where the Dolt server lives, as
// opposed to bd's own bookkeeping (the port file). Auto-start may freely
// replace a non-authoritative port; it must not silently replace an
// authoritative one (GH#4052).
func (s PortSource) IsAuthoritative() bool {
switch s {
case PortSourceEnv, PortSourceDoltConfigYaml, PortSourceConfigYaml, PortSourceMetadataJSON:
return true
default:
return false
}
}
// portSource is one step in the port-resolution precedence chain that
// DefaultConfig walks. resolve reports (port, true) when this source has a
// usable value; DefaultConfig stops at the first true.
type portSource struct {
label string
source PortSource
resolve func(beadsDir string) (int, bool)
}
// portSources is the single precedence chain for server-port resolution.
// DefaultConfig consumes it behaviorally; PortSourceLabels exposes the same
// slice's labels so callers like `bd dolt show-config` render the actual
// chain instead of hand-copying it out of sync (GH#4511).
var portSources = []portSource{
{
label: "Environment variable (BEADS_DOLT_SERVER_PORT)",
source: PortSourceEnv,
resolve: func(beadsDir string) (int, bool) {
p := os.Getenv("BEADS_DOLT_SERVER_PORT")
if p == "" {
return 0, false
}
port, err := strconv.Atoi(p)
return port, err == nil
},
},
{
// Gitignored, local-only. Elevated above the YAML sources to prevent
// git-tracked values from causing cross-project data leakage (GH#2372).
label: "Port file (.beads/dolt-server.port; shared-server dir in shared mode)",
source: PortSourcePortFile,
resolve: func(beadsDir string) (int, bool) {
p := readPortFile(beadsDir)
return p, p > 0
},
},
{
label: "Dolt server config.yaml (listener.port, in the dolt data directory)",
source: PortSourceDoltConfigYaml,
resolve: func(beadsDir string) (int, bool) {
p := configYamlPort(beadsDir)
return p, p > 0
},
},
{
// ~/.config/bd/config.yaml or project config.yaml (GH#2073). Git-tracked,
// so it ranks below the port file above.
label: "Beads config.yaml / global config (dolt.port)",
source: PortSourceConfigYaml,
resolve: func(beadsDir string) (int, bool) {
p := config.GetYamlConfig("dolt.port")
if p == "" {
return 0, false
}
port, err := strconv.Atoi(p)
return port, err == nil && port > 0
},
},
{
// Deprecated: git-tracked, propagates to all contributors, causing
// cross-project data leakage (GH#2372). Kept as a fallback so existing
// setups don't break silently.
label: "metadata.json dolt_server_port (deprecated fallback)",
source: PortSourceMetadataJSON,
resolve: func(beadsDir string) (int, bool) {
metaCfg, err := configfile.Load(beadsDir)
if err != nil || metaCfg == nil || metaCfg.DoltServerPort <= 0 {
return 0, false
}
fmt.Fprintf(os.Stderr, "Warning: dolt_server_port in metadata.json is deprecated (can cause cross-project data leakage).\n")
fmt.Fprintf(os.Stderr, " The port file (.beads/dolt-server.port) is now the primary source.\n")
fmt.Fprintf(os.Stderr, " Remove dolt_server_port from .beads/metadata.json to silence this warning.\n")
return metaCfg.DoltServerPort, true
},
},
}
// PortSourceLabels returns the operator-facing description of each step in
// the port-resolution precedence chain, in priority order.
func PortSourceLabels() []string {
labels := make([]string, len(portSources))
for i, s := range portSources {
labels[i] = s.label
}
return labels
}
// DefaultConfig returns config with sensible defaults. Port resolution walks
// portSources in priority order (see PortSourceLabels) and returns port 0
// when no source provides one, meaning Start() should allocate an ephemeral
// port from the OS.
//
// The port file (dolt-server.port) is written by Start() with the actual
// listening port, so already-running-server connections use the right port.
func DefaultConfig(beadsDir string) *Config {
// In shared mode, use the shared server directory for port resolution
sharedMode := false
if IsSharedServerMode() {
if sharedDir, err := SharedServerDir(); err == nil {
beadsDir = sharedDir
sharedMode = true
}
}
cfg := &Config{
BeadsDir: beadsDir,
Host: "127.0.0.1",
Mode: ResolveServerMode(beadsDir),
}
for _, src := range portSources {
if port, ok := src.resolve(beadsDir); ok {
cfg.Port = port
cfg.PortSource = src.source
cfg.PortSharedServer = sharedMode
break
}
}
// Port 0 means "no configured port". In shared mode, use the fixed
// shared server port. In per-project mode, Start() will allocate an
// ephemeral port from the OS (GH#2098, GH#2372).
if cfg.Port == 0 && IsSharedServerMode() {
cfg.Port = DefaultSharedServerPort // 3308 - avoids orchestrator conflict on 3307
cfg.PortSharedServer = true
}
return cfg
}
// IsRunning checks if a managed server is running for this beadsDir.
// Returns a State with Running=true if a valid dolt process is found.
func IsRunning(beadsDir string) (*State, error) {
data, err := os.ReadFile(pidPath(beadsDir))
if err != nil {
if os.IsNotExist(err) {
return &State{Running: false}, nil
}
return nil, fmt.Errorf("reading PID file: %w", err)
}
pid, err := strconv.Atoi(strings.TrimSpace(string(data)))
if err != nil {
// Corrupt PID file implies stale state; clear the port file too.
_ = os.Remove(pidPath(beadsDir))
_ = os.Remove(portPath(beadsDir))
return &State{Running: false}, nil
}
// Check if process is alive
if !isProcessAlive(pid) {
// Process is dead — clear all tracked state for this server.
_ = os.Remove(pidPath(beadsDir))
_ = os.Remove(portPath(beadsDir))
return &State{Running: false}, nil
}
// Verify it's actually a dolt sql-server process
if !isDoltProcess(pid) {
// PID was reused by another process
_ = os.Remove(pidPath(beadsDir))
_ = os.Remove(portPath(beadsDir))
return &State{Running: false}, nil
}
// Read actual port from port file; fall back to config-derived port.
port := readPortFile(beadsDir)
if port == 0 {
cfg := DefaultConfig(beadsDir)
port = cfg.Port
}
if port == 0 {
// Server is running but we can't determine its port (port file
// missing, no explicit config). Stop the orphan so that callers
// (EnsureRunning) trigger a fresh Start() with a new port file.
fmt.Fprintf(os.Stderr, "Dolt server (PID %d) running but port unknown; stopping for restart\n", pid)
if err := gracefulStop(pid, 5*time.Second); err != nil {
// Best-effort kill
if proc, findErr := os.FindProcess(pid); findErr == nil {
_ = proc.Kill()
}
}
_ = os.Remove(pidPath(beadsDir))
return &State{Running: false}, nil
}
return &State{
Running: true,
PID: pid,
Port: port,
DataDir: ResolveDoltDir(beadsDir),
}, nil
}
// EnsureRunning starts the server if it is not already running.
// This is the main auto-start entry point. Thread-safe via file lock.
// Returns the port the server is listening on.
//
// When metadata.json specifies an explicit dolt_server_port (indicating an
// external/shared server, e.g. managed by systemd), EnsureRunning will NOT
// start a new server. The external server's lifecycle is not bd's
// responsibility — starting a per-project server would conflict with (or
// kill) the shared server. See GH#2554.
func EnsureRunning(beadsDir string) (int, error) {
port, _, err := EnsureRunningDetailed(beadsDir)
return port, err
}
// EnsureRunningDetailed is like EnsureRunning but also reports whether a new
// server was started (startedByUs=true) vs. an already-running server was
// adopted (startedByUs=false). Callers that need to clean up auto-started
// servers (e.g. test teardown) should use this variant.
func EnsureRunningDetailed(beadsDir string) (port int, startedByUs bool, err error) {
serverDir := resolveServerDir(beadsDir)
// Inform when an orchestrator is also running on this machine
if IsSharedServerMode() && os.Getenv("GT_ROOT") != "" {
fmt.Fprintf(os.Stderr, "Info: Orchestrator detected (GT_ROOT set). Shared server uses port %d to avoid conflict.\n", DefaultSharedServerPort)
}
state, err := IsRunning(serverDir)
if err != nil {
return 0, false, err
}
if state.Running {
_ = EnsurePortFile(serverDir, state.Port)
return state.Port, false, nil
}
// If the server mode is External (explicit port in metadata.json,
// shared server mode, etc.), do not start a per-project server —
// it would conflict with the external one.
mode := ResolveServerMode(beadsDir)
if mode == ServerModeExternal {
cfg := DefaultConfig(beadsDir)
return 0, false, fmt.Errorf("Dolt server is not running on port %d, and auto-start is suppressed "+
"because the server is externally managed (dolt.auto-start: false or explicit port configured).\n\n"+
"Start the external server, or enable auto-start to allow bd to manage the server.\n"+
" To start manually: bd dolt start\n"+
" To check status: bd dolt status", cfg.Port)
}
// Defense-in-depth: if dolt.auto-start is explicitly disabled in
// config.yaml or env, never spawn a server even if the caller
// somehow reached this point (e.g. stale AutoStart=true in config).
if IsAutoStartDisabled() {
cfg := DefaultConfig(beadsDir)
return 0, false, fmt.Errorf("Dolt server unreachable (port %d) and auto-start is disabled "+
"(dolt.auto-start: false in config.yaml or BEADS_DOLT_AUTO_START=0).\n\n"+
"Start the server manually or enable auto-start.\n"+
" To start manually: bd dolt start\n"+
" To check status: bd dolt status", cfg.Port)
}
s, err := Start(serverDir)
if err != nil {
return 0, false, err
}
return s.Port, true, nil
}
// doltServerLogLevel is the --loglevel value passed to `dolt sql-server`.
//
// Dolt's sql-server logs every new connection and connection close at INFO
// level (`msg=NewConnection` / `msg=ConnectionClosed`). Because beads opens
// a fresh MySQL connection for each `bd` invocation, a busy project can
// produce millions of lines of connection churn noise, which in one field
// report filled dolt-server.log with ~380 MB of useless entries, generated
// significant btrfs write pressure, and buried real error signals.
//
// Raising the floor to `warning` silences that chatter while still surfacing
// warnings, errors, and fatal messages. Valid dolt levels are:
// trace, debug, info, warning, error, fatal.
const doltServerLogLevel = "warning"
// buildDoltServerArgs returns the argv passed to `dolt` (excluding argv[0]/
// the binary itself). It is factored out of Start so it can be asserted on
// in unit tests without spawning a real server.
//
// The `--loglevel` flag MUST be included here — see doltServerLogLevel for
// the rationale. If you remove or reorder these args, update the tests in
// doltserver_test.go accordingly.
//
// When debug is true, the argv begins with `--prof cpu --prof-path <profDir>`.
// These top-level dolt flags MUST appear before the `sql-server` subcommand:
// dolt's argv loop stops scanning debug flags on the first unknown token
// (see ~/cursor_src/dolt/go/cmd/dolt/dolt.go runMain). The caller must
// ensure profDir already exists — dolt panics if it does not.
//
// Debug mode also raises --loglevel from the default warning to debug;
// the connection-log spam concern that motivated the warning floor is
// the price of opting into debug.
func buildDoltServerArgs(host string, port int, debug bool, profDir string) []string {
var args []string
if debug {
args = append(args, "--prof", "cpu", "--prof-path", profDir)
}
args = append(args,
"sql-server",
"-H", host,
"-P", strconv.Itoa(port),
)
if debug {
args = append(args, "--loglevel=debug")
} else {
args = append(args, "--loglevel="+doltServerLogLevel)
}
return args
}
// doltServerConfigFileName is the YAML config Start() writes when the
// resolved external dolt binary supports auto_gc_behavior.archive_level
// (see SupportsArchiveLevelConfig). It lives alongside the other gitignored
// server state files in beadsDir, not inside the Dolt data directory.
const doltServerConfigFileName = "dolt-server-config.yaml"
func doltServerConfigPath(beadsDir string) string {
return filepath.Join(beadsDir, doltServerConfigFileName)
}
// doltCfgDirName mirrors commands.DefaultCfgDirName in the pinned dolt
// module (cmd/dolt/commands/sql.go) — the on-disk directory name Dolt looks
// in for privileges.db (users/passwords) and branch_control.db.
const doltCfgDirName = ".doltcfg"
// ErrMultipleDoltCfgDirs is returned by resolveCfgDir when both a parent
// and a data-directory .doltcfg exist. Mirrors Dolt's own ambiguous case
// (commands.ErrMultipleDoltCfgDirs in the pinned module) — guessing which
// one to use risks the same silent user/branch-control loss this function
// exists to prevent, so this is surfaced as a hard Start() failure instead.
var ErrMultipleDoltCfgDirs = errors.New("multiple .doltcfg directories detected")
// resolveCfgDir replicates Dolt's own flag-mode .doltcfg discovery
// (setupDoltConfig in cmd/dolt/commands/sqlserver/sqlserver.go, pinned
// module) for our generated --config YAML. setupDoltConfig returns
// immediately when --config is passed, so a deployment that previously ran
// this launcher in CLI-flag mode — where dolt auto-discovers a parent
// ../.doltcfg holding privileges.db/branch_control.db — would otherwise
// silently fall back to a fresh $data_dir/.doltcfg under --config mode:
// existing users and branch controls abandoned, new passwordless root
// (gastownhall/beads#4986).
//
// doltDir is the server's data directory (== process cwd, since neither
// --data-dir nor --doltcfg-dir are ever passed). Mirrors Dolt exactly:
// - parent ../.doltcfg (relative to doltDir) if it exists and is a dir
// - else doltDir/.doltcfg
// - ErrMultipleDoltCfgDirs if BOTH exist — ambiguous, matches Dolt's own
// ErrMultipleDoltCfgDirs case rather than guessing.
//
// doltDir is resolved to its physical (symlink-free) location before
// computing the parent. filepath.Join/Clean strip ".." lexically without
// touching the filesystem: if doltDir is itself a symlink, a naive
// filepath.Join(doltDir, "..") resolves to the symlink's own lexical
// parent directory, not the physical parent of whatever it points at. On
// Linux, chdir into a symlink makes the kernel track the process's cwd
// physically, so the actual `dolt` child process's own "../.doltcfg"
// lookup (a bare relative path, resolved against its real, physically
// tracked cwd — see setupDoltConfig) sees the PHYSICAL parent. Using
// filepath.EvalSymlinks here keeps this resolution consistent with what
// the child process itself does; skipping it would let a symlinked data
// directory miss the very parent .doltcfg this function exists to find,
// re-abandoning existing privileges/branch-control (gastownhall/beads#4986
// round 3).
//
// Returns an absolute path.
func resolveCfgDir(doltDir string) (string, error) {
// Fall back to the lexical doltDir on error (e.g. it does not exist
// yet) rather than failing outright — resolveCfgDir has no dependency
// on doltDir already existing beyond this best-effort symlink check,
// and ensureDoltInit already runs before this is called from Start().
physicalDoltDir := doltDir
if resolved, err := filepath.EvalSymlinks(doltDir); err == nil {
physicalDoltDir = resolved
}
parentDirCfg := filepath.Join(physicalDoltDir, "..", doltCfgDirName)
parentInfo, parentErr := os.Stat(parentDirCfg)
parentExists := parentErr == nil && parentInfo.IsDir()
currDirCfg := filepath.Join(physicalDoltDir, doltCfgDirName)
currInfo, currErr := os.Stat(currDirCfg)
currExists := currErr == nil && currInfo.IsDir()
if parentExists && currExists {
absParent, err := filepath.Abs(parentDirCfg)
if err != nil {
absParent = parentDirCfg
}
absCurr, err := filepath.Abs(currDirCfg)
if err != nil {
absCurr = currDirCfg
}
return "", fmt.Errorf("%w: %q and %q; remove or merge one before starting the managed dolt server "+
"(matches Dolt's own ambiguous-.doltcfg case; see --doltcfg-dir in `dolt sql-server --help`)",
ErrMultipleDoltCfgDirs, absParent, absCurr)
}
chosen := currDirCfg
if parentExists {
chosen = parentDirCfg
}
abs, err := filepath.Abs(chosen)
if err != nil {
return "", fmt.Errorf("resolving .doltcfg path %q: %w", chosen, err)
}
return abs, nil
}
// buildDoltServerYAMLConfig renders a minimal sql-server YAML config
// equivalent to buildDoltServerArgs' CLI flags (host, port, log level), plus
// auto_gc_behavior.archive_level: 0 so this managed server's background
// auto-GC writes classic Snappy table files instead of zstd archives
// (gastownhall/beads#4986). Auto-GC itself is left enabled (archive_level's
// sibling "enable" key is omitted, which Dolt defaults to true). cfgDir must
// be pre-resolved via resolveCfgDir — --config mode skips Dolt's own
// flag-mode .doltcfg discovery entirely, so this is the one place that
// behavior must be replicated.
//
// Callers MUST only use this when SupportsArchiveLevelConfig(doltBin)
// reports true — an older external dolt's own YAMLConfig struct may lack
// this field, and Dolt's YAML loader uses yaml.UnmarshalStrict, so an
// unrecognized key is a hard parse error at server startup, not a
// silently-ignored one.
func buildDoltServerYAMLConfig(host string, port int, debug bool, cfgDir string) ([]byte, error) {
logLevel := doltServerLogLevel
if debug {
logLevel = "debug"
}
archiveLevel := 0
yc := &servercfg.YAMLConfig{
LogLevelStr: &logLevel,
ListenerConfig: servercfg.ListenerYAMLConfig{
HostStr: &host,
PortNumber: &port,
},
CfgDirStr: &cfgDir,
BehaviorConfig: servercfg.BehaviorYAMLConfig{
AutoGCBehavior: &servercfg.AutoGCBehaviorYAMLConfig{
ArchiveLevel_: &archiveLevel,
},
},
}
return yaml.Marshal(yc)
}
// buildDoltServerArgsWithConfig is the --config counterpart to
// buildDoltServerArgs. Dolt's sql-server subcommand ignores all other
// command-line parameters when --config is present, so host/port/log-level
// must come from the YAML file at configPath (see buildDoltServerYAMLConfig)
// rather than from flags here. The top-level --prof/--prof-path flags are
// unaffected — they are consumed by the outer `dolt` command dispatcher
// before the sql-server subcommand's own arg parser ever sees --config.
func buildDoltServerArgsWithConfig(configPath string, debug bool, profDir string) []string {
var args []string
if debug {
args = append(args, "--prof", "cpu", "--prof-path", profDir)
}
args = append(args, "sql-server", "--config", configPath)
return args
}
// Start explicitly starts a dolt sql-server for the project.
// Returns the State of the started server, or an error.
func Start(beadsDir string) (*State, error) {
cfg := DefaultConfig(beadsDir)
doltDir := ResolveDoltDir(beadsDir)
// Acquire exclusive lock to prevent concurrent starts
lockF, err := os.OpenFile(lockPath(beadsDir), os.O_CREATE|os.O_RDWR, 0600)
if err != nil {
return nil, fmt.Errorf("creating lock file: %w", err)
}
defer lockF.Close()
if err := lockfile.FlockExclusiveNonBlocking(lockF); err != nil {
if lockfile.IsLocked(err) {
// Another bd process is starting the server — wait for it
if err := lockfile.FlockExclusiveBlocking(lockF); err != nil {
return nil, fmt.Errorf("waiting for server start lock: %w", err)
}
defer func() { _ = lockfile.FlockUnlock(lockF) }()
// Lock acquired — check if server is now running
state, err := IsRunning(beadsDir)
if err != nil {
return nil, err
}
if state.Running {
return state, nil
}
// Still not running — fall through to start it ourselves
} else {
return nil, fmt.Errorf("acquiring start lock: %w", err)
}
} else {
defer func() { _ = lockfile.FlockUnlock(lockF) }()
}
// Re-check after acquiring lock (double-check pattern)
if state, _ := IsRunning(beadsDir); state != nil && state.Running {
return state, nil
}
// Clean up orphaned dolt sql-server processes INSIDE the lock.
// This MUST happen under the lock to prevent a race where one process
// kills a server that another process is in the middle of starting
// (PID file not yet written). Without this, concurrent bd processes
// can cause journal corruption (GH#2430).
if killed, killErr := KillStaleServers(beadsDir); killErr == nil && len(killed) > 0 {
fmt.Fprintf(os.Stderr, "Info: cleaned up %d orphaned dolt sql-server process(es)\n", len(killed))
}
// Ensure dolt binary exists
doltBin, err := exec.LookPath("dolt")
if err != nil {
return nil, fmt.Errorf("dolt is not installed (not found in PATH)\n\nInstall from: https://docs.dolthub.com/introduction/installation")
}
// Prefer a generated YAML config (via --config) over plain CLI flags when
// the resolved external dolt is new enough to safely accept
// auto_gc_behavior.archive_level: 0. This is what keeps this managed
// server's background auto-GC producing Snappy table files instead of
// zstd archives (gastownhall/beads#4986). On an older/undetectable dolt
// we fail closed and keep the existing CLI-flag launch unchanged — never
// risk a refuse-to-start over an unrecognized YAML key.
useArchiveLevelConfig := SupportsArchiveLevelConfig(doltBin)
if !useArchiveLevelConfig {
fmt.Fprintf(os.Stderr,
"Info: external dolt at %s predates archive_level config support (need >= Dolt %s); "+
"this managed server's background auto-GC may still produce zstd archives.\n",
doltBin, MinDoltVersionForArchiveLevelConfig)
}
// Ensure dolt identity is configured
if err := ensureDoltIdentity(); err != nil {
return nil, fmt.Errorf("configuring dolt identity: %w", err)
}
// Debug mode: create the pprof output dir before exec, since dolt's
// --prof-path panics on a missing directory (see dolt/dolt.go runMain).
debug := IsDebugMode()
var profDir string
if debug {
profDir = DebugProfileDir(beadsDir)
if err := os.MkdirAll(profDir, config.BeadsDirPerm); err != nil {
return nil, fmt.Errorf("creating pprof directory %s: %w", profDir, err)
}
}
// Launch dolt sql-server.
var (
pid int
actualPort int
lastErr error
attempts int
)
{
// Ensure dolt database directory is initialized
if err := ensureDoltInit(doltDir); err != nil {
return nil, fmt.Errorf("initializing dolt database: %w", err)
}
// Rotate the log if it has grown past the configured ceiling. This is a
// startup-only check — dolt owns the fd directly once launched, so we can
// only intervene between runs. See logrotate.go for the caveat discussion.
maybeRotateLog(beadsDir)
// Resolve .doltcfg once, before the port retry loop — it does not
// depend on the port, and an ambiguous both-exist result must fail
// Start() outright rather than retry into a fresh unintended
// $data_dir/.doltcfg (see resolveCfgDir).
var cfgDir string
if useArchiveLevelConfig {
cfgDir, err = resolveCfgDir(doltDir)
if err != nil {
return nil, fmt.Errorf("resolving .doltcfg directory: %w", err)
}
}
// Open log file
logFile, err := os.OpenFile(logPath(beadsDir), os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600) //nolint:gosec // G304: logPath derives from user-configured beadsDir
if err != nil {
return nil, fmt.Errorf("opening log file: %w", err)
}
// Resolve the port to use. Explicit ports (env/config) go through
// reclaimPort for conflict detection. Port 0 means ephemeral — allocate
// a fresh port from the OS with retry for TOCTOU races.
actualPort = cfg.Port
explicitPort := actualPort > 0
if explicitPort {
// Explicit port: check for conflicts and adopt existing servers.
adoptPID, reclaimErr := reclaimPort(cfg.Host, actualPort, beadsDir)
if reclaimErr != nil {
_ = logFile.Close()
return nil, fmt.Errorf("cannot start dolt server on port %d: %w", actualPort, reclaimErr)
}
if adoptPID > 0 {
_ = logFile.Close()
_ = os.WriteFile(pidPath(beadsDir), []byte(strconv.Itoa(adoptPID)), 0600)
_ = writePortFile(beadsDir, actualPort)
return &State{Running: true, PID: adoptPID, Port: actualPort, DataDir: doltDir}, nil
}
}
// Start dolt sql-server, with retry loop for ephemeral port TOCTOU.
pid = 0
lastErr = nil
attempts = 1
if !explicitPort {
attempts = maxEphemeralPortAttempts
}
for i := range attempts {
if !explicitPort {
p, allocErr := allocateEphemeralPort(cfg.Host)
if allocErr != nil {
lastErr = allocErr
continue
}
actualPort = p
}
var cmdArgs []string
if useArchiveLevelConfig {
cfgBody, cfgErr := buildDoltServerYAMLConfig(cfg.Host, actualPort, debug, cfgDir)
if cfgErr != nil {
lastErr = fmt.Errorf("rendering managed sql-server config: %w", cfgErr)
if !explicitPort {
continue
}
break
}
// filepath.Abs defensively: the child's cwd is doltDir (cmd.Dir
// below), so a relative configPath would resolve against the
// wrong directory (matches the sibling dbproxy/server path,
// which abs's its configPath in NewDoltServer).
absConfigPath, absErr := filepath.Abs(doltServerConfigPath(beadsDir))
if absErr != nil {
lastErr = fmt.Errorf("resolving managed sql-server config path: %w", absErr)
if !explicitPort {
continue
}
break
}
if werr := os.WriteFile(absConfigPath, cfgBody, 0600); werr != nil {
lastErr = fmt.Errorf("writing managed sql-server config: %w", werr)
if !explicitPort {
continue
}
break
}
cmdArgs = buildDoltServerArgsWithConfig(absConfigPath, debug, profDir)
} else {
cmdArgs = buildDoltServerArgs(cfg.Host, actualPort, debug, profDir)
}
cmd := exec.Command(doltBin, cmdArgs...) //nolint:gosec // doltBin is resolved from PATH, not user input
cmd.Dir = doltDir
cmd.Stdout = logFile
cmd.Stderr = logFile
cmd.Stdin = nil
cmd.SysProcAttr = procAttrDetached()
cmd.Env = os.Environ()
if startErr := cmd.Start(); startErr != nil {
lastErr = startErr
if !explicitPort {
continue // retry with a new ephemeral port
}
break
}
pid = cmd.Process.Pid
_ = cmd.Process.Release()
// Quick check: did the process exit immediately (bind failure)?
// Give it a moment to fail on port bind before proceeding.
time.Sleep(200 * time.Millisecond)
if !isProcessAlive(pid) {
lastErr = fmt.Errorf("dolt sql-server exited immediately on port %d (attempt %d/%d)", actualPort, i+1, attempts)
pid = 0
if !explicitPort {
continue
}
break
}
lastErr = nil
break
}
_ = logFile.Close()
if lastErr != nil {
// GH#3290 / bd-6dnrw.6: unclean-shutdown manifest corruption is
// detected here but never auto-repaired — reinitializing .dolt is
// destructive, so repair stays behind explicit bd doctor --fix.
if dirs, detErr := detectCorruptManifest(beadsDir, doltDir); detErr == nil && len(dirs) > 0 {
return nil, fmt.Errorf("failed to start dolt server after %d attempts: %w\n"+
"Corrupt manifest with no recoverable data detected (GH#3290) in:\n %s\n"+
"Run 'bd doctor --fix' to back up the corrupt database(s) and reinitialize.\nCheck logs: %s",
attempts, lastErr, strings.Join(dirs, "\n "), logPath(beadsDir))
}
return nil, fmt.Errorf("failed to start dolt server after %d attempts: %w\nCheck logs: %s",
attempts, lastErr, logPath(beadsDir))
}
}
// Write PID and port files
if err := os.WriteFile(pidPath(beadsDir), []byte(strconv.Itoa(pid)), 0600); err != nil {
if proc, findErr := os.FindProcess(pid); findErr == nil {
_ = proc.Kill()
}
return nil, fmt.Errorf("writing PID file: %w", err)
}
if err := writePortFile(beadsDir, actualPort); err != nil {
if proc, findErr := os.FindProcess(pid); findErr == nil {
_ = proc.Kill()
}
_ = os.Remove(pidPath(beadsDir))
return nil, fmt.Errorf("writing port file: %w", err)
}
// Wait for server to accept connections
if err := waitForReady(cfg.Host, actualPort, readyTimeout()); err != nil {
if proc, findErr := os.FindProcess(pid); findErr == nil {
_ = proc.Kill()
}
_ = os.Remove(pidPath(beadsDir))
_ = os.Remove(portPath(beadsDir))
if hasJournalCorruption, logErr := logHasCorruptJournalError(logPath(beadsDir)); logErr == nil && hasJournalCorruption {
return nil, fmt.Errorf("server started (PID %d) but not accepting connections on port %d: %w\n\n%s",
pid, actualPort, err, corruptJournalRecoveryHint(beadsDir))
}
return nil, fmt.Errorf("server started (PID %d) but not accepting connections on port %d: %w\nCheck logs: %s",
pid, actualPort, err, logPath(beadsDir))
}
return &State{
Running: true,
PID: pid,
Port: actualPort,
DataDir: doltDir,
}, nil
}
// EnsureGlobalDatabase connects to the shared Dolt server and creates the
// beads_global database if it doesn't already exist. This is idempotent and
// safe to call on every shared server init. Schema initialization and config
// seeding (issue prefix, project ID) are handled by the store layer when the
// global database is first opened with CreateIfMissing=true.
//
// Returns nil if the database already exists or was successfully created.
func EnsureGlobalDatabase(host string, port int, user, password string) error {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
dsn := doltutil.ServerDSN{
Host: host,
Port: port,
User: user,
Password: password,
}.String()
db, err := sql.Open("mysql", dsn)
if err != nil {
return fmt.Errorf("ensure global db: failed to open connection: %w", err)
}
defer db.Close()
db.SetMaxOpenConns(1)
db.SetConnMaxLifetime(10 * time.Second)
if err := db.PingContext(ctx); err != nil {
return fmt.Errorf("ensure global db: server not reachable: %w", err)
}
// CREATE DATABASE IF NOT EXISTS is idempotent — safe on every call.
// GlobalDatabaseName is a constant ("beads_global"), not user input.
_, err = db.ExecContext(ctx, fmt.Sprintf("CREATE DATABASE IF NOT EXISTS `%s`", GlobalDatabaseName)) //nolint:gosec // G201: constant database name
if err != nil {
errLower := strings.ToLower(err.Error())
if !strings.Contains(errLower, "database exists") && !strings.Contains(errLower, "1007") {
return fmt.Errorf("ensure global db: failed to create %s: %w", GlobalDatabaseName, err)
}
}
return nil
}
// FlushWorkingSet connects to the running Dolt server and commits any uncommitted
// working set changes across all databases. This prevents data loss when the server
// is about to be stopped or restarted. Returns nil if there's nothing to flush or
// if the server is not reachable (best-effort).
func FlushWorkingSet(host string, port int) error {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
dsn := doltutil.ServerDSN{
Host: host,
Port: port,
User: "root",
}.String()
db, err := sql.Open("mysql", dsn)
if err != nil {
return fmt.Errorf("flush: failed to open connection: %w", err)
}
defer db.Close()
db.SetMaxOpenConns(1)
db.SetConnMaxLifetime(10 * time.Second)
if err := db.PingContext(ctx); err != nil {
return fmt.Errorf("flush: server not reachable: %w", err)
}
// List all databases, skipping system databases
rows, err := db.QueryContext(ctx, "SHOW DATABASES")
if err != nil {
return fmt.Errorf("flush: failed to list databases: %w", err)
}
var databases []string
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
continue
}
// Skip Dolt system databases
if name == "information_schema" || name == "mysql" || name == "performance_schema" {
continue
}
databases = append(databases, name)
}
_ = rows.Close()
if len(databases) == 0 {
return nil
}
var flushed int
for _, dbName := range databases {
// Check for uncommitted changes via dolt_status
var hasChanges bool
row := db.QueryRowContext(ctx, fmt.Sprintf("SELECT COUNT(*) > 0 FROM `%s`.dolt_status", dbName))
if err := row.Scan(&hasChanges); err != nil {
// dolt_status may not exist for non-beads databases; skip
continue
}
if !hasChanges {
continue
}
// Commit all uncommitted changes
_, err := db.ExecContext(ctx, fmt.Sprintf("USE `%s`", dbName))
if err != nil {
fmt.Fprintf(os.Stderr, "flush: failed to USE %s: %v\n", dbName, err)
continue
}
_, err = db.ExecContext(ctx, "CALL DOLT_COMMIT('-Am', 'auto-flush: commit working set before server stop')")
if err != nil {
errStr := strings.ToLower(err.Error())
if strings.Contains(errStr, "nothing to commit") || strings.Contains(errStr, "no changes") {
continue
}
fmt.Fprintf(os.Stderr, "flush: failed to commit %s: %v\n", dbName, err)
continue
}
flushed++
}
if flushed > 0 {
fmt.Fprintf(os.Stderr, "Flushed working set for %d database(s) before server stop\n", flushed)
}
return nil
}
// Stop is idempotent: when the server is already stopped it returns
// ErrServerNotRunning after cleaning up any leftover state files.
// Callers should use errors.Is(err, ErrServerNotRunning) to distinguish
// this expected condition from real failures.
func Stop(beadsDir string) error {
return StopWithForce(beadsDir, false)
}
// StopWithForce is like Stop but with an optional force flag.
func StopWithForce(beadsDir string, force bool) error {
state, err := IsRunning(beadsDir)
if err != nil {
return err
}
if !state.Running {
// Server not running — still clean up any leftover state files
// so bd dolt status won't report stale state (GH#2670).
// Join cleanup errors with the sentinel so callers can still use
// errors.Is(err, ErrServerNotRunning) while operators see filesystem issues.
cleanupErr := cleanupStateFiles(beadsDir)
return errors.Join(ErrServerNotRunning, cleanupErr)
}
// Flush uncommitted working set changes before stopping the server.
// This prevents data loss when changes have been written but not yet committed.
cfg := DefaultConfig(beadsDir)
if flushErr := FlushWorkingSet(cfg.Host, state.Port); flushErr != nil {
fmt.Fprintf(os.Stderr, "Warning: could not flush working set before stop: %v\n", flushErr)
}
if err := gracefulStop(state.PID, 5*time.Second); err != nil {
return errors.Join(err, cleanupStateFiles(beadsDir))
}
// In debug mode, rotate cpu.pprof → cpu-<timestamp>.pprof so the next
// server start does not overwrite this run's profile. Only meaningful
// after a graceful (SIGTERM) exit — SIGKILL skips pkg/profile's
// deferred flush, leaving nothing to rotate. Best-effort.
if IsDebugMode() {
rotateDebugProfile(beadsDir)
}
return cleanupStateFiles(beadsDir)
}
// cleanupStateFiles removes all server state files (PID and port).
// Returns a joined error for non-NotExist removal failures so callers
// can surface filesystem problems while still treating "already clean"
// as success. Logs non-NotExist errors at debug level (GH#2670).
func cleanupStateFiles(beadsDir string) error {
var errs []error
for _, path := range []string{pidPath(beadsDir), portPath(beadsDir)} {
if err := os.Remove(path); err != nil && !os.IsNotExist(err) {
debug.Logf("failed to remove server state file %s: %v", path, err)
errs = append(errs, err)
}
}
return errors.Join(errs...)
}
func StateFilePaths(beadsDir string) []string {
return []string{
pidPath(beadsDir),
portPath(beadsDir),
lockPath(beadsDir),
logPath(beadsDir),
logPath(beadsDir) + ".1",
DebugProfileDir(beadsDir),
doltServerConfigPath(beadsDir),
}
}
func RemoveStateFiles(beadsDir string) []error {
var errs []error
for _, path := range StateFilePaths(beadsDir) {
if err := os.RemoveAll(path); err != nil {
errs = append(errs, err)
}
}
return errs
}
// LogPath returns the path to the server log file.
func LogPath(beadsDir string) string {
return logPath(beadsDir)
}
func LockPath(beadsDir string) string {
return lockPath(beadsDir)
}
// killStaleServersForDir finds and kills orphan dolt sql-server processes for
// the current repo's Dolt data directory that are not tracked by the canonical
// PID file. Only processes that beads started (tracked via the PID file) are
// eligible for cleanup. Externally-managed servers are never killed.
//
// A process is considered "external" (never kill) when any of:
// - ResolveServerMode() returns ServerModeExternal (explicit port, shared server, etc.)
// - No PID file exists (beads has no record of starting a server)
func killStaleServersForDir(beadsDir string, allPIDs []int, inDir func(int, string) bool, kill func(int) error) ([]int, error) {
if len(allPIDs) == 0 {
return nil, nil
}
// If auto-start is disabled the server is externally managed (e.g., by
// systemd or a manual bd dolt start), so we must not kill any processes.
// IsAutoStartDisabled covers the BEADS_DOLT_AUTO_START env var and
// dolt.auto-start config; ResolveServerMode covers explicit port/shared
// server/embedded configurations. Both indicate "not our server" (GH#2641).
if IsAutoStartDisabled() || ResolveServerMode(beadsDir) == ServerModeExternal {
return nil, nil
}
serverDir := resolveServerDir(beadsDir)
// Read the canonical PID from the PID file. If there is no PID file,
// beads has no record of having started a server for this directory,
// so there is nothing stale to clean up. This prevents killing
// externally-started servers (systemd, other repos sharing a data dir).
var canonicalPID int
if data, readErr := os.ReadFile(pidPath(serverDir)); readErr == nil {
if pid, parseErr := strconv.Atoi(strings.TrimSpace(string(data))); parseErr == nil && pid > 0 {
canonicalPID = pid
}
}
if canonicalPID == 0 {
// No valid PID file → no beads-owned server to compare against.
// Nothing is stale from our perspective.
return nil, nil
}
// The canonical PID itself is alive and tracked — never kill it.
// Only kill OTHER dolt processes in our data dir (orphans from a
// previous beads-started server that lost its PID file tracking).
ownedDoltDir := ResolveDoltDir(serverDir)
var killed []int
for _, pid := range allPIDs {
if pid == os.Getpid() {
continue
}
if pid == canonicalPID {
continue // preserve canonical server
}
if !inDir(pid, ownedDoltDir) {
continue // preserve other repos' Dolt servers
}
if err := kill(pid); err == nil {
killed = append(killed, pid)
}
}
return killed, nil
}
// KillStaleServers finds and kills orphan dolt sql-server processes for the
// current repo's Dolt data directory that are not tracked by the canonical PID
// file. Returns the PIDs of killed processes.
//
// When auto-start is disabled (BEADS_DOLT_AUTO_START=0 or dolt.auto-start:
// false), this function is a no-op — the dolt server is externally managed
// and must not be killed by bd (GH#2641).
func KillStaleServers(beadsDir string) ([]int, error) {
if IsAutoStartDisabled() {
return nil, nil
}
allPIDs := listDoltProcessPIDs()
return killStaleServersForDir(
beadsDir,
allPIDs,
isProcessInDir,
func(pid int) error {
proc, err := os.FindProcess(pid)
if err != nil {
return err
}
return proc.Kill()
},
)
}
// waitForReady polls TCP until the server accepts connections.
func waitForReady(host string, port int, timeout time.Duration) error {
addr := net.JoinHostPort(host, strconv.Itoa(port))
deadline := time.Now().Add(timeout)
for time.Now().Before(deadline) {
conn, err := net.DialTimeout("tcp", addr, 500*time.Millisecond) //nolint:gosec // G704: addr is built from internal host+port, not user input
if err == nil {
_ = conn.Close()
return nil
}
time.Sleep(500 * time.Millisecond)
}
return fmt.Errorf("timeout after %s waiting for server at %s", timeout, addr)
}
// ensureDoltIdentity sets dolt global user identity from git config if not already set.
func ensureDoltIdentity() error {
// Check if dolt identity is already configured
nameCmd := exec.Command("dolt", "config", "--global", "--get", "user.name")
if out, err := nameCmd.Output(); err == nil && strings.TrimSpace(string(out)) != "" {
return nil // Already configured
}
// Try to get identity from git
gitName := "beads"
gitEmail := "beads@localhost"
if out, err := exec.Command("git", "config", "user.name").Output(); err == nil {
if name := strings.TrimSpace(string(out)); name != "" {
gitName = name
}
}
if out, err := exec.Command("git", "config", "user.email").Output(); err == nil {
if email := strings.TrimSpace(string(out)); email != "" {
gitEmail = email
}
}
if out, err := exec.Command("dolt", "config", "--global", "--add", "user.name", gitName).CombinedOutput(); err != nil {
return fmt.Errorf("setting dolt user.name: %w\n%s", err, out)
}
if out, err := exec.Command("dolt", "config", "--global", "--add", "user.email", gitEmail).CombinedOutput(); err != nil {
return fmt.Errorf("setting dolt user.email: %w\n%s", err, out)
}
return nil
}
// bdDoltMarker is written after a current bd process creates or acknowledges a
// local Dolt repository. Its absence in an existing .dolt/ directory indicates
// the database was created by a pre-0.56 bd version (which used embedded mode).
// Those databases are incompatible with the current server-only architecture.
const bdDoltMarker = ".bd-dolt-ok"
// MarkDoltDirCompatible writes the canonical bd compatibility marker when
// doltDir contains a local Dolt repository. It no-ops when there is no .dolt/
// directory, which lets server and repair paths call it defensively.
func MarkDoltDirCompatible(doltDir string) error {
if doltDir == "" {
return errors.New("dolt directory is required")
}
dotDolt := filepath.Join(doltDir, ".dolt")
if info, err := os.Stat(dotDolt); err != nil {
if os.IsNotExist(err) {
return nil
}
return fmt.Errorf("checking dolt metadata directory %s: %w", dotDolt, err)
} else if !info.IsDir() {
return fmt.Errorf("dolt metadata path %s is not a directory", dotDolt)
}
markerPath := filepath.Join(doltDir, bdDoltMarker)
if _, err := os.Stat(markerPath); err == nil {
return nil
} else if !os.IsNotExist(err) {
return fmt.Errorf("checking dolt compatibility marker %s: %w", markerPath, err)
}
if err := os.WriteFile(markerPath, []byte("ok\n"), 0600); err != nil {
return fmt.Errorf("writing dolt compatibility marker %s: %w", markerPath, err)
}
return nil
}
// ensureDoltInit initializes a dolt database directory if .dolt/ doesn't exist.
// If .dolt/ exists, seeds the .bd-dolt-ok marker for existing working databases.
// See GH#2137 for background on pre-0.56 database compatibility.
func ensureDoltInit(doltDir string) error {
if err := os.MkdirAll(doltDir, config.BeadsDirPerm); err != nil {
return fmt.Errorf("creating dolt directory: %w", err)
}
dotDolt := filepath.Join(doltDir, ".dolt")
if _, err := os.Stat(dotDolt); err == nil {
// .dolt/ exists — seed the marker if missing.
// This is the non-destructive path: we just mark existing databases
// as known. The destructive recovery path (RecoverPreV56DoltDir) is
// triggered separately during version upgrades.
_ = MarkDoltDirCompatible(doltDir)
return nil // Already initialized
}
cmd := exec.Command("dolt", "init")
cmd.Dir = doltDir
if out, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("dolt init: %w\n%s", err, out)
}
// Write version marker so future runs know this database is compatible.
_ = MarkDoltDirCompatible(doltDir)
return nil
}
// RecoverPreV56DoltDir removes and reinitializes a dolt database that was
// created by a pre-0.56 bd version. Call this during version upgrade detection
// (e.g., from autoMigrateOnVersionBump when previousVersion < 0.56).
//
// Pre-0.56 databases used embedded Dolt mode with a different Dolt library
// version that may produce nil DoltDB values, causing panics (GH#2137).
// The data is unrecoverable — the fix is to start fresh.
//
// Returns true if recovery was performed, false if not needed.
func RecoverPreV56DoltDir(doltDir string) (bool, error) {
dotDolt := filepath.Join(doltDir, ".dolt")
if _, err := os.Stat(dotDolt); os.IsNotExist(err) {
return false, nil // No .dolt/ directory — nothing to recover
}
markerPath := filepath.Join(doltDir, bdDoltMarker)
if _, err := os.Stat(markerPath); err == nil {
return false, nil // Marker exists — database is from 0.56+
}
fmt.Fprintf(os.Stderr, "Detected dolt database from an older bd version (pre-0.56).\n")
fmt.Fprintf(os.Stderr, "Rebuilding dolt database at %s ...\n", doltDir)
if err := os.RemoveAll(dotDolt); err != nil {
return false, fmt.Errorf("cannot remove old dolt database at %s: %w\n\n"+
"Manually delete %s and retry", dotDolt, err, dotDolt)
}
// Reinitialize
if err := ensureDoltInit(doltDir); err != nil {
return true, fmt.Errorf("recovery: %w", err)
}
return true, nil
}
// IsPreV56DoltDir returns true if doltDir contains a .dolt/ directory that
// was NOT created by bd 0.56+ (missing .bd-dolt-ok marker). These databases
// were created by the old embedded Dolt mode and may be incompatible.
// Used by doctor checks to detect potentially problematic dolt databases.
func IsPreV56DoltDir(doltDir string) bool {
dotDolt := filepath.Join(doltDir, ".dolt")
if _, err := os.Stat(dotDolt); os.IsNotExist(err) {
return false // No .dolt/ at all
}
markerPath := filepath.Join(doltDir, bdDoltMarker)
_, err := os.Stat(markerPath)
return os.IsNotExist(err)
}