beads/cmd/bd/backup_auto.go
Athos Bernardes c26cf1fb6f
fix(bd): cap auto-backup destination size to stop unbounded growth (#6071)
* fix(bd): cap auto-backup destination size to stop unbounded growth (ga-y6gjv)

ga-y6gjv asked for 4 fixes to bd's auto-backup subsystem. Three are already
merged and deployed (verified via git merge-base --is-ancestor against the
commit that built the currently-deployed ~/.local/bin/bd):
  - interval throttle persists on failure too, per-workspace: 5f685959c
  - default-off in sql-server/shared mode: 5f685959c, refined by 8b9429153
  - `bd config get backup.enabled` reports the effective value: 5f685959c

This commit is only the fourth, still-open item: "prune/cap backup
archives." BackupSync/CALL DOLT_BACKUP('sync', ...) only ever transfers new
chunks into the destination — it never prunes ones that became unreachable
on the source DB (history rewrites, superseded data) — so even with the
interval fix slowing the frequency of syncs, the destination can still only
grow, never shrink. This was the actual disk-filling mechanism in the
2026-06-19 outage (43GB backup dir from a 1.7GB store).

I first tried wiring `dolt gc` into the post-sync path (mirroring
runCompactDolt's exec.Command("dolt", "gc", ...) pattern in compact.go).
That does not work: a backup destination is a bare Dolt chunk-store
directory with no .dolt repo-root marker, so the standalone dolt CLI can't
open it ("not a valid dolt repository") no matter what its working
directory is set to, and dolt_backup.go's SQL procedures only expose
add/sync/restore — never gc — for a backup destination. Confirmed this
empirically (a real dolt gc attempt against a real backup dir, outside any
test) before shipping anything, not just reasoned about it.

Given no supported way to prune a backup destination in place, this adds a
hard size cap instead: once the destination exceeds backup.size-cap-mb
(default 2048MB), maybeAutoBackup skips the sync attempt entirely rather
than letting it grow further, and prints a throttled warning
(backup.size-warn-interval, default 24h) telling the operator to free space
or raise the cap. This directly bounds the failure mode that caused the
outage — unbounded growth — without risking any operation against the
backup store's on-disk format.

Tests (cmd/bd/backup_cap_test.go): threshold check against a real
directory, warning-throttle persistence (mirrors the existing throttle
persisted on backup failure, backup_export.go), and end-to-end proof that
maybeAutoBackup never attempts BackupDatabase once capped. Confirmed each
new test fails against the pre-fix code (wiring temporarily reverted,
re-verified, restored) before treating it as green.

Verified: go build ./... clean, gofmt clean, go vet clean, full
cmd/bd + internal/config test run green (backup/config-focused subset run
with -v; no regressions in the wider package test runs either).

* fix(bd): address PR #6071 review feedback on backup size-cap (ga-y6gjv)

bee-ghosttrack's 2026-09-03 review of the size-cap fix (3d27d7a98) requested
changes before merge -- 4 major, 2 minor, all about the pause being
invisible rather than the cap mechanism itself. This addresses all six.

Major:

1. backup.size-cap-mb: 0 now means "no cap", not "use the 2048MB default."
   effectiveSizeCapMB() centralizes the <=0-is-disabled interpretation;
   backupSizeCapExceeded and the new status helpers all go through it.
   Viper already registers a 2048 default for this key
   (internal/config/config.go), so GetInt only returns <=0 when the
   operator explicitly set 0 (or an unparseable value) -- the old code's
   own `if capMB <= 0 { capMB = 2048 }` was silently overriding that
   explicit choice, leaving no off switch for a legitimately larger
   destination.

2. `bd backup status` (and --json) now report a `size_cap` object
   (enabled, cap_mb, current_bytes, exceeded), and the human-text Config:
   line grows a "Size cap: PAUSED (cap exceeded) -- X / Y..." line when
   paused. Previously the pause was stderr-only and throttled to 24h, so
   an agent/CI caller on --json or --quiet had no way to see that
   auto-backup had silently and permanently stopped syncing.

3. The remediation message no longer says "delete the backup directory."
   Nothing in this codebase demonstrates a deleted destination is cleanly
   recreated by the next sync (the server-side Dolt backup remote stays
   registered against that path), and following the old advice would
   leave a zero-backup window while the next sync is itself
   interval-throttled. It now points at backup.size-cap-mb / `bd backup
   init <new-path>`.

4. docs/reference/configuration.md gains rows for both new keys plus an
   expanded Auto-backup section (the cap mechanism, the pause, manual
   backup staying uncapped, bd backup status reporting the pause);
   CHANGELOG.md gets an [Unreleased]/Fixed entry referencing #6071.

Minor:

5. maybeWarnBackupSizeCap's throttle timestamp is now also cached in an
   in-process map (capWarnFallback, keyed by dir) alongside the persisted
   backup_state.json. The persist write targets the directory the cap
   just declared full, so in the disk-full case this cap exists for, that
   write fails and the timestamp never sticks -- a long-lived bd process
   that keeps calling maybeAutoBackup for the same destination (e.g.
   internal/storage/dbproxy's server mode) would otherwise re-warn on
   every single call. The map is keyed by dir specifically so unrelated
   destinations -- and unrelated tests -- never share a throttle.

6. The size-cap directory walk (getDirSize's filepath.Walk) now runs
   after the interval throttle in maybeAutoBackup, not before. The
   reviewer measured 65-100ms per bd invocation at 20k files in the
   backup dir when the walk ran unconditionally on every command; moving
   it below the throttle makes it free on the common (throttled) path and
   only pays the walk cost once per interval, exactly when a sync attempt
   would occur anyway.

Also noted by the reviewer: manual `bd backup` / `bd backup sync` stay
uncapped -- now documented in configuration.md.

Tests (cmd/bd/backup_cap_test.go, cmd/bd/backup_status_test.go): one new
test per review point. Each was confirmed to fail against the pre-fix code
for the expected reason before implementing -- including catching a
vacuous pass in my own first draft of the "disabled cap" JSON assertion
(zero-value Enabled:false matched whether the size_cap key existed at all
or not; fixed to assert key presence first). The disabled-cap test uses a
sparse file (Truncate, not a real write) to cross the legacy 2048MB
threshold without allocating 2GB of real disk or memory. The reordering
test is detected indirectly: if the cap check ran despite the throttle, it
would persist LastCapWarnAt; the interval-fresh case proves it does not.

Verified: go build ./cmd/bd/... clean, gofmt clean, go vet ./cmd/bd/...
clean, golangci-lint run ./cmd/bd/... -> 0 issues, targeted backup/status
test subset green (20/20, including every pre-existing backup test -- no
regressions).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): add backup_cap sources to //cmd/bd srcs lists

The contributor branched before `cmd/bd/BUILD.bazel` existed, so the two
files this PR adds were never enumerated there. After rebasing onto a base
that hand-maintains explicit srcs lists, `//cmd/bd:bd_lib` omits
`backup_cap.go` while still compiling `backup.go` and `backup_auto.go`,
which reference four symbols it defines:

  backup.go:93       undefined: showSizeCapStatusJSON
  backup.go:141      undefined: showSizeCapStatus
  backup_auto.go:200 undefined: backupSizeCapExceeded
  backup_auto.go:203 undefined: maybeWarnBackupSizeCap

`backup_cap_test.go` was likewise absent from `//cmd/bd:bd_test`, leaving
seven of the PR's nine new tests unreachable under Bazel.

`make bazel-sync-check` does not catch this: `tools/bazel/go_srcs.py --check`
only validates the `go_srcs` filegroup globs, not the `go_library`/`go_test`
srcs enumerations.

Four unrelated test files (config_leaked_secret_test.go, init_role_*_test.go,
init_git_bootstrap_routing_test.go) are also unlisted, but they are unlisted
on the base too, so they are left alone as pre-existing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(backup): stop the size-cap walk from running on every bd command (#6071 review)

Maintainer review fixup for PR #6071 (ga-y6gjv), iterations 1 and 2.

The gating finding, filed independently by both review lanes and confirmed
empirically: the fixup that moved the getDirSize walk behind the interval
throttle does not deliver its claimed property, because state.Timestamp —
the only thing that re-arms that throttle — is written in exactly two
places, both inside runBackupExport. The two steady states that skip
before reaching it therefore pass the throttle on every single bd
invocation and pay the full filepath.Walk forever:

  IDLE    no new Dolt commit; the walk runs, then change detection skips
          and writes nothing.
  PAUSED  cap exceeded; the walk has already happened by the time
          maybeWarnBackupSizeCap persists LastCapWarnAt, which is the only
          field it wrote.

PAUSED is the pathological case: an over-cap destination is by definition
the largest one, so the feature's terminal state was its most expensive,
persisting until an operator intervened. This landed on read-only
commands too, i.e. the 65-100ms/command agent-polling regression the walk
was moved to avoid.

Both halves of the documented remedy:

  - Move the cap check after change detection, the order
    docs/reference/configuration.md already describes ("If data changed and
    the throttle interval has passed, bd checks the destination
    directory's on-disk size"). This makes that sentence true and closes
    the docs finding with no docs edit.
  - Re-arm the interval throttle (state.Timestamp) on the cap-exceeded
    skip, leaving LastDoltCommit untouched exactly as runBackupExport's own
    failure path does, so change detection still sees pending work once the
    cap is raised.

Neither half suffices alone: the reorder still walks every command in a
PAUSED workspace where data IS changing, and the persist alone still walks
forever when IDLE. Persisting on the idle skip instead would add up to one
backup.interval of backup latency, which this pairing avoids.

Also from the same review, all non-gating:

  - The cap-check error path failed open behind a debug-only line, so one
    unreadable entry could silently restore unbounded growth while
    `bd backup status --json` reported the very same error. It now warns on
    stderr, throttled by the backup interval and deliberately not by the
    LastCapWarnAt slot: spending that slot here would suppress the more
    important PAUSED notice for a full warn interval.
  - The warn timestamp is now recorded only when the message was actually
    printed, so --json/--quiet agent calls stop consuming the 24h slot
    silently.
  - Dropped capWarnFallback (map, mutex, two accessors). No reachable
    production path can read it: both call sites are in PersistentPostRunE,
    which runs once per CLI process, and the proxy child installs an empty
    PersistentPostRun. Its pinning test is repurposed to the contract that
    is real — a persist failure must not block the already-decided skip.
  - Registered backup.size-cap-mb / backup.size-warn-interval in
    YamlOnlyKeys, so the backup block is a complete inventory again.
    Functionally inert: the map has no enumerator and its one predicate
    consumer already matches on the "backup." prefix.
  - `bd backup status` prints "Size cap: unavailable (<err>)" instead of
    dropping the line, and the JSON error object carries an explicit null
    `exceeded` so a consumer cannot read a missing key as "not exceeded".

The CHANGELOG's cost claim is corrected to match the shipped order; it
described the throttle as sufficient, which is the defect above.

Iteration 2, all non-gating (that review approved):

  - The walk-error path re-arms the interval throttle itself before
    proceeding uncapped. Iteration 1 relied on the fall-through into
    runBackupExport to bound that warning to once per backup.interval,
    but not every runBackupExport exit persists state.Timestamp: a failed
    post-sync GetCurrentCommit returns without saving state, so the walk,
    the warning and the sync itself repeated on every command. The doc
    comment now states the bound the code has: once per interval whenever
    backup_state.json can be written.
  - `bd backup status` echoes the configured backup.size-cap-mb when the
    cap is disabled instead of a literal 0. viper's GetInt also resolves
    an unparseable value ("2048MB") to 0, so the line asserted a value the
    operator never wrote.
  - The status readers count only a destination that does not exist yet
    as empty. getDirSize aborts on the first per-entry error, so a chunk
    file removed mid-walk read as 0 bytes, i.e. `"exceeded": false` for a
    destination that may be over its cap. It is now reported as
    unavailable, like any other walk error.
  - docs/reference/configuration.md no longer says `--json` reports the
    string `PAUSED (cap exceeded)`; it reports `size_cap.exceeded: true`.
  - TestMaybeAutoBackup_CapCheckSkippedWhenThrottled seeded the same commit
    as its watermark, so after iteration 1's reorder change detection
    alone stopped the walk: it passed with the interval throttle deleted,
    and so did every other Backup/Throttle/Auto test in cmd/bd. It now
    seeds a changed commit, and its comment names the renamed
    pauseAutoBackupForSizeCap.

Tests: iteration 1's four new/updated pins are each mutation-proved.
Dropping the throttle re-arm reds the three throttle pins and leaves the
idle pin green; restoring the original ordering reds only the idle pin;
reverting both reds all four. Iteration 2's three new pins each go red
under their own mutation only: dropping the walk-error re-arm, restoring
the literal `=0`, and swallowing a walk ENOENT (dropping the root stat
reds the missing-destination case instead). Disabling the interval
throttle now reds the throttle test.

Base refresh before approval (rebased onto main 3f9561db2b, which had
made the PR CONFLICTING):

  - main 8269a88cbf (#6327) moved compact onto storeSizeBytes and
    deleted getDirSize from compact.go, leaving backup_cap.go its only
    caller, so the merged tree failed to build (undefined: getDirSize)
    although only CHANGELOG.md and backup_auto.go conflicted textually.
    getDirSize is restored verbatim in backup_cap.go (plus the
    path/filepath import), and the two comments that described it as
    shared with, or exercised by, compact.go are corrected.
  - main 4e4ebdb443 (#6879) added acquireBackupLock where this commit
    places the size-cap check. Both are kept, the cap check first, as
    its comment asks: a skip should not first take the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
2026-10-01 01:00:25 -07:00

268 lines
10 KiB
Go

package main
import (
"context"
"errors"
"fmt"
"os"
"strings"
"sync"
"time"
"github.com/steveyegge/beads/internal/beads"
"github.com/steveyegge/beads/internal/config"
"github.com/steveyegge/beads/internal/configfile"
"github.com/steveyegge/beads/internal/debug"
"github.com/steveyegge/beads/internal/storage"
)
// isBackupAutoEnabled returns whether backup should run.
// If user explicitly configured backup.enabled, use that.
// Otherwise auto-enable when a git remote exists — BUT only in
// embedded mode.
//
// In sql-server / shared-server mode (usesSQLServer()) the default is
// OFF: N bd clients share a single Dolt server, and the Dolt-native
// backup path (store.BackupDatabase) registers a server-side backup
// remote under one fixed name pointing at THIS client's local
// .beads/backup dir, then full-syncs the whole DB. With many clients
// that means racing remove/add of the same name plus every client
// full-syncing the entire history into its own dir — the amplifier
// behind the 2026-07 shared-dolt CPU-pin incident. Operators who want
// backups in server mode must opt in explicitly (backup.enabled=true
// / BD_BACKUP_ENABLED=1) and coordinate destinations themselves.
func isBackupAutoEnabled() bool {
if config.GetValueSource("backup.enabled") != config.SourceDefault {
return config.GetBool("backup.enabled")
}
if usesSQLServer() {
return false
}
return primeHasGitRemote()
}
// backupAutoStatusNote returns the parenthetical `bd backup status`
// prints after the effective backup.enabled value, or "" when the value
// speaks for itself. Callers pass the value isBackupAutoEnabled()
// already computed so the note cannot disagree with the number beside
// it — and so status does not re-run the git-remote probe.
//
// The note narrates the reason the decision actually used. An explicit
// backup.enabled needs no note: the source explains it, and every shape
// that reaches `bd backup status` honors it — including a managed-local
// proxied server, whose post-run arm runs auto-backup (every other proxied
// shape is refused by requireLocalProxiedBackup before status renders).
// A server-mode default is OFF whether or not a git remote exists, so it
// must not be attributed to the remote; before these arms existed, status
// did exactly that.
func backupAutoStatusNote(enabled bool) string {
if config.GetValueSource("backup.enabled") != config.SourceDefault {
return ""
}
if usesProxiedServer() {
return "auto: off in proxied-server mode; set backup.enabled=true to opt in"
}
if usesSQLServer() {
return "auto: off in sql-server mode"
}
if enabled {
return "auto: git remote detected"
}
return "auto: no git remote"
}
// clientServerShareFilesystem reports whether the configured Dolt
// server runs on a filesystem the bd client can also see — i.e.
// whether a file:// URL constructed on the client is meaningful to
// the server.
//
// Returns true when the host is empty / localhost (embedded mode or
// local server), false when the host is set to a non-localhost
// value (external server in a container or remote machine).
//
// Used by maybeAutoBackup to skip the file:// auto-register that
// would otherwise fail every command (GH#3523). External-server
// operators who want auto-backup must configure an URL scheme that
// works cross-filesystem (s3://, gs://, etc.) — auto-backup's
// hardcoded file:// path can't help them.
//
// Detection follows the same effective-host precedence as
// configfile.GetDoltServerHost / HostImpliesServerMode (env >
// metadata.json > config.yaml, GH#3545), so a workspace whose remote
// host lives only in metadata.json is classified the same way here as
// by mode inference — the operator's intent is unambiguous from the
// effective host value alone.
func clientServerShareFilesystem() bool {
host := os.Getenv("BEADS_DOLT_SERVER_HOST")
if host == "" {
if bd := beads.FindBeadsDir(); bd != "" {
if cfg, err := configfile.Load(bd); err == nil && cfg != nil {
// An explicit dolt_mode=embedded pins local storage;
// a leftover dolt_server_host is inert then (same
// gate as HostImpliesServerMode), so local
// auto-backup stays available.
if !strings.EqualFold(cfg.DoltMode, configfile.DoltModeEmbedded) {
host = cfg.DoltServerHost
}
}
}
}
if host == "" {
// Fall back to in-struct config (config.yaml dolt.host etc.).
host = config.GetString("dolt.host")
}
return configfile.IsLocalHostString(host)
}
// autoBackupSkipNoticeOnce ensures the "auto-backup skipped" INFO
// message fires at most once per process — operators running long
// bd sessions don't need a chatty repeat on every command.
var autoBackupSkipNoticeOnce sync.Once
// autoBackupBackendForCommand returns the storage this command's auto-backup
// runs against, or ok=false when there is nothing it may back up.
func autoBackupBackendForCommand() (localBackupBackend, bool) {
if usesProxiedServer() {
return proxiedAutoBackupBackend()
}
if store == nil {
return nil, false
}
if lm, ok := storage.UnwrapStore(store).(storage.LifecycleManager); ok && lm.IsClosed() {
return nil, false
}
return directLocalBackup{store: store}, true
}
// maybeAutoBackup runs a Dolt-native backup if enabled and the throttle interval has passed.
// Called from PersistentPostRun after auto-commit.
func maybeAutoBackup(ctx context.Context) {
// Skip backup entirely when running as a git hook (post-checkout, post-merge, etc.).
// Git hooks call 'bd hooks run' which goes through PersistentPostRun — without this
// guard, every git checkout/merge/rebase triggers a backup on the current branch.
if os.Getenv("BD_GIT_HOOK") == "1" {
debug.Logf("backup: skipping — running as git hook\n")
return
}
if !isBackupAutoEnabled() {
return
}
backend, ok := autoBackupBackendForCommand()
if !ok {
return
}
// GH#3523: when the Dolt server runs on a different filesystem
// from this client (operator's BEADS_DOLT_SERVER_HOST points at a
// non-localhost value), the file:// URL the auto-backup path
// constructs is meaningless to the server — register fails on
// every command. Skip cleanly with a one-time INFO so operators
// know auto-backup is silent on purpose.
//
// Proxied workspaces do not consult the host: proxiedAutoBackupBackend
// has already required managed-local, where bd spawned the server on
// this filesystem itself.
if !usesProxiedServer() && !clientServerShareFilesystem() {
autoBackupSkipNoticeOnce.Do(func() {
if !isQuiet() && !jsonOutput {
fmt.Fprintln(os.Stderr,
"Info: auto-backup skipped — server filesystem differs "+
"from client (BEADS_DOLT_SERVER_HOST is non-localhost).\n"+
" Configure backup.url=s3://... or run `bd backup` "+
"manually for cross-filesystem backups.")
}
})
debug.Logf("backup: skipping — server on remote filesystem\n")
return
}
dir, err := backupDir()
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: auto-backup skipped: %v\n", err)
return
}
state, err := loadBackupState(dir)
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: auto-backup skipped: %v\n", err)
return
}
// Throttle: skip if we backed up recently. Checked before the size-cap
// walk below so the common case — the majority of bd invocations,
// still inside the interval — pays zero directory-walk cost (ga-y6gjv
// PR #6071 review: getDirSize's filepath.Walk cost 65-100ms per
// invocation at 20k files when the cap check ran unconditionally,
// before this throttle, on every single command).
interval := config.GetDuration("backup.interval")
if interval == 0 {
interval = 15 * time.Minute
}
if !state.Timestamp.IsZero() && time.Since(state.Timestamp) < interval {
debug.Logf("backup: throttled (last backup %s ago, interval %s)\n",
time.Since(state.Timestamp).Round(time.Second), interval)
return
}
// Change detection: skip if nothing changed
currentCommit, err := backend.CurrentCommit(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "Warning: auto-backup skipped: failed to get current commit: %v\n", err)
return
}
if currentCommit == state.LastDoltCommit && state.LastDoltCommit != "" {
debug.Logf("backup: no changes since last backup\n")
return
}
// Size cap: skip entirely once the destination has grown past
// backup.size-cap-mb (ga-y6gjv) — see backupSizeCapExceeded for why a
// cap, not an in-place prune, is the safe fix here.
//
// Placed AFTER change detection, which is the order
// docs/reference/configuration.md:"How it works" documents. Ahead of
// it, the walk ran on EVERY bd invocation indefinitely in an idle
// workspace: nothing on the idle path advances state.Timestamp, so the
// interval throttle above can never re-arm while nothing is changing
// (ga-y6gjv PR #6071 review). Keep it above any lock acquisition —
// a skip should not first take a lock it is about to release.
if exceeded, size, err := backupSizeCapExceeded(dir); err != nil {
warnBackupSizeCapUnavailable(err)
// Re-arm the interval throttle before proceeding uncapped: not
// every runBackupExport exit persists state.Timestamp (see
// warnBackupSizeCapUnavailable), and without this the walk and
// its warning would repeat on every bd command.
state.Timestamp = time.Now().UTC()
if saveErr := saveBackupState(dir, state); saveErr != nil {
debug.Logf("backup: failed to persist throttle state after size cap error: %v\n", saveErr)
}
} else if exceeded {
pauseAutoBackupForSizeCap(dir, state, size)
return
}
// One backup at a time per workspace (backup_lock.go). Auto-backup never
// waits: a held lock means a backup is already running.
release, err := acquireBackupLock(0)
if err != nil {
if errors.Is(err, errBackupBusy) {
debug.Logf("backup: skipping — another backup is running\n")
return
}
fmt.Fprintf(os.Stderr, "Warning: auto-backup skipped: %v\n", err)
return
}
defer release()
// Run the backup (force=true since we already checked change detection above)
if _, err := runBackupExport(ctx, backend, true); err != nil {
if !isQuiet() && !jsonOutput {
fmt.Fprintf(os.Stderr, "Warning: auto-backup failed: %v\n", err)
}
debug.Logf("backup: error: %v\n", err)
return
}
debug.Logf("backup: completed successfully\n")
}