* fix(bd): cap auto-backup destination size to stop unbounded growth (ga-y6gjv) ga-y6gjv asked for 4 fixes to bd's auto-backup subsystem. Three are already merged and deployed (verified via git merge-base --is-ancestor against the commit that built the currently-deployed ~/.local/bin/bd): - interval throttle persists on failure too, per-workspace:5f685959c- default-off in sql-server/shared mode:5f685959c, refined by8b9429153- `bd config get backup.enabled` reports the effective value:5f685959cThis commit is only the fourth, still-open item: "prune/cap backup archives." BackupSync/CALL DOLT_BACKUP('sync', ...) only ever transfers new chunks into the destination — it never prunes ones that became unreachable on the source DB (history rewrites, superseded data) — so even with the interval fix slowing the frequency of syncs, the destination can still only grow, never shrink. This was the actual disk-filling mechanism in the 2026-06-19 outage (43GB backup dir from a 1.7GB store). I first tried wiring `dolt gc` into the post-sync path (mirroring runCompactDolt's exec.Command("dolt", "gc", ...) pattern in compact.go). That does not work: a backup destination is a bare Dolt chunk-store directory with no .dolt repo-root marker, so the standalone dolt CLI can't open it ("not a valid dolt repository") no matter what its working directory is set to, and dolt_backup.go's SQL procedures only expose add/sync/restore — never gc — for a backup destination. Confirmed this empirically (a real dolt gc attempt against a real backup dir, outside any test) before shipping anything, not just reasoned about it. Given no supported way to prune a backup destination in place, this adds a hard size cap instead: once the destination exceeds backup.size-cap-mb (default 2048MB), maybeAutoBackup skips the sync attempt entirely rather than letting it grow further, and prints a throttled warning (backup.size-warn-interval, default 24h) telling the operator to free space or raise the cap. This directly bounds the failure mode that caused the outage — unbounded growth — without risking any operation against the backup store's on-disk format. Tests (cmd/bd/backup_cap_test.go): threshold check against a real directory, warning-throttle persistence (mirrors the existing throttle persisted on backup failure, backup_export.go), and end-to-end proof that maybeAutoBackup never attempts BackupDatabase once capped. Confirmed each new test fails against the pre-fix code (wiring temporarily reverted, re-verified, restored) before treating it as green. Verified: go build ./... clean, gofmt clean, go vet clean, full cmd/bd + internal/config test run green (backup/config-focused subset run with -v; no regressions in the wider package test runs either). * fix(bd): address PR #6071 review feedback on backup size-cap (ga-y6gjv) bee-ghosttrack's 2026-09-03 review of the size-cap fix (3d27d7a98) requested changes before merge -- 4 major, 2 minor, all about the pause being invisible rather than the cap mechanism itself. This addresses all six. Major: 1. backup.size-cap-mb: 0 now means "no cap", not "use the 2048MB default." effectiveSizeCapMB() centralizes the <=0-is-disabled interpretation; backupSizeCapExceeded and the new status helpers all go through it. Viper already registers a 2048 default for this key (internal/config/config.go), so GetInt only returns <=0 when the operator explicitly set 0 (or an unparseable value) -- the old code's own `if capMB <= 0 { capMB = 2048 }` was silently overriding that explicit choice, leaving no off switch for a legitimately larger destination. 2. `bd backup status` (and --json) now report a `size_cap` object (enabled, cap_mb, current_bytes, exceeded), and the human-text Config: line grows a "Size cap: PAUSED (cap exceeded) -- X / Y..." line when paused. Previously the pause was stderr-only and throttled to 24h, so an agent/CI caller on --json or --quiet had no way to see that auto-backup had silently and permanently stopped syncing. 3. The remediation message no longer says "delete the backup directory." Nothing in this codebase demonstrates a deleted destination is cleanly recreated by the next sync (the server-side Dolt backup remote stays registered against that path), and following the old advice would leave a zero-backup window while the next sync is itself interval-throttled. It now points at backup.size-cap-mb / `bd backup init <new-path>`. 4. docs/reference/configuration.md gains rows for both new keys plus an expanded Auto-backup section (the cap mechanism, the pause, manual backup staying uncapped, bd backup status reporting the pause); CHANGELOG.md gets an [Unreleased]/Fixed entry referencing #6071. Minor: 5. maybeWarnBackupSizeCap's throttle timestamp is now also cached in an in-process map (capWarnFallback, keyed by dir) alongside the persisted backup_state.json. The persist write targets the directory the cap just declared full, so in the disk-full case this cap exists for, that write fails and the timestamp never sticks -- a long-lived bd process that keeps calling maybeAutoBackup for the same destination (e.g. internal/storage/dbproxy's server mode) would otherwise re-warn on every single call. The map is keyed by dir specifically so unrelated destinations -- and unrelated tests -- never share a throttle. 6. The size-cap directory walk (getDirSize's filepath.Walk) now runs after the interval throttle in maybeAutoBackup, not before. The reviewer measured 65-100ms per bd invocation at 20k files in the backup dir when the walk ran unconditionally on every command; moving it below the throttle makes it free on the common (throttled) path and only pays the walk cost once per interval, exactly when a sync attempt would occur anyway. Also noted by the reviewer: manual `bd backup` / `bd backup sync` stay uncapped -- now documented in configuration.md. Tests (cmd/bd/backup_cap_test.go, cmd/bd/backup_status_test.go): one new test per review point. Each was confirmed to fail against the pre-fix code for the expected reason before implementing -- including catching a vacuous pass in my own first draft of the "disabled cap" JSON assertion (zero-value Enabled:false matched whether the size_cap key existed at all or not; fixed to assert key presence first). The disabled-cap test uses a sparse file (Truncate, not a real write) to cross the legacy 2048MB threshold without allocating 2GB of real disk or memory. The reordering test is detected indirectly: if the cap check ran despite the throttle, it would persist LastCapWarnAt; the interval-fresh case proves it does not. Verified: go build ./cmd/bd/... clean, gofmt clean, go vet ./cmd/bd/... clean, golangci-lint run ./cmd/bd/... -> 0 issues, targeted backup/status test subset green (20/20, including every pre-existing backup test -- no regressions). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(bazel): add backup_cap sources to //cmd/bd srcs lists The contributor branched before `cmd/bd/BUILD.bazel` existed, so the two files this PR adds were never enumerated there. After rebasing onto a base that hand-maintains explicit srcs lists, `//cmd/bd:bd_lib` omits `backup_cap.go` while still compiling `backup.go` and `backup_auto.go`, which reference four symbols it defines: backup.go:93 undefined: showSizeCapStatusJSON backup.go:141 undefined: showSizeCapStatus backup_auto.go:200 undefined: backupSizeCapExceeded backup_auto.go:203 undefined: maybeWarnBackupSizeCap `backup_cap_test.go` was likewise absent from `//cmd/bd:bd_test`, leaving seven of the PR's nine new tests unreachable under Bazel. `make bazel-sync-check` does not catch this: `tools/bazel/go_srcs.py --check` only validates the `go_srcs` filegroup globs, not the `go_library`/`go_test` srcs enumerations. Four unrelated test files (config_leaked_secret_test.go, init_role_*_test.go, init_git_bootstrap_routing_test.go) are also unlisted, but they are unlisted on the base too, so they are left alone as pre-existing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(backup): stop the size-cap walk from running on every bd command (#6071 review) Maintainer review fixup for PR #6071 (ga-y6gjv), iterations 1 and 2. The gating finding, filed independently by both review lanes and confirmed empirically: the fixup that moved the getDirSize walk behind the interval throttle does not deliver its claimed property, because state.Timestamp — the only thing that re-arms that throttle — is written in exactly two places, both inside runBackupExport. The two steady states that skip before reaching it therefore pass the throttle on every single bd invocation and pay the full filepath.Walk forever: IDLE no new Dolt commit; the walk runs, then change detection skips and writes nothing. PAUSED cap exceeded; the walk has already happened by the time maybeWarnBackupSizeCap persists LastCapWarnAt, which is the only field it wrote. PAUSED is the pathological case: an over-cap destination is by definition the largest one, so the feature's terminal state was its most expensive, persisting until an operator intervened. This landed on read-only commands too, i.e. the 65-100ms/command agent-polling regression the walk was moved to avoid. Both halves of the documented remedy: - Move the cap check after change detection, the order docs/reference/configuration.md already describes ("If data changed and the throttle interval has passed, bd checks the destination directory's on-disk size"). This makes that sentence true and closes the docs finding with no docs edit. - Re-arm the interval throttle (state.Timestamp) on the cap-exceeded skip, leaving LastDoltCommit untouched exactly as runBackupExport's own failure path does, so change detection still sees pending work once the cap is raised. Neither half suffices alone: the reorder still walks every command in a PAUSED workspace where data IS changing, and the persist alone still walks forever when IDLE. Persisting on the idle skip instead would add up to one backup.interval of backup latency, which this pairing avoids. Also from the same review, all non-gating: - The cap-check error path failed open behind a debug-only line, so one unreadable entry could silently restore unbounded growth while `bd backup status --json` reported the very same error. It now warns on stderr, throttled by the backup interval and deliberately not by the LastCapWarnAt slot: spending that slot here would suppress the more important PAUSED notice for a full warn interval. - The warn timestamp is now recorded only when the message was actually printed, so --json/--quiet agent calls stop consuming the 24h slot silently. - Dropped capWarnFallback (map, mutex, two accessors). No reachable production path can read it: both call sites are in PersistentPostRunE, which runs once per CLI process, and the proxy child installs an empty PersistentPostRun. Its pinning test is repurposed to the contract that is real — a persist failure must not block the already-decided skip. - Registered backup.size-cap-mb / backup.size-warn-interval in YamlOnlyKeys, so the backup block is a complete inventory again. Functionally inert: the map has no enumerator and its one predicate consumer already matches on the "backup." prefix. - `bd backup status` prints "Size cap: unavailable (<err>)" instead of dropping the line, and the JSON error object carries an explicit null `exceeded` so a consumer cannot read a missing key as "not exceeded". The CHANGELOG's cost claim is corrected to match the shipped order; it described the throttle as sufficient, which is the defect above. Iteration 2, all non-gating (that review approved): - The walk-error path re-arms the interval throttle itself before proceeding uncapped. Iteration 1 relied on the fall-through into runBackupExport to bound that warning to once per backup.interval, but not every runBackupExport exit persists state.Timestamp: a failed post-sync GetCurrentCommit returns without saving state, so the walk, the warning and the sync itself repeated on every command. The doc comment now states the bound the code has: once per interval whenever backup_state.json can be written. - `bd backup status` echoes the configured backup.size-cap-mb when the cap is disabled instead of a literal 0. viper's GetInt also resolves an unparseable value ("2048MB") to 0, so the line asserted a value the operator never wrote. - The status readers count only a destination that does not exist yet as empty. getDirSize aborts on the first per-entry error, so a chunk file removed mid-walk read as 0 bytes, i.e. `"exceeded": false` for a destination that may be over its cap. It is now reported as unavailable, like any other walk error. - docs/reference/configuration.md no longer says `--json` reports the string `PAUSED (cap exceeded)`; it reports `size_cap.exceeded: true`. - TestMaybeAutoBackup_CapCheckSkippedWhenThrottled seeded the same commit as its watermark, so after iteration 1's reorder change detection alone stopped the walk: it passed with the interval throttle deleted, and so did every other Backup/Throttle/Auto test in cmd/bd. It now seeds a changed commit, and its comment names the renamed pauseAutoBackupForSizeCap. Tests: iteration 1's four new/updated pins are each mutation-proved. Dropping the throttle re-arm reds the three throttle pins and leaves the idle pin green; restoring the original ordering reds only the idle pin; reverting both reds all four. Iteration 2's three new pins each go red under their own mutation only: dropping the walk-error re-arm, restoring the literal `=0`, and swallowing a walk ENOENT (dropping the root stat reds the missing-destination case instead). Disabling the interval throttle now reds the throttle test. Base refresh before approval (rebased onto main3f9561db2b, which had made the PR CONFLICTING): - main8269a88cbf(#6327) moved compact onto storeSizeBytes and deleted getDirSize from compact.go, leaving backup_cap.go its only caller, so the merged tree failed to build (undefined: getDirSize) although only CHANGELOG.md and backup_auto.go conflicted textually. getDirSize is restored verbatim in backup_cap.go (plus the path/filepath import), and the two comments that described it as shared with, or exercised by, compact.go are corrected. - main4e4ebdb443(#6879) added acquireBackupLock where this commit places the size-cap check. Both are kept, the cap check first, as its comment asks: a skip should not first take the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com> Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
159 lines
4.7 KiB
Go
159 lines
4.7 KiB
Go
package main
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"time"
|
|
|
|
"github.com/spf13/cobra"
|
|
"github.com/steveyegge/beads/internal/config"
|
|
"github.com/steveyegge/beads/internal/metrics"
|
|
)
|
|
|
|
var backupCmd = &cobra.Command{
|
|
Use: "backup",
|
|
Short: "Back up your beads database",
|
|
Long: `Back up your beads database for off-machine recovery.
|
|
|
|
This is a Dolt-native database backup. It preserves the database state,
|
|
including tables, branches, commit history, and working-set data. This is
|
|
different from 'bd export', which writes issue records to JSONL for migration
|
|
and interoperability.
|
|
|
|
Commands:
|
|
bd backup init <path> Set up a backup destination (filesystem or DoltHub)
|
|
bd backup sync Push to configured backup destination
|
|
bd backup restore [path] Restore from a backup directory
|
|
bd backup remove Remove backup destination
|
|
bd backup status Show backup status
|
|
|
|
DoltHub is recommended for cloud backup:
|
|
bd backup init https://doltremoteapi.dolthub.com/<user>/<repo>
|
|
Set DOLT_REMOTE_USER and DOLT_REMOTE_PASSWORD for authentication.
|
|
|
|
Auto-backup default:
|
|
When backup.enabled is unset, auto-backup turns ON in embedded mode if a
|
|
git remote exists, and stays OFF in sql-server / shared-server mode. In
|
|
server mode many bd clients share one Dolt server, and each would register
|
|
a server-side backup remote under the same name pointing at its own local
|
|
dir and full-sync the whole database — a self-amplifying storm. To back up
|
|
a shared server, run 'bd backup' explicitly (or set backup.enabled=true and
|
|
coordinate destinations). 'bd config get backup.enabled' shows the effective
|
|
value and its source.
|
|
|
|
A proxied-server workspace is server mode too: auto-backup is OFF by
|
|
default and backup.enabled=true opts in. Both the opt-in and the five
|
|
backup verbs above are honored only on a managed-local proxied server —
|
|
one bd started itself; on a Dolt server bd does not own they are refused.`,
|
|
GroupID: "sync",
|
|
}
|
|
|
|
type backupSizeFunc func(context.Context) (bytes int64, available bool, err error)
|
|
|
|
var backupStatusCmd = newBackupStatusCommand(doltBackupSize)
|
|
|
|
func newBackupStatusCommand(sizeDatabase backupSizeFunc) *cobra.Command {
|
|
return &cobra.Command{
|
|
Use: "status",
|
|
Short: "Show last backup status",
|
|
SilenceUsage: true,
|
|
SilenceErrors: true,
|
|
RunE: func(cmd *cobra.Command, args []string) error {
|
|
if usesProxiedServer() {
|
|
if err := requireLocalProxiedBackup("backup status"); err != nil {
|
|
return err
|
|
}
|
|
}
|
|
evt := metrics.NewCommandEvent("backup-status")
|
|
defer func() {
|
|
if c := metrics.Global(); c != nil {
|
|
c.CloseEventAndAdd(evt)
|
|
}
|
|
}()
|
|
|
|
dir, err := backupDir()
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
state, err := loadBackupState(dir)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
databaseSize, sizeAvailable, err := sizeDatabase(cmd.Context())
|
|
if err != nil {
|
|
return HandleErrorRespectJSON("measure database size: %v", err)
|
|
}
|
|
|
|
if jsonOutput {
|
|
result := map[string]interface{}{
|
|
"backup": state,
|
|
"dolt": showDoltBackupStatusJSON(),
|
|
"size_cap": showSizeCapStatusJSON(dir),
|
|
}
|
|
if sizeAvailable {
|
|
result["database_size"] = showDBSizeJSON(databaseSize)
|
|
}
|
|
data, err := json.MarshalIndent(result, "", " ")
|
|
if err != nil {
|
|
return err
|
|
}
|
|
fmt.Println(string(data))
|
|
return nil
|
|
}
|
|
|
|
hasBackup := state.LastDoltCommit != ""
|
|
hasDolt := false
|
|
if cfg, _ := loadDoltBackupConfig(); cfg != nil {
|
|
hasDolt = true
|
|
}
|
|
|
|
if !hasBackup && !hasDolt {
|
|
fmt.Println("No backup has been performed yet.")
|
|
fmt.Println()
|
|
fmt.Println("Setup:")
|
|
fmt.Println(" bd backup init <path> Set up a backup destination")
|
|
fmt.Println(" bd backup sync Push to backup destination")
|
|
if sizeAvailable {
|
|
showDBSize(databaseSize)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
if hasBackup {
|
|
fmt.Println("Backup:")
|
|
fmt.Printf(" Last backup: %s (%s ago)\n",
|
|
state.Timestamp.Format(time.RFC3339),
|
|
time.Since(state.Timestamp).Round(time.Second))
|
|
fmt.Printf(" Dolt commit: %s\n", state.LastDoltCommit)
|
|
}
|
|
|
|
// Show config (effective values with source)
|
|
enabled := isBackupAutoEnabled()
|
|
interval := config.GetDuration("backup.interval")
|
|
enabledNote := ""
|
|
if note := backupAutoStatusNote(enabled); note != "" {
|
|
enabledNote = " (" + note + ")"
|
|
}
|
|
fmt.Printf("\nConfig: enabled=%v%s interval=%s\n",
|
|
enabled, enabledNote, interval)
|
|
showSizeCapStatus(dir)
|
|
|
|
// Show Dolt backup info
|
|
showDoltBackupStatus()
|
|
|
|
// Show database size when the active backend can measure it locally.
|
|
if sizeAvailable {
|
|
showDBSize(databaseSize)
|
|
}
|
|
|
|
return nil
|
|
},
|
|
}
|
|
}
|
|
|
|
func init() {
|
|
backupCmd.AddCommand(backupStatusCmd)
|
|
rootCmd.AddCommand(backupCmd)
|
|
}
|