Commit graph

2 commits

Author SHA1 Message Date
Julian Knutsen
8adcd25992
fix(backup): point the size-cap pause advice at levers that end it (#7074)
The PAUSED warning, the `bd backup status` PAUSED line, CHANGELOG and
configuration.md told operators to switch to a fresh destination with
`bd backup init <new-path>`. That never ends the pause: backupDir()
reads only backup.git-repo, while `bd backup init` registers the
separate destination that manual `bd backup sync` pushes to, so
auto-backup keeps targeting the over-cap directory and stays paused.

Name the two levers that do end it: raise backup.size-cap-mb, or point
backup.git-repo at a different git repository. Also correct the
backup.git-repo row in configuration.md, which called it a repo URL
although backupDir() requires a local path containing .git (anything
else falls back to .beads/backup/), so the new advice is actionable.

TestPauseAutoBackupForSizeCap_RemediationAdvice and the status PAUSED
line now require backup.git-repo and reject `bd backup init`.
TestMaybeAutoBackup_RemediationLevers runs all three for real: raising
the cap and switching backup.git-repo each resume a sync, while
`bd backup init` leaves auto-backup paused.

Post-merge review follow-up to #6071.

Agent-Signature: claude-code-claude-opus-5-5-max on behalf of Gas City Adopt-PR

Co-authored-by: Eddie the Engineer <adopt-pr@gascity.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 08:05:46 -07:00
Athos Bernardes
c26cf1fb6f
fix(bd): cap auto-backup destination size to stop unbounded growth (#6071)
* fix(bd): cap auto-backup destination size to stop unbounded growth (ga-y6gjv)

ga-y6gjv asked for 4 fixes to bd's auto-backup subsystem. Three are already
merged and deployed (verified via git merge-base --is-ancestor against the
commit that built the currently-deployed ~/.local/bin/bd):
  - interval throttle persists on failure too, per-workspace: 5f685959c
  - default-off in sql-server/shared mode: 5f685959c, refined by 8b9429153
  - `bd config get backup.enabled` reports the effective value: 5f685959c

This commit is only the fourth, still-open item: "prune/cap backup
archives." BackupSync/CALL DOLT_BACKUP('sync', ...) only ever transfers new
chunks into the destination — it never prunes ones that became unreachable
on the source DB (history rewrites, superseded data) — so even with the
interval fix slowing the frequency of syncs, the destination can still only
grow, never shrink. This was the actual disk-filling mechanism in the
2026-06-19 outage (43GB backup dir from a 1.7GB store).

I first tried wiring `dolt gc` into the post-sync path (mirroring
runCompactDolt's exec.Command("dolt", "gc", ...) pattern in compact.go).
That does not work: a backup destination is a bare Dolt chunk-store
directory with no .dolt repo-root marker, so the standalone dolt CLI can't
open it ("not a valid dolt repository") no matter what its working
directory is set to, and dolt_backup.go's SQL procedures only expose
add/sync/restore — never gc — for a backup destination. Confirmed this
empirically (a real dolt gc attempt against a real backup dir, outside any
test) before shipping anything, not just reasoned about it.

Given no supported way to prune a backup destination in place, this adds a
hard size cap instead: once the destination exceeds backup.size-cap-mb
(default 2048MB), maybeAutoBackup skips the sync attempt entirely rather
than letting it grow further, and prints a throttled warning
(backup.size-warn-interval, default 24h) telling the operator to free space
or raise the cap. This directly bounds the failure mode that caused the
outage — unbounded growth — without risking any operation against the
backup store's on-disk format.

Tests (cmd/bd/backup_cap_test.go): threshold check against a real
directory, warning-throttle persistence (mirrors the existing throttle
persisted on backup failure, backup_export.go), and end-to-end proof that
maybeAutoBackup never attempts BackupDatabase once capped. Confirmed each
new test fails against the pre-fix code (wiring temporarily reverted,
re-verified, restored) before treating it as green.

Verified: go build ./... clean, gofmt clean, go vet clean, full
cmd/bd + internal/config test run green (backup/config-focused subset run
with -v; no regressions in the wider package test runs either).

* fix(bd): address PR #6071 review feedback on backup size-cap (ga-y6gjv)

bee-ghosttrack's 2026-09-03 review of the size-cap fix (3d27d7a98) requested
changes before merge -- 4 major, 2 minor, all about the pause being
invisible rather than the cap mechanism itself. This addresses all six.

Major:

1. backup.size-cap-mb: 0 now means "no cap", not "use the 2048MB default."
   effectiveSizeCapMB() centralizes the <=0-is-disabled interpretation;
   backupSizeCapExceeded and the new status helpers all go through it.
   Viper already registers a 2048 default for this key
   (internal/config/config.go), so GetInt only returns <=0 when the
   operator explicitly set 0 (or an unparseable value) -- the old code's
   own `if capMB <= 0 { capMB = 2048 }` was silently overriding that
   explicit choice, leaving no off switch for a legitimately larger
   destination.

2. `bd backup status` (and --json) now report a `size_cap` object
   (enabled, cap_mb, current_bytes, exceeded), and the human-text Config:
   line grows a "Size cap: PAUSED (cap exceeded) -- X / Y..." line when
   paused. Previously the pause was stderr-only and throttled to 24h, so
   an agent/CI caller on --json or --quiet had no way to see that
   auto-backup had silently and permanently stopped syncing.

3. The remediation message no longer says "delete the backup directory."
   Nothing in this codebase demonstrates a deleted destination is cleanly
   recreated by the next sync (the server-side Dolt backup remote stays
   registered against that path), and following the old advice would
   leave a zero-backup window while the next sync is itself
   interval-throttled. It now points at backup.size-cap-mb / `bd backup
   init <new-path>`.

4. docs/reference/configuration.md gains rows for both new keys plus an
   expanded Auto-backup section (the cap mechanism, the pause, manual
   backup staying uncapped, bd backup status reporting the pause);
   CHANGELOG.md gets an [Unreleased]/Fixed entry referencing #6071.

Minor:

5. maybeWarnBackupSizeCap's throttle timestamp is now also cached in an
   in-process map (capWarnFallback, keyed by dir) alongside the persisted
   backup_state.json. The persist write targets the directory the cap
   just declared full, so in the disk-full case this cap exists for, that
   write fails and the timestamp never sticks -- a long-lived bd process
   that keeps calling maybeAutoBackup for the same destination (e.g.
   internal/storage/dbproxy's server mode) would otherwise re-warn on
   every single call. The map is keyed by dir specifically so unrelated
   destinations -- and unrelated tests -- never share a throttle.

6. The size-cap directory walk (getDirSize's filepath.Walk) now runs
   after the interval throttle in maybeAutoBackup, not before. The
   reviewer measured 65-100ms per bd invocation at 20k files in the
   backup dir when the walk ran unconditionally on every command; moving
   it below the throttle makes it free on the common (throttled) path and
   only pays the walk cost once per interval, exactly when a sync attempt
   would occur anyway.

Also noted by the reviewer: manual `bd backup` / `bd backup sync` stay
uncapped -- now documented in configuration.md.

Tests (cmd/bd/backup_cap_test.go, cmd/bd/backup_status_test.go): one new
test per review point. Each was confirmed to fail against the pre-fix code
for the expected reason before implementing -- including catching a
vacuous pass in my own first draft of the "disabled cap" JSON assertion
(zero-value Enabled:false matched whether the size_cap key existed at all
or not; fixed to assert key presence first). The disabled-cap test uses a
sparse file (Truncate, not a real write) to cross the legacy 2048MB
threshold without allocating 2GB of real disk or memory. The reordering
test is detected indirectly: if the cap check ran despite the throttle, it
would persist LastCapWarnAt; the interval-fresh case proves it does not.

Verified: go build ./cmd/bd/... clean, gofmt clean, go vet ./cmd/bd/...
clean, golangci-lint run ./cmd/bd/... -> 0 issues, targeted backup/status
test subset green (20/20, including every pre-existing backup test -- no
regressions).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* fix(bazel): add backup_cap sources to //cmd/bd srcs lists

The contributor branched before `cmd/bd/BUILD.bazel` existed, so the two
files this PR adds were never enumerated there. After rebasing onto a base
that hand-maintains explicit srcs lists, `//cmd/bd:bd_lib` omits
`backup_cap.go` while still compiling `backup.go` and `backup_auto.go`,
which reference four symbols it defines:

  backup.go:93       undefined: showSizeCapStatusJSON
  backup.go:141      undefined: showSizeCapStatus
  backup_auto.go:200 undefined: backupSizeCapExceeded
  backup_auto.go:203 undefined: maybeWarnBackupSizeCap

`backup_cap_test.go` was likewise absent from `//cmd/bd:bd_test`, leaving
seven of the PR's nine new tests unreachable under Bazel.

`make bazel-sync-check` does not catch this: `tools/bazel/go_srcs.py --check`
only validates the `go_srcs` filegroup globs, not the `go_library`/`go_test`
srcs enumerations.

Four unrelated test files (config_leaked_secret_test.go, init_role_*_test.go,
init_git_bootstrap_routing_test.go) are also unlisted, but they are unlisted
on the base too, so they are left alone as pre-existing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(backup): stop the size-cap walk from running on every bd command (#6071 review)

Maintainer review fixup for PR #6071 (ga-y6gjv), iterations 1 and 2.

The gating finding, filed independently by both review lanes and confirmed
empirically: the fixup that moved the getDirSize walk behind the interval
throttle does not deliver its claimed property, because state.Timestamp —
the only thing that re-arms that throttle — is written in exactly two
places, both inside runBackupExport. The two steady states that skip
before reaching it therefore pass the throttle on every single bd
invocation and pay the full filepath.Walk forever:

  IDLE    no new Dolt commit; the walk runs, then change detection skips
          and writes nothing.
  PAUSED  cap exceeded; the walk has already happened by the time
          maybeWarnBackupSizeCap persists LastCapWarnAt, which is the only
          field it wrote.

PAUSED is the pathological case: an over-cap destination is by definition
the largest one, so the feature's terminal state was its most expensive,
persisting until an operator intervened. This landed on read-only
commands too, i.e. the 65-100ms/command agent-polling regression the walk
was moved to avoid.

Both halves of the documented remedy:

  - Move the cap check after change detection, the order
    docs/reference/configuration.md already describes ("If data changed and
    the throttle interval has passed, bd checks the destination
    directory's on-disk size"). This makes that sentence true and closes
    the docs finding with no docs edit.
  - Re-arm the interval throttle (state.Timestamp) on the cap-exceeded
    skip, leaving LastDoltCommit untouched exactly as runBackupExport's own
    failure path does, so change detection still sees pending work once the
    cap is raised.

Neither half suffices alone: the reorder still walks every command in a
PAUSED workspace where data IS changing, and the persist alone still walks
forever when IDLE. Persisting on the idle skip instead would add up to one
backup.interval of backup latency, which this pairing avoids.

Also from the same review, all non-gating:

  - The cap-check error path failed open behind a debug-only line, so one
    unreadable entry could silently restore unbounded growth while
    `bd backup status --json` reported the very same error. It now warns on
    stderr, throttled by the backup interval and deliberately not by the
    LastCapWarnAt slot: spending that slot here would suppress the more
    important PAUSED notice for a full warn interval.
  - The warn timestamp is now recorded only when the message was actually
    printed, so --json/--quiet agent calls stop consuming the 24h slot
    silently.
  - Dropped capWarnFallback (map, mutex, two accessors). No reachable
    production path can read it: both call sites are in PersistentPostRunE,
    which runs once per CLI process, and the proxy child installs an empty
    PersistentPostRun. Its pinning test is repurposed to the contract that
    is real — a persist failure must not block the already-decided skip.
  - Registered backup.size-cap-mb / backup.size-warn-interval in
    YamlOnlyKeys, so the backup block is a complete inventory again.
    Functionally inert: the map has no enumerator and its one predicate
    consumer already matches on the "backup." prefix.
  - `bd backup status` prints "Size cap: unavailable (<err>)" instead of
    dropping the line, and the JSON error object carries an explicit null
    `exceeded` so a consumer cannot read a missing key as "not exceeded".

The CHANGELOG's cost claim is corrected to match the shipped order; it
described the throttle as sufficient, which is the defect above.

Iteration 2, all non-gating (that review approved):

  - The walk-error path re-arms the interval throttle itself before
    proceeding uncapped. Iteration 1 relied on the fall-through into
    runBackupExport to bound that warning to once per backup.interval,
    but not every runBackupExport exit persists state.Timestamp: a failed
    post-sync GetCurrentCommit returns without saving state, so the walk,
    the warning and the sync itself repeated on every command. The doc
    comment now states the bound the code has: once per interval whenever
    backup_state.json can be written.
  - `bd backup status` echoes the configured backup.size-cap-mb when the
    cap is disabled instead of a literal 0. viper's GetInt also resolves
    an unparseable value ("2048MB") to 0, so the line asserted a value the
    operator never wrote.
  - The status readers count only a destination that does not exist yet
    as empty. getDirSize aborts on the first per-entry error, so a chunk
    file removed mid-walk read as 0 bytes, i.e. `"exceeded": false` for a
    destination that may be over its cap. It is now reported as
    unavailable, like any other walk error.
  - docs/reference/configuration.md no longer says `--json` reports the
    string `PAUSED (cap exceeded)`; it reports `size_cap.exceeded: true`.
  - TestMaybeAutoBackup_CapCheckSkippedWhenThrottled seeded the same commit
    as its watermark, so after iteration 1's reorder change detection
    alone stopped the walk: it passed with the interval throttle deleted,
    and so did every other Backup/Throttle/Auto test in cmd/bd. It now
    seeds a changed commit, and its comment names the renamed
    pauseAutoBackupForSizeCap.

Tests: iteration 1's four new/updated pins are each mutation-proved.
Dropping the throttle re-arm reds the three throttle pins and leaves the
idle pin green; restoring the original ordering reds only the idle pin;
reverting both reds all four. Iteration 2's three new pins each go red
under their own mutation only: dropping the walk-error re-arm, restoring
the literal `=0`, and swallowing a walk ENOENT (dropping the root stat
reds the missing-destination case instead). Disabling the interval
throttle now reds the throttle test.

Base refresh before approval (rebased onto main 3f9561db2b, which had
made the PR CONFLICTING):

  - main 8269a88cbf (#6327) moved compact onto storeSizeBytes and
    deleted getDirSize from compact.go, leaving backup_cap.go its only
    caller, so the merged tree failed to build (undefined: getDirSize)
    although only CHANGELOG.md and backup_auto.go conflicted textually.
    getDirSize is restored verbatim in backup_cap.go (plus the
    path/filepath import), and the two comments that described it as
    shared with, or exercised by, compact.go are corrected.
  - main 4e4ebdb443 (#6879) added acquireBackupLock where this commit
    places the size-cap check. Both are kept, the cap check first, as
    its comment asks: a skip should not first take the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
2026-10-01 01:00:25 -07:00