beads/cmd/bd/backup_export_failure_test.go
Julian Knutsen 4e4ebdb443
feat(backup): proxied auto-backup on managed-local, serialized with bd backup sync (#6879)
* feat(backup): run opted-in auto-backup on a managed-local proxied server

#6584 routed the five `bd backup` verbs over the proxied provider on
managed-local, and #6694 made `bd backup status` admit the remaining gap:
the proxied arm of PersistentPostRunE never called runPostRunAutoBackup,
so backup.enabled=true was inert on every proxied workspace. Close it.

The proxied arm now calls the auto-backup hook before it closes the
provider, under the direct arm's maintenance gate (strict --readonly,
`bd serve`, a migration freeze) plus previews. maybeAutoBackup picks a
backend for the command: the direct store as before, or the proxied
provider's RunNonTx seam, which `bd backup sync` already uses.

The proxied backend is honored exactly where `bd backup sync` is: the
"backup sync" capability row's rule for this workspace's topology, so
managed-local only. Auto-backup registers a server-side backup remote,
which on a server bd does not own is global to every client; an implicit
trigger must not get further than the explicit verb. It never opens a
provider of its own: a command that opened none (`bd dolt stop`) or
closed it (restore) backs nothing up rather than relaunching the server.
The default stays OFF, as on every server-mode workspace.

The register/sync-to-a-local-dir sequence that DoltStore and
EmbeddedDoltStore each carried a copy of moves to
versioncontrolops.BackupToDir (and HEAD lookup to CurrentCommit), so the
proxied backend reuses it instead of adding a third copy.

Tests: TestManagedLocalProxiedAutoBackup (default off, opt-in writes a
restorable backup, change detection, `bd dolt stop` stays quiescent,
restore round trip with issue-set equality; fails against origin/main);
TestManagedLocalProxiedBackupRestoreRefusedWhileAttached pins the restore
design (refused while another bd client holds the workspace, serving
child untouched, succeeds once it detaches); unit tests for the backend
locality gate, the post-run wiring, the status note and BackupToDir.
Both managed-local tests, and the existing round trip, are added to the
proxied-local smoke lane's compiled-in list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(backup): serialize auto-backup and bd backup sync with a workspace backup lock

Review follow-up on #6879. Auto-backup and an explicit bd backup sync
both run under the command's SHARED workspace gate, so nothing stopped
them overlapping, and auto-backup's remove -> add -> sync on one fixed
remote name races a concurrent copy of itself.

A per-workspace flock (.beads/backup.lock, already gitignored by
*.lock) now serializes them in every mode. Auto-backup takes it
non-blocking and skips when held (the throttle is left alone, so a
later command retries). bd backup sync waits up to exclusiveGateWait -
the bound restore's exclusive gate acquisition uses - then fails with
"another backup is running for this workspace" before touching the
store. Restore needs no part in it: its exclusive workspace gate
already excludes every command that could hold the lock.

Tests: lock semantics (refusal while held, bounded wait, release),
auto-backup skipping while held and running once released, and sync
failing loudly without reaching the store.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 02:45:58 -07:00

96 lines
3.5 KiB
Go

package main
import (
"context"
"errors"
"os"
"path/filepath"
"testing"
"time"
"github.com/steveyegge/beads/internal/config"
"github.com/steveyegge/beads/internal/storage"
)
// failingBackupStore is a minimal DoltStorage stand-in whose Dolt-native
// backup always fails. It embeds a nil DoltStorage for identity/type
// assertion and overrides only the methods runBackupExport touches.
type failingBackupStore struct {
storage.DoltStorage
commit string
backupErr error
backupCalls int
}
func (f *failingBackupStore) GetCurrentCommit(context.Context) (string, error) { return f.commit, nil }
func (f *failingBackupStore) BackupDatabase(context.Context, string) error {
f.backupCalls++
return f.backupErr
}
func (f *failingBackupStore) BackupAdd(context.Context, string, string) error { return nil }
func (f *failingBackupStore) BackupSync(context.Context, string) error { return nil }
func (f *failingBackupStore) BackupRemove(context.Context, string) error { return nil }
func (f *failingBackupStore) RestoreDatabase(context.Context, string, bool) error { return nil }
// TestRunBackupExport_PersistsThrottleOnFailure is the wy-zrmqr repro: a
// failing Dolt backup sync must still advance the throttle timestamp so the
// next bd command is throttled instead of retrying immediately. Before the
// fix, saveBackupState ran only on success, so a slow/failing shared Dolt
// server was hammered on EVERY subsequent command — the amplifier behind the
// 2026-07 CPU-pin incident.
func TestRunBackupExport_PersistsThrottleOnFailure(t *testing.T) {
// Point backupDir() at a temp git repo via backup.git-repo so we don't
// need a live beads workspace.
repo := t.TempDir()
if err := os.MkdirAll(filepath.Join(repo, ".git"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("BD_BACKUP_GIT_REPO", repo)
config.ResetForTesting()
t.Cleanup(config.ResetForTesting)
if err := config.Initialize(); err != nil {
t.Fatalf("config.Initialize: %v", err)
}
dir, err := backupDir()
if err != nil {
t.Fatalf("backupDir: %v", err)
}
// Sanity: no prior state, so the throttle would NOT block a first attempt.
if st, _ := loadBackupState(dir); !st.Timestamp.IsZero() {
t.Fatalf("precondition: expected zero timestamp, got %v", st.Timestamp)
}
fake := &failingBackupStore{commit: "deadbeef", backupErr: errors.New("sync to backup: server too busy")}
before := time.Now().UTC()
if _, err := runBackupExport(context.Background(), directLocalBackup{store: fake}, true); err == nil {
t.Fatal("expected runBackupExport to return the sync error, got nil")
}
if fake.backupCalls != 1 {
t.Fatalf("expected exactly 1 BackupDatabase call, got %d", fake.backupCalls)
}
st, err := loadBackupState(dir)
if err != nil {
t.Fatalf("loadBackupState: %v", err)
}
if st.Timestamp.IsZero() {
t.Fatal("throttle timestamp not persisted on failure — next command would retry unthrottled (storm)")
}
if st.Timestamp.Before(before) {
t.Errorf("throttle timestamp %v predates the failed attempt %v", st.Timestamp, before)
}
// The recorded timestamp must be recent enough that the default 15m
// throttle window blocks the next attempt.
if time.Since(st.Timestamp) >= 15*time.Minute {
t.Errorf("persisted timestamp %v is already outside the throttle window", st.Timestamp)
}
// LastDoltCommit must stay empty: the backup did NOT succeed, so change
// detection must still see pending work once the failure clears.
if st.LastDoltCommit != "" {
t.Errorf("LastDoltCommit should remain empty after a failed backup, got %q", st.LastDoltCommit)
}
}