feat(backup): proxied auto-backup on managed-local, serialized with bd backup sync (#6879)
* feat(backup): run opted-in auto-backup on a managed-local proxied server #6584 routed the five `bd backup` verbs over the proxied provider on managed-local, and #6694 made `bd backup status` admit the remaining gap: the proxied arm of PersistentPostRunE never called runPostRunAutoBackup, so backup.enabled=true was inert on every proxied workspace. Close it. The proxied arm now calls the auto-backup hook before it closes the provider, under the direct arm's maintenance gate (strict --readonly, `bd serve`, a migration freeze) plus previews. maybeAutoBackup picks a backend for the command: the direct store as before, or the proxied provider's RunNonTx seam, which `bd backup sync` already uses. The proxied backend is honored exactly where `bd backup sync` is: the "backup sync" capability row's rule for this workspace's topology, so managed-local only. Auto-backup registers a server-side backup remote, which on a server bd does not own is global to every client; an implicit trigger must not get further than the explicit verb. It never opens a provider of its own: a command that opened none (`bd dolt stop`) or closed it (restore) backs nothing up rather than relaunching the server. The default stays OFF, as on every server-mode workspace. The register/sync-to-a-local-dir sequence that DoltStore and EmbeddedDoltStore each carried a copy of moves to versioncontrolops.BackupToDir (and HEAD lookup to CurrentCommit), so the proxied backend reuses it instead of adding a third copy. Tests: TestManagedLocalProxiedAutoBackup (default off, opt-in writes a restorable backup, change detection, `bd dolt stop` stays quiescent, restore round trip with issue-set equality; fails against origin/main); TestManagedLocalProxiedBackupRestoreRefusedWhileAttached pins the restore design (refused while another bd client holds the workspace, serving child untouched, succeeds once it detaches); unit tests for the backend locality gate, the post-run wiring, the status note and BackupToDir. Both managed-local tests, and the existing round trip, are added to the proxied-local smoke lane's compiled-in list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(backup): serialize auto-backup and bd backup sync with a workspace backup lock Review follow-up on #6879. Auto-backup and an explicit bd backup sync both run under the command's SHARED workspace gate, so nothing stopped them overlapping, and auto-backup's remove -> add -> sync on one fixed remote name races a concurrent copy of itself. A per-workspace flock (.beads/backup.lock, already gitignored by *.lock) now serializes them in every mode. Auto-backup takes it non-blocking and skips when held (the throttle is left alone, so a later command retries). bd backup sync waits up to exclusiveGateWait - the bound restore's exclusive gate acquisition uses - then fails with "another backup is running for this workspace" before touching the store. Restore needs no part in it: its exclusive workspace gate already excludes every command that could hold the lock. Tests: lock semantics (refusal while held, bounded wait, release), auto-backup skipping while held and running once released, and sync failing loudly without reaching the store. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
fc4d7d3958
commit
4e4ebdb443
20 changed files with 891 additions and 127 deletions
5
.github/workflows/proxied-local-smoke.yml
vendored
5
.github/workflows/proxied-local-smoke.yml
vendored
|
|
@ -23,6 +23,7 @@ on:
|
|||
- 'cmd/bd/proxied_server*.go'
|
||||
- 'cmd/bd/dolt*.go'
|
||||
- 'cmd/bd/*_proxied_server.go'
|
||||
- 'cmd/bd/backup*.go'
|
||||
- 'cmd/bd/purge*.go'
|
||||
- 'cmd/bd/*managed_local*'
|
||||
- 'cmd/bd/proxied_outage_reconnect_integration_test.go'
|
||||
|
|
@ -50,6 +51,7 @@ on:
|
|||
- 'cmd/bd/proxied_server*.go'
|
||||
- 'cmd/bd/dolt*.go'
|
||||
- 'cmd/bd/*_proxied_server.go'
|
||||
- 'cmd/bd/backup*.go'
|
||||
- 'cmd/bd/purge*.go'
|
||||
- 'cmd/bd/*managed_local*'
|
||||
- 'cmd/bd/proxied_outage_reconnect_integration_test.go'
|
||||
|
|
@ -128,6 +130,9 @@ jobs:
|
|||
TestManagedLocalProxiedDoltStartRefusesWithProxyDown \
|
||||
TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone \
|
||||
TestManagedLocalProxiedTrackerUOWConformance \
|
||||
TestManagedLocalProxiedBackupRoundTrip \
|
||||
TestManagedLocalProxiedAutoBackup \
|
||||
TestManagedLocalProxiedBackupRestoreRefusedWhileAttached \
|
||||
TestManagedLocalProxiedPurgeWispsPlaneRetention
|
||||
do
|
||||
if ! grep -qx "$test_name" <<<"$listed"; then
|
||||
|
|
|
|||
21
CHANGELOG.md
21
CHANGELOG.md
|
|
@ -190,6 +190,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
|
||||
### Added
|
||||
|
||||
- **Auto-backup runs on a managed-local proxied-server workspace.** The
|
||||
proxied arm of the post-command hook now calls auto-backup, so an explicit
|
||||
`backup.enabled=true` (or `BD_BACKUP_ENABLED=1`) takes the same throttled,
|
||||
change-detected Dolt-native backup into `.beads/backup` it takes in direct
|
||||
mode, through the proxied provider's non-transactional seam. Before this the
|
||||
opt-in was inert on every proxied workspace and `bd backup status` said so.
|
||||
The default stays OFF, as on every server-mode workspace: many bd clients
|
||||
share one server. It is honored exactly where `bd backup sync` is — a
|
||||
proxied server bd started itself — and stays inert on an external or
|
||||
team-server topology, strict `--readonly`, `bd serve`, a preview
|
||||
(`--dry-run`/`--inspect`) and a migration freeze. A command that opened no
|
||||
provider (e.g. `bd dolt stop`) backs nothing up rather than relaunching the
|
||||
server to do so. `bd backup status` now reports the proxied default as
|
||||
`auto: off in proxied-server mode; set backup.enabled=true to opt in`.
|
||||
|
||||
- **Auto-backup and `bd backup sync` no longer overlap.** A per-workspace
|
||||
backup lock (`.beads/backup.lock`) serializes them in every mode:
|
||||
auto-backup skips when another backup holds it, and `bd backup sync` waits
|
||||
up to 5 seconds (the bound restore uses for the workspace gate) and then
|
||||
fails with "another backup is running for this workspace".
|
||||
|
||||
- **`bd purge` covers an orchestrator's wisp retention sweep.** Three changes
|
||||
that together let `bd purge --wisps-plane --older-than 168h --force` replace
|
||||
a raw `DELETE FROM wisps` retention step:
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ go_library(
|
|||
"backup_auto.go",
|
||||
"backup_dolt.go",
|
||||
"backup_export.go",
|
||||
"backup_lock.go",
|
||||
"backup_proxied_server.go",
|
||||
"backup_restore.go",
|
||||
"batch.go",
|
||||
|
|
@ -492,10 +493,13 @@ go_test(
|
|||
"auto_import_upgrade_unit_test.go",
|
||||
"autocommit_embedded_test.go",
|
||||
"backend_registry_contract_test.go",
|
||||
"backup_auto_proxied_local_integration_test.go",
|
||||
"backup_auto_proxied_test.go",
|
||||
"backup_auto_test.go",
|
||||
"backup_dolt_test.go",
|
||||
"backup_embedded_test.go",
|
||||
"backup_export_failure_test.go",
|
||||
"backup_lock_test.go",
|
||||
"backup_proxied_external_integration_test.go",
|
||||
"backup_proxied_local_integration_test.go",
|
||||
"backup_proxied_policy_test.go",
|
||||
|
|
|
|||
|
|
@ -42,10 +42,10 @@ Auto-backup default:
|
|||
coordinate destinations). 'bd config get backup.enabled' shows the effective
|
||||
value and its source.
|
||||
|
||||
On a proxied-server workspace that opt-in does not apply: auto-backup has
|
||||
no post-command hook on that path, so backup.enabled=true is inert there.
|
||||
The five backup verbs above are honored on a managed-local proxied server —
|
||||
run 'bd backup sync' explicitly to back one up.`,
|
||||
A proxied-server workspace is server mode too: auto-backup is OFF by
|
||||
default and backup.enabled=true opts in. Both the opt-in and the five
|
||||
backup verbs above are honored only on a managed-local proxied server —
|
||||
one bd started itself; on a Dolt server bd does not own they are refused.`,
|
||||
GroupID: "sync",
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package main
|
|||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
|
|
@ -46,28 +47,21 @@ func isBackupAutoEnabled() bool {
|
|||
// already computed so the note cannot disagree with the number beside
|
||||
// it — and so status does not re-run the git-remote probe.
|
||||
//
|
||||
// The note narrates the reason the decision actually used. Proxied-server
|
||||
// is checked first and ignores the config source, because on that topology
|
||||
// auto-backup cannot run at all whatever backup.enabled says: the proxied
|
||||
// arm of PersistentPostRunE never calls runPostRunAutoBackup (main.go), so
|
||||
// an explicit backup.enabled=true is inert and reporting a bare
|
||||
// "enabled=true" would send the operator away believing backups happen.
|
||||
// The five backup verbs are honored on managed-local (the only proxied
|
||||
// shape that reaches this line — every other one is refused by
|
||||
// requireLocalProxiedBackup), so `bd backup sync` is the real answer there.
|
||||
//
|
||||
// Every other sql-server shape keeps its post-run hook, so there the
|
||||
// default is merely off — the reason `bd config get backup.enabled`
|
||||
// already reports for the same condition. Before this arm existed, status
|
||||
// attributed that OFF to a missing git remote, which is untrue: server
|
||||
// mode short-circuits isBackupAutoEnabled whether or not a remote exists.
|
||||
// The note narrates the reason the decision actually used. An explicit
|
||||
// backup.enabled needs no note: the source explains it, and every shape
|
||||
// that reaches `bd backup status` honors it — including a managed-local
|
||||
// proxied server, whose post-run arm runs auto-backup (every other proxied
|
||||
// shape is refused by requireLocalProxiedBackup before status renders).
|
||||
// A server-mode default is OFF whether or not a git remote exists, so it
|
||||
// must not be attributed to the remote; before these arms existed, status
|
||||
// did exactly that.
|
||||
func backupAutoStatusNote(enabled bool) string {
|
||||
if usesProxiedServer() {
|
||||
return "auto-backup does not run on proxied-server; use 'bd backup sync'"
|
||||
}
|
||||
if config.GetValueSource("backup.enabled") != config.SourceDefault {
|
||||
return ""
|
||||
}
|
||||
if usesProxiedServer() {
|
||||
return "auto: off in proxied-server mode; set backup.enabled=true to opt in"
|
||||
}
|
||||
if usesSQLServer() {
|
||||
return "auto: off in sql-server mode"
|
||||
}
|
||||
|
|
@ -125,6 +119,21 @@ func clientServerShareFilesystem() bool {
|
|||
// bd sessions don't need a chatty repeat on every command.
|
||||
var autoBackupSkipNoticeOnce sync.Once
|
||||
|
||||
// autoBackupBackendForCommand returns the storage this command's auto-backup
|
||||
// runs against, or ok=false when there is nothing it may back up.
|
||||
func autoBackupBackendForCommand() (localBackupBackend, bool) {
|
||||
if usesProxiedServer() {
|
||||
return proxiedAutoBackupBackend()
|
||||
}
|
||||
if store == nil {
|
||||
return nil, false
|
||||
}
|
||||
if lm, ok := storage.UnwrapStore(store).(storage.LifecycleManager); ok && lm.IsClosed() {
|
||||
return nil, false
|
||||
}
|
||||
return directLocalBackup{store: store}, true
|
||||
}
|
||||
|
||||
// maybeAutoBackup runs a Dolt-native backup if enabled and the throttle interval has passed.
|
||||
// Called from PersistentPostRun after auto-commit.
|
||||
func maybeAutoBackup(ctx context.Context) {
|
||||
|
|
@ -139,10 +148,8 @@ func maybeAutoBackup(ctx context.Context) {
|
|||
if !isBackupAutoEnabled() {
|
||||
return
|
||||
}
|
||||
if store == nil {
|
||||
return
|
||||
}
|
||||
if lm, ok := storage.UnwrapStore(store).(storage.LifecycleManager); ok && lm.IsClosed() {
|
||||
backend, ok := autoBackupBackendForCommand()
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
|
||||
|
|
@ -152,7 +159,11 @@ func maybeAutoBackup(ctx context.Context) {
|
|||
// constructs is meaningless to the server — register fails on
|
||||
// every command. Skip cleanly with a one-time INFO so operators
|
||||
// know auto-backup is silent on purpose.
|
||||
if !clientServerShareFilesystem() {
|
||||
//
|
||||
// Proxied workspaces do not consult the host: proxiedAutoBackupBackend
|
||||
// has already required managed-local, where bd spawned the server on
|
||||
// this filesystem itself.
|
||||
if !usesProxiedServer() && !clientServerShareFilesystem() {
|
||||
autoBackupSkipNoticeOnce.Do(func() {
|
||||
if !isQuiet() && !jsonOutput {
|
||||
fmt.Fprintln(os.Stderr,
|
||||
|
|
@ -190,7 +201,7 @@ func maybeAutoBackup(ctx context.Context) {
|
|||
}
|
||||
|
||||
// Change detection: skip if nothing changed
|
||||
currentCommit, err := store.GetCurrentCommit(ctx)
|
||||
currentCommit, err := backend.CurrentCommit(ctx)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Warning: auto-backup skipped: failed to get current commit: %v\n", err)
|
||||
return
|
||||
|
|
@ -200,8 +211,21 @@ func maybeAutoBackup(ctx context.Context) {
|
|||
return
|
||||
}
|
||||
|
||||
// One backup at a time per workspace (backup_lock.go). Auto-backup never
|
||||
// waits: a held lock means a backup is already running.
|
||||
release, err := acquireBackupLock(0)
|
||||
if err != nil {
|
||||
if errors.Is(err, errBackupBusy) {
|
||||
debug.Logf("backup: skipping — another backup is running\n")
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(os.Stderr, "Warning: auto-backup skipped: %v\n", err)
|
||||
return
|
||||
}
|
||||
defer release()
|
||||
|
||||
// Run the backup (force=true since we already checked change detection above)
|
||||
if _, err := runBackupExport(ctx, true); err != nil {
|
||||
if _, err := runBackupExport(ctx, backend, true); err != nil {
|
||||
if !isQuiet() && !jsonOutput {
|
||||
fmt.Fprintf(os.Stderr, "Warning: auto-backup failed: %v\n", err)
|
||||
}
|
||||
|
|
|
|||
240
cmd/bd/backup_auto_proxied_local_integration_test.go
Normal file
240
cmd/bd/backup_auto_proxied_local_integration_test.go
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
//go:build cgo && unix
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/storage/dbproxy/proxy"
|
||||
"github.com/steveyegge/beads/internal/storage/dbproxy/server"
|
||||
"github.com/steveyegge/beads/internal/workspacegate"
|
||||
)
|
||||
|
||||
// autoBackupOptIn is the explicit opt-in a server-mode workspace needs, with
|
||||
// the throttle collapsed so every command in a test is due.
|
||||
var autoBackupOptIn = []string{"BD_BACKUP_ENABLED=true", "BD_BACKUP_INTERVAL=1ms"}
|
||||
|
||||
// TestManagedLocalProxiedAutoBackup is the claim this slice makes: on a
|
||||
// managed-local proxied workspace, an explicit backup.enabled=true makes the
|
||||
// post-command hook take a Dolt-native backup into .beads/backup, exactly as
|
||||
// it does in direct mode — and that backup restores.
|
||||
//
|
||||
// Like the explicit-verb round trip next door, the proof is a restore, not an
|
||||
// exit code: an auto-backup that wrote a manifest nobody can restore from would
|
||||
// pass every other assertion here.
|
||||
//
|
||||
// Named TestManagedLocalProxied* so the proxied-local smoke lane runs it.
|
||||
func TestManagedLocalProxiedAutoBackup(t *testing.T) {
|
||||
requireManagedLocalProxiedEnv(t)
|
||||
|
||||
bd := buildEmbeddedBD(t)
|
||||
p := bdManagedLocalInit(t, bd, "bkauto", 5*time.Minute)
|
||||
autoDir := filepath.Join(p.beadsDir, "backup")
|
||||
manifest := filepath.Join(autoDir, "manifest")
|
||||
|
||||
// Server mode defaults auto-backup OFF: many clients share one server.
|
||||
// Nothing may be written without the opt-in.
|
||||
bdProxiedCreate(t, bd, p.dir, "no opt-in, no backup", "-p", "2")
|
||||
if _, err := os.Stat(manifest); err == nil {
|
||||
t.Fatalf("auto-backup ran without an opt-in: %s exists", manifest)
|
||||
}
|
||||
|
||||
// With the opt-in, one ordinary write is enough.
|
||||
created := proxiedAutoBackupCreate(t, bd, p, "captured by auto-backup")
|
||||
if _, err := os.Stat(manifest); err != nil {
|
||||
t.Fatalf("opted-in auto-backup produced no manifest at %s: %v", manifest, err)
|
||||
}
|
||||
state := readProxiedAutoBackupState(t, bd, p)
|
||||
if state.LastDoltCommit == "" || state.Timestamp.IsZero() {
|
||||
t.Fatalf("auto-backup did not record its watermark: %+v", state)
|
||||
}
|
||||
wantIDs := proxiedIssueIDs(t, bd, p)
|
||||
if !slices.Contains(wantIDs, created) {
|
||||
t.Fatalf("precondition: %s missing from %v", created, wantIDs)
|
||||
}
|
||||
|
||||
// A read-only command with nothing new to capture must not advance the
|
||||
// watermark: change detection runs before the backup, as in direct mode.
|
||||
if _, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, autoBackupOptIn, "list", "--json"); err != nil {
|
||||
t.Fatalf("bd list with auto-backup opted in: %v\n%s", err, stderr)
|
||||
}
|
||||
if again := readProxiedAutoBackupState(t, bd, p); again.LastDoltCommit != state.LastDoltCommit {
|
||||
t.Fatalf("auto-backup advanced with no new commit: %q -> %q", state.LastDoltCommit, again.LastDoltCommit)
|
||||
}
|
||||
|
||||
// `bd dolt stop` opens no provider, so its post-run hook has nothing to
|
||||
// back up — and must not relaunch the server it just stopped to find out.
|
||||
if _, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, autoBackupOptIn, "dolt", "stop"); err != nil {
|
||||
t.Fatalf("bd dolt stop: %v\n%s", err, stderr)
|
||||
}
|
||||
assertManagedProxiedQuiescent(t, p, "bd dolt stop with auto-backup opted in")
|
||||
|
||||
// Diverge from the backup, then restore it and compare.
|
||||
after := bdProxiedCreate(t, bd, p.dir, "created after the auto-backup", "-p", "2")
|
||||
out, err := bdProxiedRun(t, bd, p.dir, "backup", "restore", autoDir, "--force", "--json")
|
||||
if err != nil {
|
||||
t.Fatalf("bd backup restore %s --force: %v\n%s", autoDir, err, out)
|
||||
}
|
||||
gotIDs := proxiedIssueIDs(t, bd, p)
|
||||
if slices.Contains(gotIDs, after.ID) {
|
||||
t.Fatalf("restore kept %s, created after the auto-backup; got %v", after.ID, gotIDs)
|
||||
}
|
||||
if !slices.Equal(gotIDs, wantIDs) {
|
||||
t.Fatalf("restored issues differ from the auto-backed-up set:\n got %v\n want %v", gotIDs, wantIDs)
|
||||
}
|
||||
}
|
||||
|
||||
// TestManagedLocalProxiedBackupRestoreRefusedWhileAttached pins the restore
|
||||
// design choice: restore REPLACES the database, so it refuses while any other
|
||||
// bd command is attached to the workspace, and in that case touches nothing —
|
||||
// the serving child keeps running and the data stays as it was.
|
||||
//
|
||||
// "Attached" is what another bd process holds for its lifetime: a SHARED
|
||||
// workspace gate. The test holds exactly that, in-process, rather than racing
|
||||
// a second process into position.
|
||||
func TestManagedLocalProxiedBackupRestoreRefusedWhileAttached(t *testing.T) {
|
||||
requireManagedLocalProxiedEnv(t)
|
||||
|
||||
bd := buildEmbeddedBD(t)
|
||||
p := bdManagedLocalInit(t, bd, "bkatt", 5*time.Minute)
|
||||
|
||||
bdProxiedCreate(t, bd, p.dir, "in the backup", "-p", "1")
|
||||
dest := filepath.Join(t.TempDir(), "dolt-backup")
|
||||
if out, err := bdProxiedRun(t, bd, p.dir, "backup", "init", dest); err != nil {
|
||||
t.Fatalf("bd backup init: %v\n%s", err, out)
|
||||
}
|
||||
if out, err := bdProxiedRun(t, bd, p.dir, "backup", "sync"); err != nil {
|
||||
t.Fatalf("bd backup sync: %v\n%s", err, out)
|
||||
}
|
||||
after := bdProxiedCreate(t, bd, p.dir, "not in the backup", "-p", "1")
|
||||
before := proxiedIssueIDs(t, bd, p)
|
||||
|
||||
proxyPID := readManagedProxyPidFile(t, p)
|
||||
backendPID := readManagedBackendPidFile(t, p)
|
||||
if proxyPID == nil || backendPID == nil {
|
||||
t.Fatal("precondition: the managed topology should be serving")
|
||||
}
|
||||
|
||||
gates, err := buildWorkspaceGateSet(p.beadsDir)
|
||||
if err != nil {
|
||||
t.Fatalf("resolve workspace gates: %v", err)
|
||||
}
|
||||
held, err := workspacegate.AcquireAll(context.Background(), workspacegate.Shared,
|
||||
workspacegate.Options{Reason: "test: attached bd client"}, gates...)
|
||||
if err != nil {
|
||||
t.Fatalf("hold the workspace gate: %v", err)
|
||||
}
|
||||
released := false
|
||||
release := func() {
|
||||
if !released {
|
||||
released = true
|
||||
if err := held.Release(); err != nil {
|
||||
t.Errorf("release workspace gate: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
t.Cleanup(release)
|
||||
|
||||
stdout, _, err := bdProxiedRunBuffers(t, bd, p.dir, "backup", "restore", dest, "--force", "--json")
|
||||
if err == nil {
|
||||
t.Fatalf("bd backup restore succeeded while another bd client was attached:\n%s", stdout)
|
||||
}
|
||||
var refusal struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
if jerr := json.Unmarshal([]byte(stdout[strings.Index(stdout, "{"):]), &refusal); jerr != nil {
|
||||
t.Fatalf("restore refusal is not JSON on stdout: %v\n%s", jerr, stdout)
|
||||
}
|
||||
if !strings.Contains(refusal.Error, "other bd commands are using this workspace") {
|
||||
t.Fatalf("restore refusal does not say why: %q", refusal.Error)
|
||||
}
|
||||
|
||||
// Refused means untouched: the serving topology is the same processes.
|
||||
if pf := readManagedProxyPidFile(t, p); pf == nil || pf.Pid != proxyPID.Pid || !processAlive(pf.Pid) {
|
||||
t.Fatalf("refused restore disturbed the proxy: before %+v, now %+v", proxyPID, pf)
|
||||
}
|
||||
if pf := readManagedBackendPidFile(t, p); pf == nil || pf.Pid != backendPID.Pid || !processAlive(pf.Pid) {
|
||||
t.Fatalf("refused restore disturbed the dolt child: before %+v, now %+v", backendPID, pf)
|
||||
}
|
||||
|
||||
release()
|
||||
if got := proxiedIssueIDs(t, bd, p); !slices.Equal(got, before) || !slices.Contains(got, after.ID) {
|
||||
t.Fatalf("refused restore changed the data:\n got %v\n want %v", got, before)
|
||||
}
|
||||
|
||||
// Once the other client detaches, the same restore goes through.
|
||||
if out, err := bdProxiedRun(t, bd, p.dir, "backup", "restore", dest, "--force", "--json"); err != nil {
|
||||
t.Fatalf("bd backup restore after the client detached: %v\n%s", err, out)
|
||||
}
|
||||
if slices.Contains(proxiedIssueIDs(t, bd, p), after.ID) {
|
||||
t.Fatalf("restore after detach did not replace the database: %s is still here", after.ID)
|
||||
}
|
||||
}
|
||||
|
||||
// proxiedAutoBackupCreate creates an issue with auto-backup opted in and
|
||||
// returns its ID.
|
||||
func proxiedAutoBackupCreate(t *testing.T, bd string, p proxiedProject, title string) string {
|
||||
t.Helper()
|
||||
stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, autoBackupOptIn, "create", title, "-p", "2", "--silent")
|
||||
if err != nil {
|
||||
t.Fatalf("bd create with auto-backup opted in: %v\n%s", err, stderr)
|
||||
}
|
||||
if strings.Contains(stderr, "auto-backup") {
|
||||
t.Fatalf("auto-backup reported a problem:\n%s", stderr)
|
||||
}
|
||||
id := strings.TrimSpace(stdout)
|
||||
if id == "" {
|
||||
t.Fatalf("bd create --silent printed no ID; stderr:\n%s", stderr)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
// readProxiedAutoBackupState reads the auto-backup watermark through
|
||||
// `bd backup status --json`, the surface an operator uses.
|
||||
func readProxiedAutoBackupState(t *testing.T, bd string, p proxiedProject) backupState {
|
||||
t.Helper()
|
||||
out, err := bdProxiedRun(t, bd, p.dir, "backup", "status", "--json")
|
||||
if err != nil {
|
||||
t.Fatalf("bd backup status --json: %v\n%s", err, out)
|
||||
}
|
||||
start := strings.Index(string(out), "{")
|
||||
if start < 0 {
|
||||
t.Fatalf("bd backup status --json produced no JSON:\n%s", out)
|
||||
}
|
||||
var status struct {
|
||||
Backup backupState `json:"backup"`
|
||||
}
|
||||
if err := json.Unmarshal(out[start:], &status); err != nil {
|
||||
t.Fatalf("parse backup status JSON: %v\n%s", err, out)
|
||||
}
|
||||
return status.Backup
|
||||
}
|
||||
|
||||
// proxiedIssueIDs lists every issue ID in the workspace, sorted.
|
||||
func proxiedIssueIDs(t *testing.T, bd string, p proxiedProject) []string {
|
||||
t.Helper()
|
||||
var ids []string
|
||||
for _, issue := range bdProxiedListJSON(t, bd, p, "--all") {
|
||||
ids = append(ids, issue.ID)
|
||||
}
|
||||
slices.Sort(ids)
|
||||
return ids
|
||||
}
|
||||
|
||||
// assertManagedProxiedQuiescent fails when either managed-local pidfile is
|
||||
// present, i.e. something relaunched the topology.
|
||||
func assertManagedProxiedQuiescent(t *testing.T, p proxiedProject, after string) {
|
||||
t.Helper()
|
||||
for _, name := range []string{proxy.PIDFileName, server.PIDFileName} {
|
||||
if _, err := os.Stat(filepath.Join(p.proxyRoot, name)); err == nil {
|
||||
t.Fatalf("after %s, %s is present: the topology was relaunched", after, name)
|
||||
}
|
||||
}
|
||||
}
|
||||
142
cmd/bd/backup_auto_proxied_test.go
Normal file
142
cmd/bd/backup_auto_proxied_test.go
Normal file
|
|
@ -0,0 +1,142 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"github.com/steveyegge/beads/internal/configfile"
|
||||
"github.com/steveyegge/beads/internal/storage/uow"
|
||||
)
|
||||
|
||||
// fakeMaintenanceProvider is a proxied provider stand-in exposing the
|
||||
// non-transactional seam proxied auto-backup runs on. It never opens a
|
||||
// connection; the tests here only assert whether it would be used.
|
||||
type fakeMaintenanceProvider struct {
|
||||
uow.UnitOfWorkProvider
|
||||
runs int
|
||||
}
|
||||
|
||||
func (f *fakeMaintenanceProvider) RunNonTx(context.Context, func(context.Context, *sql.Conn) error) error {
|
||||
f.runs++
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeMaintenanceProvider) Close(context.Context) error { return nil }
|
||||
|
||||
// TestProxiedAutoBackupBackendFollowsLocality pins where auto-backup may run on
|
||||
// a proxied workspace: exactly where `bd backup sync` is honored. Auto-backup
|
||||
// registers a server-side backup remote, which on a server bd does not own is
|
||||
// global to every client of it — the storm the explicit verbs refuse by design
|
||||
// — so an implicit trigger must not get further than the explicit one. An
|
||||
// unreadable sidecar resolves to the unknown topology and fails closed.
|
||||
//
|
||||
// Cannot be parallel: mutates the proxiedServerMode and uowProvider globals.
|
||||
func TestProxiedAutoBackupBackendFollowsLocality(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
sidecar string
|
||||
noProvider bool
|
||||
wantBackend bool
|
||||
}{
|
||||
{name: "managed-local with an open provider", sidecar: `{}`, wantBackend: true},
|
||||
{name: "managed-local with no provider open", sidecar: `{}`, noProvider: true},
|
||||
{name: "external tcp", sidecar: `{"external":{"host":"db.example.com","port":3306}}`},
|
||||
{name: "external unix socket", sidecar: `{"external":{"socket":"/tmp/dolt.sock"}}`},
|
||||
{name: "unreadable sidecar", sidecar: `{not json`},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
beadsDir := prepareBackupStatusTest(t)
|
||||
proxiedServerMode = true
|
||||
if err := os.WriteFile(configfile.ProxiedServerClientInfoPath(beadsDir), []byte(tc.sidecar), 0o600); err != nil {
|
||||
t.Fatalf("write proxied sidecar: %v", err)
|
||||
}
|
||||
|
||||
oldProvider := uowProvider
|
||||
t.Cleanup(func() { uowProvider = oldProvider })
|
||||
fake := &fakeMaintenanceProvider{}
|
||||
uowProvider = fake
|
||||
if tc.noProvider {
|
||||
uowProvider = nil
|
||||
}
|
||||
|
||||
backend, ok := autoBackupBackendForCommand()
|
||||
if ok != tc.wantBackend {
|
||||
t.Fatalf("autoBackupBackendForCommand() ok = %v, want %v", ok, tc.wantBackend)
|
||||
}
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if _, isProxied := backend.(proxiedLocalBackup); !isProxied {
|
||||
t.Fatalf("backend = %T, want proxiedLocalBackup", backend)
|
||||
}
|
||||
if fake.runs != 0 {
|
||||
t.Fatalf("resolving the backend ran %d maintenance call(s); it must not touch the server", fake.runs)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestPersistentPostRunProxiedRunsAutoBackup pins the wiring: the proxied arm
|
||||
// of PersistentPostRunE calls the auto-backup hook, under the same gate the
|
||||
// direct arm's maintenance net uses, plus previews.
|
||||
//
|
||||
// Before this hook existed the proxied arm only pruned the journal and closed
|
||||
// the provider, so backup.enabled=true was inert on every proxied workspace.
|
||||
//
|
||||
// Cannot be parallel: mutates PersistentPostRunE's package globals.
|
||||
func TestPersistentPostRunProxiedRunsAutoBackup(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
cmd func() *cobra.Command
|
||||
readonly bool
|
||||
want int
|
||||
}{
|
||||
{name: "ordinary command", cmd: func() *cobra.Command { return &cobra.Command{Use: "list"} }, want: 1},
|
||||
{name: "strict readonly", cmd: func() *cobra.Command { return &cobra.Command{Use: "list"} }, readonly: true},
|
||||
{name: "bd serve", cmd: func() *cobra.Command { return &cobra.Command{Use: serveCmdName} }},
|
||||
{name: "preview", cmd: func() *cobra.Command {
|
||||
c := &cobra.Command{Use: "list"}
|
||||
c.Flags().Bool("dry-run", false, "")
|
||||
if err := c.Flags().Set("dry-run", "true"); err != nil {
|
||||
t.Fatalf("set --dry-run: %v", err)
|
||||
}
|
||||
return c
|
||||
}},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
t.Chdir(t.TempDir())
|
||||
t.Setenv("BEADS_DIR", "")
|
||||
|
||||
oldStore, oldProvider := store, uowProvider
|
||||
oldReadonly, oldProxied := readonlyMode, proxiedServerMode
|
||||
oldRootCtx, oldRootCancel := rootCtx, rootCancel
|
||||
oldSpan, oldProfile, oldTrace := commandSpan, profileFile, traceFile
|
||||
oldBackup := runPostRunAutoBackup
|
||||
t.Cleanup(func() {
|
||||
store, uowProvider = oldStore, oldProvider
|
||||
readonlyMode, proxiedServerMode = oldReadonly, oldProxied
|
||||
rootCtx, rootCancel = oldRootCtx, oldRootCancel
|
||||
commandSpan, profileFile, traceFile = oldSpan, oldProfile, oldTrace
|
||||
runPostRunAutoBackup = oldBackup
|
||||
})
|
||||
|
||||
calls := 0
|
||||
runPostRunAutoBackup = func(context.Context) { calls++ }
|
||||
store, uowProvider = nil, nil
|
||||
readonlyMode = tc.readonly
|
||||
proxiedServerMode = true
|
||||
rootCtx, rootCancel = context.Background(), nil
|
||||
commandSpan, profileFile, traceFile = nil, nil, nil
|
||||
|
||||
if err := rootCmd.PersistentPostRunE(tc.cmd(), nil); err != nil {
|
||||
t.Fatalf("PersistentPostRunE: %v", err)
|
||||
}
|
||||
if calls != tc.want {
|
||||
t.Fatalf("auto-backup hook calls = %d, want %d", calls, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -133,9 +133,9 @@ func TestIsBackupAutoEnabled(t *testing.T) {
|
|||
// status` reachable on a proxied server for the first time, where it rendered
|
||||
// the embedded-mode footnote "(auto: no git remote)". On any sql-server shape
|
||||
// the value is false because usesSQLServer() short-circuits, whether or not a
|
||||
// remote exists, so that reason was simply untrue; and on proxied specifically
|
||||
// even an explicit backup.enabled=true changes nothing, because the proxied
|
||||
// arm of PersistentPostRunE never calls runPostRunAutoBackup.
|
||||
// remote exists, so that reason was simply untrue. Since the proxied post-run
|
||||
// arm gained its auto-backup hook, an explicit opt-in on proxied is honored
|
||||
// and needs no caveat.
|
||||
//
|
||||
// Cannot be parallel: mutates the proxiedServerMode global, primeHasGitRemote
|
||||
// and env vars.
|
||||
|
|
@ -191,23 +191,25 @@ func TestBackupAutoStatusNote(t *testing.T) {
|
|||
wantNote: "",
|
||||
},
|
||||
{
|
||||
name: "proxied default → names the missing hook, not the remote",
|
||||
// Proxied is server mode: the default is off whether or not a
|
||||
// remote exists, and the note names the opt-in that now works.
|
||||
name: "proxied default → off, names the opt-in, not the remote",
|
||||
envVal: "\x00",
|
||||
hasRemote: true,
|
||||
proxied: true,
|
||||
wantEnabled: false,
|
||||
wantNote: "auto-backup does not run on proxied-server; use 'bd backup sync'",
|
||||
wantNote: "auto: off in proxied-server mode; set backup.enabled=true to opt in",
|
||||
},
|
||||
{
|
||||
// The dead opt-in: enabled=true with nothing to run it. The note
|
||||
// must survive an explicit source here, or status reports a bare
|
||||
// "enabled=true" and the operator believes backups are happening.
|
||||
name: "proxied explicit opt-in → still inert, still annotated",
|
||||
// The proxied post-run arm runs auto-backup on managed-local (the
|
||||
// only proxied shape status renders on), so an explicit opt-in is
|
||||
// real there and needs no caveat.
|
||||
name: "proxied explicit opt-in → honored, no note",
|
||||
envVal: "true",
|
||||
hasRemote: false,
|
||||
proxied: true,
|
||||
wantEnabled: true,
|
||||
wantNote: "auto-backup does not run on proxied-server; use 'bd backup sync'",
|
||||
wantNote: "",
|
||||
},
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -142,6 +142,20 @@ Run 'bd backup init <path>' first to configure a destination.`,
|
|||
}()
|
||||
|
||||
ctx := rootCtx
|
||||
if usesProxiedServer() {
|
||||
if err := requireLocalProxiedBackup("backup sync"); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
release, err := acquireBackupLock(backupLockWait)
|
||||
if err != nil {
|
||||
if errors.Is(err, errBackupBusy) {
|
||||
return HandleErrorRespectJSON("%v; retry when it completes", err)
|
||||
}
|
||||
return HandleErrorRespectJSON("%v", err)
|
||||
}
|
||||
defer release()
|
||||
|
||||
if usesProxiedServer() {
|
||||
return runBackupSyncProxied(ctx)
|
||||
}
|
||||
|
|
@ -453,7 +467,7 @@ backup configuration. The backup data at the destination is not deleted.`,
|
|||
}
|
||||
|
||||
// Also remove backup_export if it exists (auto-export may have created it at same URL)
|
||||
_ = bs.BackupRemove(ctx, "backup_export")
|
||||
_ = bs.BackupRemove(ctx, versioncontrolops.ExportBackupName)
|
||||
|
||||
// Remove local config
|
||||
if path, err := doltBackupConfigPath(); err == nil {
|
||||
|
|
|
|||
|
|
@ -122,9 +122,35 @@ func atomicWriteFile(path string, data []byte) error {
|
|||
return nil
|
||||
}
|
||||
|
||||
// localBackupBackend is what a Dolt-native backup into a local directory needs
|
||||
// from storage: the commit it would capture, and the backup itself. The direct
|
||||
// store supplies one (directLocalBackup) and so does the proxied provider
|
||||
// (proxiedLocalBackup, backup_proxied_server.go).
|
||||
type localBackupBackend interface {
|
||||
CurrentCommit(ctx context.Context) (string, error)
|
||||
BackupToDir(ctx context.Context, dir string) error
|
||||
}
|
||||
|
||||
// directLocalBackup is localBackupBackend over an embedded or sql-server store.
|
||||
type directLocalBackup struct {
|
||||
store storage.DoltStorage
|
||||
}
|
||||
|
||||
func (b directLocalBackup) CurrentCommit(ctx context.Context) (string, error) {
|
||||
return b.store.GetCurrentCommit(ctx)
|
||||
}
|
||||
|
||||
func (b directLocalBackup) BackupToDir(ctx context.Context, dir string) error {
|
||||
bs, ok := storage.UnwrapStore(b.store).(storage.BackupStore)
|
||||
if !ok {
|
||||
return fmt.Errorf("storage backend does not support backup operations")
|
||||
}
|
||||
return bs.BackupDatabase(ctx, dir)
|
||||
}
|
||||
|
||||
// runBackupExport performs a Dolt-native backup to .beads/backup/.
|
||||
// Returns the updated state.
|
||||
func runBackupExport(ctx context.Context, force bool) (*backupState, error) {
|
||||
func runBackupExport(ctx context.Context, backend localBackupBackend, force bool) (*backupState, error) {
|
||||
dir, err := backupDir()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -137,7 +163,7 @@ func runBackupExport(ctx context.Context, force bool) (*backupState, error) {
|
|||
|
||||
// Change detection: skip if nothing changed (unless forced)
|
||||
if !force {
|
||||
currentCommit, err := store.GetCurrentCommit(ctx)
|
||||
currentCommit, err := backend.CurrentCommit(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get current commit: %w", err)
|
||||
}
|
||||
|
|
@ -147,12 +173,7 @@ func runBackupExport(ctx context.Context, force bool) (*backupState, error) {
|
|||
}
|
||||
}
|
||||
|
||||
bs, ok := storage.UnwrapStore(store).(storage.BackupStore)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("storage backend does not support backup operations")
|
||||
}
|
||||
|
||||
if err := bs.BackupDatabase(ctx, dir); err != nil {
|
||||
if err := backend.BackupToDir(ctx, dir); err != nil {
|
||||
// Persist the attempt time even on failure so the throttle
|
||||
// interval (checked by maybeAutoBackup via state.Timestamp)
|
||||
// applies to the next command. Without this, a sync that keeps
|
||||
|
|
@ -170,7 +191,7 @@ func runBackupExport(ctx context.Context, force bool) (*backupState, error) {
|
|||
}
|
||||
|
||||
// Update watermarks
|
||||
currentCommit, err := store.GetCurrentCommit(ctx)
|
||||
currentCommit, err := backend.CurrentCommit(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get current commit for state: %w", err)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -63,13 +63,10 @@ func TestRunBackupExport_PersistsThrottleOnFailure(t *testing.T) {
|
|||
t.Fatalf("precondition: expected zero timestamp, got %v", st.Timestamp)
|
||||
}
|
||||
|
||||
oldStore := store
|
||||
fake := &failingBackupStore{commit: "deadbeef", backupErr: errors.New("sync to backup: server too busy")}
|
||||
store = fake
|
||||
t.Cleanup(func() { store = oldStore })
|
||||
|
||||
before := time.Now().UTC()
|
||||
if _, err := runBackupExport(context.Background(), true); err == nil {
|
||||
if _, err := runBackupExport(context.Background(), directLocalBackup{store: fake}, true); err == nil {
|
||||
t.Fatal("expected runBackupExport to return the sync error, got nil")
|
||||
}
|
||||
if fake.backupCalls != 1 {
|
||||
|
|
|
|||
71
cmd/bd/backup_lock.go
Normal file
71
cmd/bd/backup_lock.go
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/steveyegge/beads/internal/beads"
|
||||
"github.com/steveyegge/beads/internal/lockfile"
|
||||
)
|
||||
|
||||
// The backup lock serializes the two writers of a workspace's Dolt backup:
|
||||
// post-command auto-backup and an explicit `bd backup sync`. Both run under the
|
||||
// command's SHARED workspace gate, so the gate alone lets them overlap, and the
|
||||
// auto-backup body does remove → add → sync on one fixed remote name — two
|
||||
// concurrent runs race each other's registration and full-sync the database
|
||||
// twice. Restore needs no part in it: it holds the workspace gate EXCLUSIVELY,
|
||||
// which already excludes every command that could take this lock.
|
||||
//
|
||||
// The two callers wait differently, and deliberately:
|
||||
// - auto-backup does not wait. Another backup running means one is already
|
||||
// happening; it skips, and the throttle picks it up on a later command.
|
||||
// - `bd backup sync` waits up to backupLockWait — the same bound an exclusive
|
||||
// workspace-gate acquisition (restore) uses — and then fails with an error
|
||||
// naming the busy lock, so an operator is never told it synced when it
|
||||
// did not.
|
||||
const backupLockFileName = "backup.lock"
|
||||
|
||||
// backupLockWait bounds how long `bd backup sync` waits for another backup.
|
||||
// A var so tests can shorten it.
|
||||
var backupLockWait = exclusiveGateWait
|
||||
|
||||
// errBackupBusy is returned when another backup holds the lock past the wait.
|
||||
var errBackupBusy = errors.New("another backup is running for this workspace")
|
||||
|
||||
const backupLockPoll = 50 * time.Millisecond
|
||||
|
||||
// acquireBackupLock takes the workspace's backup lock, polling up to wait
|
||||
// (zero means one non-blocking attempt). The returned release must be called.
|
||||
func acquireBackupLock(wait time.Duration) (release func(), err error) {
|
||||
beadsDir := beads.FindBeadsDir()
|
||||
if beadsDir == "" {
|
||||
return nil, fmt.Errorf("backup lock: %s", activeWorkspaceNotFoundError())
|
||||
}
|
||||
path := filepath.Join(beadsDir, backupLockFileName)
|
||||
f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0o600) //nolint:gosec // path is the workspace's own lock file
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("backup lock: open %s: %w", path, err)
|
||||
}
|
||||
deadline := time.Now().Add(wait)
|
||||
for {
|
||||
err := lockfile.FlockExclusiveNonBlocking(f)
|
||||
if err == nil {
|
||||
return func() {
|
||||
_ = lockfile.FlockUnlock(f)
|
||||
_ = f.Close()
|
||||
}, nil
|
||||
}
|
||||
if !lockfile.IsLocked(err) {
|
||||
_ = f.Close()
|
||||
return nil, fmt.Errorf("backup lock: %s: %w", path, err)
|
||||
}
|
||||
if !time.Now().Before(deadline) {
|
||||
_ = f.Close()
|
||||
return nil, errBackupBusy
|
||||
}
|
||||
time.Sleep(backupLockPoll)
|
||||
}
|
||||
}
|
||||
133
cmd/bd/backup_lock_test.go
Normal file
133
cmd/bd/backup_lock_test.go
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestBackupLockSerializesBackups pins the lock itself: a second holder is
|
||||
// refused while the first holds it, a bounded wait gives up with errBackupBusy,
|
||||
// and releasing makes it available again.
|
||||
//
|
||||
// Cannot be parallel: prepareBackupStatusTest mutates package globals.
|
||||
func TestBackupLockSerializesBackups(t *testing.T) {
|
||||
prepareBackupStatusTest(t)
|
||||
|
||||
release, err := acquireBackupLock(0)
|
||||
if err != nil {
|
||||
t.Fatalf("first acquisition: %v", err)
|
||||
}
|
||||
if _, err := acquireBackupLock(0); !errors.Is(err, errBackupBusy) {
|
||||
t.Fatalf("non-blocking acquisition while held = %v, want errBackupBusy", err)
|
||||
}
|
||||
start := time.Now()
|
||||
if _, err := acquireBackupLock(150 * time.Millisecond); !errors.Is(err, errBackupBusy) {
|
||||
t.Fatalf("bounded wait while held = %v, want errBackupBusy", err)
|
||||
}
|
||||
if waited := time.Since(start); waited < 100*time.Millisecond {
|
||||
t.Fatalf("bounded wait returned after %s; it must poll for the wait it was given", waited)
|
||||
}
|
||||
release()
|
||||
|
||||
again, err := acquireBackupLock(0)
|
||||
if err != nil {
|
||||
t.Fatalf("acquisition after release: %v", err)
|
||||
}
|
||||
again()
|
||||
}
|
||||
|
||||
// TestAutoBackupSkipsWhileAnotherBackupRuns: auto-backup never waits on the
|
||||
// lock. With it held the backup does not run and the throttle state is left
|
||||
// alone, so the next command retries; once released, the same call backs up.
|
||||
func TestAutoBackupSkipsWhileAnotherBackupRuns(t *testing.T) {
|
||||
prepareBackupStatusTest(t)
|
||||
t.Setenv("BD_BACKUP_ENABLED", "true")
|
||||
t.Setenv("BD_BACKUP_INTERVAL", "1ms")
|
||||
t.Setenv("BD_GIT_HOOK", "")
|
||||
initConfigForTest(t)
|
||||
|
||||
oldStore := store
|
||||
fake := &failingBackupStore{commit: "c1"}
|
||||
store = fake
|
||||
t.Cleanup(func() { store = oldStore })
|
||||
|
||||
release, err := acquireBackupLock(0)
|
||||
if err != nil {
|
||||
t.Fatalf("hold the backup lock: %v", err)
|
||||
}
|
||||
maybeAutoBackup(context.Background())
|
||||
if fake.backupCalls != 0 {
|
||||
t.Fatalf("auto-backup ran %d backup(s) while another backup held the lock", fake.backupCalls)
|
||||
}
|
||||
dir, err := backupDir()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if st, _ := loadBackupState(dir); !st.Timestamp.IsZero() {
|
||||
t.Fatalf("a skipped auto-backup advanced the throttle to %v", st.Timestamp)
|
||||
}
|
||||
|
||||
release()
|
||||
maybeAutoBackup(context.Background())
|
||||
if fake.backupCalls != 1 {
|
||||
t.Fatalf("auto-backup ran %d backup(s) after the lock was released, want 1", fake.backupCalls)
|
||||
}
|
||||
}
|
||||
|
||||
// TestBackupSyncFailsWhenAnotherBackupHoldsTheLock: an explicit sync waits
|
||||
// backupLockWait for another backup and then fails loudly, before touching the
|
||||
// store — it never reports a sync it did not do.
|
||||
func TestBackupSyncFailsWhenAnotherBackupHoldsTheLock(t *testing.T) {
|
||||
prepareBackupStatusTest(t)
|
||||
oldWait := backupLockWait
|
||||
backupLockWait = 100 * time.Millisecond
|
||||
t.Cleanup(func() { backupLockWait = oldWait })
|
||||
|
||||
oldStore := store
|
||||
fake := &failingBackupStore{commit: "c1"}
|
||||
store = fake
|
||||
t.Cleanup(func() { store = oldStore })
|
||||
|
||||
release, err := acquireBackupLock(0)
|
||||
if err != nil {
|
||||
t.Fatalf("hold the backup lock: %v", err)
|
||||
}
|
||||
defer release()
|
||||
|
||||
var runErr error
|
||||
stderr := captureStderrForBackupLockTest(t, func() {
|
||||
runErr = backupSyncCmd.RunE(backupSyncCmd, nil)
|
||||
})
|
||||
if runErr == nil {
|
||||
t.Fatal("bd backup sync succeeded while another backup held the lock")
|
||||
}
|
||||
if !strings.Contains(stderr, "another backup is running") {
|
||||
t.Fatalf("sync refusal does not say why; stderr:\n%s", stderr)
|
||||
}
|
||||
if fake.backupCalls != 0 {
|
||||
t.Fatalf("refused sync still reached the store (%d backup calls)", fake.backupCalls)
|
||||
}
|
||||
}
|
||||
|
||||
func captureStderrForBackupLockTest(t *testing.T, fn func()) string {
|
||||
t.Helper()
|
||||
stdioMutex.Lock()
|
||||
defer stdioMutex.Unlock()
|
||||
r, w, err := os.Pipe()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := os.Stderr
|
||||
os.Stderr = w
|
||||
fn()
|
||||
os.Stderr = old
|
||||
_ = w.Close()
|
||||
buf := make([]byte, 1<<16)
|
||||
n, _ := r.Read(buf)
|
||||
_ = r.Close()
|
||||
return string(buf[:n])
|
||||
}
|
||||
|
|
@ -12,6 +12,7 @@ import (
|
|||
|
||||
"github.com/steveyegge/beads/internal/beads"
|
||||
"github.com/steveyegge/beads/internal/configfile"
|
||||
"github.com/steveyegge/beads/internal/debug"
|
||||
"github.com/steveyegge/beads/internal/storage"
|
||||
"github.com/steveyegge/beads/internal/storage/dbproxy/proxy"
|
||||
"github.com/steveyegge/beads/internal/storage/issueops"
|
||||
|
|
@ -47,19 +48,83 @@ const proxiedBackupTargetName = defaultDoltBackupName
|
|||
// Defense in depth, not a second policy: the rule and its wording come from the
|
||||
// same registry row the pre-provider gate consults, so the two cannot drift.
|
||||
func requireLocalProxiedBackup(path string) error {
|
||||
row, ok := LookupCapabilityRow(path, "")
|
||||
rule, ok := proxiedBackupRule(path)
|
||||
if !ok {
|
||||
// Unreachable: TestProxyCapabilityRegistryCoversCommandTree fails the
|
||||
// build for a backup path with no row. Refuse rather than guess.
|
||||
return HandleErrorRespectJSON("%s has no proxied-server capability row", path)
|
||||
}
|
||||
rule := row.ruleFor(resolveProxiedTopology(beads.FindBeadsDir()))
|
||||
if rule.Outcome == ProxyOutcomeHonored {
|
||||
return nil
|
||||
}
|
||||
return HandleProxyCapabilityError(proxyCapabilityErrorFor(rule))
|
||||
}
|
||||
|
||||
// proxiedBackupRule resolves the registry rule for a backup path on this
|
||||
// workspace's proxied topology. ok is false only for a path with no row.
|
||||
func proxiedBackupRule(path string) (proxyCapabilityRule, bool) {
|
||||
row, ok := LookupCapabilityRow(path, "")
|
||||
if !ok {
|
||||
return proxyCapabilityRule{}, false
|
||||
}
|
||||
return row.ruleFor(resolveProxiedTopology(beads.FindBeadsDir())), true
|
||||
}
|
||||
|
||||
// proxiedAutoBackupBackend returns the backend post-command auto-backup runs
|
||||
// against on a proxied-server workspace, or ok=false when it must not run.
|
||||
//
|
||||
// Auto-backup is a `bd backup sync` nobody typed, so it is honored exactly
|
||||
// where that verb is — the "backup sync" registry row, i.e. managed-local
|
||||
// only. Anywhere else the backup remote it registers would land on a server bd
|
||||
// does not own, where it is global to every client; that is the storm the
|
||||
// explicit verbs refuse by design, and an implicit trigger must not reopen it.
|
||||
//
|
||||
// It never opens a provider of its own. A command that did not open one (or
|
||||
// closed it, as restore does) has no live topology to back up, and opening one
|
||||
// here would relaunch a server the command may just have stopped.
|
||||
func proxiedAutoBackupBackend() (localBackupBackend, bool) {
|
||||
if rule, ok := proxiedBackupRule("backup sync"); !ok || rule.Outcome != ProxyOutcomeHonored {
|
||||
debug.Logf("backup: skipping auto-backup — proxied topology does not honor backup sync\n")
|
||||
return nil, false
|
||||
}
|
||||
if uowProvider == nil {
|
||||
return nil, false
|
||||
}
|
||||
mp, ok := uowProvider.(uow.MaintenanceProvider)
|
||||
if !ok {
|
||||
debug.Logf("backup: skipping auto-backup — provider %T has no maintenance seam\n", uowProvider)
|
||||
return nil, false
|
||||
}
|
||||
return proxiedLocalBackup{provider: mp}, true
|
||||
}
|
||||
|
||||
// proxiedLocalBackup is localBackupBackend over the proxied provider's
|
||||
// non-transactional seam, the one `bd backup sync` uses.
|
||||
//
|
||||
// No pre-backup commit: change detection compares HEAD, and every proxied
|
||||
// write transaction already ends in DOLT_COMMIT('-Am') (uow doltServerTx), so
|
||||
// HEAD is where the data is — the invariant direct mode gets from running
|
||||
// auto-commit just before auto-backup.
|
||||
type proxiedLocalBackup struct {
|
||||
provider uow.MaintenanceProvider
|
||||
}
|
||||
|
||||
func (b proxiedLocalBackup) CurrentCommit(ctx context.Context) (string, error) {
|
||||
var hash string
|
||||
err := b.provider.RunNonTx(ctx, func(ctx context.Context, conn *sql.Conn) error {
|
||||
var err error
|
||||
hash, err = versioncontrolops.CurrentCommit(ctx, conn)
|
||||
return err
|
||||
})
|
||||
return hash, err
|
||||
}
|
||||
|
||||
func (b proxiedLocalBackup) BackupToDir(ctx context.Context, dir string) error {
|
||||
return b.provider.RunNonTx(ctx, func(ctx context.Context, conn *sql.Conn) error {
|
||||
return versioncontrolops.BackupToDir(ctx, conn, conn, dir)
|
||||
})
|
||||
}
|
||||
|
||||
// proxiedActiveDatabase asks the server which database the provider opened,
|
||||
// rather than re-deriving it from the workspace config. The provider already
|
||||
// resolved that question (metadata.json, then --database/--db), and a second
|
||||
|
|
@ -203,7 +268,7 @@ func runBackupRemoveProxied(ctx context.Context) error {
|
|||
return err
|
||||
}
|
||||
// auto-export may have registered a second remote at the same URL.
|
||||
_ = versioncontrolops.BackupRemove(ctx, conn, "backup_export")
|
||||
_ = versioncontrolops.BackupRemove(ctx, conn, versioncontrolops.ExportBackupName)
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
|
|
|
|||
|
|
@ -2077,6 +2077,15 @@ var rootCmd = &cobra.Command{
|
|||
if shouldAutoPruneEventsJournal(cmd) {
|
||||
maybeAutoPruneEventsJournal(rootCtx, beads.FindBeadsDir())
|
||||
}
|
||||
// Auto-backup, through the provider this command opened. Same
|
||||
// gate as the direct arm's maintenance net below (strict
|
||||
// --readonly, `bd serve` and a migration freeze all skip it),
|
||||
// plus previews: a --dry-run must not register a backup remote
|
||||
// or write backup state. proxiedAutoBackupBackend decides the
|
||||
// topology, so this stays inert off managed-local.
|
||||
if runsPostCommandMaintenance(cmd.Name(), readonlyMode) && !isPreviewCommand(cmd) && !commandFreeze.Frozen() {
|
||||
runPostRunAutoBackup(rootCtx)
|
||||
}
|
||||
if uowProvider != nil {
|
||||
_ = uowProvider.Close(rootCtx)
|
||||
uowProvider = nil
|
||||
|
|
|
|||
|
|
@ -1453,44 +1453,12 @@ func (s *DoltStore) BackupRemove(ctx context.Context, name string) error {
|
|||
// BackupDatabase registers dir as a file:// Dolt backup remote and syncs
|
||||
// the full database to it, preserving complete commit history.
|
||||
func (s *DoltStore) BackupDatabase(ctx context.Context, dir string) error {
|
||||
info, err := os.Stat(dir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("backup destination does not exist: %w", err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return fmt.Errorf("backup destination is not a directory: %s", dir)
|
||||
}
|
||||
|
||||
backupURL, err := versioncontrolops.DirToFileURL(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
backupName := "backup_export"
|
||||
|
||||
syncDB, err := s.oneShotConn(0)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer syncDB.Close()
|
||||
|
||||
// Register as a backup remote (idempotent — remove first if exists).
|
||||
_ = versioncontrolops.BackupRemove(ctx, s.db, backupName)
|
||||
if err := versioncontrolops.BackupAdd(ctx, s.db, backupName, backupURL); err != nil {
|
||||
// Another backup (e.g. "default" registered by `bd backup init`) may
|
||||
// already point to this URL. In that case, sync using the existing
|
||||
// remote name rather than failing.
|
||||
if conflict := versioncontrolops.ExtractAddressConflictName(err); conflict != "" {
|
||||
if syncErr := versioncontrolops.BackupSync(ctx, syncDB, conflict); syncErr != nil {
|
||||
return fmt.Errorf("sync to backup: %w", syncErr)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("register backup remote: %w", err)
|
||||
}
|
||||
if err := versioncontrolops.BackupSync(ctx, syncDB, backupName); err != nil {
|
||||
return fmt.Errorf("sync to backup: %w", err)
|
||||
}
|
||||
return nil
|
||||
return versioncontrolops.BackupToDir(ctx, s.db, syncDB, dir)
|
||||
}
|
||||
|
||||
// RestoreDatabase restores the database from a Dolt backup at dir.
|
||||
|
|
|
|||
|
|
@ -188,12 +188,7 @@ func (s *DoltStore) ListBranches(ctx context.Context) ([]string, error) {
|
|||
// GetCurrentCommit returns the hash of the current HEAD commit.
|
||||
// Implements storage.VersionedStorage.
|
||||
func (s *DoltStore) GetCurrentCommit(ctx context.Context) (string, error) {
|
||||
var hash string
|
||||
err := s.db.QueryRowContext(ctx, "SELECT DOLT_HASHOF('HEAD')").Scan(&hash)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("failed to get current commit: %w", err)
|
||||
}
|
||||
return hash, nil
|
||||
return versioncontrolops.CurrentCommit(ctx, s.db)
|
||||
}
|
||||
|
||||
// GetStateHash returns a hash of the entire database including the working
|
||||
|
|
|
|||
|
|
@ -754,39 +754,8 @@ func (s *EmbeddedDoltStore) BackupRemove(ctx context.Context, name string) error
|
|||
// the database to it. The dir must exist locally. This preserves full Dolt
|
||||
// commit history.
|
||||
func (s *EmbeddedDoltStore) BackupDatabase(ctx context.Context, dir string) error {
|
||||
info, err := os.Stat(dir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("backup destination does not exist: %w", err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return fmt.Errorf("backup destination is not a directory: %s", dir)
|
||||
}
|
||||
|
||||
backupURL, err := versioncontrolops.DirToFileURL(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
backupName := "backup_export"
|
||||
|
||||
return s.withMutatingDBConn(ctx, func(db versioncontrolops.DBConn) error {
|
||||
// Register as a backup remote (idempotent — remove first if exists).
|
||||
_ = versioncontrolops.BackupRemove(ctx, db, backupName)
|
||||
if err := versioncontrolops.BackupAdd(ctx, db, backupName, backupURL); err != nil {
|
||||
// Another backup (e.g. "default" registered by `bd backup init`) may
|
||||
// already point to this URL. In that case, sync using the existing
|
||||
// remote name rather than failing.
|
||||
if conflict := versioncontrolops.ExtractAddressConflictName(err); conflict != "" {
|
||||
if syncErr := versioncontrolops.BackupSync(ctx, db, conflict); syncErr != nil {
|
||||
return fmt.Errorf("sync to backup: %w", syncErr)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("register backup remote: %w", err)
|
||||
}
|
||||
if err := versioncontrolops.BackupSync(ctx, db, backupName); err != nil {
|
||||
return fmt.Errorf("sync to backup: %w", err)
|
||||
}
|
||||
return nil
|
||||
return versioncontrolops.BackupToDir(ctx, db, db, dir)
|
||||
})
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -3,10 +3,16 @@ package versioncontrolops
|
|||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ExportBackupName is the Dolt backup remote the auto-backup path registers
|
||||
// for its local backup directory. `bd backup init` registers a different
|
||||
// name, so the two can coexist until they point at the same address.
|
||||
const ExportBackupName = "backup_export"
|
||||
|
||||
// BackupAdd registers a Dolt backup destination.
|
||||
func BackupAdd(ctx context.Context, db DBConn, name, url string) error {
|
||||
if _, err := db.ExecContext(ctx, "CALL DOLT_BACKUP('add', ?, ?)", name, url); err != nil {
|
||||
|
|
@ -47,6 +53,55 @@ func BackupRestore(ctx context.Context, db DBConn, url, dbName string, force boo
|
|||
return nil
|
||||
}
|
||||
|
||||
// CurrentCommit returns the hash of the current HEAD commit.
|
||||
func CurrentCommit(ctx context.Context, db DBConn) (string, error) {
|
||||
var hash string
|
||||
if err := db.QueryRowContext(ctx, "SELECT DOLT_HASHOF('HEAD')").Scan(&hash); err != nil {
|
||||
return "", fmt.Errorf("failed to get current commit: %w", err)
|
||||
}
|
||||
return hash, nil
|
||||
}
|
||||
|
||||
// BackupToDir registers dir as the ExportBackupName file:// backup remote and
|
||||
// syncs the full database to it, preserving complete commit history. dir must
|
||||
// already exist on this machine.
|
||||
//
|
||||
// register carries the remote add/remove; sync carries DOLT_BACKUP('sync'),
|
||||
// which streams the database and can outlive a pooled connection's read
|
||||
// deadline, so a caller that has one hands in a long-timeout connection there.
|
||||
// A caller with a single connection passes it for both.
|
||||
//
|
||||
// Registration is idempotent (remove, then add). When another remote — e.g.
|
||||
// the one `bd backup init` registered — already holds the same address, Dolt
|
||||
// refuses the add; the sync then goes through that remote's name instead.
|
||||
func BackupToDir(ctx context.Context, register, sync DBConn, dir string) error {
|
||||
info, err := os.Stat(dir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("backup destination does not exist: %w", err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
return fmt.Errorf("backup destination is not a directory: %s", dir)
|
||||
}
|
||||
backupURL, err := DirToFileURL(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
name := ExportBackupName
|
||||
_ = BackupRemove(ctx, register, name)
|
||||
if err := BackupAdd(ctx, register, name, backupURL); err != nil {
|
||||
conflict := ExtractAddressConflictName(err)
|
||||
if conflict == "" {
|
||||
return fmt.Errorf("register backup remote: %w", err)
|
||||
}
|
||||
name = conflict
|
||||
}
|
||||
if err := BackupSync(ctx, sync, name); err != nil {
|
||||
return fmt.Errorf("sync to backup: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// DirToFileURL resolves dir to an absolute path and returns a file:// URL.
|
||||
//
|
||||
// An input that already carries a scheme is rejected rather than prefixed.
|
||||
|
|
|
|||
|
|
@ -1,7 +1,10 @@
|
|||
package versioncontrolops
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
|
@ -74,3 +77,29 @@ func TestExtractAddressConflictName(t *testing.T) {
|
|||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestBackupToDirRefusesMissingOrFileDestination pins the local-directory
|
||||
// precondition every auto-backup caller (embedded, sql-server and proxied)
|
||||
// now shares. It fails before any SQL is issued, so nil connections are safe:
|
||||
// a regression that reached the server first would panic here.
|
||||
func TestBackupToDirRefusesMissingOrFileDestination(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
file := filepath.Join(dir, "not-a-dir")
|
||||
if err := os.WriteFile(file, nil, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
for _, tc := range []struct {
|
||||
name, dir, want string
|
||||
}{
|
||||
{name: "missing", dir: filepath.Join(dir, "absent"), want: "does not exist"},
|
||||
{name: "file", dir: file, want: "is not a directory"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
err := BackupToDir(context.Background(), nil, nil, tc.dir)
|
||||
if err == nil || !strings.Contains(err.Error(), tc.want) {
|
||||
t.Fatalf("BackupToDir(%q) error = %v, want one containing %q", tc.dir, err, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue