* feat(storage): add DoltGCFull for full, all-generation Dolt GC
Dolt GC is generational: every pass promotes the chunks reachable at that
moment into the old generation, and a default pass only visits the new
generation. Callers that rewrite history need a collection that revisits
the old generation too.
Adds versioncontrolops.DoltGCFull (CALL DOLT_GC('--full', '--archive-level',
'0')), the optional storage.FullGarbageCollector capability, and
implementations on both in-tree Dolt stores. No call sites change yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(gc): run a full Dolt GC after flatten and compact
The post-rewrite GC was Dolt's default, generational pass. Any earlier GC —
bd gc, or a previous flatten or compact — has already promoted the commit
chain into the old generation, which a default pass never revisits. The
rewrite orphans that chain, so the follow-up GC freed ~nothing on every
mature store and orphaned --as-of addresses kept resolving.
flatten, compact, and the proxied-server compact now finish with a full
collection, which frees everything a default pass frees plus the old
generation. Text mode prints a notice first (a full pass can take minutes on
large stores) and the JSON result reports gc_mode.
Unchanged: bd gc's own phase 3, bd admin compact --dolt, and the
managed-server shutdown GC all stay generational.
Fixes #5907
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(gc): add bd gc --full and hint when a default pass reclaims little
Periodic hygiene stays generational by default — a full pass costs minutes on
a large store and usually reclaims nothing extra. But phase 3 has flatten's
blind spot too: row data from decayed issues was promoted to the old
generation by an earlier GC, so a default pass cannot reclaim it. --full
selects a full collection on the embedded, server, and proxied paths.
When a default pass leaves a non-trivial store (>= 64 MiB) barely smaller
(< 1% freed), print a tip explaining the generational model and pointing at
--full. Unmeasurable sizes disable the hint.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(gc): cover suggestFullGC thresholds, GC help text, and bd gc --full
The threshold table includes the two field datapoints behind the
calibration: a large store that freed nothing hints, and the 1276 -> 971 MB
run that freed 24% does not.
The help-text assertions pin the wording in the three Cobra Long strings,
which are the source the CLI reference pages are generated from. One of them
also caught a line wrap that split "old generation" across a newline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: explain why a Dolt garbage collection can reclaim nothing
Adds a subsection to the Dolt backend page covering the generational model —
promotion to the old generation, why a default pass never revisits it, and
why a full collection is a strict superset — and extends the History Bloat
runbook's Prevention advice with an occasional full pass.
Both pages describe the pinned release (v1.2.2), so they stay on the dolt CLI
and do not name the bd behavior this branch adds; the generated CLI reference
picks that up when the pin is bumped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
301 lines
9.4 KiB
Go
301 lines
9.4 KiB
Go
package main
|
|
|
|
import (
|
|
"fmt"
|
|
"time"
|
|
|
|
"github.com/spf13/cobra"
|
|
"github.com/steveyegge/beads/internal/metrics"
|
|
"github.com/steveyegge/beads/internal/storage"
|
|
"github.com/steveyegge/beads/internal/types"
|
|
)
|
|
|
|
var (
|
|
gcDryRun bool
|
|
gcForce bool
|
|
gcOlderThan int
|
|
gcSkipDecay bool
|
|
gcSkipDolt bool
|
|
gcFull bool
|
|
)
|
|
|
|
var gcCmd = &cobra.Command{
|
|
Use: "gc",
|
|
GroupID: "maint",
|
|
Short: "Garbage collect: decay old issues, compact Dolt commits, run Dolt GC",
|
|
Long: `Full lifecycle garbage collection for standalone Beads databases.
|
|
|
|
Runs three phases in sequence:
|
|
1. DECAY — Delete closed issues older than N days (default 90)
|
|
2. COMPACT — Squash old Dolt commits into fewer commits (bd compact)
|
|
3. GC — Run Dolt garbage collection to reclaim disk space
|
|
|
|
Each phase can be skipped individually. Use --dry-run to preview all phases
|
|
without making changes.
|
|
|
|
Phase 3 runs Dolt's default, generational GC: it only examines data written
|
|
since the last GC. Data that survived an earlier GC lives in the old generation
|
|
and is never revisited, so on a long-lived store the space freed by decay may
|
|
not be reclaimed. Use --full to collect all generations (slower on large
|
|
stores).
|
|
|
|
Examples:
|
|
bd gc # Full GC with defaults (90 day decay)
|
|
bd gc --dry-run # Preview what would happen
|
|
bd gc --older-than 30 # Decay issues closed 30+ days ago
|
|
bd gc --skip-decay # Skip issue deletion, just compact+GC
|
|
bd gc --skip-dolt # Skip Dolt GC, just decay+compact
|
|
bd gc --full # Collect all storage generations
|
|
bd gc --force # Skip confirmation prompt`,
|
|
SilenceUsage: true,
|
|
SilenceErrors: true,
|
|
RunE: func(cmd *cobra.Command, _ []string) error {
|
|
if usesProxiedServer() {
|
|
return runGCProxiedServer(rootCtx)
|
|
}
|
|
evt := metrics.NewCommandEvent("gc")
|
|
defer func() {
|
|
if c := metrics.Global(); c != nil {
|
|
c.CloseEventAndAdd(evt)
|
|
}
|
|
}()
|
|
|
|
if !gcDryRun {
|
|
CheckReadonly("gc")
|
|
}
|
|
ctx := rootCtx
|
|
start := time.Now()
|
|
|
|
if gcOlderThan < 0 {
|
|
return HandleErrorRespectJSON("--older-than must be non-negative")
|
|
}
|
|
|
|
type phaseResult struct {
|
|
name string
|
|
skipped bool
|
|
detail string
|
|
}
|
|
var results []phaseResult
|
|
|
|
if gcSkipDecay {
|
|
results = append(results, phaseResult{name: "Decay", skipped: true})
|
|
} else {
|
|
if !jsonOutput {
|
|
fmt.Println("Phase 1/3: Decay (delete old closed issues)")
|
|
}
|
|
|
|
cutoffDays := gcOlderThan
|
|
cutoffTime := time.Now().UTC().AddDate(0, 0, -cutoffDays)
|
|
statusClosed := types.StatusClosed
|
|
// gc is a scripted internal sweep — opt out of BEADS_MAX_ROWS
|
|
// (designer §4.1) so a misconfigured env doesn't abort the sweep.
|
|
filter := types.IssueFilter{
|
|
Status: &statusClosed,
|
|
ClosedBefore: &cutoffTime,
|
|
MaxRows: 0,
|
|
MaxRowsSource: "",
|
|
}
|
|
|
|
closedIssues, err := store.SearchIssues(ctx, "", filter)
|
|
if err != nil {
|
|
return HandleErrorRespectJSON("searching closed issues: %v", err)
|
|
}
|
|
|
|
var stats closedDeletionCandidateStats
|
|
closedIssues, stats = filterClosedDeletionCandidates(closedIssues, &cutoffTime)
|
|
warnClosedDeletionSafetySkips(stats)
|
|
|
|
if len(closedIssues) == 0 {
|
|
detail := fmt.Sprintf(" No closed issues older than %d days", cutoffDays)
|
|
if !jsonOutput {
|
|
fmt.Println(detail)
|
|
}
|
|
results = append(results, phaseResult{name: "Decay", detail: "0 issues deleted"})
|
|
} else {
|
|
if gcDryRun {
|
|
detail := fmt.Sprintf(" Would delete %d closed issue(s)", len(closedIssues))
|
|
if !jsonOutput {
|
|
fmt.Println(detail)
|
|
}
|
|
results = append(results, phaseResult{name: "Decay", detail: fmt.Sprintf("%d issues (dry-run)", len(closedIssues))})
|
|
} else {
|
|
if !gcForce {
|
|
return HandleErrorWithHintRespectJSON(
|
|
fmt.Sprintf("would delete %d closed issue(s) older than %d days", len(closedIssues), cutoffDays),
|
|
"Use --force to confirm or --dry-run to preview.")
|
|
}
|
|
|
|
deleted := 0
|
|
for _, issue := range closedIssues {
|
|
if err := store.DeleteIssue(ctx, issue.ID); err != nil {
|
|
WarnError("failed to delete %s: %v", issue.ID, err)
|
|
} else {
|
|
deleted++
|
|
}
|
|
}
|
|
commandDidWrite.Store(true)
|
|
detail := fmt.Sprintf(" Deleted %d issue(s)", deleted)
|
|
if !jsonOutput {
|
|
fmt.Println(detail)
|
|
}
|
|
results = append(results, phaseResult{name: "Decay", detail: fmt.Sprintf("%d issues deleted", deleted)})
|
|
|
|
if deleted > 0 {
|
|
commandDidWrite.Store(true)
|
|
}
|
|
}
|
|
}
|
|
if !jsonOutput {
|
|
fmt.Println()
|
|
}
|
|
}
|
|
|
|
if !jsonOutput {
|
|
fmt.Println("Phase 2/3: Compact (Dolt commit history info)")
|
|
}
|
|
|
|
commitCount := 0
|
|
logEntries, logErr := store.Log(ctx, 0)
|
|
if logErr != nil {
|
|
WarnError("could not read Dolt commit log: %v", logErr)
|
|
} else {
|
|
commitCount = len(logEntries)
|
|
}
|
|
|
|
if commitCount <= 1 {
|
|
if !jsonOutput {
|
|
fmt.Printf(" Only %d commit(s), nothing to compact\n\n", commitCount)
|
|
}
|
|
results = append(results, phaseResult{name: "Compact", detail: "nothing to compact"})
|
|
} else {
|
|
if gcDryRun {
|
|
if !jsonOutput {
|
|
fmt.Printf(" %d commits in history (use bd flatten to squash)\n\n", commitCount)
|
|
}
|
|
results = append(results, phaseResult{name: "Compact", detail: fmt.Sprintf("%d commits (dry-run)", commitCount)})
|
|
} else {
|
|
if !jsonOutput {
|
|
fmt.Printf(" %d commits in history\n", commitCount)
|
|
fmt.Printf(" Tip: use 'bd flatten' to squash all history to one commit\n\n")
|
|
}
|
|
results = append(results, phaseResult{name: "Compact", detail: fmt.Sprintf("%d commits", commitCount)})
|
|
}
|
|
}
|
|
|
|
var gcSizeInfo map[string]interface{}
|
|
if gcSkipDolt {
|
|
results = append(results, phaseResult{name: "Dolt GC", skipped: true})
|
|
} else {
|
|
if !jsonOutput {
|
|
fmt.Println("Phase 3/3: Dolt GC (reclaim disk space)")
|
|
}
|
|
|
|
gc, ok := storage.UnwrapStore(store).(storage.GarbageCollector)
|
|
if !ok {
|
|
if !jsonOutput {
|
|
fmt.Println(" Storage backend does not support GC, skipping")
|
|
}
|
|
results = append(results, phaseResult{name: "Dolt GC", detail: "not supported"})
|
|
} else if gcDryRun {
|
|
if !jsonOutput {
|
|
if gcFull {
|
|
fmt.Println(" Would run a full DOLT_GC() (all storage generations)")
|
|
} else {
|
|
fmt.Println(" Would run DOLT_GC()")
|
|
}
|
|
}
|
|
results = append(results, phaseResult{name: "Dolt GC", detail: "dry-run"})
|
|
} else {
|
|
// bd gc runs without a preceding squash, so remote-tracking
|
|
// refs are left alone here (they cache the remote tip for the
|
|
// migrate gate); flatten/compact prune them before their GC
|
|
// (bd-agctw). Sizes are reported so a no-op reclaim is visible.
|
|
sizeBefore := storeSizeBytes(ctx)
|
|
remoteRefs, tags := listRemoteRefsAndTags(ctx)
|
|
if gcFull && !jsonOutput {
|
|
fmt.Println(" Running full Dolt GC (all generations; can take minutes on large stores)...")
|
|
}
|
|
gcMode, err := runDoltGCPass(ctx, gc, gcFull)
|
|
if err != nil {
|
|
WarnError("dolt gc failed: %v", err)
|
|
results = append(results, phaseResult{name: "Dolt GC", detail: "failed"})
|
|
} else {
|
|
sizeAfter := storeSizeBytes(ctx)
|
|
detail := "complete"
|
|
if line := gcSizeLine(sizeBefore, sizeAfter); line != "" {
|
|
detail = "complete: " + line
|
|
}
|
|
if !jsonOutput {
|
|
fmt.Printf(" Done (%s)\n", detail)
|
|
if gcMode != gcModeFull && suggestFullGC(sizeBefore, sizeAfter) {
|
|
printFullGCHint()
|
|
}
|
|
if len(remoteRefs)+len(tags) > 0 {
|
|
fmt.Printf(" Note: %d remote-tracking ref(s) and %d tag(s) anchor history;\n", len(remoteRefs), len(tags))
|
|
fmt.Printf(" after a history squash, use bd flatten / bd compact so they are pruned first.\n")
|
|
}
|
|
}
|
|
results = append(results, phaseResult{name: "Dolt GC", detail: detail})
|
|
gcSizeInfo = map[string]interface{}{
|
|
"remote_refs": len(remoteRefs),
|
|
"tags": len(tags),
|
|
"mode": gcMode,
|
|
}
|
|
addGCSizeJSON(gcSizeInfo, sizeBefore, sizeAfter)
|
|
}
|
|
}
|
|
if !jsonOutput {
|
|
fmt.Println()
|
|
}
|
|
}
|
|
|
|
elapsed := time.Since(start)
|
|
|
|
if jsonOutput {
|
|
summaryMap := make(map[string]interface{})
|
|
summaryMap["dry_run"] = gcDryRun
|
|
summaryMap["elapsed_ms"] = elapsed.Milliseconds()
|
|
phases := make([]map[string]interface{}, 0, len(results))
|
|
for _, r := range results {
|
|
p := map[string]interface{}{
|
|
"name": r.name,
|
|
"skipped": r.skipped,
|
|
}
|
|
if r.detail != "" {
|
|
p["detail"] = r.detail
|
|
}
|
|
phases = append(phases, p)
|
|
}
|
|
summaryMap["phases"] = phases
|
|
if gcSizeInfo != nil {
|
|
summaryMap["dolt_gc"] = gcSizeInfo
|
|
}
|
|
return outputJSON(summaryMap)
|
|
}
|
|
|
|
mode := "✓ GC complete"
|
|
if gcDryRun {
|
|
mode = "DRY RUN complete"
|
|
}
|
|
fmt.Printf("%s (%v)\n", mode, elapsed.Round(time.Millisecond))
|
|
for _, r := range results {
|
|
if r.skipped {
|
|
fmt.Printf(" %s: skipped\n", r.name)
|
|
} else {
|
|
fmt.Printf(" %s: %s\n", r.name, r.detail)
|
|
}
|
|
}
|
|
return nil
|
|
},
|
|
}
|
|
|
|
func init() {
|
|
gcCmd.Flags().BoolVar(&gcDryRun, "dry-run", false, "Preview without making changes")
|
|
gcCmd.Flags().BoolVarP(&gcForce, "force", "f", false, "Skip confirmation prompts")
|
|
gcCmd.Flags().IntVar(&gcOlderThan, "older-than", 90, "Delete closed issues older than N days")
|
|
gcCmd.Flags().BoolVar(&gcSkipDecay, "skip-decay", false, "Skip issue deletion phase")
|
|
gcCmd.Flags().BoolVar(&gcSkipDolt, "skip-dolt", false, "Skip Dolt garbage collection phase")
|
|
gcCmd.Flags().BoolVar(&gcFull, "full", false, "Run a full Dolt GC (all generations; slower, reclaims space default passes cannot)")
|
|
|
|
rootCmd.AddCommand(gcCmd)
|
|
}
|