beads/cmd/bd/gc.go
Julian Knutsen beb3b5d7d2
fix(gc): flatten/compact run full GC after the rewrite; bd gc --full; low-reclaim hint (#5907) (#6057)
* feat(storage): add DoltGCFull for full, all-generation Dolt GC

Dolt GC is generational: every pass promotes the chunks reachable at that
moment into the old generation, and a default pass only visits the new
generation. Callers that rewrite history need a collection that revisits
the old generation too.

Adds versioncontrolops.DoltGCFull (CALL DOLT_GC('--full', '--archive-level',
'0')), the optional storage.FullGarbageCollector capability, and
implementations on both in-tree Dolt stores. No call sites change yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(gc): run a full Dolt GC after flatten and compact

The post-rewrite GC was Dolt's default, generational pass. Any earlier GC —
bd gc, or a previous flatten or compact — has already promoted the commit
chain into the old generation, which a default pass never revisits. The
rewrite orphans that chain, so the follow-up GC freed ~nothing on every
mature store and orphaned --as-of addresses kept resolving.

flatten, compact, and the proxied-server compact now finish with a full
collection, which frees everything a default pass frees plus the old
generation. Text mode prints a notice first (a full pass can take minutes on
large stores) and the JSON result reports gc_mode.

Unchanged: bd gc's own phase 3, bd admin compact --dolt, and the
managed-server shutdown GC all stay generational.

Fixes #5907

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(gc): add bd gc --full and hint when a default pass reclaims little

Periodic hygiene stays generational by default — a full pass costs minutes on
a large store and usually reclaims nothing extra. But phase 3 has flatten's
blind spot too: row data from decayed issues was promoted to the old
generation by an earlier GC, so a default pass cannot reclaim it. --full
selects a full collection on the embedded, server, and proxied paths.

When a default pass leaves a non-trivial store (>= 64 MiB) barely smaller
(< 1% freed), print a tip explaining the generational model and pointing at
--full. Unmeasurable sizes disable the hint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(gc): cover suggestFullGC thresholds, GC help text, and bd gc --full

The threshold table includes the two field datapoints behind the
calibration: a large store that freed nothing hints, and the 1276 -> 971 MB
run that freed 24% does not.

The help-text assertions pin the wording in the three Cobra Long strings,
which are the source the CLI reference pages are generated from. One of them
also caught a line wrap that split "old generation" across a newline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: explain why a Dolt garbage collection can reclaim nothing

Adds a subsection to the Dolt backend page covering the generational model —
promotion to the old generation, why a default pass never revisits it, and
why a full collection is a strict superset — and extends the History Bloat
runbook's Prevention advice with an occasional full pass.

Both pages describe the pinned release (v1.2.2), so they stay on the dolt CLI
and do not name the bd behavior this branch adds; the generated CLI reference
picks that up when the pin is bumped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 05:19:06 +00:00

301 lines
9.4 KiB
Go

package main
import (
"fmt"
"time"
"github.com/spf13/cobra"
"github.com/steveyegge/beads/internal/metrics"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/types"
)
var (
gcDryRun bool
gcForce bool
gcOlderThan int
gcSkipDecay bool
gcSkipDolt bool
gcFull bool
)
var gcCmd = &cobra.Command{
Use: "gc",
GroupID: "maint",
Short: "Garbage collect: decay old issues, compact Dolt commits, run Dolt GC",
Long: `Full lifecycle garbage collection for standalone Beads databases.
Runs three phases in sequence:
1. DECAY — Delete closed issues older than N days (default 90)
2. COMPACT — Squash old Dolt commits into fewer commits (bd compact)
3. GC — Run Dolt garbage collection to reclaim disk space
Each phase can be skipped individually. Use --dry-run to preview all phases
without making changes.
Phase 3 runs Dolt's default, generational GC: it only examines data written
since the last GC. Data that survived an earlier GC lives in the old generation
and is never revisited, so on a long-lived store the space freed by decay may
not be reclaimed. Use --full to collect all generations (slower on large
stores).
Examples:
bd gc # Full GC with defaults (90 day decay)
bd gc --dry-run # Preview what would happen
bd gc --older-than 30 # Decay issues closed 30+ days ago
bd gc --skip-decay # Skip issue deletion, just compact+GC
bd gc --skip-dolt # Skip Dolt GC, just decay+compact
bd gc --full # Collect all storage generations
bd gc --force # Skip confirmation prompt`,
SilenceUsage: true,
SilenceErrors: true,
RunE: func(cmd *cobra.Command, _ []string) error {
if usesProxiedServer() {
return runGCProxiedServer(rootCtx)
}
evt := metrics.NewCommandEvent("gc")
defer func() {
if c := metrics.Global(); c != nil {
c.CloseEventAndAdd(evt)
}
}()
if !gcDryRun {
CheckReadonly("gc")
}
ctx := rootCtx
start := time.Now()
if gcOlderThan < 0 {
return HandleErrorRespectJSON("--older-than must be non-negative")
}
type phaseResult struct {
name string
skipped bool
detail string
}
var results []phaseResult
if gcSkipDecay {
results = append(results, phaseResult{name: "Decay", skipped: true})
} else {
if !jsonOutput {
fmt.Println("Phase 1/3: Decay (delete old closed issues)")
}
cutoffDays := gcOlderThan
cutoffTime := time.Now().UTC().AddDate(0, 0, -cutoffDays)
statusClosed := types.StatusClosed
// gc is a scripted internal sweep — opt out of BEADS_MAX_ROWS
// (designer §4.1) so a misconfigured env doesn't abort the sweep.
filter := types.IssueFilter{
Status: &statusClosed,
ClosedBefore: &cutoffTime,
MaxRows: 0,
MaxRowsSource: "",
}
closedIssues, err := store.SearchIssues(ctx, "", filter)
if err != nil {
return HandleErrorRespectJSON("searching closed issues: %v", err)
}
var stats closedDeletionCandidateStats
closedIssues, stats = filterClosedDeletionCandidates(closedIssues, &cutoffTime)
warnClosedDeletionSafetySkips(stats)
if len(closedIssues) == 0 {
detail := fmt.Sprintf(" No closed issues older than %d days", cutoffDays)
if !jsonOutput {
fmt.Println(detail)
}
results = append(results, phaseResult{name: "Decay", detail: "0 issues deleted"})
} else {
if gcDryRun {
detail := fmt.Sprintf(" Would delete %d closed issue(s)", len(closedIssues))
if !jsonOutput {
fmt.Println(detail)
}
results = append(results, phaseResult{name: "Decay", detail: fmt.Sprintf("%d issues (dry-run)", len(closedIssues))})
} else {
if !gcForce {
return HandleErrorWithHintRespectJSON(
fmt.Sprintf("would delete %d closed issue(s) older than %d days", len(closedIssues), cutoffDays),
"Use --force to confirm or --dry-run to preview.")
}
deleted := 0
for _, issue := range closedIssues {
if err := store.DeleteIssue(ctx, issue.ID); err != nil {
WarnError("failed to delete %s: %v", issue.ID, err)
} else {
deleted++
}
}
commandDidWrite.Store(true)
detail := fmt.Sprintf(" Deleted %d issue(s)", deleted)
if !jsonOutput {
fmt.Println(detail)
}
results = append(results, phaseResult{name: "Decay", detail: fmt.Sprintf("%d issues deleted", deleted)})
if deleted > 0 {
commandDidWrite.Store(true)
}
}
}
if !jsonOutput {
fmt.Println()
}
}
if !jsonOutput {
fmt.Println("Phase 2/3: Compact (Dolt commit history info)")
}
commitCount := 0
logEntries, logErr := store.Log(ctx, 0)
if logErr != nil {
WarnError("could not read Dolt commit log: %v", logErr)
} else {
commitCount = len(logEntries)
}
if commitCount <= 1 {
if !jsonOutput {
fmt.Printf(" Only %d commit(s), nothing to compact\n\n", commitCount)
}
results = append(results, phaseResult{name: "Compact", detail: "nothing to compact"})
} else {
if gcDryRun {
if !jsonOutput {
fmt.Printf(" %d commits in history (use bd flatten to squash)\n\n", commitCount)
}
results = append(results, phaseResult{name: "Compact", detail: fmt.Sprintf("%d commits (dry-run)", commitCount)})
} else {
if !jsonOutput {
fmt.Printf(" %d commits in history\n", commitCount)
fmt.Printf(" Tip: use 'bd flatten' to squash all history to one commit\n\n")
}
results = append(results, phaseResult{name: "Compact", detail: fmt.Sprintf("%d commits", commitCount)})
}
}
var gcSizeInfo map[string]interface{}
if gcSkipDolt {
results = append(results, phaseResult{name: "Dolt GC", skipped: true})
} else {
if !jsonOutput {
fmt.Println("Phase 3/3: Dolt GC (reclaim disk space)")
}
gc, ok := storage.UnwrapStore(store).(storage.GarbageCollector)
if !ok {
if !jsonOutput {
fmt.Println(" Storage backend does not support GC, skipping")
}
results = append(results, phaseResult{name: "Dolt GC", detail: "not supported"})
} else if gcDryRun {
if !jsonOutput {
if gcFull {
fmt.Println(" Would run a full DOLT_GC() (all storage generations)")
} else {
fmt.Println(" Would run DOLT_GC()")
}
}
results = append(results, phaseResult{name: "Dolt GC", detail: "dry-run"})
} else {
// bd gc runs without a preceding squash, so remote-tracking
// refs are left alone here (they cache the remote tip for the
// migrate gate); flatten/compact prune them before their GC
// (bd-agctw). Sizes are reported so a no-op reclaim is visible.
sizeBefore := storeSizeBytes(ctx)
remoteRefs, tags := listRemoteRefsAndTags(ctx)
if gcFull && !jsonOutput {
fmt.Println(" Running full Dolt GC (all generations; can take minutes on large stores)...")
}
gcMode, err := runDoltGCPass(ctx, gc, gcFull)
if err != nil {
WarnError("dolt gc failed: %v", err)
results = append(results, phaseResult{name: "Dolt GC", detail: "failed"})
} else {
sizeAfter := storeSizeBytes(ctx)
detail := "complete"
if line := gcSizeLine(sizeBefore, sizeAfter); line != "" {
detail = "complete: " + line
}
if !jsonOutput {
fmt.Printf(" Done (%s)\n", detail)
if gcMode != gcModeFull && suggestFullGC(sizeBefore, sizeAfter) {
printFullGCHint()
}
if len(remoteRefs)+len(tags) > 0 {
fmt.Printf(" Note: %d remote-tracking ref(s) and %d tag(s) anchor history;\n", len(remoteRefs), len(tags))
fmt.Printf(" after a history squash, use bd flatten / bd compact so they are pruned first.\n")
}
}
results = append(results, phaseResult{name: "Dolt GC", detail: detail})
gcSizeInfo = map[string]interface{}{
"remote_refs": len(remoteRefs),
"tags": len(tags),
"mode": gcMode,
}
addGCSizeJSON(gcSizeInfo, sizeBefore, sizeAfter)
}
}
if !jsonOutput {
fmt.Println()
}
}
elapsed := time.Since(start)
if jsonOutput {
summaryMap := make(map[string]interface{})
summaryMap["dry_run"] = gcDryRun
summaryMap["elapsed_ms"] = elapsed.Milliseconds()
phases := make([]map[string]interface{}, 0, len(results))
for _, r := range results {
p := map[string]interface{}{
"name": r.name,
"skipped": r.skipped,
}
if r.detail != "" {
p["detail"] = r.detail
}
phases = append(phases, p)
}
summaryMap["phases"] = phases
if gcSizeInfo != nil {
summaryMap["dolt_gc"] = gcSizeInfo
}
return outputJSON(summaryMap)
}
mode := "✓ GC complete"
if gcDryRun {
mode = "DRY RUN complete"
}
fmt.Printf("%s (%v)\n", mode, elapsed.Round(time.Millisecond))
for _, r := range results {
if r.skipped {
fmt.Printf(" %s: skipped\n", r.name)
} else {
fmt.Printf(" %s: %s\n", r.name, r.detail)
}
}
return nil
},
}
func init() {
gcCmd.Flags().BoolVar(&gcDryRun, "dry-run", false, "Preview without making changes")
gcCmd.Flags().BoolVarP(&gcForce, "force", "f", false, "Skip confirmation prompts")
gcCmd.Flags().IntVar(&gcOlderThan, "older-than", 90, "Delete closed issues older than N days")
gcCmd.Flags().BoolVar(&gcSkipDecay, "skip-decay", false, "Skip issue deletion phase")
gcCmd.Flags().BoolVar(&gcSkipDolt, "skip-dolt", false, "Skip Dolt garbage collection phase")
gcCmd.Flags().BoolVar(&gcFull, "full", false, "Run a full Dolt GC (all generations; slower, reclaims space default passes cannot)")
rootCmd.AddCommand(gcCmd)
}