beads/cmd/bd/gc_prune.go
Julian Knutsen beb3b5d7d2
fix(gc): flatten/compact run full GC after the rewrite; bd gc --full; low-reclaim hint (#5907) (#6057)
* feat(storage): add DoltGCFull for full, all-generation Dolt GC

Dolt GC is generational: every pass promotes the chunks reachable at that
moment into the old generation, and a default pass only visits the new
generation. Callers that rewrite history need a collection that revisits
the old generation too.

Adds versioncontrolops.DoltGCFull (CALL DOLT_GC('--full', '--archive-level',
'0')), the optional storage.FullGarbageCollector capability, and
implementations on both in-tree Dolt stores. No call sites change yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(gc): run a full Dolt GC after flatten and compact

The post-rewrite GC was Dolt's default, generational pass. Any earlier GC —
bd gc, or a previous flatten or compact — has already promoted the commit
chain into the old generation, which a default pass never revisits. The
rewrite orphans that chain, so the follow-up GC freed ~nothing on every
mature store and orphaned --as-of addresses kept resolving.

flatten, compact, and the proxied-server compact now finish with a full
collection, which frees everything a default pass frees plus the old
generation. Text mode prints a notice first (a full pass can take minutes on
large stores) and the JSON result reports gc_mode.

Unchanged: bd gc's own phase 3, bd admin compact --dolt, and the
managed-server shutdown GC all stay generational.

Fixes #5907

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(gc): add bd gc --full and hint when a default pass reclaims little

Periodic hygiene stays generational by default — a full pass costs minutes on
a large store and usually reclaims nothing extra. But phase 3 has flatten's
blind spot too: row data from decayed issues was promoted to the old
generation by an earlier GC, so a default pass cannot reclaim it. --full
selects a full collection on the embedded, server, and proxied paths.

When a default pass leaves a non-trivial store (>= 64 MiB) barely smaller
(< 1% freed), print a tip explaining the generational model and pointing at
--full. Unmeasurable sizes disable the hint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(gc): cover suggestFullGC thresholds, GC help text, and bd gc --full

The threshold table includes the two field datapoints behind the
calibration: a large store that freed nothing hints, and the 1276 -> 971 MB
run that freed 24% does not.

The help-text assertions pin the wording in the three Cobra Long strings,
which are the source the CLI reference pages are generated from. One of them
also caught a line wrap that split "old generation" across a newline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs: explain why a Dolt garbage collection can reclaim nothing

Adds a subsection to the Dolt backend page covering the generational model —
promotion to the old generation, why a default pass never revisits it, and
why a full collection is a strict superset — and extends the History Bloat
runbook's Prevention advice with an occasional full pass.

Both pages describe the pinned release (v1.2.2), so they stay on the dolt CLI
and do not name the bd behavior this branch adds; the generated CLI reference
picks that up when the pin is bumped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-29 05:19:06 +00:00

178 lines
6 KiB
Go

package main
import (
"context"
"fmt"
"strings"
"github.com/steveyegge/beads/internal/storage"
)
// storeSizeBytes returns the approximate on-disk size of the active database,
// or -1 when that database cannot be measured. It deliberately does not fall
// back to StoreLocator.Path: both concrete locator paths can contain sibling
// databases whose activity is unrelated to the current GC operation.
func storeSizeBytes(ctx context.Context) int64 {
return storeSizeBytesForStore(ctx, store)
}
func storeSizeBytesForStore(ctx context.Context, candidate storage.DoltStorage) int64 {
sizer, ok := storage.UnwrapStore(candidate).(storage.ActiveDatabaseSizer)
if !ok {
return -1
}
size, err := sizer.ActiveDatabaseSize(ctx)
if err != nil {
return -1
}
return size
}
// pruneRemoteRefsForGC deletes cached remote-tracking refs ahead of a
// post-squash GC — they anchor the pre-squash chain, and with them in place
// DOLT_GC reclaims nothing on any workspace that has ever pushed or fetched
// (bd-agctw). Also returns tags, which anchor history the same way but are
// user-created and therefore only warned about. Failures are warnings: GC
// still runs, it just reclaims less.
func pruneRemoteRefsForGC(ctx context.Context) (pruned, tags []string) {
pruner, ok := storage.UnwrapStore(store).(storage.RemoteRefPruner)
if !ok {
return nil, nil
}
var err error
pruned, err = pruner.PruneRemoteRefs(ctx)
if err != nil {
WarnError("pruning remote-tracking refs before GC: %v (GC may reclaim little)", err)
}
tags, err = pruner.ListTags(ctx)
if err != nil {
WarnError("listing tags before GC: %v", err)
}
return pruned, tags
}
// listRemoteRefsAndTags is the read-only companion for dry runs and bd gc.
func listRemoteRefsAndTags(ctx context.Context) (refs, tags []string) {
pruner, ok := storage.UnwrapStore(store).(storage.RemoteRefPruner)
if !ok {
return nil, nil
}
refs, _ = pruner.ListRemoteRefs(ctx)
tags, _ = pruner.ListTags(ctx)
return refs, tags
}
// printPruneReport prints the outcome of pruneRemoteRefsForGC (text mode only).
func printPruneReport(pruned, tags []string) {
if len(pruned) > 0 {
fmt.Printf(" Pruned %d remote-tracking ref(s): %s\n", len(pruned), strings.Join(pruned, ", "))
fmt.Printf(" (local cache only — the next push/fetch re-creates them at the new tip)\n")
}
if len(tags) > 0 {
fmt.Printf(" Warning: %d tag(s) still anchor old history: %s\n", len(tags), strings.Join(tags, ", "))
fmt.Printf(" GC cannot reclaim commits reachable from tags; delete unwanted tags and re-run GC.\n")
}
}
// Dolt GC pass kinds, reported as the "mode"/"gc_mode" JSON field.
const (
gcModeGenerational = "generational"
gcModeFull = "full"
)
// runDoltGCPass runs the requested Dolt GC pass and reports which pass actually
// ran. Dolt GC is generational: a default pass only visits data written since
// the last GC, while a full pass also collects the old generation. A store that
// does not implement FullGarbageCollector degrades to a generational pass with
// a warning.
func runDoltGCPass(ctx context.Context, gc storage.GarbageCollector, full bool) (string, error) {
if full {
if fullGC, ok := gc.(storage.FullGarbageCollector); ok {
return gcModeFull, fullGC.DoltGCFull(ctx)
}
WarnError("storage backend does not support a full GC; running a generational pass, " +
"which cannot reclaim data an earlier GC moved to the old generation")
}
return gcModeGenerational, gc.DoltGC(ctx)
}
// runPostRewriteGC runs the GC that follows a history rewrite and reports which
// pass ran ("" when the backend has no GC at all). It is always a full pass:
// the rewrite orphans a commit chain that any earlier GC already promoted to
// the old generation, and a generational pass never revisits it. Failures are
// warnings — the rewrite itself already succeeded.
func runPostRewriteGC(ctx context.Context, op string) string {
gc, ok := storage.UnwrapStore(store).(storage.GarbageCollector)
if !ok {
return ""
}
if !jsonOutput {
fmt.Println(" Running full Dolt GC (all generations; can take minutes on large stores)...")
}
mode, err := runDoltGCPass(ctx, gc, true)
if err != nil {
WarnError("dolt gc after %s failed: %v", op, err)
}
return mode
}
const (
// fullGCHintMinStoreBytes keeps the hint off small stores, where a few
// reclaimed kilobytes are not worth a multi-minute full pass.
fullGCHintMinStoreBytes = 64 << 20
// fullGCHintMaxFreedFraction is what counts as "reclaimed ~nothing".
fullGCHintMaxFreedFraction = 0.01
)
// suggestFullGC reports whether a completed generational GC pass freed so
// little of a non-trivial store that the caller should suggest --full.
// before/after are ActiveDatabaseSize measurements; negative means unmeasurable
// and disables the hint.
func suggestFullGC(before, after int64) bool {
if before < 0 || after < 0 || before < fullGCHintMinStoreBytes {
return false
}
freed := before - after
if freed < 0 {
freed = 0
}
return float64(freed) < fullGCHintMaxFreedFraction*float64(before)
}
// printFullGCHint explains why a default pass can reclaim nothing (text mode).
func printFullGCHint() {
fmt.Println(" Tip: little space was reclaimed. A default Dolt GC never revisits data an")
fmt.Println(" earlier GC moved to the old generation — run 'bd gc --full' to collect all")
fmt.Println(" generations.")
}
// gcSizeLine formats a before/after size pair as "X → Y (freed Z)", or ""
// when either measurement failed.
func gcSizeLine(before, after int64) string {
if before < 0 || after < 0 {
return ""
}
freed := before - after
if freed < 0 {
freed = 0
}
return fmt.Sprintf("%s → %s (freed %s)", formatBytes(before), formatBytes(after), formatBytes(freed))
}
// addGCSizeJSON adds size measurements to a JSON output map (fields omitted
// when a measurement failed).
func addGCSizeJSON(m map[string]interface{}, before, after int64) {
if before >= 0 {
m["size_before_bytes"] = before
}
if after >= 0 {
m["size_after_bytes"] = after
}
if before >= 0 && after >= 0 {
freed := before - after
if freed < 0 {
freed = 0
}
m["freed_bytes"] = freed
}
}