beads/internal/storage/embeddeddolt/cache_cleanup.go
matt wilkie 813fdfabf8
fix(ci): unblock main — gosec G122 + credentials_test signatures (#3389)
* fix(lint): silence G122 on dolt cache cleanup remove

golangci-lint v2.11.4 (released recently) enabled a new gosec check:

    G122: Filesystem operation in filepath.Walk/WalkDir callback uses
    race-prone path; consider root-scoped APIs (e.g. os.Root) to prevent
    symlink TOCTOU traversal (gosec)

This flags the `os.Remove(path)` inside cleanGitRemoteCacheGarbage.
Main went red on the Lint job with no code change to blame — it broke
on every PR that rebased past the golangci-lint bump.

The warning is not applicable here: the walk root is
`<beadsDir>/<database>/.dolt/git-remote-cache/`, owned by the user
running bd. An attacker who can symlink-swap a tmp_pack_*/tmp_idx_*
file mid-walk already has write access to the Dolt data directory,
which is a much bigger compromise than cache_cleanup.go doing anything
unsafe.

Add a targeted `#nosec G122` with justification rather than refactor
to `os.Root`-scoped APIs, which would be a larger change against a
file that exists to clean up a 102 GB leak (GH#3354) and shouldn't
take on additional complexity for a hypothetical threat. Matches the
project's existing pattern of per-site gosec exceptions with rationale.

Verification:
    golangci-lint run --timeout=5m --build-tags=gms_pure_go ./...
    # 0 issues.

* fix(dolt): update shared-server credential tests for new per-remote signatures

Two tests added by #3223 (f9ec9859, 2026-04-13) were not updated when
#3211 (73d72ec0, 2026-04-20) widened the credential-routing guards to
take per-remote parameters. The resulting signature drift landed on
main and broke the build on ubuntu-latest and macos-latest:

    internal/storage/dolt/credentials_test.go:445:39: not enough
      arguments in call to store.shouldUseCLIForCredentials
    internal/storage/dolt/credentials_test.go:486:6: not enough
      arguments in call to store.shouldUseCLIForCloudAuth

Fix:
- TestCredentialCLIRoutingSharedServerUsesSharedDoltRoot:
  pass store.remote and store.mainRemoteCredentials() to match the
  new shouldUseCLIForCredentials(ctx, remote, creds) signature,
  matching the pattern already used by the sibling
  TestCredentialCLIRoutingExternalServer/NoRemote tests.

- TestCloudAuthCLIRoutingSharedServerUsesSharedDoltRoot:
  pass store.remote to the new shouldUseCLIForCloudAuth(remote)
  signature, and change the dolt remote URL from https:// to az://.
  The old global "any cloud env var triggers CLI routing" heuristic
  was replaced by per-scheme matching (AZURE_STORAGE_* only triggers
  for az:// remotes, etc. — see the positive/negative cases in
  TestCloudAuthCLIRouting). The test sets AZURE_STORAGE_ACCOUNT, so
  the remote must be az:// for the assertion to hold.

Verification: both tests pass locally (`go test -tags gms_pure_go
-run 'SharedServerUsesSharedDoltRoot' ./internal/storage/dolt/`).
2026-04-20 19:02:11 -07:00

94 lines
2.8 KiB
Go

//go:build cgo
package embeddeddolt
import (
"os"
"path/filepath"
"strings"
"sync"
"time"
)
// cleanGitRemoteCacheGarbage removes orphaned tmp_pack_* files from the
// Dolt git-remote-cache. These files are created by `git fetch` (invoked
// by Dolt's GitBlobstore) and should be renamed to final .pack/.idx files
// on success or deleted on failure. In practice, failed or interrupted
// fetches leave them behind indefinitely, and Dolt's built-in periodic
// git gc (maybeRunGC, gated to once per 24h) either never runs or cannot
// keep up with the accumulation rate.
//
// On a real machine with normal beads usage, this leak consumed 102 GB
// (412 files) in 7 days. See https://github.com/gastownhall/beads/issues/3354
//
// This function is safe to call concurrently and is rate-limited to avoid
// unnecessary filesystem walks on hot paths.
func (s *EmbeddedDoltStore) cleanGitRemoteCacheGarbage() {
if !cacheCleanupThrottle.shouldRun() {
return
}
cacheBase := filepath.Join(s.dataDir, s.database, ".dolt", "git-remote-cache")
if _, err := os.Stat(cacheBase); os.IsNotExist(err) {
return
}
cutoff := time.Now().Add(-tmpPackMinAge)
_ = filepath.WalkDir(cacheBase, func(path string, d os.DirEntry, err error) error {
if err != nil {
return nil // best-effort: skip unreadable entries
}
if d.IsDir() {
return nil
}
name := d.Name()
if !strings.HasPrefix(name, "tmp_pack_") && !strings.HasPrefix(name, "tmp_idx_") {
return nil
}
info, err := d.Info()
if err != nil {
return nil
}
if info.ModTime().Before(cutoff) {
// #nosec G122 -- path is under .dolt/git-remote-cache/ which is
// owned by the user running bd. A TOCTOU symlink swap would
// require write access to that directory; in that case the
// attacker already controls the Dolt data. The tmp_pack_/tmp_idx_
// prefix check further narrows the scope to files Dolt itself writes.
_ = os.Remove(path)
}
return nil
})
}
const (
// tmpPackMinAge is the minimum age before a tmp_pack file is considered
// garbage. Files younger than this may belong to an in-progress fetch.
tmpPackMinAge = 5 * time.Minute
// cacheCleanupInterval is how often cleanGitRemoteCacheGarbage actually
// walks the filesystem when called repeatedly.
cacheCleanupInterval = 10 * time.Minute
)
// throttle gates a function to run at most once per interval.
type throttle struct {
mu sync.Mutex
interval time.Duration
lastRun time.Time
}
func (t *throttle) shouldRun() bool {
t.mu.Lock()
defer t.mu.Unlock()
if time.Since(t.lastRun) < t.interval {
return false
}
t.lastRun = time.Now()
return true
}
// cacheCleanupThrottle is a package-level throttle shared across all
// EmbeddedDoltStore instances in the same process.
var cacheCleanupThrottle = &throttle{interval: cacheCleanupInterval}