beads/internal/storage/dbproxy
matt wilkie d7b9f4fc52
Use Snappy output for Dolt garbage collection (#4986)
* Use Snappy output for Dolt garbage collection

Pass archive level 0 through every Beads-owned GC path so new table files use Snappy instead of zstd archives. Existing zstd archive decoding remains unchanged.

Refs #4249.

Agent-Signature: amp-unknown-model-unknown-reasoning on behalf of matt wilkie
Amp-Thread-ID: https://ampcode.com/threads/T-019f8d27-e480-774e-bd5a-7051e8e6e378

* fix(dolt-gc): gate archive_level on managed server auto-GC to avoid bricking older external dolt

Paired review on gastownhall/beads#4986 flagged that Beads-managed dolt
sql-server processes still auto-GC into zstd archives: the pinned Dolt
defaults background auto-GC to archive_level 1, and neither server-launch
path set auto_gc_behavior.archive_level: 0.

Both managed-server launch paths now set archive_level: 0 (auto-GC stays
enabled) when the resolved external dolt binary is new enough to support
the key:

- internal/storage/dbproxy/server (via cmd/bd/proxied_server.go's
  config.yaml generator) already wrote a YAML config; it now includes
  auto_gc_behavior when supported.
- internal/doltserver's legacy launcher previously used CLI flags only
  (-H/-P/--loglevel) with no YAML config at all — Dolt's own
  commandLineServerConfig.AutoGCBehavior() is a stub that hardcodes
  archive_level 1 regardless of flags, so CLI-flag mode can never express
  this. It now optionally switches to `--config <generated yaml>` (Dolt
  ignores all other sql-server flags once --config is present, so
  host/port/log-level move into the YAML alongside archive_level).

Both paths gate on internal/doltserver.SupportsArchiveLevelConfig, which
probes `dolt version` and fails closed (omits the key, keeps prior
behavior) on any parse/exec error. This matters because Dolt's YAML loader
uses yaml.UnmarshalStrict (servercfg/yaml_config.go): an older external
dolt binary whose own compiled-in YAMLConfig struct predates
auto_gc_behavior.archive_level (minver 1.52.1) would otherwise hit a hard
parse error and refuse to start, not silently ignore the key. Existing
config.yaml files are left untouched (only fresh bootstraps set the key).

Refs gastownhall/beads#4986.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(compact): fall back to plain dolt gc on older external dolt --archive-level rejection

Paired review on gastownhall/beads#4986 flagged that `bd compact --dolt`
shells out to the PATH `dolt` binary with no version guarantee, unlike the
in-process GC paths pinned by go.mod. Passing --archive-level 0
unconditionally meant an external dolt predating the flag would abort
compact with an unknown-flag error where plain `dolt gc` used to succeed.

runCompactDolt now retries with plain `dolt gc` when the first attempt's
combined output matches isUnknownArchiveLevelFlagError, printing a one-line
notice that the older dolt may still produce zstd archives. The classifier
is deliberately narrow (requires both an "unknown flag"-style phrase and
the archive-level flag name in the output) so a genuine GC failure is never
misclassified as a missing flag and swallowed — real failures still fail
compact as before.

Refs gastownhall/beads#4986.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dolt-gc): replicate .doltcfg discovery in --config mode; memoize version probe

Delta review round 2 on gastownhall/beads#4986 flagged that the legacy
launcher's new --config mode silently drops Dolt's own .doltcfg discovery:
Dolt's setupDoltConfig (cmd/dolt/commands/sqlserver/sqlserver.go, pinned
module) returns immediately when --config is passed, so the parent
../.doltcfg lookup (privileges.db users, branch_control.db) that flag mode
performs never runs. A deployment previously run in CLI-flag mode that had
a parent .doltcfg would, once the resolved dolt becomes new enough to use
--config mode, silently fall back to a fresh $data_dir/.doltcfg: existing
users and branch controls abandoned, new passwordless root.

internal/doltserver.resolveCfgDir replicates Dolt's own flag-mode discovery
exactly (parent ../.doltcfg if present, else $data_dir/.doltcfg, error via
ErrMultipleDoltCfgDirs if both exist rather than guess) and Start() now
resolves it once per launch and writes it into the generated YAML's
cfg_dir. servercfg.YAMLConfig.CfgDirStr carries no minver tag in the pinned
module (i.e. always supported), so the existing 1.52.1 archive_level gate
stays as the single version gate for --config mode.

Also: filepath.Abs the --config path before exec (the child's cwd is the
data dir, matching the sibling dbproxy/server path's existing defensive
abs), and memoize SupportsArchiveLevelConfig per doltBin (sync.Map) so
Start()/UOW-provider creation does not re-fork `dolt version` on every
call within a process.

Refs gastownhall/beads#4986.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dolt-gc): reconcile pre-existing managed configs on load; warn on custom ones

Delta review round 2 on gastownhall/beads#4986 flagged that pre-existing
config.yaml files bypass the archive_level plumbing in both directions: an
already-bootstrapped install's config.yaml silently keeps zstd auto-GC
forever (missing key never gets added without a manual regenerate), and a
config already carrying the key can hard-fail a later dolt downgrade via
yaml.UnmarshalStrict on an unrecognized key.

Fresh config.yaml renders now carry a one-line YAML comment marker
identifying them as Beads-managed (invisible to Dolt's own parser, which
treats it as a comment). ensureProxiedServerConfig uses the marker to
decide how to handle an existing default-path file:

  - marker present (Beads-managed): reconcile at load via
    reconcileManagedProxiedServerConfig, which parses the existing YAML
    into the servercfg structs, adds or strips auto_gc_behavior to match
    the current archive-level probe result, and re-renders — preserving
    every other field (including port) via the struct round trip rather
    than string surgery. A no-op when already aligned, so this costs
    nothing on the steady-state path.
  - marker absent (a pre-marker file from an older Beads version, or a
    genuinely user-supplied custom config path): never rewritten. Emits a
    one-line stderr warning instead, covering both directions (zstd may
    still be active with a hint how to opt in; or the file may cause a
    refuse-to-start on this dolt).

Refs gastownhall/beads#4986.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dolt-gc): preserve sibling auto_gc fields, interpolate custom configs, write atomically

Verification review of the round-2 commits on gastownhall/beads#4986 found
three confirmed issues in cmd/bd/proxied_server.go:

- reconcileManagedProxiedServerConfig tested cfg.BehaviorConfig.AutoGCBehavior
  != nil (the whole block) instead of ArchiveLevel_ specifically. A managed
  config whose block carried only a user-set "enable" neither gained
  archive_level when the binary supported it, nor kept "enable" when the
  whole block was stripped on an unsupported binary. Now inspects and
  mutates only ArchiveLevel_, creating the block if needed to add it and
  removing it only if clearing archive_level leaves it completely empty.

- Validating a custom (env var / sidecar) config via servercfg.NewYamlConfig
  on raw bytes skips Dolt's own environment-variable interpolation
  (servercfg/env_interpolate.go, applied by YamlConfigFromFile before
  parsing) — confirmed by reading the pinned module source. A valid custom
  config using ${VAR}-style values was being rejected before the server
  ever started. ensureProxiedServerConfig and warnUnmanagedProxiedServerConfig
  now parse any unmanaged file (custom path, or a pre-marker default-path
  file) via YamlConfigFromFile; only a Beads-managed file, which never
  contains interpolation syntax, still uses the raw-bytes/NewYamlConfig
  path (inside reconcileManagedProxiedServerConfig).

- Reconciliation rewrote config.yaml with a truncating os.WriteFile, which
  could leave a partial file — blocking every future dolt sql-server
  start — if the process crashed mid-write. Both the reconcile-rewrite and
  the fresh-render write paths now go through the existing atomicWriteFile
  helper (temp file in the same directory, fsync, close, rename), already
  used elsewhere in cmd/bd for exactly this reason.

Tests: TestReconcileManagedProxiedServerConfig_PreservesSiblingFields
(all three directions requested by the review), and
TestEnsureProxiedServerConfig_CustomConfigEnvVarInterpolation.

Refs gastownhall/beads#4986.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dolt-gc): resolve .doltcfg physically through symlinks; key version cache by file identity

Verification review of the round-2 commits on gastownhall/beads#4986 found
two more confirmed issues:

- resolveCfgDir computed the parent .doltcfg via filepath.Join(doltDir,
  "..", ...), which cleans ".." lexically without touching the filesystem.
  If doltDir is itself a symlink, that yields the symlink's own lexical
  parent directory, not the physical parent of whatever it actually points
  at. Reproduced directly: on Linux, chdir into a symlink makes the kernel
  track the process's cwd physically, so the real `dolt` child process's
  own "../.doltcfg" lookup (a bare relative path resolved against its
  actual, physically-tracked cwd) sees the PHYSICAL parent — diverging
  from the lexical one a naive Go-side filepath.Join would compute. A
  symlinked data directory could therefore miss the very parent .doltcfg
  this whole fix exists to find, re-abandoning existing
  privileges/branch-control. resolveCfgDir now resolves doltDir via
  filepath.EvalSymlinks (falling back to the lexical path on error) before
  computing the parent. TestResolveCfgDir_SymlinkedDataDirFindsPhysicalParent
  reproduces the exact layout and was confirmed to fail without the fix.

- SupportsArchiveLevelConfig's memoization cache was keyed by doltBin path
  alone, so an in-place dolt upgrade or downgrade at that same path kept
  the stale verdict for the rest of the process's lifetime. The cache key
  now includes size+mtime (archiveLevelCacheKey), falling back to an
  uncached probe when the binary can't be stat'd. The test that previously
  codified the staleness (rewriting the binary at the same path and
  asserting the cached result "sticks") is replaced with
  TestSupportsArchiveLevelConfig_MemoizesUnchangedBinary (same identity ->
  still cached, proven via an embedded call counter, not just the
  returned bool) and
  TestSupportsArchiveLevelConfig_InvalidatesOnFileIdentityChange (changed
  mtime -> re-probed).

Refs gastownhall/beads#4986.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(dolt-gc): preserve symlinked configs, field-level warn check, graceful degrade on hand-edited managed files

Verification review of the round-3 commits on gastownhall/beads#4986 found
three narrower issues, all in cmd/bd/proxied_server.go:

- atomicWriteFile's os.Rename destination does not follow a symlink — it
  replaces whatever is directly at that path. Reconciling (or freshly
  rendering) a managed config.yaml that was itself a symlink therefore
  silently replaced the symlink with a regular file at the same location,
  losing the indirection and orphaning whatever it pointed at. Both write
  call sites now resolve the target through filepath.EvalSymlinks first
  (resolveConfigWriteTarget), falling back to the given path when it does
  not exist yet (which also covers a dangling symlink — self-healed by
  writing directly at its location). Test:
  TestEnsureProxiedServerConfig_ManagedConfigBehindSymlinkPreservesSymlink
  reconciles a config.yaml symlink and asserts the symlink itself survives
  while the physical target gets the reconciled content; confirmed it
  fails without the fix (reverted, watched fail, restored).

- warnUnmanagedProxiedServerConfig tested cfg.AutoGCBehavior() != nil (the
  whole block), not archive_level specifically — the same bug the
  reconciler had before round 3, now applied to the warn path. An
  enable-only custom config missed the zstd-risk warning on a capable
  binary and got a false incompatibility warning on an incapable one. Now
  goes through the same field-level check via a new autoGCArchiveLevelIsSet
  helper (type-asserts to the concrete *servercfg.YAMLConfig the pinned
  module's loaders always return, falling back to the block-level check if
  that assertion ever fails). Test:
  TestWarnUnmanagedProxiedServerConfig_EnableOnlyFieldLevel covers both
  directions.

- A managed file's marker only proves Beads generated it ORIGINALLY, not
  that it hasn't been hand-edited since (e.g. a ${VAR} placeholder added
  by an operator). reconcileManagedProxiedServerConfig's raw-bytes parse
  is deliberately non-interpolating, so that now fails to unmarshal.
  Verified the prior behavior first: that parse error propagated as a
  hard error out of ensureProxiedServerConfig, which
  newManagedProxiedServerUOWProvider (cmd/bd/uow_factory.go) returns
  directly — aborting the ENTIRE bd command before ever attempting to
  start the managed server, even though a real dolt process would happily
  interpolate and start against the same file. Now degrades gracefully:
  logs a one-line warning and falls through to the same
  warn-only/never-rewrite treatment as an unmanaged file. Test:
  TestEnsureProxiedServerConfig_ManagedFileWithPlaceholderDegradesGracefully.

Refs gastownhall/beads#4986.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(backup): narrow atomicWriteFile's crash-safe claim instead of changing behavior

Verification review of the round-3 commits on gastownhall/beads#4986 noted
the "crash-safe" doc comment overstates: only the temp file's own contents
are fsynced (not the parent directory entry, so a crash between rename
and a directory fsync could still lose the rename on some filesystems),
and os.Rename's atomic-replace guarantee is a POSIX/Unix property, not
guaranteed on Windows. This helper has an existing caller
(saveBackupState) whose expectations should not change silently, so this
narrows the doc comment to state those caveats precisely rather than
strengthening the implementation. Also notes the symlink-replacement
behavior a caller needs to know about, pointing at
cmd/bd/proxied_server.go's resolveConfigWriteTarget for a case that
resolves the physical target first.

No behavior change.

Refs gastownhall/beads#4986.

Agent-Signature: claude-fable-5-high on behalf of matt wilkie
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 08:55:29 -07:00
..
pidfile /{cmd,internal}: surface mismatches 2026-05-27 16:20:38 -07:00
proxy /{cmd,internal}: get proxy working with remote and tls 2026-07-16 16:44:13 -07:00
server Use Snappy output for Dolt garbage collection (#4986) 2026-07-23 08:55:29 -07:00
util more commands 2026-07-20 14:41:07 -07:00