beads/internal/storage/versioncontrolops/clone.go
Stephen Hodges 3efcd3571c
feat(storage): carry the git data ref of a git-backed Dolt remote (#6521)
* feat(storage): carry the git data ref of a git-backed Dolt remote

Dolt lets a git-backed remote keep its data on a ref other than
refs/dolt/data (the git_ref remote parameter, --ref on dolt remote add,
dolt clone, DOLT_REMOTE, and DOLT_CLONE). The storage layer never passed
it through, so a host that only accepts pushes under refs/heads/, or one
repository holding several databases on refs such as
refs/dolt/units/<key>, could not be used. Part of #6261; the commands
and the config key follow in a stacked PR.

- storage.RemoteInfo gains Ref (empty = refs/dolt/data);
  RemoteRefsMatch, EffectiveGitDataRef, and GitRefFromParamsJSON give
  every package one reading of it, pinned against dolt's constants.
- RemoteStore.AddRemoteWithRef on both stores, and CreateRemoteWithRef
  on the proxied-server remote use case, pass a non-empty ref as
  DOLT_REMOTE's --ref. AddRemote is unchanged; UpdateRemote keeps the
  ref through replacement and restore.
- versioncontrolops.DoltCloneWithRef and dolt.BootstrapFromRemoteWithDBRef
  pass the ref to DOLT_CLONE and to the dolt clone argv.
- ListRemotes reads the ref back from dolt_remotes.params (git_ref) in
  both remote repositories; PersistedRemotes reads it from
  repo_state.json. Malformed params is an error, never a silent default.
- The server-mode CLI mirror is compared on URL and ref and
  re-materialized with dolt remote add --ref when the ref differs. Where
  the CLI is proxied to the running server: the check reads the server's
  own repo_state.json when the listing is empty, the cold-start fallback
  re-adds a ref remote over SQL (which accepts --ref) so the server knows
  it, and a throwaway probe remote proves --ref works before an existing
  mirror remote is removed; probe remotes left by runs no longer alive
  are swept before any change to the mirror. For a server this bd does
  not own, the route stops and names the DOLT_REMOTE call to run
  against that server through bd sql, as one shell-quoted word.
- A ref is checked for a space, an ASCII control character, DEL, and a
  leading dash before it becomes a dolt argument; no other ref-name rule
  applies at this layer.
- Tests carry both ref shapes; the embedded-engine test proves add, list,
  push, and clone on the real engine; sqlmock tests cover the route
  decider and the cold-start re-add at the store seam; the dolt-CLI tests
  skip without a dolt binary.

* fix(storage): make both git_ref readers fail closed; pin the edges (#6521 review)

Maintainer review fixup for PR #6521. The review found no blocker or major;
this is the non-gating minor/nit set plus the test pins the synthesis named as
the residual defects.

- One reader for git_ref. storage.GitRefFromParams is now the single decoder
  for both planes (the SQL params column through GitRefFromParamsJSON, and
  repo_state.json through doltutil.PersistedRemotes), and a git_ref that is
  present but not a string is an error naming the remote instead of silently
  the default ref. Reading it as the default is how a mirror pinned to a custom
  ref would be kept on refs/dolt/data -- the wrong-ref push this PR closes,
  reached from the opposite direction.

- CreateRemoteWithRef checks the ref against what the dolt argv boundary will
  accept, so the recordable ref set is no longer wider than the routable one
  and a bad ref is refused at add time rather than diagnosed at push time.
  UpdateRemote deliberately does not re-check: it carries and restores refs
  that are already recorded and must stay able to put one back.

- BootstrapFromRemoteWithDBRef validates the ref before the already-exists
  early return, so the same --ref is accepted or refused the same way on a
  fresh host and on a bootstrapped one.

- sweepLeftoverProbes warns when it cannot remove another run's leftover probe
  remote, following PersistedRemoteInfos' precedent: the remote stays visible
  in `bd remote list` until some run succeeds, and nothing ever said why.

- EnsureCLIRemote's doc comment records that escalating a state-file read error
  is intentional for default-ref remotes too, and
  TestCLIMirrorUnreadableStateIsAnError now pins the ref == "" direction that a
  later "helpful" relaxation would gate away. The suggested URL-match
  short-circuit is deliberately NOT applied: when the state file is unreadable
  currentRef is unknown, so returning nil on a URL match alone would route a
  push onto whatever ref the mirror is secretly pinned to.

- AddCLIRemote's doc comment records that it is kept deliberately as the
  default-ref spelling now that EnsureCLIRemote uses AddCLIRemoteWithRef.

Not applied: widening probeCLIRemoteRef to `ref != "" || currentRef != ""` and
probing the restore shape. On a proxied server that turns a working custom-ref
-> default-ref transition into a hard failure (the target add is plain and
succeeds today, while probing the restore shape would be refused), and in the
one window that survives calibration -- a non-proxied plain-add failure from
disk/lock contention -- a capability probe predicts nothing about it.

Each new pin was mutation-proved: reverting the fix it guards reds that pin,
and only that pin.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Julian Knutsen <julianknutsen@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Julian Knutsen <julian@gastownhall.dev>
2026-09-27 15:56:18 -07:00

58 lines
1.9 KiB
Go

package versioncontrolops
import (
"context"
"fmt"
"net/url"
"strings"
)
// DoltClone clones a Dolt database from a remote URL.
// conn must be a non-transactional database connection.
// The database parameter specifies the local database name for the clone.
// If user is non-empty, authenticates with that user. Dolt reads the remote
// password from DOLT_REMOTE_PASSWORD.
func DoltClone(ctx context.Context, conn DBConn, remoteURL, database, user string) error {
return DoltCloneWithRef(ctx, conn, remoteURL, database, user, "")
}
// DoltCloneWithRef is DoltClone for a git-backed remote whose Dolt data lives
// on the git ref ref. A non-empty ref is passed as DOLT_CLONE's --ref option;
// Dolt reads the data from that ref and records it on the clone's origin
// remote, so later push and pull use it. An empty ref is DoltClone.
func DoltCloneWithRef(ctx context.Context, conn DBConn, remoteURL, database, user, ref string) error {
var parts []string
var args []any
if user != "" {
parts = append(parts, "'--user', ?")
args = append(args, user)
}
if ref = strings.TrimSpace(ref); ref != "" {
parts = append(parts, "'--ref', ?")
args = append(args, ref)
}
parts = append(parts, "?, ?")
args = append(args, remoteURL, database)
query := "CALL DOLT_CLONE(" + strings.Join(parts, ", ") + ")"
// GH#4272: the initial fetch runs git hooks just like push/pull; disable
// them for the clone window too (see remotes.go for the full rationale).
return withRemoteEnvGuards(func() error {
if _, err := conn.ExecContext(ctx, query, args...); err != nil {
return fmt.Errorf("dolt clone %s: %w", sanitizeURL(remoteURL), err)
}
return nil
})
}
// sanitizeURL removes credentials from a URL for safe error reporting.
func sanitizeURL(raw string) string {
parsed, err := url.Parse(raw)
if err != nil {
return raw
}
parsed.User = nil
parsed.RawQuery = ""
parsed.Fragment = ""
return parsed.String()
}