fix(doltremote): treat s3:// as a native Dolt remote scheme (#5948)

* fix(doltremote): treat s3:// as a native Dolt remote scheme

s3:// URLs previously survived Normalize only by falling through, and the
SCP heuristic treated any "@" in a URL as an SCP-style git remote, so an
s3 URL with "@" in its path or query string would be rewritten into a
bogus git+ssh remote. Add s3:// to NativeSchemes and anchor SCP detection
to inputs without a "://" scheme that match a user@host:path grammar.

* test(doltremote): pin the s3:// entry and the dotless host:pa@th case

Follow-up to the #5948 review. Deleting "s3://" from NativeSchemes made
no test fail, so TestNativeSchemesContainsEachNativeScheme now asserts
every entry the list holds; removing the s3:// line fails its s3://
subtest.

The base change also flipped a fourth behavior its description did not
list: a dotless pre-colon token whose only "@" sits after the colon
("host:pa@th", "alias:repo@v1") used to classify as SCP-style on the
bare strings.Contains(url, "@") check and normalize to
git+ssh://host/pa@th. It now classifies false and passes through
Normalize unchanged. Both inputs are pinned as rows in
TestIsSCPStyleGitURLRejectsNonSCPInputs; against origin/main's
remote.go they fail.

Normalize's doc comment now lists s3:// with the other native schemes.

* build(bazel): add go_test target for internal/doltremote

The PR adds remote_test.go and remote_invariant_test.go to a package whose
BUILD.bazel (added on main by #6798) declares only a go_library. Generated by
bazel run //:gazelle; no hand-written lines.

* fix(doltremote): guard FromGitURL's SCP split, cross-reference the SCP grammars

Maintainer review fixes for the four non-gating findings in the iteration-1
synthesis (0 blockers, 0 majors; quality scorecard 986/approve). The PR's own
fix is unchanged.

Contract fidelity, the one gate-impact-minor finding: FromGitURL is exported and
cmd/bd's gitURLToDoltRemote calls it directly, bypassing Normalize's
classification. Its first-colon split still rebuilt s3://bucket/team@prod/db
into git+ssh://s3///bucket/team@prod/db, the exact string this PR removes from
Normalize. That split now obeys the same "://" rule as isSCPStyleGitURL.

This takes the finding's second offered arm over its first. Gating on
isSCPStyleGitURL(url) instead changes 17 of 31 inputs across the package's own
corpus, against 9 for the "://" guard; the 8 extra are the SCP hosts that
predicate deliberately declines to classify - IDN hosts, non-ASCII userinfo and
dotless aliases - which FromGitURL converts correctly today.
TestFromGitURLStillConvertsSCPForms pins them, and mutating to the rejected arm
reds exactly those three rows while the new regression pin still passes.

NativeSchemes: the finding's "add az:// and oci:// for consistency" arm is
declined and the documentation arm taken instead. The PR body scopes this change
to s3:// and defers az:// to #6227 explicitly; #6227 is open, and it pairs
NativeSchemes with dolt.IsBackupURL - the site where the gap actually changes
behavior - so adding the entry here would have overridden a tracked contributor
decision while closing only half the bug. oci:// is in neither authority's
native list. The list's doc comment now records that narrowing, points at
remotecache.remoteSchemes as the fuller authority, and states that membership is
inert: every entry contains "://", so isSCPStyleGitURL rejects it and
Normalize's closing "return url" returns the same bytes listed or not. Dropping
gs:// reds one census leaf and no behavioral test, which measures that claim
rather than asserting it. The new FromGitURL table covers az:// and oci://
precisely because the guard is keyed on "://" and not on list membership.

Duplicate SCP grammar: scpStyleGitURLPattern's first alternative is
byte-identical to remotecache.gitSSHPattern, and the whole pattern is a superset
by exactly the user-less dotted-host form - the form on which the two already
disagree in production. Both regexes now cross-reference each other and the
divergence is pinned from each side. Exporting a shared matcher was the other
offered arm; it would have widened remotecache.IsRemoteURL's accept set, a
behavior change in a second package for a non-gating finding.

normalizeRemoteURL's doc comment restated Normalize's rules and had drifted
twice: it never gained s3://, and it named gitURLToDoltRemote as the converter
when the body calls doltremote.Normalize directly. Collapsed it to a reference
so the surface cannot drift again.

Codex's prescribed remedy for the hollow NativeSchemes census subtest is
declined, as the synthesis directs: asserting Normalize(scheme+"bucket/x") per
scheme passes against an empty list, for the same inertness reason above. The
subtest's comment, which claimed to protect a fast path it cannot observe, is
corrected instead.

go test ./internal/doltremote/ ./internal/remotecache/ green; cmd/bd named
selector 56 pass, 0 fail, 0 skip; go build ./..., go vet and gofmt clean; bazel
test passes on both packages; gazelle and go_srcs.py --check report no BUILD
drift.

---------

Co-authored-by: Julian Knutsen <julian@gastownhall.com>
Co-authored-by: Gas City Adopt-PR <adopt-pr@gascity.local>
This commit is contained in:
Jones 2026-09-29 15:28:57 +02:00 • committed by GitHub
commit 89348699b0
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
7 changed files with 347 additions and 31 deletions

View file

@ -60,10 +60,11 @@ func commitBeadsConfigForActiveRepo(ctx context.Context, msg string) {
}
// normalizeRemoteURL converts a remote URL to a Dolt-compatible format.
// Dolt-native URLs (dolthub://, file://, aws://, gs://, git+...) are
// returned as-is. Git URLs (https://, ssh://, git@...) are converted
// via gitURLToDoltRemote. Unknown schemes are returned as-is and let
// dolt clone decide.
// See doltremote.Normalize for the scheme rules - this wrapper adds nothing and
// exists only so call sites in this package read against one local name.
// Restating those rules here drifts: this comment still omitted s3:// after the
// source gained it, and named gitURLToDoltRemote as the converter when the call
// below goes to doltremote.Normalize directly.
func normalizeRemoteURL(url string) string {
return doltremote.Normalize(url)
}

View file

@ -1,4 +1,4 @@
load("@rules_go//go:def.bzl", "go_library")
load("@rules_go//go:def.bzl", "go_library", "go_test")
go_library(
name = "doltremote",
@ -6,3 +6,16 @@ go_library(
importpath = "github.com/steveyegge/beads/internal/doltremote",
visibility = ["//:__subpackages__"],
)
go_test(
name = "doltremote_test",
srcs = [
"remote_invariant_test.go",
"remote_test.go",
],
embed = [":doltremote"],
deps = [
"//internal/remotecache",
"//internal/storage/doltutil",
],
)

View file

@ -1,14 +1,29 @@
package doltremote
import "strings"
import (
"regexp"
"strings"
)
// NativeSchemes are URL schemes that Dolt understands natively and should not
// be converted through FromGitURL.
//
// This list is deliberately narrower than remotecache.remoteSchemes, the repo's
// fuller scheme authority: az:// and oci:// are absent here. For az:// that gap
// is tracked in #6227, which also covers dolt.IsBackupURL - the site where the
// gap actually changes behavior - so completing this list on its own would
// close only half of it.
//
// Membership is not load-bearing either way. Every entry contains "://", which
// isSCPStyleGitURL rejects outright, so Normalize's closing "return url"
// already hands back any of these URLs byte-identical whether or not they are
// listed. The list records intent; the "://" guard is what preserves the bytes.
var NativeSchemes = []string{
"dolthub://",
"file://",
"aws://",
"gs://",
"s3://",
"git+https://",
"git+ssh://",
"git+http://",
@ -16,9 +31,14 @@ var NativeSchemes = []string{
}
// Normalize converts a remote URL to a Dolt-compatible format.
// Dolt-native URLs (dolthub://, file://, aws://, gs://, git+...) are returned
// Dolt-native URLs (dolthub://, file://, aws://, gs://, s3://, git+...) are returned
// as-is. Git URLs (https://, ssh://, git@...) are converted via FromGitURL.
// Unknown schemes are returned as-is and let dolt clone decide.
//
// The NativeSchemes loop is only a shortcut. What actually keeps every scheme
// URL intact is the "://" guard in isSCPStyleGitURL: it stops a path or query
// containing "@" from being read as an SCP-style git remote, so an unlisted
// scheme still falls through to the closing "return url".
func Normalize(url string) string {
for _, scheme := range NativeSchemes {
if strings.HasPrefix(url, scheme) {
@ -43,6 +63,15 @@ func Normalize(url string) string {
// SCP-style SSH URLs are converted: git@host:path -> git+ssh://git@host/path
// SSH URLs get "git+" prefix: ssh://... -> git+ssh://...
// URLs that already have "git+" prefix are returned as-is.
//
// The SCP split obeys the same "://" rule as isSCPStyleGitURL, so a scheme URL
// handed to this exported entry point directly - rather than routed here by
// Normalize - cannot be rebuilt into git+ssh://s3///bucket/team@prod/db. The
// narrower shape test stays in isSCPStyleGitURL, which is what Normalize
// classifies on. Gating here on that predicate instead would additionally stop
// converting the SCP hosts it deliberately declines to classify (IDN hosts,
// non-ASCII userinfo, dotless aliases) - measured at 8 further inputs changed
// over the package's own corpus, none of them the bug this guard closes.
func FromGitURL(url string) string {
if strings.HasPrefix(url, "git+") {
return url
@ -56,37 +85,37 @@ func FromGitURL(url string) string {
if isWindowsDrivePath(url) {
return "git+" + url
}
if idx := strings.Index(url, ":"); idx > 0 && !strings.Contains(url[:idx], "/") {
return "git+ssh://" + url[:idx] + "/" + url[idx+1:]
if !strings.Contains(url, "://") {
if idx := strings.Index(url, ":"); idx > 0 && !strings.Contains(url[:idx], "/") {
return "git+ssh://" + url[:idx] + "/" + url[idx+1:]
}
}
return "git+" + url
}
// scpStyleGitURLPattern matches SCP-style git remotes. Its first alternative
// (user@host) is byte-identical to remotecache.gitSSHPattern in
// internal/remotecache/url.go; this pattern is a superset by exactly its second
// alternative, the user-less dotted-host form ("github.com:org/repo.git"),
// which remotecache.IsRemoteURL intentionally reports as false. The two are
// otherwise one grammar with no cross-reference, so keep them in sync; the
// divergence is pinned by TestSCPGrammarDivergesFromRemotecacheOnlyOnDottedHost
// here and by TestIsRemoteURLRejectsUserlessDottedHost on the remotecache side.
var scpStyleGitURLPattern = regexp.MustCompile(`^(?:[a-zA-Z0-9._-]+@[a-zA-Z0-9][a-zA-Z0-9._-]*|[a-zA-Z0-9][a-zA-Z0-9._-]*\.[a-zA-Z0-9._-]*):[^\x00-\x1f\x7f]+$`)
// isSCPStyleGitURL reports whether url looks like an SCP-style git remote:
// either the classic user form (git@host:path) or the user-less form
// (host:path). The user-less form is only recognized when the pre-colon
// token looks like a hostname (contains a ".") so that dotless tokens such
// as a Windows drive letter ("C:foo") are not mistaken for a host.
// user@host:path, or host:path when the pre-colon token contains a "." so a
// Windows drive letter ("C:foo") is not mistaken for a host. Anything that
// carries a "://" scheme is never SCP-style, whatever else it contains, so a
// query or path with "@" in it cannot turn an s3:// URL into a git remote.
//
// Known limitation: git itself also accepts dotless SSH config aliases in
// this position (e.g. "github:org/repo.git", "localhost:repo.git"), which
// the "." check above rejects. Those origins pass through unconverted and
// won't canonically match an equivalent git+ssh://alias/... or
// user@alias:path Dolt remote. This is a deliberate trade-off to keep
// isWindowsDrivePath's single-letter-drive check from being the only guard
// against misreading "C:foo" as a host; widening the rule (e.g. following
// git's own convention of treating anything host:path as SCP-style unless
// the pre-colon token is a single-letter drive) is tracked as a follow-up
// rather than fixed here.
// Known limitations: git also accepts dotless SSH config aliases
// ("github:org/repo.git"); those pass through unconverted and will not
// canonically match an equivalent git+ssh:// form. User and host are limited
// to [a-zA-Z0-9._-], so non-ASCII userinfo and IDN hosts are not recognized
// either.
func isSCPStyleGitURL(url string) bool {
idx := strings.Index(url, ":")
if idx <= 0 || strings.Contains(url[:idx], "/") {
return false
}
if strings.Contains(url, "@") {
return true
}
return strings.Contains(url[:idx], ".")
return !strings.Contains(url, "://") && scpStyleGitURLPattern.MatchString(url)
}
// CanonicalForComparison returns a form of url suitable for equality checks

View file

@ -0,0 +1,76 @@
package doltremote_test
import (
"testing"
"github.com/steveyegge/beads/internal/doltremote"
"github.com/steveyegge/beads/internal/remotecache"
"github.com/steveyegge/beads/internal/storage/doltutil"
)
func TestS3RemotePreservesInternalBoundaries(t *testing.T) {
// The "@" in the path is what the old SCP heuristic tripped on.
const raw = "s3://bucket/team@prod/db?endpoint=https://acct.r2.example&region=auto&path-style=true"
if err := remotecache.ValidateRemoteURL(raw); err != nil {
t.Fatalf("ValidateRemoteURL(%q): %v", raw, err)
}
if got := doltremote.Normalize(raw); got != raw {
t.Errorf("Normalize(%q) = %q, want unchanged", raw, got)
}
if doltutil.IsGitProtocolURL(raw) {
t.Errorf("IsGitProtocolURL(%q) = true, want false", raw)
}
}
// TestSCPGrammarDivergesFromRemotecacheOnlyOnDottedHost pins the one form on
// which doltremote's SCP grammar and remotecache's intentionally disagree.
//
// scpStyleGitURLPattern's first alternative is byte-identical to
// remotecache.gitSSHPattern; the whole pattern is a superset by exactly its
// second alternative, the user-less dotted-host form. Neither file referenced
// the other before this test, so silent drift was one edit away in either. This
// asserts the relationship through each package's public behavior rather than
// by comparing the two regexes, which would just be a third copy of the grammar.
func TestSCPGrammarDivergesFromRemotecacheOnlyOnDottedHost(t *testing.T) {
// The divergence itself: doltremote converts it, remotecache does not
// classify it as a remote URL at all.
const dottedHost = "github.com:org/repo.git"
if got, want := doltremote.Normalize(dottedHost), "git+ssh://github.com/org/repo.git"; got != want {
t.Errorf("Normalize(%q) = %q, want %q", dottedHost, got, want)
}
if remotecache.IsRemoteURL(dottedHost) {
t.Errorf("IsRemoteURL(%q) = true, want false - remotecache omits the dotted-host alternative", dottedHost)
}
// Everywhere else the two grammars agree, which is what makes the line
// above the *only* divergence rather than one of several.
agree := []string{
"git@github.com:org/repo.git",
"deploy@myserver.com:beads/data",
"user.name@host.com:path",
}
for _, raw := range agree {
t.Run(raw, func(t *testing.T) {
if !remotecache.IsRemoteURL(raw) {
t.Errorf("IsRemoteURL(%q) = false, want true", raw)
}
if got := doltremote.Normalize(raw); got == raw {
t.Errorf("Normalize(%q) returned it unconverted, want an SCP conversion", raw)
}
})
}
// And neither treats these as SCP-style git remotes.
reject := []string{
"host:pa@th",
"s3://bucket/team@prod/db",
}
for _, raw := range reject {
t.Run(raw, func(t *testing.T) {
if got := doltremote.Normalize(raw); got != raw {
t.Errorf("Normalize(%q) = %q, want unchanged", raw, got)
}
})
}
}

View file

@ -0,0 +1,157 @@
package doltremote
import (
"slices"
"testing"
)
func TestIsSCPStyleGitURLRecognizesValidForms(t *testing.T) {
tests := []string{
"git@github.com:org/repo.git",
"deploy@myserver.com:beads/data",
"git@github:org/repo.git",
"github.com:org/repo.git",
// Dots in the user token are inside the accepted charset.
"user.name@host.com:path",
}
for _, raw := range tests {
t.Run(raw, func(t *testing.T) {
if !isSCPStyleGitURL(raw) {
t.Errorf("isSCPStyleGitURL(%q) = false, want true", raw)
}
})
}
}
func TestIsSCPStyleGitURLRejectsNonSCPInputs(t *testing.T) {
tests := []string{
"s3://bucket/team@prod/beads",
"s3://bucket/db?endpoint=https://minio.local/api@v1",
`C:\Users\alice\beads`,
"C:/Users/alice/beads",
// Empty path. The "@" alone used to classify this as SCP-style; the
// anchored grammar requires at least one path character.
"git@host.com:",
// Dotless host with the only "@" after the colon. The "@" alone used
// to classify these as SCP-style (host:pa@th -> git+ssh://host/pa@th);
// the anchored grammar wants user@host or a dotted host before the colon.
"host:pa@th",
"alias:repo@v1",
// Non-ASCII userinfo is outside [a-zA-Z0-9._-]; the URL passes
// through unconverted instead of being rewritten to git+ssh://.
"usér@host.com:path",
// IDN host, same charset limit.
"git@bücher.example:repo",
}
for _, raw := range tests {
t.Run(raw, func(t *testing.T) {
if isSCPStyleGitURL(raw) {
t.Errorf("isSCPStyleGitURL(%q) = true, want false", raw)
}
})
}
}
// TestNativeSchemesContainsEachNativeScheme guards the documented contents of
// NativeSchemes, nothing more. It deliberately asserts no behavior: the list is
// a fast path with no observable effect, because every entry contains "://",
// isSCPStyleGitURL rejects those outright, and Normalize's closing "return url"
// therefore hands back the same bytes for an entry that is present and for one
// that has been deleted. Removing an entry is caught here, as a documentation
// regression; no assertion over Normalize can catch it while that guard stands.
// The behavior this package actually relies on is pinned by the
// isSCPStyleGitURL tables above and by remote_invariant_test.go.
func TestNativeSchemesContainsEachNativeScheme(t *testing.T) {
tests := []string{
"dolthub://",
"file://",
"aws://",
"gs://",
"s3://",
"git+https://",
"git+ssh://",
"git+http://",
"git+file://",
}
for _, scheme := range tests {
t.Run(scheme, func(t *testing.T) {
if !slices.Contains(NativeSchemes, scheme) {
t.Errorf("slices.Contains(NativeSchemes, %q) = false, want true", scheme)
}
})
}
}
// TestFromGitURLLeavesSchemeURLsUnsplit pins the exported entry point against
// the misparse the PR removed from Normalize. Normalize never routes a scheme
// URL here, but FromGitURL is exported and gitURLToDoltRemote (cmd/bd) calls it
// directly, so the refusal has to live in FromGitURL itself rather than in its
// caller.
func TestFromGitURLLeavesSchemeURLsUnsplit(t *testing.T) {
tests := []struct {
raw string
want string
}{
// Before the "://" guard this returned git+ssh://s3///bucket/team@prod/db:
// the first colon sits at index 2 with no "/" before it, so the SCP split
// fired and fabricated "s3" as the host.
{"s3://bucket/team@prod/db", "git+s3://bucket/team@prod/db"},
{
"s3://bucket/db?endpoint=https://acct.r2.example&region=auto",
"git+s3://bucket/db?endpoint=https://acct.r2.example&region=auto",
},
{"gs://bucket/team@prod/db", "git+gs://bucket/team@prod/db"},
// az:// and oci:// are deliberately absent from NativeSchemes (#6227).
// The guard is keyed on "://", not on list membership, so they are
// protected here exactly as the listed schemes are - which is why
// completing that list is not what closes this hole.
{
"az://account.blob.core.windows.net/container/beads",
"git+az://account.blob.core.windows.net/container/beads",
},
{"oci://registry.example/tenant@prod/db", "git+oci://registry.example/tenant@prod/db"},
}
for _, tt := range tests {
t.Run(tt.raw, func(t *testing.T) {
if got := FromGitURL(tt.raw); got != tt.want {
t.Errorf("FromGitURL(%q) = %q, want %q", tt.raw, got, tt.want)
}
})
}
}
// TestFromGitURLStillConvertsSCPForms is the opposite-polarity twin of
// TestFromGitURLLeavesSchemeURLsUnsplit. The guard added there is scoped to
// "://" on purpose: gating the split on isSCPStyleGitURL instead would also
// stop converting the SCP hosts that predicate deliberately declines to
// classify, and the last three rows are exactly those. They reach FromGitURL
// only through a direct call, never through Normalize, and converting them is
// pre-existing behavior this change must not quietly drop.
func TestFromGitURLStillConvertsSCPForms(t *testing.T) {
tests := []struct {
raw string
want string
}{
{"git@github.com:org/repo.git", "git+ssh://git@github.com/org/repo.git"},
{"deploy@myserver.com:beads/data", "git+ssh://deploy@myserver.com/beads/data"},
{"github.com:org/repo.git", "git+ssh://github.com/org/repo.git"},
// Non-ASCII userinfo: outside scpStyleGitURLPattern's charset.
{"usér@host.com:path", "git+ssh://usér@host.com/path"},
// IDN host: same charset limit.
{"git@bücher.example:repo", "git+ssh://git@bücher.example/repo"},
// Dotless SSH config alias: no "@", no "." before the colon.
{"github:org/repo.git", "git+ssh://github/org/repo.git"},
}
for _, tt := range tests {
t.Run(tt.raw, func(t *testing.T) {
if got := FromGitURL(tt.raw); got != tt.want {
t.Errorf("FromGitURL(%q) = %q, want %q", tt.raw, got, tt.want)
}
})
}
}

View file

@ -49,6 +49,13 @@ var allowedSchemes = map[string]bool{
// gitSSHPattern matches SCP-style git remote URLs (user@host:path).
// The path portion excludes control characters (0x00-0x1f, 0x7f).
//
// This is byte-identical to the first alternative of
// doltremote.scpStyleGitURLPattern in internal/doltremote/remote.go. That one
// is a superset: it also accepts the user-less dotted-host form
// ("github.com:org/repo.git"), which IsRemoteURL intentionally reports as false
// (pinned by TestIsRemoteURLRejectsUserlessDottedHost). The two are one grammar
// maintained in two places, so keep them in sync.
var gitSSHPattern = regexp.MustCompile(`^[a-zA-Z0-9._-]+@[a-zA-Z0-9][a-zA-Z0-9._-]*:[^\x00-\x1f\x7f]+$`)
// validRemoteNameRegex matches valid remote names: starts with a letter,

View file

@ -47,6 +47,39 @@ func TestIsRemoteURL(t *testing.T) {
}
}
// TestIsRemoteURLRejectsUserlessDottedHost pins this package's side of the one
// intentional disagreement with doltremote.scpStyleGitURLPattern, whose first
// alternative is byte-identical to gitSSHPattern but which additionally accepts
// the user-less dotted-host form. doltremote.Normalize converts that form to
// git+ssh://; IsRemoteURL treats it as a local path. Changing this behavior
// means reconciling both files, not just this one.
func TestIsRemoteURLRejectsUserlessDottedHost(t *testing.T) {
userless := []string{
"github.com:org/repo.git",
"myserver.com:beads/data",
}
for _, raw := range userless {
t.Run(raw, func(t *testing.T) {
if IsRemoteURL(raw) {
t.Errorf("IsRemoteURL(%q) = true, want false", raw)
}
})
}
// The shared alternative: identical charset, and both packages accept it.
withUser := []string{
"git@github.com:org/repo.git",
"deploy@myserver.com:beads/data",
}
for _, raw := range withUser {
t.Run(raw, func(t *testing.T) {
if !IsRemoteURL(raw) {
t.Errorf("IsRemoteURL(%q) = false, want true", raw)
}
})
}
}
func TestValidateRemoteURL(t *testing.T) {
tests := []struct {
name string