beads/internal/storage/embeddeddolt/federation.go
Steve Yegge 387959695d
fix(dolt): pre-pull commit must include config (kv.memory.* pull-wedge) (#4412)
* fix(dolt): pre-pull commit must include config; auto-resolve convergent kv.memory.* config conflicts

bd remember stores persistent memories as kv.memory.* rows in the synced
config table. Commit() deliberately excludes config (GH#2455, to avoid
sweeping a concurrent writer's stale issue_prefix into an unrelated commit),
so the GH#2474 pre-pull auto-commit at pullFromRemote (store.go) and PullFrom
(federation.go) leaves config permanently dirty once any memory exists. The
"clean the working set before merging" step then leaves config dirty and
DOLT_MERGE refuses to start ("cannot merge with uncommitted changes"). Two
correct PRs colliding over config. Diagnosed from a live server-mode repro
(Wyvern wy-71t); see PROPOSAL-pull-config-wedge.md.

Fix (server mode): add commitBeforePull, which stages config explicitly via
DOLT_ADD rather than relying on CommitWithConfig's DOLT_COMMIT('-Am'). The
proposal author observed -Am not staging config reliably under the server-mode
stored-procedure path, and the new embedded test TestEmbeddedConfigCommitStagesConfig
confirms -Am DOES stage config in the embedded engine — so the unreliability is
server-mode-specific and explicit staging is the robust choice. Commit() and
commitBeforePull now share commitWorkingSet(includeConfig bool).

Hardening (both modes): making config a committed/synced table means two clones
editing the same memory key collide on a config merge conflict. Extend
TryAutoResolveMergeConflicts to auto-resolve config conflicts with --theirs ONLY
when every conflicted row is a kv.memory.* key (machine-convergent, same policy
as metadata); any other config key (issue_prefix above all) is left for the
operator. Without this the primary fix would trade a silent pull wedge for an
operator-visible conflict wedge on the exact bd remember workload.

Embedded pre-pull (CommitPending -> -Am) already stages config and is unchanged;
it gains the resolver hardening.

Tests: server-mode pull_config_wedge_test.go (commit-includes-config + resolver
positive/negative, run under Docker in CI) and embedded pull_config_wedge_test.go
(in-process: -Am-stages-config probe + resolver positive/negative, all green).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: add pull-config-wedge diagnosis/proposal (PR #4412)

The diagnosis-and-patch write-up behind this PR: how GH#2455 (Commit excludes
config) and GH#2474 (pre-pull auto-commit) collide over the synced config table,
leaving kv.memory.* rows permanently uncommitted so DOLT_MERGE refuses to start.
Includes the live-DB validation (Wyvern wy-71t) and the rejected alternatives.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(dolt): adopt-pr maintainer fixes for pull/sync config wedge

Maintainer review fixups for PR #4412, squashed into one commit on top of
the contributor's branch.

Pre-merge config wedge (sync): bd federation sync committed with the
config-excluding Commit before its merge while this PR switched the pull
paths to commitBeforePull, so any DB that ran `bd remember` carried dirty
kv.memory.* config rows and hit the "cannot merge with uncommitted changes"
wedge. Sync now uses commitBeforePull too, with a regression test.

Pre-pull auto-commit safety (GH#2455): commitBeforePull no longer stages the
whole config table. It auto-commits only convergent memory rows
(kv.memory.*); if any other config key is dirty (issue_prefix above all) it
refuses with operator guidance instead of sweeping a stale value into the
merge basis, restoring the GH#2455 protection the broad pre-pull staging had
reopened.

Sync --strategy config commit: a config-only conflict resolution committed
through the config-excluding Commit, so it could report success without
creating the merge-resolution commit and re-wedge the next sync. Sync now
commits resolutions through a config-inclusive path (commitMergeResolution).

Embedded sync parity: EmbeddedDoltStore.Sync fetched and merged without the
pre-merge CommitPending guard already used by embedded Pull/PullRemote/PullFrom
and the new server Sync, so the same dirty kv.memory.* working set could still
wedge `bd federation sync` in embedded mode. Sync now commits pending changes
before fetch/merge (embedded Commit is DOLT_COMMIT('-Am'), which stages
config), with an embedded regression test that drives Sync with a dirty
kv.memory.* row.

KV/memory namespace collision: `bd kv set memory.*` stored to kv.memory.*,
indistinguishable from a `bd remember` memory the merge resolver auto-resolves
with --theirs. validateKVKey now reserves the memory. prefix so a generic KV
value can't be silently overridden by the remote on pull.

Shared prefix + tests: centralize the kv.memory. prefix in
internal/storage/kvkeys (used by cmd/bd and the merge resolver) with a
contract test; route the remaining cmd/bd kv. prefix literals (the doctor KV
check, the validateKVKey nested-prefix guard, and the memory listing) through
the kvkeys constants; and strengthen the server-mode non-memory config
conflict test to prove a conflict materialized before the resolver declines it.

Maintainer review fixup for PR #4412.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Eddie the Engineer <ci@beads.test>
2026-06-19 21:31:33 -07:00

353 lines
10 KiB
Go

//go:build cgo
package embeddeddolt
import (
"context"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"database/sql"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
"github.com/steveyegge/beads/internal/storage"
"github.com/steveyegge/beads/internal/storage/issueops"
"github.com/steveyegge/beads/internal/storage/versioncontrolops"
)
// credentialKeyFile is the filename for the random encryption key.
const credentialKeyFile = ".beads-credential-key" //nolint:gosec // G101: filename, not a credential
// ensureCredentialKey lazily initializes the credential encryption key.
func (s *EmbeddedDoltStore) ensureCredentialKey() error {
if s.credentialKey != nil {
return nil
}
if s.beadsDir == "" {
return fmt.Errorf("beads directory not set; credential encryption unavailable")
}
keyPath := filepath.Join(s.beadsDir, credentialKeyFile)
// Try to load existing key.
key, err := os.ReadFile(keyPath) //nolint:gosec // G304: keyPath derived from trusted beadsDir
if err == nil && len(key) == 32 {
s.credentialKey = key
return nil
}
// Generate new random 32-byte key (AES-256).
key = make([]byte, 32)
if _, err := io.ReadFull(rand.Reader, key); err != nil {
return fmt.Errorf("generate credential key: %w", err)
}
if err := os.WriteFile(keyPath, key, 0600); err != nil {
return fmt.Errorf("write credential key: %w", err)
}
s.credentialKey = key
return nil
}
func (s *EmbeddedDoltStore) encryptPassword(password string) ([]byte, error) {
if password == "" {
return nil, nil
}
if err := s.ensureCredentialKey(); err != nil {
return nil, err
}
block, err := aes.NewCipher(s.credentialKey)
if err != nil {
return nil, err
}
gcm, err := cipher.NewGCM(block)
if err != nil {
return nil, err
}
nonce := make([]byte, gcm.NonceSize())
if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
return nil, err
}
return gcm.Seal(nonce, nonce, []byte(password), nil), nil
}
func (s *EmbeddedDoltStore) decryptPassword(encrypted []byte) (string, error) {
if len(encrypted) == 0 {
return "", nil
}
if err := s.ensureCredentialKey(); err != nil {
return "", err
}
block, err := aes.NewCipher(s.credentialKey)
if err != nil {
return "", err
}
gcm, err := cipher.NewGCM(block)
if err != nil {
return "", err
}
nonceSize := gcm.NonceSize()
if len(encrypted) < nonceSize {
return "", fmt.Errorf("ciphertext too short")
}
nonce, ciphertext := encrypted[:nonceSize], encrypted[nonceSize:]
plaintext, err := gcm.Open(nil, nonce, ciphertext, nil)
if err != nil {
return "", err
}
return string(plaintext), nil
}
// ---------------------------------------------------------------------------
// FederationStore implementation
// ---------------------------------------------------------------------------
func (s *EmbeddedDoltStore) AddFederationPeer(ctx context.Context, peer *storage.FederationPeer) error {
encryptedPwd, err := s.encryptPassword(peer.Password)
if err != nil {
return fmt.Errorf("encrypt password: %w", err)
}
if err := s.withConn(ctx, true, func(tx *sql.Tx) error {
if err := issueops.AddFederationPeerInTx(ctx, tx, peer, encryptedPwd); err != nil {
return err
}
// Also add the Dolt remote.
return issueops.AddRemoteIfNotExists(ctx, tx, peer.Name, peer.RemoteURL)
}); err != nil {
return err
}
return nil
}
func (s *EmbeddedDoltStore) GetFederationPeer(ctx context.Context, name string) (*storage.FederationPeer, error) {
var row *issueops.FederationPeerRow
err := s.withConn(ctx, false, func(tx *sql.Tx) error {
var err error
row, err = issueops.GetFederationPeerInTx(ctx, tx, name)
return err
})
if err != nil {
return nil, err
}
if len(row.EncryptedPwd) > 0 {
row.Peer.Password, err = s.decryptPassword(row.EncryptedPwd)
if err != nil {
return nil, fmt.Errorf("decrypt password: %w", err)
}
}
return &row.Peer, nil
}
func (s *EmbeddedDoltStore) ListFederationPeers(ctx context.Context) ([]*storage.FederationPeer, error) {
var rows []*issueops.FederationPeerRow
err := s.withConn(ctx, false, func(tx *sql.Tx) error {
var err error
rows, err = issueops.ListFederationPeersInTx(ctx, tx)
return err
})
if err != nil {
return nil, err
}
peers := make([]*storage.FederationPeer, 0, len(rows))
for _, row := range rows {
if len(row.EncryptedPwd) > 0 {
pwd, err := s.decryptPassword(row.EncryptedPwd)
if err != nil {
return nil, fmt.Errorf("decrypt password for peer %s: %w", row.Peer.Name, err)
}
row.Peer.Password = pwd
}
peers = append(peers, &row.Peer)
}
return peers, nil
}
func (s *EmbeddedDoltStore) RemoveFederationPeer(ctx context.Context, name string) error {
if err := s.withConn(ctx, true, func(tx *sql.Tx) error {
return issueops.RemoveFederationPeerInTx(ctx, tx, name)
}); err != nil {
return err
}
// Also remove the Dolt remote (best-effort).
if rmErr := s.RemoveRemote(ctx, name); rmErr != nil {
if !strings.Contains(rmErr.Error(), "not found") {
// Silently ignore "not found" — the remote may not exist.
_ = rmErr
}
}
return nil
}
// ---------------------------------------------------------------------------
// SyncStore implementation
// ---------------------------------------------------------------------------
// Sync performs a full bidirectional sync with a peer:
// 1. Fetch from peer
// 2. Merge peer's changes (handling conflicts per strategy)
// 3. Push local changes to peer
func (s *EmbeddedDoltStore) Sync(ctx context.Context, peer string, strategy string) (*storage.SyncResult, error) {
result := &storage.SyncResult{
Peer: peer,
StartTime: time.Now(),
}
// GH#2474 / bd-578h9.2: commit pending changes before the merge, matching
// embedded Pull/PullRemote/PullFrom and server-mode Sync. Embedded Commit is
// DOLT_COMMIT('-Am'), so it stages config — where kv.memory.* memories live —
// and a leftover dirty working set (e.g. a `bd remember` write) would
// otherwise make DOLT_MERGE refuse to start ("cannot merge with uncommitted
// changes"). CommitPending is a no-op when the working set is already clean.
if _, err := s.CommitPending(ctx, "beads"); err != nil {
result.Error = fmt.Errorf("commit pending before sync: %w", err)
return result, result.Error
}
// Step 1: Fetch
if err := s.Fetch(ctx, peer); err != nil {
result.Error = fmt.Errorf("fetch failed: %w", err)
return result, result.Error
}
result.Fetched = true
// Step 2: Get commit before merge for change detection
beforeCommit, _ := s.GetCurrentCommit(ctx)
// Step 3: Merge peer's branch
remoteBranch := fmt.Sprintf("%s/%s", peer, s.branch)
conflicts, err := s.Merge(ctx, remoteBranch)
if err != nil {
result.Error = fmt.Errorf("merge failed: %w", err)
return result, result.Error
}
// Step 4: Handle conflicts
if len(conflicts) > 0 {
result.Conflicts = conflicts
if strategy == "" {
result.Error = fmt.Errorf("merge conflicts require resolution (use --strategy ours|theirs)")
return result, result.Error
}
for _, c := range conflicts {
if err := s.ResolveConflicts(ctx, c.Field, strategy); err != nil {
result.Error = fmt.Errorf("conflict resolution failed for %s: %w", c.Field, err)
return result, result.Error
}
}
result.ConflictsResolved = true
if err := s.Commit(ctx, fmt.Sprintf("Resolve conflicts from %s using %s strategy", peer, strategy)); err != nil {
result.Error = fmt.Errorf("commit conflict resolution: %w", err)
return result, result.Error
}
// bd-578h9.11: the conflicted merge skipped the automatic is_blocked
// recompute (unresolved rows would have fed it garbage); now that the
// resolution is committed, cover the whole merge+resolution window.
if err := s.RecomputeBlockedAfterMerge(ctx, beforeCommit); err != nil {
result.Error = fmt.Errorf("conflicts resolved but is_blocked recompute failed: %w", err)
return result, result.Error
}
}
result.Merged = true
afterCommit, _ := s.GetCurrentCommit(ctx)
if beforeCommit != afterCommit {
result.PulledCommits = 1
}
// Step 5: Push
if err := s.PushTo(ctx, peer); err != nil {
result.PushError = err
} else {
result.Pushed = true
}
// Record last sync time in metadata.
_ = s.setLastSyncTime(ctx, peer)
result.EndTime = time.Now()
return result, nil
}
// SyncStatus returns the synchronization status with a peer.
func (s *EmbeddedDoltStore) SyncStatus(ctx context.Context, peer string) (*storage.SyncStatus, error) {
status := &storage.SyncStatus{
Peer: peer,
}
// Get ahead/behind counts by comparing refs.
// Dolt's AS OF requires a literal ref, not a parameterized expression.
remoteRef := peer + "/" + s.branch
if err := issueops.ValidateRef(remoteRef); err != nil {
status.LocalAhead = -1
status.LocalBehind = -1
} else if err := s.withDBConn(ctx, func(db versioncontrolops.DBConn) error {
query := fmt.Sprintf(`
SELECT
(SELECT COUNT(*) FROM dolt_log WHERE commit_hash NOT IN
(SELECT commit_hash FROM dolt_log AS OF '%s')) as ahead,
(SELECT COUNT(*) FROM dolt_log AS OF '%s' WHERE commit_hash NOT IN
(SELECT commit_hash FROM dolt_log)) as behind
`, remoteRef, remoteRef)
if err := db.QueryRowContext(ctx, query).
Scan(&status.LocalAhead, &status.LocalBehind); err != nil {
// Remote branch may not exist locally yet.
status.LocalAhead = -1
status.LocalBehind = -1
}
return nil
}); err != nil {
return nil, err
}
// Check for conflicts.
conflicts, err := s.GetConflicts(ctx)
if err == nil && len(conflicts) > 0 {
status.HasConflicts = true
}
// Get last sync time.
status.LastSync = s.getLastSyncTime(ctx, peer)
return status, nil
}
// setLastSyncTime records the last sync time for a peer in metadata.
func (s *EmbeddedDoltStore) setLastSyncTime(ctx context.Context, peer string) error {
key := "last_sync_" + peer
value := time.Now().Format(time.RFC3339)
return s.withConn(ctx, true, func(tx *sql.Tx) error {
_, err := tx.ExecContext(ctx,
"REPLACE INTO metadata (`key`, value) VALUES (?, ?)", key, value)
return err
})
}
// getLastSyncTime retrieves the last sync time for a peer from metadata.
func (s *EmbeddedDoltStore) getLastSyncTime(ctx context.Context, peer string) time.Time {
key := "last_sync_" + peer
var value string
err := s.withConn(ctx, false, func(tx *sql.Tx) error {
return tx.QueryRowContext(ctx, "SELECT value FROM metadata WHERE `key` = ?", key).Scan(&value)
})
if err != nil {
return time.Time{}
}
t, err := time.Parse(time.RFC3339, value)
if err != nil {
return time.Time{}
}
return t
}