From f62cc4f072d456479c8269611a5b88f1a583089b Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 16 Sep 2026 01:12:57 -0700 Subject: [PATCH 01/45] fix(scripts): make check-doc-flags portable to BSD grep and fail loud on grep errors (#6026) (#6581) (cherry picked from commit 4c4be78b5c935ae8f819be84b2a636476293f267) Co-authored-by: postoso Co-authored-by: postoso --- scripts/check-doc-flags.sh | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/scripts/check-doc-flags.sh b/scripts/check-doc-flags.sh index 193ed1579..96ce8d39a 100755 --- a/scripts/check-doc-flags.sh +++ b/scripts/check-doc-flags.sh @@ -93,8 +93,16 @@ echo "" # --- Check 2: bd init flags --- echo "=== Check 2: bd init flags ===" -# Get actual init flags -INIT_FLAGS=$($BD init --help 2>&1 | grep -oP '^\s+--[a-z][a-z0-9-]*' | sed 's/^\s*//' || true) +# Get actual init flags. Match both long-only rows and rows with a short alias. +INIT_FLAGS_OK=1 +if ! INIT_FLAGS=$("$BD" init --help 2>&1 \ + | grep -E '^[[:space:]]+(-[[:alnum:]], )?--[a-z]' \ + | grep -oE -- '--[a-z][a-z0-9-]*' \ + | sort -u); then + echo "FAIL: Could not extract flags from 'bd init --help'" + ERRORS=$((ERRORS + 1)) + INIT_FLAGS_OK=0 +fi # Check for --branch on init (removed) BRANCH_REFS=$(grep -rn 'bd init.*--branch' \ @@ -107,7 +115,11 @@ BRANCH_REFS=$(grep -rn 'bd init.*--branch' \ | grep -v 'CHANGELOG\|removed\|was removed\|no longer\|deprecated' \ || true) -if [ -n "$BRANCH_REFS" ]; then +if [ "$INIT_FLAGS_OK" -ne 1 ]; then + echo "SKIP: Cannot validate 'bd init --branch' without the live init flag list" +elif printf '%s\n' "$INIT_FLAGS" | grep -qx -- '--branch'; then + echo "PASS: 'bd init --branch' is a live flag" +elif [ -n "$BRANCH_REFS" ]; then echo "FAIL: Found references to removed 'bd init --branch' flag:" echo "$BRANCH_REFS" | head -20 ERRORS=$((ERRORS + 1)) From 87c7375d5ff62756163011bd8c561f50860f422b Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 16 Sep 2026 01:13:07 -0700 Subject: [PATCH 02/45] fix(dolt): contain `bd dolt start` on proxied workspaces; make `bd dolt status` truthful [release/1.3.0] (#6580) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Align proxied-local-smoke Dolt CLI pin with measured 2.2.0 (refs be-egcj2.3) (#6002) * test(feat): red — Align dolt-sql-server test container version with host DOLT_VERSION pin (refs be-egcj2.3) * feat: green — Align dolt-sql-server test container version with host DOLT_VERSION pin (refs be-egcj2.3) * chore(ci): retrigger CI after transient golangci-lint schema-fetch timeout (refs be-fjj0i) (cherry picked from commit ea89b160fbf00c0ccf0da154a8d0cf690d00235e) * fix(dolt): contain `bd dolt start` on proxied workspaces; make `bd dolt status` truthful `bd dolt start` knew nothing about proxied-server mode, and the default proxied root IS `.beads/dolt` (internal/doltserver/physical_root.go:113-127). With the proxy tree down it launched an unsupervised `dolt sql-server` directly over that root; the next ordinary bd command then failed with `invalid connection` because the relaunched proxy found a foreign server on its port. With the tree up it adopted the proxy's own dolt child into bd's classic PID/port records, leaving two managers for one process. `bd dolt status` read the classic pidfile, which proxied mode never writes, and reported "not running" against a healthy backend — the wrong answer that sent operators to `bd dolt start` in the first place. start is now a typed refusal (`proxy.dolt_start.conflict`): the proxy owns its child's lifecycle, and an idle reaper means a "started" postcondition bd could not honor beyond the idle window. status reads the proxy's own records and reports the proxy and its backend separately, starting nothing. Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Jim Wordelman Co-authored-by: Claude Fable 5 --- .github/workflows/proxied-local-smoke.yml | 14 +- CHANGELOG.md | 29 ++ cmd/bd/dolt.go | 51 ++- cmd/bd/dolt_lifecycle_proxied_server_test.go | 108 ++++++ cmd/bd/dolt_proxied_lifecycle.go | 144 ++++++++ cmd/bd/dolt_proxied_lifecycle_test.go | 187 ++++++++++ cmd/bd/proxied_local_dolt_lifecycle_test.go | 321 ++++++++++++++++++ internal/storage/dbproxy/proxy/read_status.go | 62 ++++ scripts/ci_workflow_test.go | 26 ++ 9 files changed, 936 insertions(+), 6 deletions(-) create mode 100644 cmd/bd/dolt_lifecycle_proxied_server_test.go create mode 100644 cmd/bd/dolt_proxied_lifecycle.go create mode 100644 cmd/bd/dolt_proxied_lifecycle_test.go create mode 100644 cmd/bd/proxied_local_dolt_lifecycle_test.go create mode 100644 internal/storage/dbproxy/proxy/read_status.go diff --git a/.github/workflows/proxied-local-smoke.yml b/.github/workflows/proxied-local-smoke.yml index 93a0eab2d..e19f4f8dd 100644 --- a/.github/workflows/proxied-local-smoke.yml +++ b/.github/workflows/proxied-local-smoke.yml @@ -21,6 +21,7 @@ on: paths: - 'cmd/bd/proxied_local_*' - 'cmd/bd/proxied_server*.go' + - 'cmd/bd/dolt*.go' - 'cmd/bd/*_proxied_server.go' - 'cmd/bd/db_proxy_child.go' - 'cmd/bd/init*.go' @@ -39,6 +40,7 @@ on: paths: - 'cmd/bd/proxied_local_*' - 'cmd/bd/proxied_server*.go' + - 'cmd/bd/dolt*.go' - 'cmd/bd/*_proxied_server.go' - 'cmd/bd/db_proxy_child.go' - 'cmd/bd/init*.go' @@ -65,8 +67,10 @@ jobs: env: CGO_ENABLED: "1" # Pinned so the lane tests a known dolt CLI, not whatever the installer - # resolves on a given day. Bump deliberately. - DOLT_VERSION: 2.2.2 + # resolves on a given day. Bump deliberately, and only in lockstep with + # the measured pin in scripts/ci/install-dolt.sh (see "Which Dolt + # version to install" in docs/architecture/dolt.md). + DOLT_VERSION: 2.2.0 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v6 @@ -102,7 +106,11 @@ jobs: TestManagedLocalProxiedLegacyRecordFailClosed \ TestManagedLocalProxiedOrphanBackendReaped \ TestManagedLocalProxiedNProcessColdStartConvergence \ - TestManagedLocalProxiedStopStartInterleave + TestManagedLocalProxiedStopStartInterleave \ + TestManagedLocalProxiedDoltStatusReportsLiveTopology \ + TestManagedLocalProxiedDoltStartRefusesOverLiveProxy \ + TestManagedLocalProxiedDoltStartRefusesWithProxyDown \ + TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone do if ! grep -qx "$test_name" <<<"$listed"; then echo "managed-local lifecycle test $test_name is missing; got: '$listed'" >&2 diff --git a/CHANGELOG.md b/CHANGELOG.md index a560f470b..442ce035b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- **`bd dolt start` no longer puts a second sql-server over a proxied + workspace's data directory.** On a proxied-server workspace the default root + IS `.beads/dolt`, and `bd dolt start` knew nothing about proxied mode: with + the proxy quiesced it launched an unsupervised `dolt sql-server` directly + over that root, after which ordinary bd commands failed (`invalid + connection`) because the relaunched proxy found a foreign server on its + port; with the proxy live it instead adopted the proxy's own dolt child into + bd's classic PID/port records, leaving two managers for one process. It is + now a typed refusal with code `proxy.dolt_start.conflict`. The proxy owns + its backend's lifecycle — it starts on demand and `bd dolt stop` shuts it + down. Direct-server and embedded workspaces are unaffected. + +- **`bd dolt status` tells the truth on a proxied workspace.** It read the + classic PID file, which proxied mode never writes, and so reported `Dolt + server: not running` while the proxy was serving CRUD — the wrong answer + that sent operators to `bd dolt start` in the first place. It now reads the + proxy's own records and reports the proxy and its dolt backend separately, + without starting either. **JSON output shape change**: on a proxied + workspace `bd dolt status --json` now emits + `{"mode": "proxied-server", "root": ..., "running": , "proxy_pid": + ..., "proxy_port": ..., "backend_managed": ..., "backend_running": ..., + "backend_pid": ..., "backend_port": ..., "idle_timeout": ...}` instead of + the always-false `{"running": false, "pid": 0, "port": 0}`. `running` + describes the proxy, the endpoint every bd command connects through; + `backend_managed` is false on external proxied topologies, where the dolt + server is not bd's process to report on. + ## [1.3.0] - 2026-09-15 The first tested release off `main` since the 1.1 line. [1.2.2] was a recovery diff --git a/cmd/bd/dolt.go b/cmd/bd/dolt.go index 174748254..009b15bb6 100644 --- a/cmd/bd/dolt.go +++ b/cmd/bd/dolt.go @@ -773,7 +773,12 @@ The server runs in the background on a per-project port derived from the project path. PID and logs are stored in .beads/. The server auto-starts transparently when needed, so manual start is rarely -required. Use this command for explicit control or diagnostics.`, +required. Use this command for explicit control or diagnostics. + +Not available in proxied-server mode: there the proxy owns its dolt backend's +lifecycle and starts it on demand, so a server started here would be a second, +unsupervised one over the same data directory. Use 'bd dolt status' to see what +is running and 'bd dolt stop' to shut it down.`, RunE: func(cmd *cobra.Command, args []string) error { beadsDir := selectedDoltBeadsDir() if beadsDir == "" { @@ -783,6 +788,26 @@ required. Use this command for explicit control or diagnostics.`, if err != nil { return HandleError("%v", err) } + // A proxied workspace already has a server manager. The proxy spawns + // its dolt child on demand and reaps the whole tree when idle, so + // there is no state for bd to add here — only a second one to + // collide with. With the tree down, doltserver.Start resolves the + // proxied root's own configured port and launches an unsupervised + // sql-server directly over .beads/dolt, and the next ordinary bd + // command then fails because the relaunched proxy finds a foreign + // server on its port; with the tree up, Start instead adopts the + // proxy's child into bd's classic PID/port records, leaving two + // managers for one process. Refused rather than routed: the proxy + // owns its child's lifecycle, and an idle reaper means a "started" + // postcondition bd cannot honor for more than the idle window. + // + // Both the process-wide mode and the selected workspace's own + // metadata are consulted, and this runs before the embedded guard: + // over-refusing costs an operator one error message, under-refusing + // costs them the datastore. + if usesProxiedServer() || fileCfg.IsDoltProxiedServerMode() { + return HandleProxyCapabilityError(proxiedDoltStartRefusal()) + } if !usesSQLServer() { return HandleError("'bd dolt start' is not supported in embedded mode (no Dolt server)") } @@ -790,7 +815,7 @@ required. Use this command for explicit control or diagnostics.`, // server lifecycle (GH#3545/GH#3518): starting a repo-local // server here would write local PID/port state that shadows the // configured remote endpoint. - if host := fileCfg.GetDoltServerHost(); !usesProxiedServer() && !configfile.IsLocalHostString(host) { + if host := fileCfg.GetDoltServerHost(); !configfile.IsLocalHostString(host) { return HandleError("the configured Dolt server host is remote (%s); 'bd dolt start' only manages a local server.\nStart the server on that host, or clear dolt_server_host / dolt.host / BEADS_DOLT_SERVER_HOST to run one locally", host) } serverDir := doltserver.ResolveServerDir(beadsDir) @@ -1067,7 +1092,12 @@ PID, port, and data directory from the local PID file. For externally- managed servers — a shared server (dolt.shared-server: true), a remote dolt_server_host, or a local server managed outside bd (dolt.auto-start: false, e.g. an orchestrator-shared sql-server) — pings the configured -endpoint via SQL and reports reachability, server version, and database.`, +endpoint via SQL and reports reachability, server version, and database. + +In proxied-server mode, reports the proxy (the endpoint every bd command +connects through) and the dolt backend behind it separately, read from the +proxy's own process records. Reporting starts nothing: a quiesced workspace +is shown as not running, and the next bd command launches it.`, RunE: func(cmd *cobra.Command, args []string) error { beadsDir := selectedDoltBeadsDir() if beadsDir == "" { @@ -1090,6 +1120,21 @@ endpoint via SQL and reports reachability, server version, and database.`, fmt.Printf("Backend: %s (no Dolt engine)\n", cfg.GetBackend()) return nil } + // Neither branch below describes a proxied workspace. It writes no + // classic PID file, so the bd-managed path reports "not running" + // while the proxy is serving CRUD, and shouldUseExternalDoltStatus + // deliberately excludes proxied so the SQL-probe path never sees it + // either. Keyed on the loaded config rather than the process-wide + // mode so `bd --db dolt status` describes the + // workspace it was pointed at. + if cfg.IsDoltProxiedServerMode() { + status, err := collectProxiedDoltStatus(beadsDir) + if err != nil { + return HandleError("%v", err) + } + renderProxiedDoltStatus(status) + return nil + } if !usesSQLServer() { showEmbeddedDoltStatus(beadsDir) return nil diff --git a/cmd/bd/dolt_lifecycle_proxied_server_test.go b/cmd/bd/dolt_lifecycle_proxied_server_test.go new file mode 100644 index 000000000..8bcb76b76 --- /dev/null +++ b/cmd/bd/dolt_lifecycle_proxied_server_test.go @@ -0,0 +1,108 @@ +//go:build cgo + +package main + +// The external-tcp half of the `bd dolt status` / `bd dolt start` coverage. +// It lives in the TestProxiedServer* lane, which runs on every PR, so the +// corruption guard is enforced there and not only in the path-filtered +// managed-local lane. +// +// The distinction this topology exercises: bd runs a proxy but spawns no dolt, +// so "the backend is not running" is a claim bd has no standing to make. + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/doltserver" + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" +) + +func TestProxiedServerDoltLifecycle(t *testing.T) { + bd := buildEmbeddedBD(t) + p := newSharedProxiedProject(t, bd, "extdolt") + t.Cleanup(func() { + serverDir := doltserver.ResolveServerDir(p.beadsDir) + if _, err := os.Stat(filepath.Join(serverDir, doltserver.PIDFileName)); err != nil { + return + } + t.Logf("stray classic dolt-server.pid under %s; stopping it", serverDir) + if err := doltserver.Stop(serverDir); err != nil { + t.Logf("doltserver.Stop(%s): %v", serverDir, err) + } + }) + bdProxiedCreate(t, bd, p.dir, "external lifecycle sentinel") + + proxyPid := readProxyPidFileOrFail(t, p) + + t.Run("status reports the proxy and disclaims the backend", func(t *testing.T) { + stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, []string{"BEADS_JSON=1"}, "--json", "dolt", "status") + if err != nil { + t.Fatalf("bd dolt status failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + var got struct { + Mode string `json:"mode"` + Running bool `json:"running"` + ProxyPID int `json:"proxy_pid"` + BackendManaged bool `json:"backend_managed"` + BackendRunning bool `json:"backend_running"` + BackendEndpoint string `json:"backend_endpoint"` + } + if jsonErr := json.Unmarshal([]byte(strings.TrimSpace(stdout)), &got); jsonErr != nil { + t.Fatalf("decode dolt status JSON: %v\nstdout:\n%s", jsonErr, stdout) + } + if got.Mode != "proxied-server" || !got.Running { + t.Errorf("status=%+v, want mode=proxied-server running=true", got) + } + if got.ProxyPID != proxyPid { + t.Errorf("proxy_pid=%d, want %d", got.ProxyPID, proxyPid) + } + if got.BackendManaged { + t.Error("backend_managed=true on an external topology; bd spawns no dolt there") + } + if got.BackendRunning { + t.Error("backend_running=true on an external topology; bd has no record to claim that from") + } + if !strings.HasPrefix(got.BackendEndpoint, "127.0.0.1:") { + t.Errorf("backend_endpoint=%q, want the configured external endpoint", got.BackendEndpoint) + } + }) + + t.Run("start is refused", func(t *testing.T) { + stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, []string{"BEADS_JSON=1"}, "--json", "dolt", "start") + if err == nil { + t.Errorf("bd dolt start succeeded on an external proxied workspace:\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + var got struct { + Code string `json:"code"` + Mutates bool `json:"mutates"` + } + if jsonErr := json.Unmarshal([]byte(strings.TrimSpace(stdout)), &got); jsonErr != nil { + t.Fatalf("refusal is not typed JSON (%v)\nstdout:\n%s\nstderr:\n%s", jsonErr, stdout, stderr) + } + if got.Code != proxyDoltStartConflictCode || got.Mutates { + t.Errorf("refusal=%+v, want code=%q mutates=false", got, proxyDoltStartConflictCode) + } + for _, name := range []string{doltserver.PIDFileName, doltserver.PortFileName} { + if _, statErr := os.Stat(filepath.Join(p.beadsDir, name)); statErr == nil { + t.Errorf("%s exists after a refused start", name) + } + } + }) + + if out, err := bdProxiedRun(t, bd, p.dir, "create", "--json", "post-refusal write"); err != nil { + t.Fatalf("workspace unusable after the lifecycle probes: %v\n%s", err, out) + } +} + +func readProxyPidFileOrFail(t *testing.T, p proxiedProject) int { + t.Helper() + running, pid := proxy.IsRunning(p.proxyRoot) + if !running { + t.Fatalf("expected a live proxy under %s after a create", p.proxyRoot) + } + return pid +} diff --git a/cmd/bd/dolt_proxied_lifecycle.go b/cmd/bd/dolt_proxied_lifecycle.go new file mode 100644 index 000000000..31b7db6cb --- /dev/null +++ b/cmd/bd/dolt_proxied_lifecycle.go @@ -0,0 +1,144 @@ +package main + +// `bd dolt start` and `bd dolt status` for proxied-server workspaces. +// +// Both commands predate proxied mode and assume bd owns the sql-server: they +// read and write .beads/dolt-server.pid. A proxied workspace never has one — +// the proxy writes proxy.pid and proxy-child.pid under the proxied root — so +// status used to report "not running" against a healthy backend, and start +// used to act on that by putting a second sql-server over the live data +// directory. This file is the proxied-aware half of both. + +import ( + "fmt" + "net" + "os" + "strconv" + + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" +) + +// Stable identity of the `bd dolt start` refusal on a proxied workspace. +// Codes are the part of the refusal contract downstream consumers branch on, +// so this one is frozen; the message beside it is informative. +const proxyDoltStartConflictCode = "proxy.dolt_start.conflict" + +const proxyDoltStartConflictMessage = "dolt start is not supported in proxied-server mode: " + + "the proxy owns its dolt backend, and a second sql-server over the same data directory risks corrupting it; " + + "the proxy starts on demand — run 'bd dolt status' to see what is running, or 'bd dolt stop' to shut it down" + +// proxiedDoltStartRefusal is enforced inside the command rather than by the +// pre-provider capability gate because `bd dolt` is a noDbCommands entry: its +// subcommands skip store init and so never reach the gate in main.go. Folding +// this row into the consolidated registry is S2's job; the corruption guard +// should not wait for it. +func proxiedDoltStartRefusal() *ProxyCapabilityError { + return &ProxyCapabilityError{ + Code: proxyDoltStartConflictCode, + Message: proxyDoltStartConflictMessage, + ExitCode: 1, + } +} + +// proxiedDoltStatus is the `bd dolt status --json` payload for a proxied +// workspace. It reports the two processes separately because they fail +// separately and an operator's next move differs: Running describes the proxy, +// which is the endpoint every bd command connects through, while the Backend* +// fields describe the dolt server behind it. +type proxiedDoltStatus struct { + Mode string `json:"mode"` + Root string `json:"root"` + Running bool `json:"running"` + ProxyPID int `json:"proxy_pid,omitempty"` + ProxyPort int `json:"proxy_port,omitempty"` + // BackendManaged is false on an external proxied topology, where the + // dolt server is somebody else's process and BackendRunning says nothing + // about it. Without this flag a caller cannot tell "the backend is down" + // from "bd never had a backend to report". + BackendManaged bool `json:"backend_managed"` + BackendRunning bool `json:"backend_running"` + BackendPID int `json:"backend_pid,omitempty"` + BackendPort int `json:"backend_port,omitempty"` + BackendEndpoint string `json:"backend_endpoint,omitempty"` + IdleTimeout string `json:"idle_timeout,omitempty"` +} + +func collectProxiedDoltStatus(beadsDir string) (proxiedDoltStatus, error) { + root, err := resolveProxiedServerRootPath(beadsDir) + if err != nil { + return proxiedDoltStatus{}, err + } + live := proxy.ReadStatus(root) + status := proxiedDoltStatus{ + Mode: configfile.DoltModeProxiedServer, + Root: root, + Running: live.ProxyRunning, + ProxyPID: live.ProxyPID, + ProxyPort: live.ProxyPort, + BackendManaged: true, + BackendRunning: live.BackendRunning, + BackendPID: live.BackendPID, + BackendPort: live.BackendPort, + } + + info, err := configfile.LoadProxiedServerClientInfo(beadsDir) + if err != nil { + return proxiedDoltStatus{}, err + } + // A missing sidecar means the workspace predates it or was hand-edited; + // managed-local is the default topology, and the pid records already read + // above are the authority on what is actually running either way. + if info != nil { + if info.IdleTimeout > 0 { + status.IdleTimeout = info.IdleTimeout.String() + } + if info.External != nil { + status.BackendManaged = false + status.BackendEndpoint = externalDoltEndpoint(*info.External) + } + } + return status, nil +} + +func externalDoltEndpoint(cfg configfile.ExternalDoltConfig) string { + if cfg.Socket != "" { + return "unix:" + cfg.Socket + } + return net.JoinHostPort(cfg.Host, strconv.Itoa(cfg.Port)) +} + +func renderProxiedDoltStatus(status proxiedDoltStatus) { + if jsonOutput { + if err := outputJSON(status); err != nil { + fmt.Fprintf(os.Stderr, "Error: %v\n", err) + } + return + } + + if status.Running { + fmt.Printf("Dolt server: running (%s)\n", status.Mode) + fmt.Printf(" Proxy PID: %d\n", status.ProxyPID) + fmt.Printf(" Proxy port: %d\n", status.ProxyPort) + } else { + fmt.Printf("Dolt server: not running (%s)\n", status.Mode) + } + fmt.Printf(" Root: %s\n", status.Root) + switch { + case !status.BackendManaged: + fmt.Printf(" Backend: external at %s (not managed by bd)\n", status.BackendEndpoint) + case status.BackendRunning: + fmt.Printf(" Backend: running (dolt PID %d, port %d)\n", status.BackendPID, status.BackendPort) + default: + fmt.Println(" Backend: not running") + } + if status.IdleTimeout != "" { + fmt.Printf(" Idle timeout: %s\n", status.IdleTimeout) + } + if !status.Running { + fmt.Println(" The proxy starts on demand; the next bd command launches it.") + } + if isDoltLocalOnly() { + fmt.Println(" Remote sync: disabled (dolt.local-only=true)") + } +} diff --git a/cmd/bd/dolt_proxied_lifecycle_test.go b/cmd/bd/dolt_proxied_lifecycle_test.go new file mode 100644 index 000000000..d12f87d0b --- /dev/null +++ b/cmd/bd/dolt_proxied_lifecycle_test.go @@ -0,0 +1,187 @@ +package main + +import ( + "encoding/json" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/configfile" +) + +// TestProxiedDoltStartRefusalContract pins the parts downstream consumers +// branch on. The code is frozen; the message is informative and may be +// reworded, but it has to keep naming the mode and pointing somewhere useful, +// because "start refused" with no onward path is what sends an operator +// looking for a manual workaround. +func TestProxiedDoltStartRefusalContract(t *testing.T) { + err := proxiedDoltStartRefusal() + if err.Code != "proxy.dolt_start.conflict" { + t.Errorf("code = %q, want %q", err.Code, "proxy.dolt_start.conflict") + } + if err.ExitCode != 1 { + t.Errorf("exit code = %d, want 1", err.ExitCode) + } + if err.Mutates { + t.Error("mutates = true; a refused start changes nothing") + } + for _, want := range []string{"proxied-server mode", "bd dolt status", "bd dolt stop"} { + if !strings.Contains(err.Message, want) { + t.Errorf("message does not mention %q: %q", want, err.Message) + } + } +} + +func TestExternalDoltEndpoint(t *testing.T) { + for _, tc := range []struct { + name string + cfg configfile.ExternalDoltConfig + want string + }{ + {"tcp", configfile.ExternalDoltConfig{Host: "127.0.0.1", Port: 3310}, "127.0.0.1:3310"}, + {"ipv6", configfile.ExternalDoltConfig{Host: "::1", Port: 3310}, "[::1]:3310"}, + {"socket wins", configfile.ExternalDoltConfig{Socket: "/tmp/dolt.sock"}, "unix:/tmp/dolt.sock"}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := externalDoltEndpoint(tc.cfg); got != tc.want { + t.Errorf("externalDoltEndpoint = %q, want %q", got, tc.want) + } + }) + } +} + +// TestCollectProxiedDoltStatusQuiescentRoot covers the case an operator hits +// most: nothing is running yet. The answer has to be "not running" WITHOUT +// starting anything — a status command that spawns the topology it is +// reporting on would defeat the idle reaper it is reporting about. +func TestCollectProxiedDoltStatusQuiescentRoot(t *testing.T) { + beadsDir := t.TempDir() + root := filepath.Join(beadsDir, "dolt") + t.Setenv("BEADS_PROXIED_SERVER_ROOT_PATH", root) + + status, err := collectProxiedDoltStatus(beadsDir) + if err != nil { + t.Fatalf("collectProxiedDoltStatus: %v", err) + } + if status.Mode != "proxied-server" { + t.Errorf("mode = %q, want %q", status.Mode, "proxied-server") + } + if status.Root != root { + t.Errorf("root = %q, want %q", status.Root, root) + } + if status.Running || status.BackendRunning { + t.Errorf("reported something running for an empty root: %+v", status) + } + if !status.BackendManaged { + t.Error("backend_managed = false with no external sidecar; managed-local is the default topology") + } + if entries, _ := filepath.Glob(filepath.Join(root, "*")); len(entries) != 0 { + t.Errorf("reading status created %v under the proxied root", entries) + } +} + +func TestCollectProxiedDoltStatusExternalTopology(t *testing.T) { + beadsDir := t.TempDir() + root := filepath.Join(beadsDir, "dolt") + t.Setenv("BEADS_PROXIED_SERVER_ROOT_PATH", root) + if err := configfile.SaveProxiedServerClientInfo(beadsDir, &configfile.ProxiedServerClientInfo{ + IdleTimeout: 45 * time.Second, + External: &configfile.ExternalDoltConfig{Host: "10.0.0.5", Port: 3306}, + }); err != nil { + t.Fatalf("SaveProxiedServerClientInfo: %v", err) + } + + status, err := collectProxiedDoltStatus(beadsDir) + if err != nil { + t.Fatalf("collectProxiedDoltStatus: %v", err) + } + if status.BackendManaged { + t.Error("backend_managed = true for an external topology; bd spawns no dolt there") + } + if status.BackendEndpoint != "10.0.0.5:3306" { + t.Errorf("backend_endpoint = %q, want %q", status.BackendEndpoint, "10.0.0.5:3306") + } + if status.IdleTimeout != "45s" { + t.Errorf("idle_timeout = %q, want %q", status.IdleTimeout, "45s") + } +} + +func TestRenderProxiedDoltStatus(t *testing.T) { + running := proxiedDoltStatus{ + Mode: "proxied-server", Root: "/w/.beads/dolt", Running: true, + ProxyPID: 111, ProxyPort: 40001, + BackendManaged: true, BackendRunning: true, BackendPID: 222, BackendPort: 40002, + IdleTimeout: "30s", + } + + t.Run("text reports both processes", func(t *testing.T) { + defer restoreJSONOutput(t)() + jsonOutput = false + out := captureStdout(t, func() error { renderProxiedDoltStatus(running); return nil }) + for _, want := range []string{ + "Dolt server: running (proxied-server)", + "Proxy PID: 111", + "Proxy port: 40001", + "Root: /w/.beads/dolt", + "Backend: running (dolt PID 222, port 40002)", + "Idle timeout: 30s", + } { + if !strings.Contains(out, want) { + t.Errorf("missing %q in:\n%s", want, out) + } + } + }) + + t.Run("text says how to start it when down", func(t *testing.T) { + defer restoreJSONOutput(t)() + jsonOutput = false + down := running + down.Running, down.ProxyPID, down.ProxyPort = false, 0, 0 + down.BackendRunning, down.BackendPID, down.BackendPort = false, 0, 0 + out := captureStdout(t, func() error { renderProxiedDoltStatus(down); return nil }) + for _, want := range []string{ + "Dolt server: not running (proxied-server)", + "Backend: not running", + "The proxy starts on demand", + } { + if !strings.Contains(out, want) { + t.Errorf("missing %q in:\n%s", want, out) + } + } + }) + + t.Run("text marks an external backend as somebody else's", func(t *testing.T) { + defer restoreJSONOutput(t)() + jsonOutput = false + external := running + external.BackendManaged, external.BackendRunning = false, false + external.BackendEndpoint = "10.0.0.5:3306" + out := captureStdout(t, func() error { renderProxiedDoltStatus(external); return nil }) + if !strings.Contains(out, "Backend: external at 10.0.0.5:3306 (not managed by bd)") { + t.Errorf("external backend not described:\n%s", out) + } + if strings.Contains(out, "Backend: not running") { + t.Errorf("external backend reported as down; bd has no record to say that from:\n%s", out) + } + }) + + t.Run("json carries the process records", func(t *testing.T) { + defer restoreJSONOutput(t)() + jsonOutput = true + out := captureStdout(t, func() error { renderProxiedDoltStatus(running); return nil }) + var got proxiedDoltStatus + if err := json.Unmarshal([]byte(strings.TrimSpace(out)), &got); err != nil { + t.Fatalf("decode %q: %v", out, err) + } + if got != running { + t.Errorf("round-trip mismatch:\ngot %+v\nwant %+v", got, running) + } + }) +} + +func restoreJSONOutput(t *testing.T) func() { + t.Helper() + orig := jsonOutput + return func() { jsonOutput = orig } +} diff --git a/cmd/bd/proxied_local_dolt_lifecycle_test.go b/cmd/bd/proxied_local_dolt_lifecycle_test.go new file mode 100644 index 000000000..97568590d --- /dev/null +++ b/cmd/bd/proxied_local_dolt_lifecycle_test.go @@ -0,0 +1,321 @@ +//go:build cgo && unix + +package main + +// Front-door lifecycle coverage for `bd dolt status` and `bd dolt start` on a +// MANAGED-LOCAL proxied workspace, plus the direct/embedded control that keeps +// the proxied guard from leaking into topologies bd really does manage. +// +// These live in the managed-local lane (TestManagedLocalProxied* runs under +// proxied-local-smoke.yml) because they are the only lane with a real proxy +// supervising a real dolt child: the defects being covered are about which +// process record bd reads and which process bd spawns, and neither is +// observable against an external testcontainer. + +import ( + "encoding/json" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/doltserver" + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" + "github.com/steveyegge/beads/internal/storage/dbproxy/server" +) + +// classicServerArtifacts are the files bd writes when IT owns a dolt +// sql-server. On a proxied workspace their appearance is the signature of the +// second-writer hazard: bd has spawned or adopted a server over the root the +// proxy serves, and now holds lifecycle state for a process it does not own. +var classicServerArtifacts = []string{ + doltserver.PIDFileName, + doltserver.PortFileName, + "dolt-server-config.yaml", +} + +// stopStrayClassicServer tears down a classic sql-server if one was recorded +// for beadsDir. It exists so that a RED run of these tests — where `bd dolt +// start` still spawns its own server over the proxied root — cannot leak that +// process past the test. Teardown goes through doltserver.Stop, which signals +// only the PID bd itself recorded. +func stopStrayClassicServer(t *testing.T, beadsDir string) { + t.Helper() + serverDir := doltserver.ResolveServerDir(beadsDir) + if _, err := os.Stat(filepath.Join(serverDir, doltserver.PIDFileName)); err != nil { + return + } + t.Logf("stray classic dolt-server.pid found under %s; stopping it", serverDir) + if err := doltserver.Stop(serverDir); err != nil { + t.Logf("doltserver.Stop(%s): %v", serverDir, err) + } +} + +func assertNoClassicServerArtifacts(t *testing.T, beadsDir, when string) { + t.Helper() + for _, name := range classicServerArtifacts { + path := filepath.Join(beadsDir, name) + if _, err := os.Stat(path); err == nil { + body, _ := os.ReadFile(path) // #nosec G304 - test-owned temp path + t.Errorf("%s: %s exists on a proxied workspace (contents %q); bd took ownership of a server over the proxied root", + when, name, strings.TrimSpace(string(body))) + } + } +} + +// TestManagedLocalProxiedDoltStatusReportsLiveTopology pins the truthfulness +// half of the fix: with a proxy and its dolt child both live, `bd dolt status` +// must report them. Before the fix it read the classic pidfile, which a +// proxied workspace never writes, and answered "not running" — a confident +// lie, and the reason an operator would reach for `bd dolt start` next. +func TestManagedLocalProxiedDoltStatusReportsLiveTopology(t *testing.T) { + requireManagedLocalProxiedEnv(t) + bd := buildEmbeddedBD(t) + p := bdManagedLocalInit(t, bd, "mlstat", 5*time.Minute) + bdProxiedCreate(t, bd, p.dir, "status sentinel") + + proxyPid := readManagedProxyPidFile(t, p) + if proxyPid == nil || !processAlive(proxyPid.Pid) { + t.Fatalf("expected a live proxy after a create; pidfile=%+v", proxyPid) + } + backendPid := readManagedBackendPidFile(t, p) + if backendPid == nil || !processAlive(backendPid.Pid) { + t.Fatalf("expected a live dolt backend after a create; pidfile=%+v", backendPid) + } + + stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, []string{"BEADS_JSON=1"}, "--json", "dolt", "status") + if err != nil { + t.Fatalf("bd dolt status failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + var got struct { + Mode string `json:"mode"` + Running bool `json:"running"` + ProxyPID int `json:"proxy_pid"` + ProxyPort int `json:"proxy_port"` + BackendRunning bool `json:"backend_running"` + BackendPID int `json:"backend_pid"` + BackendPort int `json:"backend_port"` + Root string `json:"root"` + } + if jsonErr := json.Unmarshal([]byte(strings.TrimSpace(stdout)), &got); jsonErr != nil { + t.Fatalf("decode dolt status JSON: %v\nstdout:\n%s", jsonErr, stdout) + } + if got.Mode != "proxied-server" { + t.Errorf("mode=%q, want %q (status must name the topology it is describing)", got.Mode, "proxied-server") + } + if !got.Running { + t.Errorf("running=false while proxy pid %d is alive; status is reporting the wrong process record", proxyPid.Pid) + } + if got.ProxyPID != proxyPid.Pid { + t.Errorf("proxy_pid=%d, want %d (from %s)", got.ProxyPID, proxyPid.Pid, proxy.PIDFileName) + } + if got.ProxyPort != proxyPid.Port { + t.Errorf("proxy_port=%d, want %d (from %s)", got.ProxyPort, proxyPid.Port, proxy.PIDFileName) + } + if !got.BackendRunning { + t.Errorf("backend_running=false while dolt pid %d is alive", backendPid.Pid) + } + if got.BackendPID != backendPid.Pid { + t.Errorf("backend_pid=%d, want %d (from %s)", got.BackendPID, backendPid.Pid, server.PIDFileName) + } + if got.BackendPort != backendPid.Port { + t.Errorf("backend_port=%d, want %d (from %s)", got.BackendPort, backendPid.Port, server.PIDFileName) + } + if got.Root != p.proxyRoot { + t.Errorf("root=%q, want %q", got.Root, p.proxyRoot) + } + + textOut, textErr, err := bdProxiedRunBuffers(t, bd, p.dir, "dolt", "status") + if err != nil { + t.Fatalf("bd dolt status (text) failed: %v\nstdout:\n%s\nstderr:\n%s", err, textOut, textErr) + } + if strings.Contains(textOut, "Dolt server: not running") { + t.Errorf("text status still claims the server is not running:\n%s", textOut) + } + for _, want := range []string{"proxied-server", "Backend"} { + if !strings.Contains(textOut, want) { + t.Errorf("text status missing %q:\n%s", want, textOut) + } + } + + // Reading status must not disturb what it is reporting on. + if after := readManagedProxyPidFile(t, p); after == nil || after.Pid != proxyPid.Pid { + t.Errorf("proxy pidfile changed across a status read: before=%+v after=%+v", proxyPid, after) + } + assertNoClassicServerArtifacts(t, p.beadsDir, "after bd dolt status") +} + +// TestManagedLocalProxiedDoltStartRefusesOverLiveProxy covers the hazard in +// its live-topology shape: with the proxy up, `bd dolt start` used to adopt +// the proxy's own dolt child into bd's classic server bookkeeping, leaving two +// managers for one process. +func TestManagedLocalProxiedDoltStartRefusesOverLiveProxy(t *testing.T) { + requireManagedLocalProxiedEnv(t) + bd := buildEmbeddedBD(t) + p := bdManagedLocalInit(t, bd, "mlstart", 5*time.Minute) + t.Cleanup(func() { stopStrayClassicServer(t, p.beadsDir) }) + bdProxiedCreate(t, bd, p.dir, "start sentinel") + + proxyBefore := readManagedProxyPidFile(t, p) + backendBefore := readManagedBackendPidFile(t, p) + if proxyBefore == nil || backendBefore == nil { + t.Fatalf("expected a live proxy and backend; proxy=%+v backend=%+v", proxyBefore, backendBefore) + } + + stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, []string{"BEADS_JSON=1"}, "--json", "dolt", "start") + if err == nil { + t.Errorf("bd dolt start succeeded on a live proxied workspace:\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + assertProxyStartRefusal(t, stdout, stderr) + assertNoClassicServerArtifacts(t, p.beadsDir, "after bd dolt start") + + proxyAfter := readManagedProxyPidFile(t, p) + backendAfter := readManagedBackendPidFile(t, p) + if proxyAfter == nil || proxyAfter.Pid != proxyBefore.Pid { + t.Errorf("proxy record changed across a refused start: before=%+v after=%+v", proxyBefore, proxyAfter) + } + if backendAfter == nil || backendAfter.Pid != backendBefore.Pid { + t.Errorf("backend record changed across a refused start: before=%+v after=%+v", backendBefore, backendAfter) + } +} + +// TestManagedLocalProxiedDoltStartRefusesWithProxyDown covers the same hazard +// in its damaging shape. A proxied workspace whose proxy has idled out still +// has a proxied ROOT, and the classic start path resolves that root's port +// from its config.yaml: `bd dolt start` used to launch a real, unmanaged dolt +// sql-server directly over the proxy's data directory. The next ordinary bd +// command then failed, because the proxy came up and found a foreign server on +// its port. The workspace staying usable afterwards is the assertion that +// matters most here. +func TestManagedLocalProxiedDoltStartRefusesWithProxyDown(t *testing.T) { + requireManagedLocalProxiedEnv(t) + bd := buildEmbeddedBD(t) + p := bdManagedLocalInit(t, bd, "mldown", 5*time.Minute) + t.Cleanup(func() { stopStrayClassicServer(t, p.beadsDir) }) + bdProxiedCreate(t, bd, p.dir, "down sentinel") + + if err := proxy.Shutdown(p.proxyRoot); err != nil { + t.Fatalf("proxy.Shutdown(%s): %v", p.proxyRoot, err) + } + + stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, []string{"BEADS_JSON=1"}, "--json", "dolt", "start") + if err == nil { + t.Errorf("bd dolt start succeeded against a quiesced proxied root:\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + assertProxyStartRefusal(t, stdout, stderr) + assertNoClassicServerArtifacts(t, p.beadsDir, "after bd dolt start with the proxy down") + + // The workspace must still work. Pre-fix this create failed with + // "invalid connection": the proxy relaunched and adopted the foreign + // server `bd dolt start` had left on its port. + out, createErr := bdProxiedRun(t, bd, p.dir, "create", "--json", "post-refusal write") + if createErr != nil { + t.Fatalf("workspace unusable after a refused dolt start: %v\n%s", createErr, out) + } +} + +func assertProxyStartRefusal(t *testing.T, stdout, stderr string) { + t.Helper() + var got struct { + Code string `json:"code"` + Error string `json:"error"` + Mutates bool `json:"mutates"` + } + if err := json.Unmarshal([]byte(strings.TrimSpace(stdout)), &got); err != nil { + t.Errorf("dolt start refusal is not typed JSON (%v)\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + return + } + if got.Code != proxyDoltStartConflictCode { + t.Errorf("refusal code=%q, want %q", got.Code, proxyDoltStartConflictCode) + } + if got.Mutates { + t.Errorf("refusal reported mutates=true; a refused start changes nothing") + } + if !strings.Contains(got.Error, "proxied-server mode") { + t.Errorf("refusal message does not name the mode: %q", got.Error) + } +} + +// TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone is the +// non-regression control for the two tests above: the proxied guard keys on +// the workspace's own mode, so an embedded workspace and a bd-managed direct +// local server must behave exactly as they did before. It runs in this lane +// rather than a direct-server one because this is the lane that installs the +// dolt CLI the direct case needs. +func TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone(t *testing.T) { + requireManagedLocalProxiedEnv(t) + bd := buildEmbeddedBD(t) + + t.Run("embedded", func(t *testing.T) { + dir, _, _ := bdInit(t, bd) + + statusOut, statusErr, err := runDirectBD(t, bd, dir, false, "dolt", "status") + if err != nil { + t.Fatalf("bd dolt status failed on embedded: %v\nstdout:\n%s\nstderr:\n%s", err, statusOut, statusErr) + } + if !strings.Contains(statusOut, "Dolt engine: embedded (in-process, no server)") { + t.Errorf("embedded status changed:\n%s", statusOut) + } + + startOut, startErr, err := runDirectBD(t, bd, dir, false, "dolt", "start") + if err == nil { + t.Fatalf("bd dolt start unexpectedly succeeded on embedded:\n%s", startOut) + } + if !strings.Contains(startErr, "'bd dolt start' is not supported in embedded mode (no Dolt server)") { + t.Errorf("embedded start refusal changed:\nstdout:\n%s\nstderr:\n%s", startOut, startErr) + } + }) + + t.Run("direct-local-server", func(t *testing.T) { + dir := t.TempDir() + initGitRepoAt(t, dir) + if out, errOut, err := runDirectBD(t, bd, dir, true, "init", "--quiet", "--server", "--prefix", "dls"); err != nil { + t.Fatalf("bd init --server failed: %v\nstdout:\n%s\nstderr:\n%s", err, out, errOut) + } + beadsDir := filepath.Join(dir, ".beads") + t.Cleanup(func() { stopStrayClassicServer(t, beadsDir) }) + + startOut, startErr, err := runDirectBD(t, bd, dir, true, "dolt", "start") + if err != nil { + t.Fatalf("bd dolt start failed on a direct local server: %v\nstdout:\n%s\nstderr:\n%s", err, startOut, startErr) + } + if !strings.Contains(startOut, "Dolt server started (PID ") { + t.Fatalf("direct start output changed:\n%s", startOut) + } + + statusOut, statusErr, err := runDirectBD(t, bd, dir, true, "dolt", "status") + if err != nil { + t.Fatalf("bd dolt status failed on a direct local server: %v\nstdout:\n%s\nstderr:\n%s", err, statusOut, statusErr) + } + if !strings.Contains(statusOut, "Dolt server: running") { + t.Errorf("direct status did not report the server bd just started:\n%s", statusOut) + } + + stopOut, stopErr, err := runDirectBD(t, bd, dir, true, "dolt", "stop") + if err != nil { + t.Fatalf("bd dolt stop failed on a direct local server: %v\nstdout:\n%s\nstderr:\n%s", err, stopOut, stopErr) + } + if !strings.Contains(stopOut, "Dolt server stopped.") { + t.Errorf("direct stop output changed:\n%s", stopOut) + } + }) +} + +// runDirectBD runs bd against a NON-proxied workspace. autoStart controls +// BEADS_DOLT_AUTO_START: the direct-server case needs it on, because +// bd dolt status routes a local server with auto-start disabled to the +// external SQL-probe path instead of the pidfile path under test. +func runDirectBD(t *testing.T, bd, dir string, autoStart bool, args ...string) (string, string, error) { + t.Helper() + env := bdEnv(dir) + if autoStart { + env = append(envWithout(env, "BEADS_DOLT_AUTO_START"), "BEADS_DOLT_AUTO_START=1") + } + cmd := exec.Command(bd, args...) + cmd.Dir = dir + cmd.Env = env + stdout, stderr, err := runCommandBuffers(t, cmd) + return stdout.String(), stderr.String(), err +} diff --git a/internal/storage/dbproxy/proxy/read_status.go b/internal/storage/dbproxy/proxy/read_status.go new file mode 100644 index 000000000..14c996db2 --- /dev/null +++ b/internal/storage/dbproxy/proxy/read_status.go @@ -0,0 +1,62 @@ +package proxy + +import ( + "github.com/steveyegge/beads/internal/procid" + "github.com/steveyegge/beads/internal/storage/dbproxy/pidfile" + "github.com/steveyegge/beads/internal/storage/dbproxy/server" +) + +// Status is a point-in-time description of the process tree under a proxied +// root: the proxy bd connects through, and the dolt backend the proxy +// supervises on a managed-local topology. +type Status struct { + ProxyRunning bool + ProxyPID int + ProxyPort int + // BackendRunning describes the proxy-child record only. An external + // proxied topology never writes one — bd spawns no dolt there — so a + // false value means "bd supervises no backend here", which is not the + // same claim as "the database is down". Callers that can tell the + // topologies apart must say which one they are reporting. + BackendRunning bool + BackendPID int + BackendPort int +} + +// ReadStatus answers what is running under rootDir without starting, +// stopping, or adopting anything: it reads the two pid records and verifies +// the recorded processes are still the ones that wrote them. Safe for +// diagnostics, which is the whole point — a reporting command must not have +// lifecycle side effects. +func ReadStatus(rootDir string) Status { + var st Status + if discovery := readAndDial(rootDir); discovery.status == adoptionAdopted { + st.ProxyRunning = true + st.ProxyPID = discovery.pidfile.Pid + st.ProxyPort = discovery.pidfile.Port + } + st.BackendRunning, st.BackendPID, st.BackendPort = readBackendRecord(rootDir) + return st +} + +// readBackendRecord verifies the supervised dolt backend from its pidfile. +// Unlike the proxy there is no control channel to interrogate — the backend is +// a plain dolt sql-server — so identity stops at the birth token. That still +// rejects a recycled pid, which a bare liveness check would not, and it is the +// same first gate the proxy's own adoption path applies. A record that fails +// any check reports nothing rather than a pid a caller might act on: a stale +// record's pid is precisely the one that must not be signaled. +func readBackendRecord(rootDir string) (running bool, pid int, port int) { + pf, err := pidfile.Read(rootDir, server.PIDFileName) + if err != nil || pf == nil { + return false, 0, 0 + } + if err := pf.ValidateV2(pidfile.KindDoltBackend); err != nil { + return false, 0, 0 + } + matched, err := verifyProcessIdentity(pf.Pid, procid.Token(pf.Birth)) + if err != nil || !matched { + return false, 0, 0 + } + return true, pf.Pid, pf.Port +} diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index 18ffc071f..b92665797 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -1294,6 +1294,32 @@ func TestPinnedDoltCLIMatchesContainerImage(t *testing.T) { } } +// TestProxiedLocalSmokeMatchesPinnedDoltVersion keeps the proxied-local-smoke +// lane's standalone Dolt CLI install on the same release as the rest of the +// suite. That lane downloads its own dolt binary straight from GitHub +// releases instead of going through scripts/ci/install-dolt.sh, so nothing +// else catches it drifting off the measured pin (see "Which Dolt version to +// install" in docs/architecture/dolt.md for why the pin is not just "latest"). +func TestProxiedLocalSmokeMatchesPinnedDoltVersion(t *testing.T) { + root := sourceRepoRoot(t) + + installer, err := os.ReadFile(filepath.Join(root, "scripts", "ci", "install-dolt.sh")) + if err != nil { + t.Fatal(err) + } + cliVersion := captureOne(t, `(?m)^readonly version="([0-9]+\.[0-9]+\.[0-9]+)"$`, string(installer), "scripts/ci/install-dolt.sh") + + workflow, err := os.ReadFile(filepath.Join(root, ".github", "workflows", "proxied-local-smoke.yml")) + if err != nil { + t.Fatal(err) + } + smokeVersion := captureOne(t, `(?m)^\s*DOLT_VERSION:\s*([0-9]+\.[0-9]+\.[0-9]+)\s*$`, string(workflow), "proxied-local-smoke.yml:DOLT_VERSION") + + if cliVersion != smokeVersion { + t.Errorf("dolt pins disagree: CLI %s, proxied-local-smoke.yml DOLT_VERSION %s", cliVersion, smokeVersion) + } +} + func captureOne(t *testing.T, pattern, body, source string) string { t.Helper() From 76860ebdfbbb7a8676675a9a9df589877f9c3db8 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 16 Sep 2026 02:52:51 -0700 Subject: [PATCH 03/45] fix(config): dotted keys round-trip through SetYamlConfigInDir/UnsetYamlConfig (bd-zj95, 1.3.1) [release/1.3.0] (#6578) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(config): make a dotted key round-trip through SetYamlConfigInDir (bd-zj95) SetYamlConfigInDir writes dotted keys; GetStringFromDir reads them by splitting on the dot and walking nested mappings; UnsetYamlConfig removes them. Those are three directions of ONE round-trip property, and set/get was broken in two ways, both reachable from ordinary files: - An empty or comment-only config.yaml has no mapping node, so the nested writer declined and the caller appended a literal 'dolt.host: ...' line — a key whose NAME contains a dot. Nothing that splits on the dot can ever find it. - A file that already carried such a flat key had it updated in place, which preserved the unreadable shape forever. So a caller could write a value and immediately be unable to read it back. Two consumers found it the hard way: bd's ownership handoff, whose commit write-set refused itself on dolt.auto-start, and a Gas City managed city, whose .beads/config.yaml is entirely flat dotted keys. Six shipped call sites write a dotted key through this path, and every one of them is a write into a .beads/config.yaml that may be empty or comment-only — exactly the shape that produced the unreadable spelling: cmd/bd/bootstrap.go:1108 sync.remote cmd/bd/dolt.go:481 sync.remote cmd/bd/init.go:3386, :3394 sync.remote cmd/bd/migrate_dolt_mode.go:810 dolt.shared-server cmd/bd/migrate_dolt_mode.go:1168 dolt.shared-server An empty document now gets a mapping made for it, and a flat key of the name being written is migrated to the nested form rather than updated in place. Only the key being written is touched: a dotted key this call does not own belongs to someone else and is left exactly as they wrote it. Table test over nine file shapes — absent, empty, comment-only, flat, flat commented, existing section, unrelated nested, unrelated flat, and mixed — each asserting both that every key reads back and that no unreadable flat spelling survives. (cherry picked from commit cbe9bd6ca03416a76441e560bdec9fa67add99c5) Co-Authored-By: Claude Fable 5.1 * fix(config): unset a dotted key in the shape the writer produces Writing a dotted key nested and then being unable to unset it is the same round-trip break in the other direction. Set, get and unset are three directions of ONE property, and all three have to agree on the shape: UnsetYamlConfig comments out the line matching the key, and its pattern only ever matched a FLAT 'sync.remote:' line. Once the writer nests, an unset silently does nothing and the value stays live — for sync.remote that means bd keeps a remote the operator asked it to forget. commentOutYamlKey now walks the nesting, tracking indentation per segment so a key is commented only when it sits under its OWN parents rather than under some other section sharing a leaf name. The flat spelling is still handled, because files written by older bds carry it. This is what closes the loop for the shipped unset paths — cmd/bd/config.go:578 ('bd config unset'), cmd/bd/dolt.go:1761 and cmd/bd/dolt_remote_proxied_server.go:34 — against the six writers the previous commit nests: sync.remote at cmd/bd/bootstrap.go:1108, cmd/bd/dolt.go:481 and cmd/bd/init.go:3386/:3394, and dolt.shared-server at cmd/bd/migrate_dolt_mode.go:810/:1168. TestFinalizeSyncedBootstrapWritesConfigFiles moves with it. It required a literal 'sync.remote: ' line — a key whose name contains a dot, the one shape GetStringFromDir and viper can never find, because both split on the dot and walk nested mappings. That assertion passed for exactly as long as the value was unreadable and failed the moment it became readable, so it now asserts what bootstrap actually owes its caller: a remote that reads back. (cherry picked from the internal/config half of commit 7f1ec4799a3e8af0a89b8f97c28ec30082fedbb7) Co-Authored-By: Claude Fable 5.1 * test(cmd/bd): assert the sync.remote round trip, not a YAML spelling Six cmd/bd test sites baked the flat spelling. They passed BECAUSE the bug was present, and that is the part worth recording. Viper finds a literal dotted key as well as a nested one; GetStringFromDir does not, because it splits on the dot and walks mappings. So these tests exercised config.yaml through ONE reader while the value was invisible to the other. They were green for exactly as long as the value was unreadable, and TestFinalizeSyncedBootstrapWritesConfigFiles failed on macOS and Linux CI the moment it became readable. A test that only passes while a bug is live does not merely fail to catch it — it defends it. bootstrap_guard_test.go:84 seed bootstrap_test.go:358, :410 seed bootstrap_test.go:1330 assert (accepted 'sync.remote: ' OR 'sync-remote: ', neither readable) dolt_worktree_remote_test.go:55 assert init_remote_test.go:128 seed All six now assert the round trip through GetStringFromDir, or the caller's own observable, rather than any YAML text — flat or nested. Asserting the nested spelling instead would have been the same mistake facing the other way. The three seeds go through config.SetYamlConfigInDir, the real writer, via a shared seedSyncRemote helper that asserts the read-back. A fixture can no longer hand bd a config.yaml that bd itself would never have written. These are the tests standing over the shipped writers this series fixes: sync.remote at cmd/bd/bootstrap.go:1108, cmd/bd/dolt.go:481 and cmd/bd/init.go:3386/:3394, and dolt.shared-server at cmd/bd/migrate_dolt_mode.go:810/:1168. (cherry picked from commit b0df00b59d2d68ee5fee313c05dc2300ad66fcef) Co-Authored-By: Claude Fable 5.1 * fix(config): close the dotted-key round trip the rest of the way Review of the first three commits found the round trip still open in two places, both reachable from ordinary files. An empty or comment-only config.yaml parses to no YAML nodes at all, and yaml.v3 keeps no trace of its text either — not even the comments. The writer fabricated a mapping for that case and marshalled it, so the file it wrote back was the new key and nothing else. `bd init` writes a template that is comments and nothing else and then persists sync.remote into it, so every remote-connected init shipped a config.yaml with its whole documentation deleted. config.yaml is git-tracked: the operator sees that as a committed diff they did not make. Append the rendered key to the existing text instead, so nothing already in the file is parsed, re-emitted or reformatted, and it survives byte for byte. An unset comments the leaf out and leaves the section holding nothing — `sync:` and no more. A null is not a mapping, so the next set refused it, fell through to the flat writer, and put a literal `sync.remote:` back into the file: the unreadable shape, re-created one command after being fixed, with bd reporting success while GetStringFromDir saw nothing. A section with no value under it has nothing to lose, so treat it as the empty mapping it looks like. Two shapes that are genuinely unwritable now say so instead of writing the flat key and claiming success: a parent that already holds a value (`sync: enabled`), and a file whose top level is not a mapping. Both returned nil and produced a value no reader could see. Unset also matched lines inside literal block scalars, so unsetting sync.remote rewrote text the operator had written under a `notes: |`. Skip a block scalar's body: it is data, not structure. Finally, drop normalizeYamlKey and its empty keyAliases map. It has been an identity function whose comment claimed it renamed keys. * docs(changelog): record the dotted config key fix under Unreleased It is the most user-visible fix in 1.3.1 and it was not in the changelog, so it would not have reached the release notes or `bd upgrade review`. --------- Co-authored-by: Claude Fable 5.1 --- CHANGELOG.md | 21 + cmd/bd/bootstrap_guard_test.go | 8 +- cmd/bd/bootstrap_test.go | 22 +- cmd/bd/dolt_worktree_remote_test.go | 14 +- cmd/bd/init_remote_test.go | 56 ++- cmd/bd/sync_remote_fixture_test.go | 48 +++ internal/config/yaml_config.go | 223 +++++++--- internal/config/yaml_config_test.go | 20 - internal/config/yaml_nested_roundtrip_test.go | 399 ++++++++++++++++++ 9 files changed, 714 insertions(+), 97 deletions(-) create mode 100644 cmd/bd/sync_remote_fixture_test.go create mode 100644 internal/config/yaml_nested_roundtrip_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 442ce035b..375762a21 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `backend_managed` is false on external proxied topologies, where the dolt server is not bd's process to report on. +- **A dotted config key now round-trips: what `bd config set` writes, + `bd config get` and bd's own readers find** + ([#6578](https://github.com/gastownhall/beads/pull/6578), bd-zj95). Setting a + key like `sync.remote` or `dolt.host` into a `config.yaml` with no matching + section appended a key whose *name* contained the dot — a top-level + `sync.remote: "..."` — instead of nesting it under `sync:`. Viper finds a key + spelled that way; the direct reader `GetStringFromDir` splits on the dot and + walks nested mappings, so it does not. The value was set and invisible at the + same time depending on which reader asked, and `bd init --remote` in a fresh + workspace was the ordinary way in: the remote was recorded and then not found. + Dotted keys are written nested now, an existing flat spelling of the key being + written is migrated to the nested one, and `bd config unset` removes the + nested form — it only ever matched the flat spelling, so unsetting + `sync.remote` silently left the remote live. Keys the write does not own, + including dotted ones somebody else put there, are left exactly as they are. + + Two things that used to succeed now report an error instead, because both + produced a value no reader could see: setting a key under a parent that + already holds a value (`sync: enabled`, then `bd config set sync.remote`), and + setting one in a file whose top level is not a mapping. + ## [1.3.0] - 2026-09-15 The first tested release off `main` since the 1.1 line. [1.2.2] was a recovery diff --git a/cmd/bd/bootstrap_guard_test.go b/cmd/bd/bootstrap_guard_test.go index 6a3df5560..40df292a3 100644 --- a/cmd/bd/bootstrap_guard_test.go +++ b/cmd/bd/bootstrap_guard_test.go @@ -79,13 +79,7 @@ func TestIsGitCodeRepoURL(t *testing.T) { // Returns a cleanup function that resets config state. func setupSyncRemoteConfig(t *testing.T, beadsDir, remote string) func() { t.Helper() - if err := os.WriteFile( - filepath.Join(beadsDir, "config.yaml"), - []byte("sync.remote: "+remote+"\n"), - 0o644, - ); err != nil { - t.Fatalf("write config.yaml: %v", err) - } + seedSyncRemote(t, beadsDir, remote) config.ResetForTesting() t.Setenv("BEADS_DIR", beadsDir) t.Setenv("BEADS_TEST_IGNORE_REPO_CONFIG", "1") diff --git a/cmd/bd/bootstrap_test.go b/cmd/bd/bootstrap_test.go index a8d944165..c590f6d24 100644 --- a/cmd/bd/bootstrap_test.go +++ b/cmd/bd/bootstrap_test.go @@ -355,9 +355,7 @@ func TestDetectBootstrapAction_ExplicitSyncRemotePreservesRemotesAPIURL(t *testi t.Fatal(err) } const syncRemote = "http://myserver:7007/mydb" - if err := os.WriteFile(filepath.Join(beadsDir, "config.yaml"), []byte("sync.remote: "+syncRemote+"\n"), 0o644); err != nil { - t.Fatal(err) - } + seedSyncRemote(t, beadsDir, syncRemote) t.Setenv("BEADS_DIR", beadsDir) t.Setenv("BEADS_TEST_IGNORE_REPO_CONFIG", "1") if err := config.Initialize(); err != nil { @@ -407,9 +405,7 @@ func TestDetectBootstrapAction_ExistingEmbeddedDBWithSyncRemoteIsNoOp(t *testing t.Fatal(err) } const syncRemote = "http://myserver:7007/mydb" - if err := os.WriteFile(filepath.Join(beadsDir, "config.yaml"), []byte("sync.remote: "+syncRemote+"\n"), 0o644); err != nil { - t.Fatal(err) - } + seedSyncRemote(t, beadsDir, syncRemote) t.Setenv("BEADS_DIR", beadsDir) t.Setenv("BEADS_TEST_IGNORE_REPO_CONFIG", "1") if err := config.Initialize(); err != nil { @@ -1327,8 +1323,18 @@ func TestFinalizeSyncedBootstrapWritesConfigFiles(t *testing.T) { // sync.remote must be persisted so subsequent fresh clones (and // bootstrap retries) can rediscover the remote without re-probing // origin refs. - if !strings.Contains(yaml, "sync.remote: ") && !strings.Contains(yaml, "sync-remote: ") { - t.Errorf("config.yaml does not contain sync.remote entry:\n%s", yaml) + // + // Asserted by READING it back rather than by matching a spelling. This used + // to require a literal `sync.remote: ` line, which is a key whose name + // contains a dot — the shape config.GetStringFromDir and viper can never + // find, because both split on the dot and walk nested mappings. The writer + // now nests (bd-zj95), so the old assertion passed for exactly as long as + // the value was unreadable and failed the moment it became readable. What + // bootstrap owes its caller is a remote that can be read back, so that is + // what this checks. + if got := config.GetStringFromDir(beadsDir, "sync.remote"); got != syncRemote { + t.Errorf("config.GetStringFromDir(%q, \"sync.remote\") = %q, want %q:\n%s", + beadsDir, got, syncRemote, yaml) } if !strings.Contains(yaml, syncRemote) { t.Errorf("config.yaml does not contain sync remote URL %q:\n%s", syncRemote, yaml) diff --git a/cmd/bd/dolt_worktree_remote_test.go b/cmd/bd/dolt_worktree_remote_test.go index fd9de5b7c..3ad6ea3e5 100644 --- a/cmd/bd/dolt_worktree_remote_test.go +++ b/cmd/bd/dolt_worktree_remote_test.go @@ -8,9 +8,9 @@ import ( "path/filepath" "runtime" "strconv" - "strings" "testing" + "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/testutil" ) @@ -47,13 +47,21 @@ func TestDoltRemoteAddPersistsSyncRemoteToSharedWorktreeConfig(t *testing.T) { t.Fatalf("bd dolt remote add from bare-parent worktree failed: %v\n%s", err, out) } + // Asserted by reading it back, not by matching YAML text. The literal + // `sync.remote: ""` this used to require is a key whose NAME contains + // a dot — the shape config.GetStringFromDir can never find, because it + // splits on the dot and walks nested mappings. Viper finds a dotted key + // either way, so this assertion was green while the value was invisible to + // bd's other reader; it passed BECAUSE bd-zj95 was present. What the remote + // add owes its caller is a remote that reads back, so that is what this + // checks. configPath := filepath.Join(bareBeadsDir, "config.yaml") content, err := os.ReadFile(configPath) if err != nil { t.Fatalf("failed to read shared config.yaml: %v", err) } - if !strings.Contains(string(content), `sync.remote: "`+remoteURL+`"`) { - t.Fatalf("expected shared config.yaml to contain sync.remote, got:\n%s", string(content)) + if got := config.GetStringFromDir(bareBeadsDir, "sync.remote"); got != remoteURL { + t.Fatalf("shared config.yaml sync.remote = %q, want %q; contents:\n%s", got, remoteURL, content) } if _, err := os.Stat(filepath.Join(worktreeDir, ".beads")); !os.IsNotExist(err) { diff --git a/cmd/bd/init_remote_test.go b/cmd/bd/init_remote_test.go index 0243689dd..71aed22bf 100644 --- a/cmd/bd/init_remote_test.go +++ b/cmd/bd/init_remote_test.go @@ -6,6 +6,7 @@ import ( "strings" "testing" + "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/configfile" ) @@ -125,9 +126,8 @@ func TestPersistInitSyncRemoteExplicitRemoteWritesTargetDir(t *testing.T) { } callerConfig := filepath.Join(callerBeadsDir, "config.yaml") - if err := os.WriteFile(callerConfig, []byte("sync.remote: git+ssh://git@example.com/wrong/repo.git\n"), 0o600); err != nil { - t.Fatal(err) - } + const callerRemote = "git+ssh://git@example.com/wrong/repo.git" + seedSyncRemote(t, callerBeadsDir, callerRemote) targetConfig := filepath.Join(targetBeadsDir, "config.yaml") if err := os.WriteFile(targetConfig, []byte("# Beads Config\n"), 0o600); err != nil { t.Fatal(err) @@ -162,6 +162,56 @@ func TestPersistInitSyncRemoteExplicitRemoteWritesTargetDir(t *testing.T) { if strings.Contains(string(callerBytes), remote) { t.Fatalf("caller config.yaml was modified instead of target:\n%s", callerBytes) } + // Stronger than the absence above: the caller still resolves to what it had. + // A write that landed somewhere unreadable would satisfy "does not contain + // the new remote" while having quietly destroyed the old one. + if got := config.GetStringFromDir(callerBeadsDir, "sync.remote"); got != callerRemote { + t.Fatalf("caller sync.remote = %q, want it untouched at %q:\n%s", got, callerRemote, callerBytes) + } +} + +// `bd init --remote` runs createConfigYaml and then immediately writes +// sync.remote into what it just created. That template is comments and nothing +// else, which is the one document shape yaml.v3 parses to no nodes at all — so a +// writer that round-trips a node tree has nothing to write back but the key it +// added, and the operator's first commit after init carries a config.yaml with +// its entire documentation deleted. It is git-tracked, so they see it. +// +// This runs the real pair in the real order rather than asserting on a replica +// of the template, so it stays true if either side moves. +func TestInitPersistingSyncRemoteKeepsTheConfigTemplate(t *testing.T) { + beadsDir := filepath.Join(t.TempDir(), ".beads") + if err := os.MkdirAll(beadsDir, 0o750); err != nil { + t.Fatal(err) + } + if err := createConfigYaml(beadsDir, false, ""); err != nil { + t.Fatalf("createConfigYaml: %v", err) + } + template, err := os.ReadFile(filepath.Join(beadsDir, "config.yaml")) + if err != nil { + t.Fatal(err) + } + + const remote = "git+ssh://git@example.com/right/repo.git" + if err := persistInitSyncRemote(beadsDir, remote, remote, false, true, false); err != nil { + t.Fatalf("persistInitSyncRemote: %v", err) + } + + body, err := os.ReadFile(filepath.Join(beadsDir, "config.yaml")) + if err != nil { + t.Fatal(err) + } + for _, line := range strings.Split(string(template), "\n") { + if strings.TrimSpace(line) == "" { + continue + } + if !strings.Contains(string(body), line) { + t.Errorf("init dropped a line of its own config template: %q\ngot:\n%s", line, body) + } + } + if got := config.GetStringFromDir(beadsDir, "sync.remote"); got != remote { + t.Errorf("sync.remote = %q, want %q\n%s", got, remote, body) + } } func TestInitTimeCloneConfigExternalDefaultsAreSelfContained(t *testing.T) { diff --git a/cmd/bd/sync_remote_fixture_test.go b/cmd/bd/sync_remote_fixture_test.go new file mode 100644 index 000000000..d5870dfba --- /dev/null +++ b/cmd/bd/sync_remote_fixture_test.go @@ -0,0 +1,48 @@ +package main + +import ( + "os" + "path/filepath" + "testing" + + "github.com/steveyegge/beads/internal/config" +) + +// seedSyncRemote records a sync remote in beadsDir the way bd records one, and +// proves it is readable afterwards. +// +// The tests that use this used to hand-write `sync.remote: ` into +// config.yaml — a key whose NAME contains a dot. That spelling is not what bd's +// writer produces and not what config.GetStringFromDir can read: it splits on +// the dot and walks nested mappings. Viper happens to find a literal dotted key +// as well as a nested one, so those fixtures worked through one reader while +// being invisible to the other — the exact asymmetry bd-zj95 was, and the +// reason the fixtures were green while the bug was live. +// +// Writing through the real writer and asserting the read-back means these tests +// can no longer encode a spelling at all. If the round trip breaks again, the +// fixture fails here, naming the property, rather than some downstream +// assertion failing for a reason that looks unrelated. +func seedSyncRemote(t *testing.T, beadsDir, remote string) { + t.Helper() + path := filepath.Join(beadsDir, "config.yaml") + if _, err := os.Stat(path); os.IsNotExist(err) { + // SetYamlConfigInDir edits an existing file; it declines to create one, + // which is a deliberate guard against writing config into a directory + // that is not a workspace. + if writeErr := os.WriteFile(path, nil, 0o644); writeErr != nil { //nolint:gosec // test fixture + t.Fatalf("create config.yaml: %v", writeErr) + } + } else if err != nil { + t.Fatalf("stat config.yaml: %v", err) + } + + if err := config.SetYamlConfigInDir(beadsDir, "sync.remote", remote); err != nil { + t.Fatalf("record sync.remote: %v", err) + } + if got := config.GetStringFromDir(beadsDir, "sync.remote"); got != remote { + body, _ := os.ReadFile(path) //nolint:gosec,errcheck // diagnostic only + t.Fatalf("sync.remote does not read back after being written: got %q, want %q\nconfig.yaml:\n%s", + got, remote, body) + } +} diff --git a/internal/config/yaml_config.go b/internal/config/yaml_config.go index f86b0ef2c..2581f4090 100644 --- a/internal/config/yaml_config.go +++ b/internal/config/yaml_config.go @@ -247,23 +247,8 @@ func checkSecretGitTracked(configPath, key string) error { ) } -// keyAliases maps alternative key names to their canonical yaml form. -// This ensures consistency when users use different formats (dot vs hyphen). -var keyAliases = map[string]string{} - -// normalizeYamlKey converts a key to its canonical yaml format. -// Some keys have aliases (e.g., sync.branch -> sync-branch) to handle -// different input formats consistently. -func normalizeYamlKey(key string) string { - if canonical, ok := keyAliases[key]; ok { - return canonical - } - return key -} - // SetYamlConfig sets a configuration value in the project's config.yaml file. // It handles both adding new keys and updating existing (possibly commented) keys. -// Keys are normalized to their canonical yaml format (e.g., sync.branch -> sync-branch). func SetYamlConfig(key, value string) error { // Validate specific keys (GH#995) if err := validateYamlConfigValue(key, value); err != nil { @@ -524,8 +509,6 @@ func UnsetUserYamlConfig(key string) error { if err != nil { return err } - normalizedKey := normalizeYamlKey(key) - content, err := os.ReadFile(configPath) //nolint:gosec // configPath is a validated absolute user config path if err != nil { if os.IsNotExist(err) { @@ -534,7 +517,7 @@ func UnsetUserYamlConfig(key string) error { return fmt.Errorf("failed to read user config.yaml: %w", err) } - newContent := commentOutYamlKey(string(content), normalizedKey) + newContent := commentOutYamlKey(string(content), key) // Preserve the owner-private 0600 posture every other user-global writer // uses (SetUserYamlConfig, setYamlConfigAtPath, the metrics bootstrap); @@ -568,10 +551,6 @@ func SetUserYamlConfig(key, value string) error { } func setYamlConfigAtPath(configPath, key, value string) error { - - // Normalize key to canonical yaml format - normalizedKey := normalizeYamlKey(key) - // Read existing config content, err := os.ReadFile(configPath) //nolint:gosec // configPath is from findProjectConfigYaml if err != nil { @@ -579,7 +558,7 @@ func setYamlConfigAtPath(configPath, key, value string) error { } // Update or add the key - newContent, err := updateYamlKey(string(content), normalizedKey, value) + newContent, err := updateYamlKey(string(content), key, value) if err != nil { return err } @@ -594,13 +573,11 @@ func setYamlConfigAtPath(configPath, key, value string) error { // GetYamlConfig gets a configuration value from config.yaml. // Returns empty string if key is not found or is commented out. -// Keys are normalized to their canonical yaml format (e.g., sync.branch -> sync-branch). func GetYamlConfig(key string) string { if v == nil { return "" } - normalizedKey := normalizeYamlKey(key) - return v.GetString(normalizedKey) + return v.GetString(key) } // UnsetYamlConfig removes a configuration value from the project's config.yaml file. @@ -611,14 +588,12 @@ func UnsetYamlConfig(key string) error { return err } - normalizedKey := normalizeYamlKey(key) - content, err := os.ReadFile(configPath) //nolint:gosec // configPath is from findProjectConfigYaml if err != nil { return fmt.Errorf("failed to read config.yaml: %w", err) } - newContent := commentOutYamlKey(string(content), normalizedKey) + newContent := commentOutYamlKey(string(content), key) if err := os.WriteFile(configPath, []byte(newContent), 0600); err != nil { //nolint:gosec // configPath is validated return fmt.Errorf("failed to write config.yaml: %w", err) @@ -706,8 +681,6 @@ func findProjectBeadsDir() string { // updateYamlKey updates a key in yaml content, handling commented-out keys. // If the key exists (commented or not), it updates it in place. // If the key doesn't exist, it appends it at the end. -// -//nolint:unparam // error return kept for future validation func updateYamlKey(content, key, value string) (string, error) { if strings.Contains(key, ".") { if updated, ok, err := updateNestedYamlKey(content, key, value); err != nil { @@ -768,20 +741,39 @@ func updateNestedYamlKey(content, key, value string) (string, bool, error) { return "", false, err } if len(root.Content) == 0 { - return "", false, nil + // An empty or comment-only document parses to no nodes at all: yaml.v3 + // keeps no trace of its text, not even the comments. So there is nothing + // to nest into AND nothing to marshal back — fabricating a mapping here + // would emit the new key and silently delete everything else in the + // file, and `bd init`'s default template is exactly this shape, comments + // and nothing else. Append the rendered key to the text instead, which + // leaves the document byte for byte intact. Falling through to the flat + // writer is not an option either: that is what produced a key literally + // named "dolt.host", which GetStringFromDir — splitting on the dot and + // looking for a nested mapping — can never read back. + appended, err := appendNestedYamlKey(content, parts, value) + if err != nil { + return "", false, err + } + return appended, true, nil } mapping := root.Content[0] if mapping.Kind != yaml.MappingNode { - return "", false, nil + return "", false, fmt.Errorf("cannot set %q: the top level of this config file is not a mapping", key) } - if findMappingChild(mapping, key) != -1 { - return "", false, nil + // A flat key of this exact name is the unreadable shape, whether an older + // bd wrote it or the file arrived that way. Migrate it: drop the flat entry + // and write the value nested, so the round trip holds from here on. Only the + // key being written is touched — a dotted key this call does not own is + // someone else's and stays exactly as they wrote it. + if idx := findMappingChild(mapping, key); idx != -1 { + mapping.Content = append(mapping.Content[:idx], mapping.Content[idx+2:]...) } - leaf, ok := findOrCreateNestedScalar(mapping, parts) - if !ok { - return "", false, nil + leaf, err := findOrCreateNestedScalar(mapping, parts) + if err != nil { + return "", false, err } leaf.Kind = yaml.ScalarNode @@ -796,12 +788,9 @@ func updateNestedYamlKey(content, key, value string) (string, bool, error) { return string(out), true, nil } -func findOrCreateNestedScalar(mapping *yaml.Node, parts []string) (*yaml.Node, bool) { +func findOrCreateNestedScalar(mapping *yaml.Node, parts []string) (*yaml.Node, error) { current := mapping for i, part := range parts { - if current.Kind != yaml.MappingNode { - return nil, false - } idx := findMappingChild(current, part) isLeaf := i == len(parts)-1 if idx == -1 { @@ -814,21 +803,60 @@ func findOrCreateNestedScalar(mapping *yaml.Node, parts []string) (*yaml.Node, b } current.Content = append(current.Content, keyNode, valNode) if isLeaf { - return valNode, true + return valNode, nil } current = valNode continue } child := current.Content[idx+1] if isLeaf { - return child, true + return child, nil + } + if child.Tag == "!!null" { + // A section with nothing left under it — `sync:` and no more, which + // is exactly what an unset leaves behind once it has commented the + // last leaf out. It holds no value to lose, so treat it as the empty + // mapping it looks like. Refusing here sent the caller to the flat + // writer, so set -> unset -> set put the unreadable `sync.remote:` + // spelling back into the file this whole fix exists to keep out. + child.Kind = yaml.MappingNode + child.Tag = "!!map" + child.Value = "" + child.Style = 0 } if child.Kind != yaml.MappingNode { - return nil, false + return nil, fmt.Errorf("cannot set %q: %q already holds a value, so there is no section to nest under it", + strings.Join(parts, "."), strings.Join(parts[:i+1], ".")) } current = child } - return nil, false + // Unreachable: every path through the loop returns on the last part. + return nil, fmt.Errorf("cannot set %q: no key to write", strings.Join(parts, ".")) +} + +// appendNestedYamlKey renders just the key being written and appends it to the +// document text. Used when the document has no nodes to walk, where the text is +// the only copy of the file's contents that exists. +func appendNestedYamlKey(content string, parts []string, value string) (string, error) { + node := &yaml.Node{Kind: yaml.ScalarNode, Style: scalarStyleFor(value), Value: value} + for i := len(parts) - 1; i >= 0; i-- { + node = &yaml.Node{Kind: yaml.MappingNode, Tag: "!!map", Content: []*yaml.Node{ + {Kind: yaml.ScalarNode, Tag: "!!str", Value: parts[i]}, + node, + }} + } + rendered, err := yaml.Marshal(node) + if err != nil { + return "", err + } + + // Same spacing the flat writer uses: a blank line between whatever was + // already in the file and the key being added. + existing := strings.TrimRight(content, "\n") + if existing == "" { + return string(rendered), nil + } + return existing + "\n\n" + string(rendered), nil } func findMappingChild(mapping *yaml.Node, name string) int { @@ -862,28 +890,111 @@ func scalarStyleFor(value string) yaml.Style { } func commentOutYamlKey(content, key string) string { - keyPattern := regexp.MustCompile(`^(\s*)` + regexp.QuoteMeta(key) + `\s*:`) + // The flat spelling first — a key literally named "sync.remote", which + // older files carry and which this function has always handled. + flatPattern := regexp.MustCompile(`^(\s*)` + regexp.QuoteMeta(key) + `\s*:`) + // And the nested one, which is what the writer produces. Missing this half + // made an unset silently do nothing once the writer started nesting: the + // value stayed live, so bd kept a setting the operator had asked it to + // forget. Unset is the other direction of the same round-trip property as + // set and get, and all three have to agree on the shape. + segments := strings.Split(key, ".") var result []string + // depth tracks how many segments of the key have been matched so far, and + // indents holds the indentation each was found at, so a key is only + // commented out when it is nested under its OWN parents rather than under + // some other section that happens to share a leaf name. + depth := 0 + var indents []int + // blockIndent is the indentation of the key that opened a literal or folded + // block scalar, or -1 outside one. Everything indented past that key is the + // value the user typed, not structure: `notes: |` with an indented + // `remote: keep-this` inside it is prose, and commenting it out edits their + // data. Matching by line has to skip those lines to stay honest. + blockIndent := -1 scanner := bufio.NewScanner(strings.NewReader(content)) for scanner.Scan() { line := scanner.Text() - if keyPattern.MatchString(line) { - matches := keyPattern.FindStringSubmatch(line) - indent := "" - if len(matches) > 1 { - indent = matches[1] + + if blockIndent >= 0 { + if strings.TrimSpace(line) == "" || lineIndent(line) > blockIndent { + result = append(result, line) + continue } - // Comment out the line, preserving indentation - result = append(result, indent+"# "+strings.TrimLeft(line, " \t")) - } else { - result = append(result, line) + blockIndent = -1 } + blockIndent = blockScalarIndent(line) + + if matches := flatPattern.FindStringSubmatch(line); matches != nil { + result = append(result, matches[1]+"# "+strings.TrimLeft(line, " \t")) + continue + } + + if len(segments) > 1 { + if name, indent, ok := yamlKeyOnLine(line); ok { + // Leaving a block: drop every segment matched at an indent at + // or deeper than this line's. + for depth > 0 && indent <= indents[depth-1] { + depth-- + indents = indents[:depth] + } + if depth < len(segments) && name == segments[depth] { + if depth == len(segments)-1 { + result = append(result, strings.Repeat(" ", indent)+"# "+strings.TrimLeft(line, " \t")) + depth, indents = 0, nil + continue + } + indents = append(indents, indent) + depth++ + } + } + } + result = append(result, line) } return strings.Join(result, "\n") } +// blockScalarIndent reports the indentation of a key whose value is a literal +// or folded block scalar (`notes: |`), or -1 when the line opens no block. +func blockScalarIndent(line string) int { + _, indent, ok := yamlKeyOnLine(line) + if !ok { + return -1 + } + _, rest, _ := strings.Cut(strings.TrimLeft(line, " \t"), ":") + rest = strings.TrimSpace(rest) + // "|" and ">" are only ever block indicators in value position; a plain + // scalar cannot start with either. + if rest == "" || (rest[0] != '|' && rest[0] != '>') { + return -1 + } + return indent +} + +func lineIndent(line string) int { + return len(line) - len(strings.TrimLeft(line, " \t")) +} + +// yamlKeyOnLine reports the key a mapping line declares and its indentation. +// Comments, list items and blank lines declare nothing. +func yamlKeyOnLine(line string) (name string, indent int, ok bool) { + trimmed := strings.TrimLeft(line, " \t") + if trimmed == "" || strings.HasPrefix(trimmed, "#") || strings.HasPrefix(trimmed, "- ") { + return "", 0, false + } + key, _, found := strings.Cut(trimmed, ":") + if !found { + return "", 0, false + } + key = strings.TrimSpace(key) + if key == "" || strings.ContainsAny(key, " \t") { + return "", 0, false + } + return key, lineIndent(line), true +} + // formatYamlValue formats a value appropriately for YAML. func formatYamlValue(value string) string { // Boolean values diff --git a/internal/config/yaml_config_test.go b/internal/config/yaml_config_test.go index d6fd9522c..45061490d 100644 --- a/internal/config/yaml_config_test.go +++ b/internal/config/yaml_config_test.go @@ -273,26 +273,6 @@ func TestFormatYamlValue(t *testing.T) { } } -func TestNormalizeYamlKey(t *testing.T) { - tests := []struct { - input string - expected string - }{ - {"no-db", "no-db"}, // no alias, unchanged - {"json", "json"}, // no alias, unchanged - {"routing.mode", "routing.mode"}, // no alias for this one - } - - for _, tt := range tests { - t.Run(tt.input, func(t *testing.T) { - got := normalizeYamlKey(tt.input) - if got != tt.expected { - t.Errorf("normalizeYamlKey(%q) = %q, want %q", tt.input, got, tt.expected) - } - }) - } -} - func TestSetYamlConfig(t *testing.T) { oldBeadsDir := os.Getenv("BEADS_DIR") if err := os.Unsetenv("BEADS_DIR"); err != nil { diff --git a/internal/config/yaml_nested_roundtrip_test.go b/internal/config/yaml_nested_roundtrip_test.go new file mode 100644 index 000000000..197a248f6 --- /dev/null +++ b/internal/config/yaml_nested_roundtrip_test.go @@ -0,0 +1,399 @@ +package config + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +// A dotted key must round-trip: whatever SetYamlConfigInDir writes, +// GetStringFromDir must read back. That is the whole contract between the two, +// and it is not a property either function can hold alone. +// +// It was broken in two directions, and both were reachable from ordinary files: +// +// - An empty or comment-only config.yaml has no mapping node, so the nested +// writer declined and the caller appended a literal `dolt.host: ...` line — +// a key whose NAME contains a dot. GetStringFromDir splits on the dot and +// looks for a nested mapping, so it never finds it. +// - A file that already carried such a flat key had it updated in place, +// which preserved the unreadable shape forever. +// +// The observable consequence was a caller writing a value and immediately being +// unable to read it back. Both consumers found it the hard way (bd-zj95). +func TestDottedKeysRoundTripThroughEveryConfigShape(t *testing.T) { + cases := []struct { + name string + seed string + // present is true when the file should exist before the write. + present bool + }{ + {name: "absent file", present: false}, + {name: "empty file", seed: "", present: true}, + {name: "comment only", seed: "# a workspace config\n", present: true}, + {name: "flat dotted keys", seed: "dolt.host: 10.0.0.1\ndolt.port: 3307\n", present: true}, + {name: "flat commented key", seed: "# dolt.host: 10.0.0.1\n", present: true}, + {name: "existing dolt section", seed: "dolt:\n host: 10.0.0.1\n", present: true}, + {name: "unrelated nested section", seed: "sync:\n branch: beads-sync\n", present: true}, + {name: "unrelated flat key", seed: "node_id: somewhere\n", present: true}, + {name: "mixed flat and nested", seed: "dolt.host: 10.0.0.1\ndolt:\n port: 3307\n", present: true}, + } + + writes := map[string]string{ + "dolt.host": "127.0.0.1", + "dolt.port": "45678", + "dolt.auto-start": "true", + "dolt.mode": "server", + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "config.yaml") + if tc.present { + if err := os.WriteFile(path, []byte(tc.seed), 0o600); err != nil { + t.Fatalf("seed: %v", err) + } + } else { + // SetYamlConfigInDir refuses a workspace with no config.yaml at + // all; that refusal is deliberate and not what this test is + // about, so give it the empty file it asks for. + if err := os.WriteFile(path, nil, 0o600); err != nil { + t.Fatalf("create: %v", err) + } + } + + for key, value := range writes { + if err := SetYamlConfigInDir(dir, key, value); err != nil { + t.Fatalf("SetYamlConfigInDir(%q): %v", key, err) + } + } + + body, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read: %v", err) + } + for key, want := range writes { + if got := GetStringFromDir(dir, key); got != want { + t.Errorf("GetStringFromDir(%q) = %q, want %q\nfile:\n%s", key, got, want, body) + } + } + // No LIVE flat key may survive: it is the unreadable shape, and a + // reader that happens to prefer it finds a stale answer. A commented + // one is inert — nothing reads it — and it is the operator's text, + // which the assertion below says bd has to leave alone. + for _, line := range strings.Split(string(body), "\n") { + if line != strings.TrimSpace(line) || strings.HasPrefix(line, "#") { + continue // indented: inside a mapping. commented: not a key. + } + name, _, isKeyValue := strings.Cut(line, ":") + if isKeyValue && writes[strings.TrimSpace(name)] != "" { + t.Errorf("a live flat %q survived the write:\n%s", strings.TrimSpace(name), body) + } + } + // Comments the file arrived with are the operator's, and a write that + // drops them is a committed diff nobody asked for. `bd init` writes a + // template that is nothing BUT comments, so this is the common case, + // not an exotic one. + for _, line := range strings.Split(tc.seed, "\n") { + line = strings.TrimSpace(line) + if !strings.HasPrefix(line, "#") { + continue + } + if !strings.Contains(string(body), line) { + t.Errorf("the write dropped the comment %q:\n%s", line, body) + } + } + }) + } +} + +// Keys the caller owns are left exactly as written. Migrating a flat key is only +// correct for the key being written; rewriting the whole file would be bd +// deciding how someone else's config should look. +func TestDottedWriteLeavesOtherKeysAlone(t *testing.T) { + dir := t.TempDir() + seed := "# keep this comment\nsync.branch: keep-me\ndolt.host: 10.0.0.1\nnode_id: mini\n" + if err := os.WriteFile(filepath.Join(dir, "config.yaml"), []byte(seed), 0o600); err != nil { + t.Fatalf("seed: %v", err) + } + + if err := SetYamlConfigInDir(dir, "dolt.host", "127.0.0.1"); err != nil { + t.Fatalf("set: %v", err) + } + + body, err := os.ReadFile(filepath.Join(dir, "config.yaml")) + if err != nil { + t.Fatalf("read: %v", err) + } + text := string(body) + if !strings.Contains(text, "sync.branch: keep-me") { + t.Errorf("a flat key the write does not own was rewritten:\n%s", text) + } + if !strings.Contains(text, "node_id: mini") { + t.Errorf("an unrelated key was lost:\n%s", text) + } + if got := GetStringFromDir(dir, "dolt.host"); got != "127.0.0.1" { + t.Errorf("dolt.host reads back as %q\n%s", got, text) + } +} + +// A single-segment key has no nesting to do and must keep working exactly as it +// did: this is the shape most of bd's config keys have. +func TestUndottedKeysAreUnaffected(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "config.yaml"), []byte("# seed\n"), 0o600); err != nil { + t.Fatalf("seed: %v", err) + } + if err := SetYamlConfigInDir(dir, "node_id", "mini"); err != nil { + t.Fatalf("set: %v", err) + } + if got := GetStringFromDir(dir, "node_id"); got != "mini" { + body, _ := os.ReadFile(filepath.Join(dir, "config.yaml")) //nolint:errcheck // diagnostic + t.Fatalf("node_id reads back as %q\n%s", got, body) + } +} + +// Writing a dotted key nested and then being unable to unset it is the same +// round-trip break in the other direction: UnsetYamlConfig comments out the +// line matching the key, and its pattern only ever matched a FLAT +// `sync.remote:` line. Once the writer nests, an unset silently does nothing +// and the value stays live — which for sync.remote means bd keeps a remote the +// operator asked it to forget. +func TestUnsetRemovesADottedKeyInEveryShape(t *testing.T) { + cases := []struct { + name string + seed string + }{ + {name: "nested", seed: "sync:\n remote: \"file:///origin.git\"\n"}, + {name: "nested among siblings", seed: "sync:\n branch: beads-sync\n remote: \"file:///origin.git\"\n"}, + {name: "legacy flat", seed: "sync.remote: \"file:///origin.git\"\n"}, + {name: "nested with other sections", seed: "dolt:\n port: 3307\nsync:\n remote: \"file:///origin.git\"\n"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "config.yaml") + if err := os.WriteFile(path, []byte(tc.seed), 0o600); err != nil { + t.Fatalf("seed: %v", err) + } + + t.Setenv("BEADS_DIR", dir) + if err := UnsetYamlConfig("sync.remote"); err != nil { + t.Fatalf("UnsetYamlConfig: %v", err) + } + + body, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read: %v", err) + } + if got := GetStringFromDir(dir, "sync.remote"); got != "" { + t.Errorf("sync.remote still reads back as %q after unset:\n%s", got, body) + } + // The key is preserved as documentation, which is this function's + // stated contract, so it must still be visible — commented. + if !strings.Contains(string(body), "remote:") { + t.Errorf("unset removed the key instead of commenting it out:\n%s", body) + } + // A sibling under the same section is none of the unset's business. + if tc.name == "nested among siblings" { + if got := GetStringFromDir(dir, "sync.branch"); got != "beads-sync" { + t.Errorf("unset took a sibling with it: sync.branch = %q\n%s", got, body) + } + } + if tc.name == "nested with other sections" { + if got := GetStringFromDir(dir, "dolt.port"); got != "3307" { + t.Errorf("unset touched an unrelated section: dolt.port = %q\n%s", got, body) + } + } + }) + } +} + +// set, unset, set is one command sequence, not three independent ones, and the +// round trip has to survive all of it. An unset comments the leaf out and leaves +// the section behind holding nothing — `sync:` and no more. The writer used to +// refuse that section, because a null is not a mapping, and fell through to the +// flat writer, which put a literal `sync.remote:` key back into the file: the +// unreadable shape, re-created one command after being fixed. bd said "Set +// sync.remote = ..." while GetStringFromDir saw nothing. +func TestSettingADottedKeyAgainAfterUnsetStaysNested(t *testing.T) { + cases := []struct { + name string + seed string + }{ + // The section is left holding nothing at all once its only leaf is + // commented out. This is the shape that broke. + {name: "section empties out", seed: "node_id: mini\n"}, + // A sibling keeps the section a mapping, so this shape always worked. + // Pin it anyway: the fix must not trade one for the other. + {name: "sibling keeps the section", seed: "sync:\n branch: beads-sync\n"}, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "config.yaml") + if err := os.WriteFile(path, []byte(tc.seed), 0o600); err != nil { + t.Fatalf("seed: %v", err) + } + t.Setenv("BEADS_DIR", dir) + + if err := SetYamlConfigInDir(dir, "sync.remote", "file:///a.git"); err != nil { + t.Fatalf("first set: %v", err) + } + if err := UnsetYamlConfig("sync.remote"); err != nil { + t.Fatalf("unset: %v", err) + } + if got := GetStringFromDir(dir, "sync.remote"); got != "" { + body, _ := os.ReadFile(path) //nolint:errcheck // diagnostic + t.Fatalf("unset left sync.remote at %q:\n%s", got, body) + } + if err := SetYamlConfigInDir(dir, "sync.remote", "file:///b.git"); err != nil { + t.Fatalf("second set: %v", err) + } + + body, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read: %v", err) + } + if got := GetStringFromDir(dir, "sync.remote"); got != "file:///b.git" { + t.Errorf("sync.remote reads back as %q after set/unset/set:\n%s", got, body) + } + for _, line := range strings.Split(string(body), "\n") { + if strings.HasPrefix(strings.TrimSpace(line), "sync.remote:") { + t.Errorf("the unreadable flat spelling came back:\n%s", body) + } + } + if tc.name == "sibling keeps the section" { + if got := GetStringFromDir(dir, "sync.branch"); got != "beads-sync" { + t.Errorf("the sibling was lost: sync.branch = %q\n%s", got, body) + } + } + }) + } +} + +// A config.yaml that is nothing but comments is not an edge case: it is what +// `bd init` writes, so it is the shape every fresh workspace's FIRST dotted +// write lands on — and config.yaml is git-tracked, so losing it is a committed +// diff the operator never asked for. +// +// yaml.v3 parses such a document to no nodes at all and keeps none of its text, +// so a writer that marshals a node tree has nothing to write back but the key it +// just added. Asserting the value reads back is not enough to catch that; the +// file is only correct if everything that was in it is still in it. +func TestFirstDottedWriteKeepsACommentOnlyFile(t *testing.T) { + seed := `# Beads Configuration File +# This file configures default behavior for all bd commands in this repository + +# Issue prefix for this repository (used by bd init) +# issue-prefix: "" + +# Use no-db mode: JSONL-only, no Dolt database +# no-db: false + +# Default actor for audit trails (overridden by BEADS_ACTOR or --actor) +# actor: "" +` + + dir := t.TempDir() + path := filepath.Join(dir, "config.yaml") + if err := os.WriteFile(path, []byte(seed), 0o600); err != nil { + t.Fatalf("seed: %v", err) + } + + // Two writes: the first lands on the comment-only file, the second on + // whatever the first produced. Both have to keep the comments. + if err := SetYamlConfigInDir(dir, "sync.remote", "file:///origin.git"); err != nil { + t.Fatalf("set sync.remote: %v", err) + } + if err := SetYamlConfigInDir(dir, "dolt.port", "3307"); err != nil { + t.Fatalf("set dolt.port: %v", err) + } + + body, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read: %v", err) + } + text := string(body) + for _, line := range strings.Split(strings.TrimRight(seed, "\n"), "\n") { + if line == "" { + continue + } + if !strings.Contains(text, line) { + t.Errorf("the write dropped a line that was already in the file: %q\ngot:\n%s", line, text) + } + } + if got := GetStringFromDir(dir, "sync.remote"); got != "file:///origin.git" { + t.Errorf("sync.remote reads back as %q\n%s", got, text) + } + if got := GetStringFromDir(dir, "dolt.port"); got != "3307" { + t.Errorf("dolt.port reads back as %q\n%s", got, text) + } +} + +// When a key's own parent already holds a value there is no section to nest +// under it, and bd has nothing correct to write. It used to write the flat +// `sync.remote:` spelling and report success, so the operator was told the +// value was set and every reader that splits on the dot saw nothing. Say so +// instead, and leave the file alone. +func TestSettingUnderAScalarParentIsAnError(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "config.yaml") + seed := "sync: enabled\n" + if err := os.WriteFile(path, []byte(seed), 0o600); err != nil { + t.Fatalf("seed: %v", err) + } + + err := SetYamlConfigInDir(dir, "sync.remote", "file:///origin.git") + if err == nil { + body, _ := os.ReadFile(path) //nolint:errcheck // diagnostic + t.Fatalf("setting sync.remote under a scalar sync reported success\n%s", body) + } + if !strings.Contains(err.Error(), "sync") { + t.Errorf("the error does not name the key in the way: %v", err) + } + + body, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read: %v", err) + } + if string(body) != seed { + t.Errorf("a refused write still changed the file:\n%s", body) + } +} + +// Unset matches by line, so it has to know which lines are YAML and which are +// somebody's prose. A literal block scalar's body is data: text indented under +// `notes: |` only looks like a mapping. Commenting inside it edits the value the +// operator wrote. +func TestUnsetLeavesBlockScalarTextAlone(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "config.yaml") + seed := "notes: |\n sync:\n remote: keep-this-text\nsync:\n remote: \"file:///origin.git\"\n" + if err := os.WriteFile(path, []byte(seed), 0o600); err != nil { + t.Fatalf("seed: %v", err) + } + + t.Setenv("BEADS_DIR", dir) + if err := UnsetYamlConfig("sync.remote"); err != nil { + t.Fatalf("UnsetYamlConfig: %v", err) + } + + body, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read: %v", err) + } + text := string(body) + if !strings.Contains(text, " remote: keep-this-text") { + t.Errorf("unset rewrote text inside the notes block:\n%s", text) + } + if got := GetStringFromDir(dir, "sync.remote"); got != "" { + t.Errorf("sync.remote still reads back as %q after unset:\n%s", got, text) + } + if got := GetStringFromDir(dir, "notes"); got != "sync:\n remote: keep-this-text\n" { + t.Errorf("the notes value changed: %q\n%s", got, text) + } +} From fa9543900666ed4360cded0b484786c5362e0cb8 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 16 Sep 2026 02:53:27 -0700 Subject: [PATCH 04/45] test(topology): front-door capability matrix across storage topologies (proxied-parity S0) [release/1.3.0] (#6579) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(topology): front-door capability matrix across storage topologies (S0) Characterization test for the proxied-parity work. Records what bd does TODAY across embedded, direct-local-server, direct-external-server, proxied-local and proxied-external-tcp — including the behavior that is wrong — so every later slice flips named rows and the flip is the review artifact. 18 command probes x 5 topologies = 90 cells. Outcomes are classified so a reader can tell the classes apart: REFUSED_TYPED carries reason=design (permanent policy) or reason=unimplemented (a gap a later slice closes), separately from WRONG_ANSWER, DEGRADED, ERROR and HONORED. Cells that record a bug carry a knownBad string naming it. Refusal rows additionally assert the two side-effect invariants from the existing history matrix: no proxy pidfile materializes (the provider was never constructed) and durable artifacts are byte-identical. Findings this matrix pins down beyond the design inventory: - `bd doctor` exits 0 on EMBEDDED having run no checks, not just proxied. - root `bd compact` is HONORED proxied; its refusal row is unreachable because the gate short-circuits when the command has no --dolt flag. - `bd dolt status` is correct on both direct server topologies; proxied is the sole outlier (running=false, pid 0, while the proxy serves CRUD). Reuses the existing fixtures rather than adding a harness: bdInit, newServerModeProject, bdManagedLocalInit, newSharedProxiedProject, buildEmbeddedBD, runCommandBuffers and snapshotHistoryArtifacts. The only new fixture is direct-local-server, which had no named helper. Gating: named TestProxiedServer* so proxied-test-shard.sh discovers it, which lands it in pr-risk.yml's test-proxied-cmd lane — a lane that runs on pull_request, not only push-to-main. managed-local is gated on BEADS_TEST_PROXIED_LOCAL, which only the proxied-local smoke lane sets, so it gets a second entry point named/filed to match that lane's filters. No production code changes. Co-Authored-By: Claude Fable 5 * test(topology): tell the post-#6580 truth about proxied dolt start/status #6580 makes `bd dolt status` truthful on a proxied workspace and turns `bd dolt start` there into a typed refusal. This matrix still encoded the pre-#6580 behaviour, so the two PRs are individually green and jointly red: merged together, `dolt status` on both proxied topologies fails the `"running": false` marker this table asserted. - flip the proxied `dolt status` cell from WRONG_ANSWER/`"running": false` to HONORED/`"running": true` and drop its knownBad — the defect is fixed, not deferred - add the `dolt start` row the stale deliberatelyNotProbed entry promised: REFUSED_TYPED `proxy.dolt_start.conflict` on both proxied topologies, REFUSED_UNTYPED on embedded (same command, no code a consumer can branch on), HONORED on direct-local where bd really does own the server - delete deliberatelyNotProbed["dolt start"], which described the hazard #6580 fixed as if it were current - add probe.skipOn so one row can be excused from one topology with the reason printed under the table; direct-external is excused from `dolt start` because running it there would leave a bd-owned sql-server the fixture has no teardown for This commit depends on #6580. Against a base that lacks it the flipped cells fail, because there `bd dolt status` still lies. Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Julian Knutsen Co-authored-by: Claude Fable 5 --- .../proxied_local_capability_matrix_test.go | 29 + ...ogy_capability_matrix_direct_local_test.go | 106 ++ cmd/bd/topology_capability_matrix_test.go | 1019 +++++++++++++++++ 3 files changed, 1154 insertions(+) create mode 100644 cmd/bd/proxied_local_capability_matrix_test.go create mode 100644 cmd/bd/topology_capability_matrix_direct_local_test.go create mode 100644 cmd/bd/topology_capability_matrix_test.go diff --git a/cmd/bd/proxied_local_capability_matrix_test.go b/cmd/bd/proxied_local_capability_matrix_test.go new file mode 100644 index 000000000..01b886318 --- /dev/null +++ b/cmd/bd/proxied_local_capability_matrix_test.go @@ -0,0 +1,29 @@ +//go:build cgo && unix + +package main + +// Managed-local entry point for the front-door capability matrix. +// +// Why this exists as a second entry point rather than a topology of the +// primary one: the managed-local fixture is gated on BEADS_TEST_PROXIED_LOCAL, +// and the only lane that sets it is .github/workflows/proxied-local-smoke.yml, +// which selects tests with -test.run "^TestManagedLocalProxied". The primary +// matrix runs in the proxied shard, which does not set that gate, so +// managed-local would be silently skipped there and nowhere else. +// +// Both the file name (matching the lane's `cmd/bd/proxied_local_*` path +// filter) and the test name are load-bearing. The smoke lane runs inside a +// loopback-only network namespace, so this entry point must build only +// topologies that need no Docker: embedded and managed-local. +// +// The table itself lives in topology_capability_matrix_test.go. This file +// adds no expectations of its own. + +import "testing" + +func TestManagedLocalProxiedCapabilityMatrix(t *testing.T) { + runCapabilityMatrix(t, []topologySpec{ + embeddedTopology(), + proxiedLocalTopology(), + }) +} diff --git a/cmd/bd/topology_capability_matrix_direct_local_test.go b/cmd/bd/topology_capability_matrix_direct_local_test.go new file mode 100644 index 000000000..65ee2ea22 --- /dev/null +++ b/cmd/bd/topology_capability_matrix_direct_local_test.go @@ -0,0 +1,106 @@ +//go:build cgo && unix + +package main + +// Fixture for the direct LOCAL server topology: `bd init --server` with no +// port, where bd spawns and owns a `dolt sql-server` under .beads. +// +// This is the one topology in the matrix with no existing named helper — the +// two tests that use it today build it inline behind the `integration` build +// tag (dolt_autostart_lifecycle_integration_test.go, +// migrate_dolt_mode_frontdoor_integration_test.go). This file factors out the +// two pieces those tests had to get right, and nothing else. + +import ( + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "syscall" + "testing" + "time" +) + +// directLocalServerEnv builds a subprocess environment for a bd-owned local +// dolt sql-server. +// +// Neither bdEnv nor bdProxiedEnv works here, for opposite reasons: +// - bdEnv sets BEADS_DOLT_AUTO_START=0, which stops bd from ever launching +// the server this topology is about. +// - the inherited env carries the cmd/bd TestMain's BEADS_TEST_MODE=1, which +// disables auto-start AND rewrites an unresolved port to 1. +// +// So: strip every BEADS_*/BD_* var and redirect HOME, exactly as the existing +// front-door integration tests do, and add nothing back. +func directLocalServerEnv(home string) []string { + env := make([]string, 0, len(os.Environ())) + for _, kv := range os.Environ() { + key := kv + if i := strings.IndexByte(kv, '='); i >= 0 { + key = kv[:i] + } + if strings.HasPrefix(key, "BEADS_") || strings.HasPrefix(key, "BD_") || + key == "HOME" || key == "USERPROFILE" { + continue + } + env = append(env, kv) + } + return append(env, "HOME="+home, "USERPROFILE="+home) +} + +// stopDirectLocalServer tears the workspace's server down through `bd dolt +// stop`, then verifies the PID bd recorded for it is actually gone. +// +// Safety contract: the ONLY process this may stop is the one named in this +// workspace's own .beads/dolt-server.pid. It never scans, never matches by +// process name, and never signals a PID it did not read from that file. If +// the server survives, the test reports it loudly rather than escalating — +// a stray kill on a host running unrelated dolt servers is far worse than a +// noisy test. +func stopDirectLocalServer(t *testing.T, bd, dir, beadsDir string, env []string) { + t.Helper() + + pid := readRecordedServerPID(t, beadsDir) + + cmd := exec.Command(bd, "dolt", "stop") + cmd.Dir = dir + cmd.Env = env + if stdout, stderr, err := runCommandBuffers(t, cmd); err != nil { + t.Logf("bd dolt stop (direct local): %v\nstdout:\n%s\nstderr:\n%s", err, stdout.String(), stderr.String()) + } + + if pid <= 0 { + return + } + for i := 0; i < 50; i++ { + if !processStillAlive(pid) { + return + } + time.Sleep(100 * time.Millisecond) + } + t.Errorf("direct-local dolt sql-server pid %d survived `bd dolt stop`; "+ + "stop it by that PID before re-running. This test deliberately does not "+ + "kill by process name: this host runs unrelated dolt servers.", pid) +} + +func readRecordedServerPID(t *testing.T, beadsDir string) int { + t.Helper() + raw, err := os.ReadFile(filepath.Join(beadsDir, "dolt-server.pid")) + if err != nil { + return 0 + } + pid, err := strconv.Atoi(strings.TrimSpace(string(raw))) + if err != nil { + return 0 + } + return pid +} + +func processStillAlive(pid int) bool { + proc, err := os.FindProcess(pid) + if err != nil { + return false + } + return proc.Signal(syscall.Signal(0)) == nil +} diff --git a/cmd/bd/topology_capability_matrix_test.go b/cmd/bd/topology_capability_matrix_test.go new file mode 100644 index 000000000..3e00ceb20 --- /dev/null +++ b/cmd/bd/topology_capability_matrix_test.go @@ -0,0 +1,1019 @@ +//go:build cgo && unix + +package main + +// Front-door capability matrix across storage topologies. +// +// THIS IS A CHARACTERIZATION TEST. Every expectation below records what bd +// does TODAY, including the behavior that is wrong. It is green because it +// describes the present, not because the present is good. Rows carrying a +// knownBad string are tracked defects; the slice that fixes one must flip its +// expectation in the same diff, and that flip is the review artifact proving +// the fix landed. +// +// Do NOT "fix" a failing row by relaxing it. A red row means behavior moved; +// either the move was intended (flip the row) or it is a regression. +// +// Reading the output: each cell is classified by `matrixOutcome` (what +// happened) and `matrixReason` (why it is allowed to be that way). +// REFUSED_TYPED + design is a deliberate, permanent policy refusal. +// REFUSED_TYPED + unimplemented is a capability gap a later slice closes. +// WRONG_ANSWER and DEGRADED are bugs bd currently ships. The distinction is +// the whole point of this file: the refusal surface today mixes all three and +// a reader cannot otherwise tell them apart. +// +// The `reason` column is the test's own authority. Production does not yet +// carry a Reason field on its refusals; when it does, this test should assert +// that the two agree. + +import ( + "encoding/json" + "errors" + "flag" + "fmt" + "os" + "os/exec" + "path/filepath" + "sort" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" + "github.com/steveyegge/beads/internal/testutil" +) + +// --------------------------------------------------------------------------- +// Outcome taxonomy +// --------------------------------------------------------------------------- + +type matrixOutcome string + +const ( + // outcomeHonored: the command ran and the topology did not stand in its way. + outcomeHonored matrixOutcome = "HONORED" + // outcomeRefusedTyped: front-door refusal carrying a stable proxy.* code. + outcomeRefusedTyped matrixOutcome = "REFUSED_TYPED" + // outcomeRefusedUntyped: refused with a bare string and no machine-readable + // code. A downstream JSON consumer cannot classify these. + outcomeRefusedUntyped matrixOutcome = "REFUSED_UNTYPED" + // outcomeWrongAnswer: exited successfully and reported something false. + outcomeWrongAnswer matrixOutcome = "WRONG_ANSWER" + // outcomeDegraded: exited successfully having silently skipped the work. + outcomeDegraded matrixOutcome = "DEGRADED" + // outcomeError: reached the command, which then failed on its own merits. + // Not a topology refusal — the front door let it through. + outcomeError matrixOutcome = "ERROR" +) + +type matrixReason string + +const ( + reasonNA = matrixReason("-") + // reasonDesign: refusing is correct and is expected to stay that way. + reasonDesign = matrixReason("design") + // reasonUnimplemented: refusing is a capability gap, not a policy. + reasonUnimplemented = matrixReason("unimplemented") +) + +// expectation is the observed contract for one command on one topology. +type expectation struct { + outcome matrixOutcome + // code is the proxy.* capability code, for outcomeRefusedTyped only. + code string + // substr must appear in the combined output. + substr string + // absent, when set, must NOT appear in the combined output. This is how a + // silent omission (config show dropping database-backed settings) is + // pinned down as a wrong answer rather than a formatting difference. + absent string + // knownBad names the defect this cell documents. Required for + // WRONG_ANSWER and DEGRADED; a later slice deletes it when it flips. + knownBad string + reason matrixReason +} + +// probe is one command exercised across every topology that can safely run it. +// +// Adding a command to the matrix is a single row here. `direct` applies to +// every direct-class topology and `proxied` to every proxied-class one, which +// mirrors bd's own ProxyMode axis; `override` pins a single topology when the +// two-class default is not enough. When a later slice makes managed-local +// honor something external topologies still refuse, that lands as one +// override entry, not new plumbing. +type probe struct { + name string + args []string + direct expectation + proxied expectation + override map[string]expectation + // skipOn excuses named topologies from this row, with the reason. The only + // admissible reason is that RUNNING the command there would leave process + // state this matrix cannot tear down; an inconvenient answer is never one. + // Excused cells print as "(not probed)" and the reason prints under the + // table, so absence is not mistaken for coverage. + skipOn map[string]string +} + +func (p probe) want(topology string, class topologyClass) expectation { + if e, ok := p.override[topology]; ok { + return e + } + if class == classProxied { + return p.proxied + } + return p.direct +} + +// --------------------------------------------------------------------------- +// The matrix +// --------------------------------------------------------------------------- + +const ( + topoEmbedded = "embedded" + topoDirectLocal = "direct-local-server" + topoDirectExternal = "direct-external-server" + topoProxiedLocal = "proxied-local" + topoProxiedTCP = "proxied-external-tcp" +) + +// capabilityProbes is the table. One row per command family from the 1.3.1 +// priority order, plus control rows that must be honored everywhere. +var capabilityProbes = []probe{ + // --- control: plain CRUD must work on every topology ------------------- + { + name: "list", + args: []string{"list"}, + direct: expectation{outcome: outcomeHonored, reason: reasonNA}, + proxied: expectation{outcome: outcomeHonored, reason: reasonNA}, + }, + { + // `list` is the one command that carries --max-rows through the + // proxied reader (proxy_capability.go:230). + name: "list --max-rows", + args: []string{"list", "--max-rows", "5"}, + direct: expectation{outcome: outcomeHonored, reason: reasonNA}, + proxied: expectation{outcome: outcomeHonored, reason: reasonNA}, + }, + + // --- dolt server lifecycle --------------------------------------------- + { + // The same command, three refusal surfaces. A proxied workspace's dolt + // backend belongs to the proxy, which spawns it on demand and reaps it + // when idle, so bd refuses with a code a JSON consumer can branch on. + // Embedded refuses too — there is no server to start — but with a bare + // string and no code. Direct-local is the one topology where bd owns + // the server and the command is its own honored idempotent no-op. + name: "dolt start", + args: []string{"dolt", "start"}, + direct: expectation{ + // The fixture's `bd init --server` already started it, so this is + // the adopt path: doltserver.Start finds the recorded server live + // and returns it rather than launching a second one. + outcome: outcomeHonored, substr: "Dolt server started", reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.dolt_start.conflict", + substr: "dolt start is not supported in proxied-server mode", + reason: reasonDesign, + }, + override: map[string]expectation{ + topoEmbedded: { + outcome: outcomeRefusedUntyped, + substr: "'bd dolt start' is not supported in embedded mode", + reason: reasonDesign, + }, + }, + skipOn: map[string]string{ + // The configured host is 127.0.0.1, so the remote-host ownership + // guard does not fire and doltserver.Start goes on to put a + // bd-owned sql-server over this workspace's own .beads/dolt — for + // a workspace whose server bd does not own. That is worth + // recording, but not by running it: the external-server fixture + // tears down only its proxy, so whatever the start leaves behind + // outlives the test, and this matrix must not leak a dolt server + // onto a shared host. Probe it when the fixture can stop one. + topoDirectExternal: "`bd dolt start` would launch a bd-owned sql-server over the workspace's .beads/dolt " + + "(host is 127.0.0.1, so the remote-host guard does not fire) and the external-server fixture has no " + + "`bd dolt stop` teardown for it, so the process would outlive the test", + }, + }, + { + // Every topology reports the process it actually depends on. Proxied + // used to be the outlier: status read the CLASSIC pidfile, which + // proxied mode never writes, and answered "not running" while the + // proxy was serving CRUD. It now reads the proxy's own records and + // reports the proxy and its dolt backend separately. Direct-local and + // direct-external both report a live server, one bd-owned and one not. + name: "dolt status", + args: []string{"dolt", "status"}, + direct: expectation{ + outcome: outcomeHonored, substr: `"running": true`, reason: reasonNA, + }, + proxied: expectation{ + // `running` describes the PROXY — the endpoint every bd command + // connects through — and the honored rows ahead of this one have + // just started it. The rest of the payload (mode, proxy pid/port, + // backend_managed on an external backend) is asserted field by + // field by the dolt lifecycle tests; what this cell pins is the + // cross-topology claim, that no topology's status lies about + // whether the thing bd talks to is up. + outcome: outcomeHonored, substr: `"running": true`, reason: reasonNA, + }, + override: map[string]expectation{ + topoEmbedded: { + outcome: outcomeHonored, substr: `"mode": "embedded"`, reason: reasonNA, + }, + }, + }, + + // --- the verified wrong answer ----------------------------------------- + { + // collectDatabaseEntries calls ensureDirectMode, which fails under + // proxied, and the error is swallowed with `return nil`. Every + // database-backed setting silently disappears with no marker. + name: "config show", + args: []string{"config", "show"}, + direct: expectation{ + outcome: outcomeHonored, substr: `"source": "database"`, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeWrongAnswer, substr: `"source": "default"`, absent: `"source": "database"`, + knownBad: "config_show.go collectDatabaseEntries swallows the proxied error and omits every DB-stored setting (design 1.5 / slice S6)", + reason: reasonNA, + }, + }, + + // --- doctor ------------------------------------------------------------- + { + name: "doctor", + args: []string{"doctor"}, + direct: expectation{ + outcome: outcomeHonored, substr: `"checks"`, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.doctor.unsupported", + substr: "doctor is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + override: map[string]expectation{ + // Embedded does not refuse: it prints guidance and exits 0, so a + // CI gate running `bd doctor` scores a pass having checked nothing. + topoEmbedded: { + outcome: outcomeDegraded, substr: "not yet supported in embedded mode", + knownBad: "doctor exits 0 on embedded without running checks; a CI gate cannot tell this from success (design 1.5 names the proxied twin; the embedded one is found by this matrix)", + reason: reasonNA, + }, + }, + }, + + // --- backup family (consumer priority #1, slice S3) --------------------- + { + name: "backup status", + args: []string{"backup", "status"}, + direct: expectation{ + outcome: outcomeHonored, substr: `"configured"`, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.backup.unsupported", + substr: "backup status is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + }, + { + name: "backup sync", + args: []string{"backup", "sync"}, + direct: expectation{ + outcome: outcomeError, substr: "no backup destination configured", reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.backup.unsupported", + substr: "backup sync is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + }, + + // --- version-control verbs (slice S4) ----------------------------------- + { + name: "dolt commit", + args: []string{"dolt", "commit", "-m", "matrix probe"}, + direct: expectation{ + outcome: outcomeHonored, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.dolt_commit.unsupported", + substr: "dolt commit is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + }, + { + name: "dolt push", + args: []string{"dolt", "push"}, + direct: expectation{ + outcome: outcomeHonored, substr: "No remote is configured", reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.dolt_push.unsupported", + substr: "dolt push is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + }, + { + name: "dolt remote list", + args: []string{"dolt", "remote", "list"}, + direct: expectation{ + // Renders an empty JSON array; there is no stable marker beyond + // "exit 0 and no refusal", which is what this row asserts. + outcome: outcomeHonored, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.dolt_remote.unsupported", + substr: "dolt remote list is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + }, + { + name: "sync", + args: []string{"sync"}, + direct: expectation{ + outcome: outcomeHonored, substr: `"status": "no-remote"`, reason: reasonNA, + }, + proxied: expectation{ + // Refused three independent ways today: the maintenance table, the + // history class, and an inline RunE string (design 1.3). + outcome: outcomeRefusedTyped, code: "proxy.sync.unsupported", + substr: "sync is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + }, + + // --- refused by design: shared-history semantics are undefined ---------- + { + name: "branch", + args: []string{"branch"}, + direct: expectation{ + outcome: outcomeHonored, substr: `"branches"`, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.branch.unsupported", + substr: "branch is not supported in proxied-server mode", + reason: reasonDesign, + }, + }, + { + name: "flatten", + args: []string{"flatten"}, + direct: expectation{ + // Refuses without --force, which is its own safety interlock and + // nothing to do with topology. + outcome: outcomeError, substr: "irreversible", reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.flatten.unsupported", + substr: "flatten is not supported in proxied-server mode", + reason: reasonDesign, + }, + }, + + // --- transform family (slice S7) ---------------------------------------- + { + name: "rename", + args: []string{"rename", "zz-000000", "zz-111111"}, + direct: expectation{ + outcome: outcomeError, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.transform.unsupported", + substr: "rename is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + }, + + // --- formula verbs (slice S7, demand-gated) ------------------------------ + { + name: "cook", + args: []string{"cook", "matrix-probe-no-such-formula.yaml"}, + direct: expectation{ + outcome: outcomeError, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.formula.unsupported", + substr: "cook is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + }, + + // --- flag-level capability ------------------------------------------------ + { + // The cap is refused rather than silently dropped, which is correct. + // The fix is threading it through the UOW reader as `list` already + // does, not lifting the refusal. + name: "ready --max-rows", + args: []string{"ready", "--max-rows", "5"}, + direct: expectation{ + outcome: outcomeHonored, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeRefusedTyped, code: "proxy.max_rows.unsupported", + substr: "--max-rows / BEADS_MAX_ROWS is not supported in proxied-server mode", + reason: reasonUnimplemented, + }, + }, + + // --- compact ------------------------------------------------------------- + { + // HONORED proxied, which is NOT what proxyMaintenanceRefusals["compact"] + // suggests on a quick read. validateProxyMaintenanceBeforeProvider + // short-circuits to nil when the command has no --dolt flag + // (proxy_capability.go:168-172), and root `bd compact` — the Dolt + // history compaction command — has no such flag. So that table row is + // unreachable from this command path. Recorded here because a reader + // auditing the table would otherwise expect a refusal. + name: "compact", + args: []string{"compact"}, + direct: expectation{ + outcome: outcomeHonored, reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeHonored, reason: reasonNA, + }, + }, + + // --- migrate schema: already lifted in 1.3.0 ----------------------------- + { + // The one family the 1.3.0 release already lifted. It is HONORED + // proxied, but reports no applied count — S6 makes it report one. + name: "migrate schema", + args: []string{"migrate", "schema"}, + direct: expectation{ + outcome: outcomeHonored, substr: "Schema", reason: reasonNA, + }, + proxied: expectation{ + outcome: outcomeDegraded, substr: "reconciled during provider open", + knownBad: "proxied migrate schema reports no applied count (design 1.5 / slice S6)", + reason: reasonNA, + }, + }, +} + +// deliberatelyNotProbed records commands this matrix will not run at all, and +// why. Leaving them undocumented would let a reader mistake absence for +// coverage. A command excused from only SOME topologies belongs in that +// probe's skipOn instead. +var deliberatelyNotProbed = map[string]string{ + "restore": "`bd restore` restores an ISSUE, not a backup; the backup verb is `bd backup restore`, which is already " + + "covered by the backup rows. Probing it adds a row that says nothing about the backup family.", + "backup restore": "would need a real backup destination to distinguish a topology refusal from a missing-backup error. " + + "Covered indirectly by `backup status` / `backup sync`; revisit in slice S3 when the family gains a proxied route.", +} + +// --------------------------------------------------------------------------- +// Topologies +// --------------------------------------------------------------------------- + +type topologyClass string + +const ( + classDirect topologyClass = "direct" + classProxied topologyClass = "proxied" +) + +// topologyWorkspace is one initialized bd workspace plus everything needed to +// run the real binary against it and to assert proxied side effects. +type topologyWorkspace struct { + name string + class topologyClass + dir string + beadsDir string + proxyRoot string // empty for non-proxied topologies + env []string +} + +// run executes the real bd binary and returns stdout, stderr and the exit code. +func (w topologyWorkspace) run(t *testing.T, bd string, args ...string) (string, string, int) { + t.Helper() + cmd := exec.Command(bd, args...) + cmd.Dir = w.dir + cmd.Env = w.env + stdout, stderr, err := runCommandBuffers(t, cmd) + code := 0 + if err != nil { + var ee *exec.ExitError + if !errors.As(err, &ee) { + t.Fatalf("[%s] bd %s could not run: %v\nstdout:\n%s\nstderr:\n%s", + w.name, strings.Join(args, " "), err, stdout.String(), stderr.String()) + } + code = ee.ExitCode() + } + return stdout.String(), stderr.String(), code +} + +type topologySpec struct { + name string + class topologyClass + // make builds the workspace, or calls t.Skip with a reason. Construction + // is lazy so one unavailable topology never hides the others. + make func(t *testing.T, bd string) topologyWorkspace +} + +// embeddedTopology is always available: no server, no docker, no dolt binary. +func embeddedTopology() topologySpec { + return topologySpec{ + name: topoEmbedded, class: classDirect, + make: func(t *testing.T, bd string) topologyWorkspace { + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "mtxe") + return topologyWorkspace{ + name: topoEmbedded, class: classDirect, + dir: dir, beadsDir: beadsDir, env: bdEnv(dir), + } + }, + } +} + +// directLocalServerTopology is `bd init --server` with no port: bd spawns and +// owns a dolt sql-server under .beads on an OS-allocated ephemeral port. +// +// It needs an env that does NOT carry the package TestMain's BEADS_TEST_MODE +// (which disables auto-start and pins the port to 1) and does NOT carry +// bdEnv's BEADS_DOLT_AUTO_START=0. Teardown is `bd dolt stop`, which reads +// .beads/dolt-server.pid — the only PID this test is entitled to stop. +func directLocalServerTopology() topologySpec { + return topologySpec{ + name: topoDirectLocal, class: classDirect, + make: func(t *testing.T, bd string) topologyWorkspace { + testutil.RequireDoltBinary(t) + dir := t.TempDir() + initGitRepoAt(t, dir) + beadsDir := filepath.Join(dir, ".beads") + env := directLocalServerEnv(dir) + + t.Cleanup(func() { stopDirectLocalServer(t, bd, dir, beadsDir, env) }) + + cmd := exec.Command(bd, "init", "--quiet", "--backend", "dolt", "--server", + "--prefix", "mtxdl", "--non-interactive", "--skip-agents", "--skip-hooks") + cmd.Dir = dir + cmd.Env = env + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + t.Fatalf("bd init --server (direct local) failed: %v\nstdout:\n%s\nstderr:\n%s", + err, stdout.String(), stderr.String()) + } + return topologyWorkspace{ + name: topoDirectLocal, class: classDirect, + dir: dir, beadsDir: beadsDir, env: env, + } + }, + } +} + +// directExternalServerTopology reuses newServerModeProject: `bd init --server` +// pointed at the shared test Dolt container, which bd did not start. +func directExternalServerTopology() topologySpec { + return topologySpec{ + name: topoDirectExternal, class: classDirect, + make: func(t *testing.T, bd string) topologyWorkspace { + p := newServerModeProject(t, bd, "mtxdx") + return topologyWorkspace{ + name: topoDirectExternal, class: classDirect, + dir: p.dir, beadsDir: p.beadsDir, env: p.env, + } + }, + } +} + +// proxiedLocalTopology is the managed-local proxy plus its child dolt server. +func proxiedLocalTopology() topologySpec { + return topologySpec{ + name: topoProxiedLocal, class: classProxied, + make: func(t *testing.T, bd string) topologyWorkspace { + requireManagedLocalProxiedEnv(t) + p := bdManagedLocalInit(t, bd, "mtxpl", 5*time.Minute) + return topologyWorkspace{ + name: topoProxiedLocal, class: classProxied, + dir: p.dir, beadsDir: p.beadsDir, proxyRoot: p.proxyRoot, env: bdProxiedEnv(p.dir), + } + }, + } +} + +// proxiedExternalTCPTopology fronts the shared Dolt container with a proxy. +func proxiedExternalTCPTopology() topologySpec { + return topologySpec{ + name: topoProxiedTCP, class: classProxied, + make: func(t *testing.T, bd string) topologyWorkspace { + requireProxiedServerEnv(t) + p := newSharedProxiedProject(t, bd, "mtxpx") + return topologyWorkspace{ + name: topoProxiedTCP, class: classProxied, + dir: p.dir, beadsDir: p.beadsDir, proxyRoot: p.proxyRoot, env: bdProxiedEnv(p.dir), + } + }, + } +} + +// --------------------------------------------------------------------------- +// Entry points +// --------------------------------------------------------------------------- + +// TestProxiedServerTopologyCapabilityMatrix is the primary entry point. +// +// Named TestProxiedServer* so .github/scripts/proxied-test-shard.sh discovers +// it, which puts it in pr-risk.yml's `test-proxied-cmd` lane — a lane that +// runs on pull_request, not only on push-to-main. +// +// It covers four of the five topologies. managed-local is gated on +// BEADS_TEST_PROXIED_LOCAL, which that lane does not set; it is covered by +// TestManagedLocalProxiedCapabilityMatrix in the proxied-local smoke lane +// instead. Both entry points share this table. +func TestProxiedServerTopologyCapabilityMatrix(t *testing.T) { + runCapabilityMatrix(t, []topologySpec{ + embeddedTopology(), + directLocalServerTopology(), + directExternalServerTopology(), + proxiedLocalTopology(), + proxiedExternalTCPTopology(), + }) +} + +// --------------------------------------------------------------------------- +// Runner +// --------------------------------------------------------------------------- + +type cellResult struct { + topology string + probe string + outcome matrixOutcome + reason matrixReason + knownBad bool +} + +func runCapabilityMatrix(t *testing.T, topologies []topologySpec) { + bd := buildEmbeddedBD(t) + + var results []cellResult + exercised := map[string]bool{} + skipped := map[string]string{} + + for _, spec := range topologies { + t.Run(spec.name, func(t *testing.T) { + // A topology that cannot be built here must say so out loud. A + // silently skipped topology is the main way this matrix could + // look green while proving nothing. + defer func() { + if t.Skipped() && skipped[spec.name] == "" { + skipped[spec.name] = "gated off in this environment" + } + }() + + w := spec.make(t, bd) + exercised[spec.name] = true + + // Refusal rows run first, behind the no-side-effect invariants: + // the provider must never have been constructed. Ordering matters + // because a later honored probe legitimately starts the proxy. + refusals, others := partitionProbes(capabilityProbes, spec.name, spec.class) + + if w.class == classProxied && len(refusals) > 0 { + if err := proxy.Shutdown(w.proxyRoot); err != nil { + t.Logf("pre-refusal proxy.Shutdown(%s): %v", w.proxyRoot, err) + } + } + before := w.snapshotArtifacts(t) + + for _, p := range refusals { + p := p + t.Run(p.name, func(t *testing.T) { + want := p.want(spec.name, spec.class) + results = append(results, assertCell(t, w, bd, p, want)) + w.assertNoProviderSideEffects(t, p, before) + }) + } + for _, p := range others { + p := p + t.Run(p.name, func(t *testing.T) { + want := p.want(spec.name, spec.class) + results = append(results, assertCell(t, w, bd, p, want)) + }) + } + + assertCRUDControl(t, w, bd) + }) + } + + reportMatrix(t, topologies, results, exercised, skipped) +} + +// partitionProbes puts the refusal rows for this topology first, preserving +// table order within each group. +func partitionProbes(all []probe, topology string, class topologyClass) (refusals, others []probe) { + for _, p := range all { + if _, excused := p.skipOn[topology]; excused { + continue + } + switch p.want(topology, class).outcome { + case outcomeRefusedTyped, outcomeRefusedUntyped: + refusals = append(refusals, p) + default: + others = append(others, p) + } + } + return refusals, others +} + +// assertCell runs one probe and asserts the observed outcome class. +func assertCell(t *testing.T, w topologyWorkspace, bd string, p probe, want expectation) cellResult { + t.Helper() + + // --json before the subcommand, plus BEADS_JSON=1: a few nested front + // doors do not inherit persistent flags placed after their args. + args := append([]string{"--json"}, p.args...) + stdout, stderr, code := w.run(t, bd, args...) + combined := stdout + stderr + + fail := func(format string, a ...interface{}) { + t.Fatalf("[%s] bd %s\n want %s (reason=%s)%s\n %s\n exit=%d\n stdout:\n%s\n stderr:\n%s", + w.name, strings.Join(p.args, " "), want.outcome, want.reason, + knownBadSuffix(want), fmt.Sprintf(format, a...), code, stdout, stderr) + } + + switch want.outcome { + case outcomeRefusedTyped: + if code == 0 { + fail("expected a typed refusal, but the command succeeded") + } + var got struct { + Code string `json:"code"` + Error string `json:"error"` + Mutates bool `json:"mutates"` + } + if err := json.Unmarshal([]byte(strings.TrimSpace(stdout)), &got); err == nil && got.Code != "" { + if got.Code != want.code { + fail("refusal code = %q, want %q", got.Code, want.code) + } + if got.Mutates { + fail("refusal reported mutates=true; refusals must never mutate") + } + if want.substr != "" && !strings.Contains(got.Error, want.substr) { + fail("refusal message %q does not contain %q", got.Error, want.substr) + } + } else { + // Documented fallback: nested commands render the typed refusal + // through the text path before config-backed JSON is applied. + if want.substr != "" && !strings.Contains(combined, want.substr) { + fail("no typed refusal JSON and output lacks %q", want.substr) + } + } + + case outcomeRefusedUntyped: + if code == 0 { + fail("expected an untyped refusal, but the command succeeded") + } + if strings.Contains(stdout, `"code"`) { + fail("refusal carried a typed code; this row should be REFUSED_TYPED") + } + if want.substr != "" && !strings.Contains(combined, want.substr) { + fail("output lacks %q", want.substr) + } + + case outcomeHonored: + if code != 0 { + fail("expected success") + } + if strings.Contains(combined, "is not supported in proxied-server mode") { + fail("exit 0 but the output carries a proxied refusal") + } + if want.substr != "" && !strings.Contains(combined, want.substr) { + fail("output lacks %q", want.substr) + } + + case outcomeWrongAnswer, outcomeDegraded: + if want.knownBad == "" { + t.Fatalf("[%s] probe %q is %s but carries no knownBad; every bug this matrix records must name itself", + w.name, p.name, want.outcome) + } + if code != 0 { + fail("expected exit 0 (that is what makes it a silent wrong answer)") + } + if want.substr != "" && !strings.Contains(combined, want.substr) { + fail("output lacks the marker %q", want.substr) + } + + case outcomeError: + if code == 0 { + fail("expected a non-zero exit from the command itself") + } + if strings.Contains(combined, "is not supported in proxied-server mode") { + fail("this row expects the command's own error, but it was refused by topology policy") + } + if want.substr != "" && !strings.Contains(combined, want.substr) { + fail("output lacks %q", want.substr) + } + + default: + t.Fatalf("unhandled outcome %q", want.outcome) + } + + if want.absent != "" && strings.Contains(combined, want.absent) { + fail("output unexpectedly contains %q", want.absent) + } + + return cellResult{ + topology: w.name, probe: p.name, + outcome: want.outcome, reason: want.reason, knownBad: want.knownBad != "", + } +} + +// assertCRUDControl proves the workspace is genuinely functional. Without it +// a topology whose store was broken would report a tidy column of refusals +// and look like a successful characterization. +func assertCRUDControl(t *testing.T, w topologyWorkspace, bd string) { + t.Helper() + t.Run("control create+show", func(t *testing.T) { + stdout, stderr, code := w.run(t, bd, "--json", "create", "matrix control probe", "-p", "2") + if code != 0 { + t.Fatalf("[%s] control create failed (exit %d)\nstdout:\n%s\nstderr:\n%s", w.name, code, stdout, stderr) + } + var created struct { + ID string `json:"id"` + } + if err := json.Unmarshal([]byte(strings.TrimSpace(stdout)), &created); err != nil || created.ID == "" { + t.Fatalf("[%s] control create did not return an id: %v\nstdout:\n%s", w.name, err, stdout) + } + showOut, showErr, showCode := w.run(t, bd, "--json", "show", created.ID) + if showCode != 0 { + t.Fatalf("[%s] control show %s failed (exit %d)\nstdout:\n%s\nstderr:\n%s", + w.name, created.ID, showCode, showOut, showErr) + } + if !strings.Contains(showOut, created.ID) { + t.Fatalf("[%s] control show did not round-trip %s\nstdout:\n%s", w.name, created.ID, showOut) + } + }) +} + +// --------------------------------------------------------------------------- +// Side-effect invariants +// --------------------------------------------------------------------------- + +// snapshotArtifacts reuses the existing history-matrix snapshot so both tests +// agree on what "durable artifacts" means. +func (w topologyWorkspace) snapshotArtifacts(t *testing.T) []byte { + t.Helper() + if w.class != classProxied { + return nil + } + return snapshotHistoryArtifacts(t, proxiedProject{beadsDir: w.beadsDir, proxyRoot: w.proxyRoot}) +} + +// assertNoProviderSideEffects is the regression net the design calls for: a +// refusal must be decided before any workspace side effect, so the provider +// must never have been constructed and nothing durable may have changed. +func (w topologyWorkspace) assertNoProviderSideEffects(t *testing.T, p probe, before []byte) { + t.Helper() + if w.class != classProxied { + return + } + if _, err := os.Stat(filepath.Join(w.proxyRoot, proxy.PIDFileName)); err == nil { + t.Fatalf("[%s] %s: proxy started for a refused command; the refusal must precede provider construction", + w.name, p.name) + } + if after := w.snapshotArtifacts(t); string(before) != string(after) { + t.Fatalf("[%s] %s: durable artifacts changed during a refusal", w.name, p.name) + } +} + +// --------------------------------------------------------------------------- +// Reporting +// --------------------------------------------------------------------------- + +// reportMatrix prints the matrix so a reader can tell the classes apart +// without reading the source. This is the completion signal every later slice +// checks itself against. +func reportMatrix(t *testing.T, topologies []topologySpec, results []cellResult, exercised map[string]bool, skipped map[string]string) { + t.Helper() + + var b strings.Builder + b.WriteString("\n=== front-door capability matrix (observed behavior, not desired) ===\n\n") + + byProbe := map[string]map[string]cellResult{} + for _, r := range results { + if byProbe[r.probe] == nil { + byProbe[r.probe] = map[string]cellResult{} + } + byProbe[r.probe][r.topology] = r + } + + var names []string + for _, spec := range topologies { + names = append(names, spec.name) + } + + // Wide enough for the longest label, "REFUSED_TYPED/unimplemented !". + const cellWidth = 29 + + fmt.Fprintf(&b, "%-22s", "COMMAND") + for _, n := range names { + fmt.Fprintf(&b, " | %-*s", cellWidth, n) + } + b.WriteString("\n") + b.WriteString(strings.Repeat("-", 22+len(names)*(cellWidth+3)) + "\n") + + for _, p := range capabilityProbes { + fmt.Fprintf(&b, "%-22s", p.name) + for _, n := range names { + cell, ok := byProbe[p.name][n] + if !ok { + missing := "(not exercised)" + if _, excused := p.skipOn[n]; excused { + missing = "(not probed)" + } + fmt.Fprintf(&b, " | %-*s", cellWidth, missing) + continue + } + label := string(cell.outcome) + if cell.reason != reasonNA && cell.reason != "" { + label += "/" + string(cell.reason) + } + if cell.knownBad { + label += " !" + } + fmt.Fprintf(&b, " | %-*s", cellWidth, label) + } + b.WriteString("\n") + } + + counts := map[matrixOutcome]int{} + designRefusals, unimplementedRefusals, knownBads := 0, 0, 0 + for _, r := range results { + counts[r.outcome]++ + if r.outcome == outcomeRefusedTyped || r.outcome == outcomeRefusedUntyped { + switch r.reason { + case reasonDesign: + designRefusals++ + case reasonUnimplemented: + unimplementedRefusals++ + } + } + if r.knownBad { + knownBads++ + } + } + + fmt.Fprintf(&b, "\ncells: %d over %d commands x %d exercised topologies\n", + len(results), len(capabilityProbes), len(exercised)) + var outcomes []string + for o, c := range counts { + outcomes = append(outcomes, fmt.Sprintf("%s=%d", o, c)) + } + sort.Strings(outcomes) + fmt.Fprintf(&b, "outcomes: %s\n", strings.Join(outcomes, " ")) + fmt.Fprintf(&b, "refusals: %d refused-by-design, %d not-yet-implemented\n", designRefusals, unimplementedRefusals) + fmt.Fprintf(&b, "tracked defects (knownBad, marked !): %d cells\n", knownBads) + + b.WriteString("\ntopologies exercised:\n") + for _, spec := range topologies { + if exercised[spec.name] { + fmt.Fprintf(&b, " [exercised] %s\n", spec.name) + } else { + fmt.Fprintf(&b, " [NOT EXERCISED] %s — %s\n", spec.name, skipped[spec.name]) + } + } + b.WriteString("\ndeliberately not probed:\n") + var notProbed []string + for k, why := range deliberatelyNotProbed { + notProbed = append(notProbed, fmt.Sprintf(" %s — %s", k, why)) + } + for _, p := range capabilityProbes { + for topo, why := range p.skipOn { + notProbed = append(notProbed, fmt.Sprintf(" %s on %s — %s", p.name, topo, why)) + } + } + sort.Strings(notProbed) + for _, line := range notProbed { + b.WriteString(line + "\n") + } + + t.Log(b.String()) + + // A run that built no topology at all proves nothing, and `go test` would + // otherwise print "ok". The embedded topology needs no server, no docker + // and no dolt binary, so an empty run means something is broken rather + // than gated — unless the operator narrowed the run themselves. + if len(exercised) == 0 && !subtestFilterActive() { + t.Fatal("no topology could be built; this run proves nothing. See the report above.") + } +} + +// subtestFilterActive reports whether -run narrows below the top-level test, +// in which case topologies are absent by operator request rather than by +// breakage. +func subtestFilterActive() bool { + f := flag.Lookup("test.run") + return f != nil && strings.Contains(f.Value.String(), "/") +} + +func knownBadSuffix(want expectation) string { + if want.knownBad == "" { + return "" + } + return "\n knownBad: " + want.knownBad +} From d487b9796e87d4db0d89535d88595c98b80e3bd6 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Fri, 18 Sep 2026 17:38:56 -0700 Subject: [PATCH 05/45] release: prep v1.3.1-rc.1 (#6593) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * release: prep v1.3.1-rc.1 Sanctioned RC bump per RELEASING.md "Cut an RC", on the #6298 model: ./scripts/update-versions.sh 1.3.1-rc.1, which carries the stamp into version.go, the three plugin.json manifests, marketplace.json, the npm package.json, the MCP pyproject.toml/__init__.py/uv.lock, default.nix, the ten .githooks managed-section markers and the winres strings. scripts/check-versions.sh passes all 24 gates at 1.3.1-rc.1. Unlike the 1.3.0 RCs, this bump moves the BASE version, so cmd/bd/winres/manifest.xml changes too (1.3.0.0 -> 1.3.1.0) along with the winres numeric file_version/product_version (1.3.0 -> 1.3.1). Those PE fields must be purely numeric; the FileVersion/ProductVersion strings carry the full 1.3.1-rc.1, and uv.lock carries the PEP 440 normalization 1.3.1rc1 written by `uv lock`, never by hand. This has to land before the tag: release.yml's verify-version-consistency compares ${GITHUB_REF_NAME#v} byte-for-byte against the Version string in cmd/bd/version.go, and every stamp on this branch still read 1.3.0. Simulated against the working tree with the workflow's own grep/sed pipeline: the extracted string is exactly "1.3.1-rc.1", on a single matching line. The tag must be spelled v1.3.1-rc.1. CHANGELOG: creates the [1.3.1] section dated 2026-09-16 and leaves [Unreleased] empty but present, which is what #6038 (b3ef65c85) did when it created [1.3.0] at branch-cut, and what #6298's fixup established for an RC tag — an entry parked under [Unreleased] ships in no release by Keep-a-Changelog semantics, yet the RC demonstrably ships it. The three Fixed bullets already on the branch (#6580 x2, #6578) move under [1.3.1] by the rename alone; their wording is untouched, and the shipped [1.3.0] section and everything below it is byte-identical (sha256 of the [1.3.0]-to-EOF region is unchanged across this commit). The 2026-09-16 date is provisional. The final 1.3.1 prep must hand-restamp it to the real release date rather than rerun the beads-release formula's stamp-changelog step, which sed-inserts a fresh header under [Unreleased] and never re-dates an existing section — running it against an already dated [1.3.1] duplicates the heading. Same trap #6298 documented and #6564 then worked around. cmd/bd/info.go: adds the 1.3.1-rc.1 digest at index 0 of versionChanges, dated 2026-09-16 so the array stays newest-first with monotonic dates for TestGetVersionsSinceOrder. It leads with the no-migration note (a 1.3.0 user's upgrade is a binary swap) and then digests the two #6580 changes and the two #6578 ones. Per the #6298 lesson this key can only reach rc.1 users by shipping inside the rc.1 binary; omit it and `bd upgrade review` dumps the whole history at them. docs/cli-docs.pin is still v1.2.2 on this branch — 1.3.0's docs-pin step never happened. check-versions.sh explicitly excludes it, so it is not tag-blocking and is deliberately left for the post-tag docs step. Co-Authored-By: Claude Fable 5 * docs(release): reclassify 1.3.1 behaviour changes and document the config migration Documentation-only amendment to the v1.3.1-rc.1 prep. No version stamp is touched; ./scripts/check-versions.sh stays 24/24 at 1.3.1-rc.1. Three corrections, all inside the `## [1.3.1]` section and its `cmd/bd/info.go` `versionChanges` twin. 1. The flat->nested config migration is now documented (#6594). #6578 made `bd config set ` write a nested mapping, and where a flat top-level key of the same literal name exists it deletes that entry and re-emits the value nested (`updateNestedYamlKey`). `.beads/config.yaml` is not exclusively bd's, so another writer that reads its dotted keys flat stops seeing any value migrated this way — and if it re-adds the flat key, bd's own readers disagree: Viper-backed reads return the flat value while GetStringFromDir returns the nested one. Verified both halves empirically against this tree (dual-spelling probe: viper="flatvalue", GetStringFromDir="nestedvalue", independent of document order; migration probe: `dolt.host` flat entry deleted, sibling `dolt.port` and `other.thing` byte-intact). Realistic collisions are `dolt.*`, `export.auto`, `backup.enabled`. 2. Two genuine behaviour changes moved out of `### Fixed` into `### Changed`, following the 1.3.0-rc.2 precedent in c1e5e49e3: the `bd dolt status --json` shape change (#6580), which a parser of the old payload meets on upgrade, and the two `bd config set` success->failure cases (#6578). The fix content stays in `Fixed` with a pointer; neither statement appears in both sections. 3. `backend_endpoint` added to the documented key list in both CHANGELOG.md and info.go. It is a real `omitempty` field on `proxiedDoltStatus` (cmd/bd/dolt_proxied_lifecycle.go:63), set only when the sidecar reports an external backend — the one topology where `backend_managed` is false and nothing else names the server. Both surfaces now carry the same 11 keys as the struct. Validated: `## [1.3.0]`->EOF is byte-identical before and after (sha256 798a2ee0b0f0b7a3f0fc017ae01c919807740348a34f5ea8719b82a008533d00); exactly one `## [1.3.1]` heading, `## [Unreleased]` present and empty; check-versions.sh 24/24; go build/vet ./...; go test ./cmd/bd -run 'TestGetVersionsSinceOrder|TestVersionChangesStructure|TestVersionChangesCoverage|TestVersionChangesJSON|TestGetVersionsSince|TestDisplayVersion' all PASS, no skips; scripts/ci/pr-lint.sh and scripts/ci/pr-policy.sh green; `bd info --whats-new` surfaces both amendments end-to-end. Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Fable 5 --- .claude-plugin/marketplace.json | 2 +- .githooks/post-checkout | 4 +- .githooks/post-merge | 4 +- .githooks/pre-commit | 4 +- .githooks/pre-push | 4 +- .githooks/prepare-commit-msg | 4 +- CHANGELOG.md | 73 ++++++++++++++----- cmd/bd/info.go | 12 +++ cmd/bd/version.go | 2 +- cmd/bd/winres/manifest.xml | 2 +- cmd/bd/winres/winres.json | 8 +- default.nix | 2 +- integrations/beads-mcp/pyproject.toml | 2 +- .../beads-mcp/src/beads_mcp/__init__.py | 2 +- integrations/beads-mcp/uv.lock | 2 +- npm-package/package.json | 2 +- plugins/beads/.claude-plugin/plugin.json | 2 +- plugins/beads/.codex-plugin/plugin.json | 2 +- plugins/beads/.copilot-plugin/plugin.json | 2 +- 19 files changed, 93 insertions(+), 42 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index aad41ecfb..575163c11 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ "name": "beads", "source": "./plugins/beads", "description": "AI-supervised issue tracker for coding workflows", - "version": "1.3.0" + "version": "1.3.1-rc.1" } ] } diff --git a/.githooks/post-checkout b/.githooks/post-checkout index 19d309b87..3e06c7b41 100755 --- a/.githooks/post-checkout +++ b/.githooks/post-checkout @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.0 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.0 --- +# --- END BEADS INTEGRATION v1.3.1-rc.1 --- diff --git a/.githooks/post-merge b/.githooks/post-merge index 4d315045b..039402e0b 100755 --- a/.githooks/post-merge +++ b/.githooks/post-merge @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.0 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.0 --- +# --- END BEADS INTEGRATION v1.3.1-rc.1 --- diff --git a/.githooks/pre-commit b/.githooks/pre-commit index 46e710a33..a667ef2b0 100755 --- a/.githooks/pre-commit +++ b/.githooks/pre-commit @@ -40,7 +40,7 @@ if [ -n "$staged_go_files" ]; then echo "$staged_go_files" | xargs git add fi -# --- BEGIN BEADS INTEGRATION v1.3.0 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -97,4 +97,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.0 --- +# --- END BEADS INTEGRATION v1.3.1-rc.1 --- diff --git a/.githooks/pre-push b/.githooks/pre-push index 8bcc2483b..096439496 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -39,7 +39,7 @@ fi exec < <(printf '%s\n' "$push_refs") -# --- BEGIN BEADS INTEGRATION v1.3.0 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -96,4 +96,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.0 --- +# --- END BEADS INTEGRATION v1.3.1-rc.1 --- diff --git a/.githooks/prepare-commit-msg b/.githooks/prepare-commit-msg index bf8cfb8ff..1c24adc7c 100755 --- a/.githooks/prepare-commit-msg +++ b/.githooks/prepare-commit-msg @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.0 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.0 --- +# --- END BEADS INTEGRATION v1.3.1-rc.1 --- diff --git a/CHANGELOG.md b/CHANGELOG.md index 375762a21..8a4489a31 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,56 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.3.1] - 2026-09-16 + +### Changed + +- **`bd dolt status --json` changes shape on a proxied workspace** + ([#6580](https://github.com/gastownhall/beads/pull/6580)). It now emits + `{"mode": "proxied-server", "root": ..., "running": , "proxy_pid": + ..., "proxy_port": ..., "backend_managed": ..., "backend_running": ..., + "backend_pid": ..., "backend_port": ..., "backend_endpoint": ..., + "idle_timeout": ...}` instead of the always-false `{"running": false, "pid": + 0, "port": 0}` — `pid` and `port` are gone, replaced by the `proxy_*` and + `backend_*` pairs. `running` no longer reports a bd-managed dolt PID file; it + describes the proxy, the endpoint every bd command connects through, so it is + now usually true where it used to be always false. `backend_managed` is false + on external proxied topologies, where the dolt server is not bd's process to + report on and `backend_endpoint` names it instead. Anything parsing the old + payload changes behaviour on upgrade. Direct-server, shared, embedded and + externally-managed workspaces keep the output they had. + +- **`bd config set ` deletes a flat top-level key of the same + literal name** ([#6578](https://github.com/gastownhall/beads/pull/6578)). + Dotted keys are now written as a nested mapping (see **Fixed** below for why), + and where the file already carried the flat spelling — a top-level key whose + *name* contains the dot, such as `dolt.host: "10.0.0.1"` — that entry is + removed and the value re-emitted under `dolt:`. Only the key being written is + touched; other dotted keys, including ones bd did not write, are left exactly + as they are. + + `.beads/config.yaml` is not exclusively bd's, so this can be visible outside + bd: an external tool that both writes *and* reads flat dotted keys there will + stop seeing any value bd migrates this way. Any dotted key bd stores in + `config.yaml` rather than the database can collide; in practice that means + `dolt.*`, `export.auto`, and `backup.enabled`. If the other writer then + re-adds its flat key, the file carries both spellings and bd's own readers + disagree — Viper-backed reads (`bd config get`, and the `dolt.auto-start` + lookup on the store-open path) return the **flat** value, while + `GetStringFromDir` returns the **nested** one, so a `bd config set` can be + shadowed by a stale flat entry. Tracked as + [#6594](https://github.com/gastownhall/beads/issues/6594). Until that lands, + in a workspace whose config another tool maintains, let that tool own its + keys, or check `.beads/config.yaml` for a duplicated key after running + `bd config set`. + +- **Two `bd config set` writes that used to exit 0 now exit 1** + ([#6578](https://github.com/gastownhall/beads/pull/6578)), because both wrote + a value no reader could see: setting a dotted key under a parent that already + holds a scalar (`sync: enabled`, then `bd config set sync.remote`), and + setting one in a file whose top level is not a mapping. A script that + tolerated the old exit 0 now has to handle the failure. + ### Fixed - **`bd dolt start` no longer puts a second sql-server over a proxied @@ -26,15 +76,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 server: not running` while the proxy was serving CRUD — the wrong answer that sent operators to `bd dolt start` in the first place. It now reads the proxy's own records and reports the proxy and its dolt backend separately, - without starting either. **JSON output shape change**: on a proxied - workspace `bd dolt status --json` now emits - `{"mode": "proxied-server", "root": ..., "running": , "proxy_pid": - ..., "proxy_port": ..., "backend_managed": ..., "backend_running": ..., - "backend_pid": ..., "backend_port": ..., "idle_timeout": ...}` instead of - the always-false `{"running": false, "pid": 0, "port": 0}`. `running` - describes the proxy, the endpoint every bd command connects through; - `backend_managed` is false on external proxied topologies, where the dolt - server is not bd's process to report on. + without starting either. The `--json` payload changes shape to carry that — + see **Changed** above before parsing it. - **A dotted config key now round-trips: what `bd config set` writes, `bd config get` and bd's own readers find** @@ -46,16 +89,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 walks nested mappings, so it does not. The value was set and invisible at the same time depending on which reader asked, and `bd init --remote` in a fresh workspace was the ordinary way in: the remote was recorded and then not found. - Dotted keys are written nested now, an existing flat spelling of the key being - written is migrated to the nested one, and `bd config unset` removes the - nested form — it only ever matched the flat spelling, so unsetting + Dotted keys are written nested now, and `bd config unset` removes the nested + form — it only ever matched the flat spelling, so unsetting `sync.remote` silently left the remote live. Keys the write does not own, including dotted ones somebody else put there, are left exactly as they are. - - Two things that used to succeed now report an error instead, because both - produced a value no reader could see: setting a key under a parent that - already holds a value (`sync: enabled`, then `bd config set sync.remote`), and - setting one in a file whose top level is not a mapping. + Two consequences of the nested write — the flat-key migration and two new + failure exits — are under **Changed** above. ## [1.3.0] - 2026-09-15 diff --git a/cmd/bd/info.go b/cmd/bd/info.go index e8fee1860..d33378517 100644 --- a/cmd/bd/info.go +++ b/cmd/bd/info.go @@ -220,6 +220,18 @@ type VersionChange struct { // versionChanges contains agent-actionable changes for recent versions var versionChanges = []VersionChange{ + { + Version: "1.3.1-rc.1", + Date: "2026-09-16", + Changes: []string{ + "RC: first candidate for the 1.3.1 patch line, on top of 1.3.0. It carries fixes only — no schema migration, so upgrading from 1.3.0 is a binary swap. If you are coming from 1.2.2 or earlier, the [1.3.0] entry below still describes everything you meet, including the migration and backup notes; read it first.", + "CHANGE: 'bd dolt start' on a PROXIED workspace is now a typed refusal with code 'proxy.dolt_start.conflict' instead of starting a second sql-server over the proxy's own data directory. It previously either stranded ordinary bd commands on 'invalid connection' or adopted the proxy's dolt child into bd's classic PID/port records, leaving two managers for one process. The proxy owns its backend's lifecycle: it starts on demand, and 'bd dolt stop' shuts it down. Direct-server and embedded workspaces are unaffected.", + "CHANGE: 'bd dolt status --json' changes shape on a proxied workspace. It now reports the proxy and its dolt backend separately — {\"mode\": \"proxied-server\", \"running\": , \"proxy_pid\", \"proxy_port\", \"backend_managed\", \"backend_running\", \"backend_pid\", \"backend_port\", \"backend_endpoint\", \"idle_timeout\", \"root\"} — instead of the always-false {\"running\": false, \"pid\": 0, \"port\": 0} it emitted by reading a PID file proxied mode never writes. 'pid' and 'port' are gone, replaced by the proxy_*/backend_* pairs. 'running' now describes the proxy, the endpoint every bd command connects through, so it is usually true where it used to be always false; 'backend_managed' is false on external proxied topologies, where 'backend_endpoint' names the server bd does not manage. A script parsing the old payload changes behavior on upgrade.", + "FIX: a dotted config key now round-trips — what 'bd config set' writes is what 'bd config get' and bd's own readers find. Setting a key like 'sync.remote' or 'dolt.host' into a config.yaml with no matching section used to append a top-level key whose NAME contained the dot, which Viper finds and the direct reader GetStringFromDir does not; 'bd init --remote' in a fresh workspace was the ordinary way in, recording a remote no reader could locate. Dotted keys are written nested now, an existing flat spelling is migrated, and 'bd config unset' removes the nested form (it only ever matched the flat one, so unsetting 'sync.remote' silently left the remote live).", + "CHANGE: writing a dotted key nested has a consequence outside bd — 'bd config set ' DELETES a flat top-level key of the same literal name. Writing 'dolt.host' into a file that already carries a top-level 'dolt.host: \"10.0.0.1\"' removes that entry and re-emits the value under 'dolt:'; only the key being written is touched. .beads/config.yaml is not exclusively bd's, so an external tool that both writes AND reads flat dotted keys there stops seeing any value bd migrates this way. Realistic collisions are 'dolt.*', 'export.auto', and 'backup.enabled'. If that tool re-adds its flat key the file carries both spellings and bd's own readers disagree: Viper-backed reads ('bd config get', and the dolt.auto-start lookup on store open) return the FLAT value while GetStringFromDir returns the NESTED one, so a 'bd config set' can be shadowed by a stale flat entry. Tracked as #6594; until it lands, let the other tool own its keys, or check .beads/config.yaml for a duplicated key after running 'bd config set'.", + "CHANGE: two config writes that used to exit 0 now exit 1, because both produced a value no reader could see — setting a dotted key under a parent that already holds a scalar ('sync: enabled', then 'bd config set sync.remote'), and setting one in a file whose top level is not a mapping.", + }, + }, { Version: "1.3.0", Date: "2026-09-15", diff --git a/cmd/bd/version.go b/cmd/bd/version.go index 3bf32c13f..be648b593 100644 --- a/cmd/bd/version.go +++ b/cmd/bd/version.go @@ -16,7 +16,7 @@ import ( var ( // Version is the current version of bd (overridden by ldflags at build time) - Version = "1.3.0" + Version = "1.3.1-rc.1" // Build can be set via ldflags at compile time Build = "dev" // Commit and branch the git revision the binary was built from (optional ldflag) diff --git a/cmd/bd/winres/manifest.xml b/cmd/bd/winres/manifest.xml index 5623c89ce..aae5b5d6b 100644 --- a/cmd/bd/winres/manifest.xml +++ b/cmd/bd/winres/manifest.xml @@ -3,7 +3,7 @@ beads - AI-supervised issue tracker diff --git a/cmd/bd/winres/winres.json b/cmd/bd/winres/winres.json index b3c9a8137..c497b8a90 100644 --- a/cmd/bd/winres/winres.json +++ b/cmd/bd/winres/winres.json @@ -9,8 +9,8 @@ "#1": { "0000": { "fixed": { - "file_version": "1.3.0", - "product_version": "1.3.0", + "file_version": "1.3.1", + "product_version": "1.3.1", "type": "App" }, "info": { @@ -18,12 +18,12 @@ "Comments": "AI-supervised issue tracker for coding workflows", "CompanyName": "Steve Yegge", "FileDescription": "beads - AI-supervised issue tracker", - "FileVersion": "1.3.0", + "FileVersion": "1.3.1-rc.1", "InternalName": "bd", "LegalCopyright": "Copyright (c) 2024-2026 Steve Yegge. MIT License.", "OriginalFilename": "bd.exe", "ProductName": "beads", - "ProductVersion": "1.3.0" + "ProductVersion": "1.3.1-rc.1" } } } diff --git a/default.nix b/default.nix index 6bbc63ec4..cfe3c35bd 100644 --- a/default.nix +++ b/default.nix @@ -7,7 +7,7 @@ }: buildGoModule { pname = "beads"; - version = "1.3.0"; + version = "1.3.1-rc.1"; src = self; diff --git a/integrations/beads-mcp/pyproject.toml b/integrations/beads-mcp/pyproject.toml index 412c678c5..5832ba943 100644 --- a/integrations/beads-mcp/pyproject.toml +++ b/integrations/beads-mcp/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "beads-mcp" -version = "1.3.0" +version = "1.3.1-rc.1" description = "MCP server for beads issue tracker." readme = "README.md" requires-python = ">=3.10" diff --git a/integrations/beads-mcp/src/beads_mcp/__init__.py b/integrations/beads-mcp/src/beads_mcp/__init__.py index 2c222f1f7..8118d4656 100644 --- a/integrations/beads-mcp/src/beads_mcp/__init__.py +++ b/integrations/beads-mcp/src/beads_mcp/__init__.py @@ -4,4 +4,4 @@ This package provides an MCP (Model Context Protocol) server that exposes beads (bd) issue tracker functionality to MCP Clients. """ -__version__ = "1.3.0" +__version__ = "1.3.1-rc.1" diff --git a/integrations/beads-mcp/uv.lock b/integrations/beads-mcp/uv.lock index 78ff50cd8..1ccf25835 100644 --- a/integrations/beads-mcp/uv.lock +++ b/integrations/beads-mcp/uv.lock @@ -175,7 +175,7 @@ wheels = [ [[package]] name = "beads-mcp" -version = "1.3.0" +version = "1.3.1rc1" source = { editable = "." } dependencies = [ { name = "fastmcp" }, diff --git a/npm-package/package.json b/npm-package/package.json index eedd8c5da..c9382b627 100644 --- a/npm-package/package.json +++ b/npm-package/package.json @@ -1,6 +1,6 @@ { "name": "@beads/bd", - "version": "1.3.0", + "version": "1.3.1-rc.1", "description": "Beads issue tracker - lightweight memory system for coding agents with native binary support", "main": "bin/bd.js", "bin": { diff --git a/plugins/beads/.claude-plugin/plugin.json b/plugins/beads/.claude-plugin/plugin.json index f861043e2..0ee54c8c7 100644 --- a/plugins/beads/.claude-plugin/plugin.json +++ b/plugins/beads/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.0", + "version": "1.3.1-rc.1", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" diff --git a/plugins/beads/.codex-plugin/plugin.json b/plugins/beads/.codex-plugin/plugin.json index bf76223b3..96958575f 100644 --- a/plugins/beads/.codex-plugin/plugin.json +++ b/plugins/beads/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "beads", - "version": "1.3.0", + "version": "1.3.1-rc.1", "description": "AI-supervised issue tracking for coding workflows with agent skills.", "author": { "name": "Steve Yegge", diff --git a/plugins/beads/.copilot-plugin/plugin.json b/plugins/beads/.copilot-plugin/plugin.json index 90a496a43..dd5cd6eff 100644 --- a/plugins/beads/.copilot-plugin/plugin.json +++ b/plugins/beads/.copilot-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.0", + "version": "1.3.1-rc.1", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" From d0063ac894b41cb6b38d67724e4f3b9d89092103 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 21 Sep 2026 00:23:37 -0700 Subject: [PATCH 06/45] release/1.3.0: eleven cherry-picks for 1.3.1-rc.2 (#6653) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(storage): include custom active-category statuses in bd ready (#5831) (#5918) bd ready pins WorkFilter.Status to StatusOpen. BuildReadyWorkWhere treated that pin as an exact match, so issues in a custom status with category active never appeared — even though bd statuses --help, the ready_issues view (migration 0025), and ClaimableSourceStatusesInTx all treat that category as ready-eligible. Widen the singular StatusOpen clause to the same predicate the view uses: status = ? OR status IN (SELECT name FROM custom_statuses WHERE category = 'active') Non-open singular statuses stay exact. Built-in open still matches. Custom wip/done/frozen/unspecified statuses stay out, matching claim's GH-3570 anti-steal set. Reported by Justin Clemens (@justin-candor). Addresses #5831 (cherry picked from commit 66aa73bb992399a5feeeb900933784f00f027a2d) * fix(cli): `bd list --wisp-type` cannot match a row — register the plane knob it needs (#6098) `bd list --wisp-type heartbeat` returns an empty listing for EVERY input, and `bd list` exposes no flag that changes that. ListRequest.WispType is a predicate, not a plane selector — its own doc says so, and RunReaderListWispTypeNarrowsTheAdmittedPlaneRatherThanAdmittingIt pins it. That contract is right and this does not touch it. The doc also names the way out: The combination that answers with rows is IncludeEphemeral (or IncludeInfra) plus this, and it is lawful rather than refused. `bd list` registers neither `--include-ephemeral` nor `--all-types`. So of the two combinations the doc names, the CLI can express only IncludeInfra — which admits the plane AND lifts the infra-type exclusions AND changes the template treatment, three decisions a caller who wanted one classification did not ask for. The narrow knob the doc names FIRST has no flag at all. That is the whole defect: the API composes correctly, and the CLI is missing the other half of the pair. ## What this does 1. Registers `--include-ephemeral` on `bd list`, wired to `ListRequest.IncludeEphemeral` — the plane knob alone, off by default, so the default listing is byte-identical. 2. Refuses `--wisp-type` when no plane is admitted, with the fix in the hint, instead of printing an empty listing. The refusal is at the CLI layer ONLY. At the API layer the combination stays lawful and still composes to an empty page: that is the pinned contract, and a programmatic caller composing filters may legitimately land on it. A human typing `--wisp-type heartbeat` cannot have meant "prove to me that durable rows carry no classification", so the CLI is where the guess is safe to make. An explicit `--type` is deliberately left alone: an infra type routes to the plane by itself, so the request may well be satisfiable, and this layer cannot tell which types those are without loading the workspace's infra vocabulary. Refusing there would reject lawful, answerable requests. ## Verification --- PASS: TestListRegistersIncludeEphemeral --- PASS: TestGatherListInput_WispTypeWithoutAPlaneIsRefused --- PASS: TestGatherListInput_WispTypeWithAPlaneIsAccepted (3 subtests) --- PASS: TestGatherListInput_IncludeEphemeralReachesTheRequest The pair discriminates rather than being trivially green: a guard that refused every `--wisp-type` would pass the refusal test and fail all three acceptance subtests, one per route that admits the plane. `ok github.com/steveyegge/beads/cmd/bd`, `ok github.com/steveyegge/beads/internal/workapi`, `go build ./...` and `go vet` clean. `internal/workapi/list.go` and the golden corpus are untouched. docs/CLI_REFERENCE.md is not regenerated here: the docs pipeline builds from the release tag in docs/cli-docs.pin (v1.2.2), not from this checkout, so the new flag appears at the next release bump. Claude-Session: https://claude.ai/code/session_01X1jMLQTSrCwtzFRS1xeUC6 (cherry picked from commit cbf84cb1c007dfcb2beb7d64f795e1ba8b76ed17) * fix(list): honor --status when combined with --ready (#5924) * fix(list): honor --status when combined with --ready (#5832) `bd list --status X --ready` silently dropped --status and returned the unfiltered ready set (typically open issues). BuildListFilter pinned StatusOpen whenever ReadyFlag was set, and ReadyFilterFromIssueFilter hard-coded the same pin, so a custom status or in_progress selector never reached GetReadyWork. --ready without --status still defaults to open. An explicit --status is now the ready-work status selector, matching how other list filters already compose with --ready. `--status all` remains the no-filter spelling and keeps the open default. * fix(list): status-aware --ready footer and honor-path contract docs (#5832) The query intersection from the prior commit is unchanged. Round-1 review asked the ReadyFlag/Statuses contract comments and the text footer to describe the honor path: default --ready is still open, an explicit --status is named instead of "excludes in_progress". (cherry picked from commit f5a20fb66b6fd0893682eb9ed83d93320e0c809d) * fix(doctor): ignore the generated dolt-server-config.yaml (#5978) doltserver.Start() writes .beads/dolt-server-config.yaml when the resolved dolt binary supports auto_gc_behavior.archive_level. Its own doc comment says the file "lives alongside the other gitignored server state files in beadsDir", but the hyphenated name falls outside the "dolt-server." prefix shared by the other five entries, so both GitignoreTemplate and requiredPatterns missed it. The file holds an absolute cfg_dir and a per-machine port, so committing it breaks every other clone -- the same reason the template already gives for 'redirect'. It shows up as untracked in every consumer repo whose dolt supports archive_level. This repo does not see it because its own root .gitignore has a blanket .beads/* rule. Add the pattern to requiredPatterns as well as the template: the template only helps repos initialised after this change, while requiredPatterns is what missingGitignorePatterns() iterates, so it is what lets bd doctor --fix append the rule to an existing .beads/.gitignore. Co-authored-by: Adam Spiers (cherry picked from commit 671db6560dcfd312b82669c5909ef045c296a644) * fix(storage): stop recommending an inert config.yaml key for a missing prefix (#5936) ReadConfigPrefix gates on the database's config table row `issue_prefix`. When that row is absent its error told the operator that, "if using config.yaml, use key 'issue-prefix', not 'issue_prefix'" — but this gate never consults config.yaml. The YAML value is read one line above the error solely to populate a debug.Logf diagnostic; it is discarded, so setting `issue-prefix` in config.yaml leaves the gate failing exactly as before. Following the advice produces a tracked config field that changes nothing and misleads the next reader. Point the message at what actually writes the row — `bd init --prefix` or `bd bootstrap`, both of which it already named — and say plainly that the check reads the database's config table so the config.yaml key does not satisfy it. No resolution behavior changes; this is the error text only. Reported-by: @ncljesus Closes #5916 (cherry picked from commit 61ec85181f5baa3343aa53b166e30ba9490f4241) * fix(create): say --file is one issue per ## heading (#5921) bd create --file/-f is a batch-import route: it creates one issue per H2 ("##") heading, and ### subsections attach fields to that issue. The old Usage was "Create multiple issues from markdown file", which never named those semantics or --body-file. Callers who wanted "one issue, description loaded from a file" reached for the shortest flag and got a silent bulk import instead -- reported by @seanmartinsmith with a 3-section briefing file (## Problem / ## Context / ## Acceptance Criteria) that became 3 issues instead of one. gatherCreateInput already rejects a positional title combined with --file. That is the reporter's literal repro, so this keeps the rejection and adds a --body-file hint on the error they actually see. The remaining trap -- --file used alone on a section-headed file -- is unchanged, because bd cannot tell that apart from a genuine batch file by content. Tests: parseMarkdownFile reproduces the trap file; gatherCreateInput locks in the title+--file rejection and --body-file hint; flag Usage names the per-heading behavior. The existing TestCreateFromMarkdownFile binary path already covers well-formed --file creates. (cherry picked from commit 67b066b58e1fa9bff3ea4ec8eeb6ec28d0e35c9f) * fix(list): stop truncation hint padding a blank TTY line (#5927) printTruncationHint fed surrounding newlines through ui.RenderWarn. lipgloss treats that as a block, pads the blank lines to the widest-line width (alignTextHorizontal shortAmount = widestLine - lineWidth), and drops the trailing newline (GH#5685). The next shell prompt then prints on the same line as a run of spaces. Keep the newlines outside RenderWarn so the hint is one colored line with a real trailing newline. Tests construct an explicitly styled lipgloss Style so the padded blank-line shape fails if the helper is reverted. Fixes #5685. (cherry picked from commit 766c0bd15a7e368cf4541a48989cd328fed22ea1) * fix(setup/claude): keep settings.json's trailing newline and skip no-op rewrites (#5944) * fix(setup/claude): keep settings.json's trailing newline and skip no-op rewrites (#5693) bd setup claude (and bd init, which calls it) marshalled the Claude settings map with json.MarshalIndent and wrote it unconditionally. MarshalIndent emits no trailing newline, so every run stripped the one Claude Code's own writes leave behind, and re-wrote the file even when no hook changed. Because bd init git-adds .claude/settings.json, that surfaced as a staged diff with no semantic content. The same marshal-then-write pattern appears at four call sites in claude.go: installClaude's settings write and its settings.local.json migration, and removeClaude's settings write and its settings.local.json cleanup. Fixing one would have left the bug live in the other three, so both halves are now shared helpers -- marshalSettings appends the newline, writeSettingsIfChanged compares against the bytes already on disk and returns early when they match -- and all four sites go through them. Reported by @maphew. * chore(setup/claude): use the repo spelling "marshaled" in the new comments The misspell linter flags British "marshalled"; CI pins golangci-lint v2.10.1 and runs it with --new-from-merge-base, so both new comment lines are in scope. Comment text only, no behaviour change. (cherry picked from commit cad7e17d70d1df34890aae2547e4d428bf5b5c7f) * fix(gate): evaluate cross-rig bead gates via prefix routes (#5859) * fix(gate): route cross-rig bead checks (#5857) Evaluate historical rig:bead gate targets through the current read-only prefix routing seam in direct and proxied-server modes. Preserve local bead gates, reject malformed cross-rig values, and cover the real two-store route with a Dolt integration test. Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Klas Hesselman * fix(gate): address cross-rig review feedback (#5857) Preserve the original local not-found error when prefix routing is unavailable, document the user-visible behavior, and lock the proxied error contract with a regression test. Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Klas Hesselman (cherry picked from commit cbfc505e39a60514c57dcdb5afe155c8659647ba) * fix(storage/uow): retry transient ping failures during openDB bootstrap (#6003) * test(uow): red — pingWithRetry does not exist yet for openDB's boot-time ping Adds TestPingWithRetryRecoversFromATransientBadConn (be-x49l6): a fake pinger exercising three cases — transient error retries then recovers, non-transient error fails without retrying, and an already-cancelled context fails promptly. Fails to compile until pingWithRetry/pinger are added to dolt_sql_provider.go. * feat: green — Fix: unretried ping in uow openDB turns a transient Dolt connection drop into a hard CI failure (refs be-x49l6) * refactor: drop what-comment on pingWithRetry (self-review, refs be-x49l6) Self-review flagged it as restating the function's own signature/behavior rather than explaining a non-obvious why; the pinger and isTransientPingError comments stay since they carry rationale the code doesn't otherwise convey. * test(uow): red — a hung boot ping is not bounded per attempt Adds the two behaviours review #6003 asked for, both failing: - a_hung_ping_is_capped_per_attempt_and_retried: blockingPinger stalls until its context is cancelled. backoff.Retry bounds only the gap between attempts, so with the caller's context handed straight to PingContext the call never returns at all. The subtest runs the call off-goroutine so this reports as "never returned" instead of a package timeout. - connection-level_net_error_retries_then_recovers: a *net.OpError (connection refused / reset / broken pipe) is currently classified non-transient and fails the boot ping outright. the_caller's_own_deadline_fails_without_retrying passes already and is kept as the regression guard for the trap #6000 fell into: making context.DeadlineExceeded retryable must not make the caller's own expiry retryable too. pingWithRetry takes attemptTimeout here but does not honour it yet, so the red is behavioural rather than a compile error. Refs be-g1i7n. * fix(uow): bound each boot ping attempt and widen the transient classifier Review #6003, point 1. backoff.Retry bounds the gap between attempts but cannot interrupt one in flight, and the DSN sets a dial Timeout (5s) with no ReadTimeout — so a server that accepts TCP and then stalls mid-handshake blocked for the caller's entire context, which is the exact scenario the PR summary claims to fix. Each attempt now runs under its own context.WithTimeout (pingAttemptTimeout, 10s — above the 5s dial timeout so a real dial timeout still surfaces as itself), and context.DeadlineExceeded is classified transient so the retry loop actually continues. pingWithRetry checks the caller's context *before* classifying. That is what keeps the retryable DeadlineExceeded from swallowing the caller's own expiry — the inverse of the regression #6000 introduced by capping each attempt and then treating the resulting DeadlineExceeded as permanent. Covered from both sides: a_hung_ping_is_capped_per_attempt_and_retried and the_caller's_own_deadline_fails_without_retrying. Review #6003, point 2. isTransientPingError stays separate from internal/storage/dolt's isRetryableError, and now says why in full: that one is a query-time classifier matching on error text, admitting migration-lock, read-only and MySQL-1105 shapes a boot ping cannot be in because nothing has executed on the connection yet. The coverage gap the review named is closed on this side instead — *net.OpError is transient, which is refused/reset/broken-pipe portably and by type rather than by substring, with permanent DNS failures excluded so an unresolvable host still fails fast instead of spending the whole budget. Refs be-g1i7n. (cherry picked from commit c0d8da42de5fd15c95adac85e342ba4a121da0fb) * fix(deps): bump dolt/go past upstream manifest-missing-chunk defense (#4070) (#5931) * fix(deps): bump dolt/go past upstream manifest-missing-chunk defense (#4070) Pin github.com/dolthub/dolt/go to 796d074977417da931df65202a79e0a7f532a3db. That hop includes: - 23ccbf9442275166f168939277a5b4a348a63c9d — reject a file manifest update that references a missing table file (the defense itself) - 796d074977417da931df65202a79e0a7f532a3db — doc/test follow-up to the defense commit (no production-semantics change) A non-atomic backup sync can leave the backup manifest ahead of chunk files; subsequent syncs then fail with "table file not found" and stay broken. The upstream defense turns that silent corruption into a loud failed sync. Beads' previous pin (a6690826d767, 2026-07-15) predates both commits. Minimal-churn pin: do not bump to today's dolt tip. go mod tidy did not force a later version. This is a mitigation, not a root-cause fix: backup file+manifest publish remains non-atomic, and backups already in the manifest-ahead state are not repaired by this change. Mitigates #4070 * fix(nix): update vendorHash for the dolt/go bump The go.mod/go.sum bump in this PR invalidates the pinned vendorHash, so `nix build .#default` fails with a fixed-output hash mismatch. Update it to the value CI computed: specified: sha256-D+KDoLFJHpt9D9rCrtE3NMfL4fgK7UxpdKlCxp5yz5Y= got: sha256-wDU1tw/4t2CetwQl4+zUhlDI70dSasLLI2Gy66ZyhQQ= The repo documents this coupling in .github/dependabot.yml; the existing "Update vendorHash for dependabot Go bumps" workflow only fires for dependabot PRs, so a manual bump is needed here. (cherry picked from commit 851aa2f20ab7556c97f5650cf0976cd6a6946dd4) * docs(changelog): record the 1.3.1 cherry-pick behaviour deltas Documents all eleven picks under the existing `## [1.3.1]` heading. The six behaviour deltas go under Changed, including the two that are not pure wins and that a user is most likely to hit without knowing why: - bd doctor now warns on every pre-existing workspace until --fix appends the new .beads/dolt-server-config.yaml gitignore rule. - opening a store against a dead server takes up to ~30s to fail instead of failing instantly, because the boot ping is retried. Also called out as non-obvious: the ready set grows where custom active-category statuses exist, bare --wisp-type becomes a refusal rather than an empty success, and the Dolt bump converts a quietly corrupting backup sync into a loud failing one without repairing backups already in that state. The dated `## [1.3.0]` section is shipped and is left byte-identical: the diff is 165 insertions and 0 deletions, and the [1.3.0]-to-EOF region still hashes to 798a2ee0b0f0b7a3f0fc017ae01c919807740348a34f5ea8719b82a008533d00. The `## [1.3.1]` date is left for a later release prep to restamp. * fix(nix): recompute vendorHash for the release-branch dependency set The #5931 cherry-pick (510675948) bumps four dolt module pins, which invalidates the pinned vendorHash. Neither side's value was usable: the release branch already carried its own dependency bump (d594092eb), so this pick lands on a vendor set that has existed on neither release/1.3.0 nor main, and #5931's own hash was computed against main's graph. Taken from the `got:` value the nix-build lane reported for this branch: specified: sha256-wDU1tw/4t2CetwQl4+zUhlDI70dSasLLI2Gy66ZyhQQ= got: sha256-DQdauEx5g48Xbxrz4wGLx4vkrQoYQX3FUx+7N/Y6YV4= go mod tidy after the pick was a no-op, and the dependency delta against the release tip is exactly the four intended pins, so this hash covers that change and nothing else. * docs(changelog): scope the #6003 open-latency entry and correct its bound The 1.3.1 entry for the openDB bootstrap-ping retry said "up to ~30s" and named no topology. Both were misleading: - 30s is the retry BUDGET, not the wall-clock bound. backoff stops scheduling new attempts at 30s (MaxElapsedTime, dolt_sql_provider.go) but cannot cancel one in flight, and pingAttemptTimeout caps an attempt at 10s — so a single open can take ~40s. openAndInitSchema pings twice, each with a fresh budget, so the narrow case where the backend satisfies the first ping and the schema init and is then unreachable for the second reaches ~80s. - Only unit-of-work provider construction takes that ping, which is a minority of topologies: proxied-server (local and external) on every command, plus `bd serve` in server/external-server/shared-server workspaces, which funnels through newSQLServerUOWProvider. Embedded never reaches it (bd serve refuses embedded outright), and an ordinary CLI command in a server or shared-server workspace opens a DoltStore that takes no such ping. An embedded reader of the old entry would have triaged a slowdown that cannot affect them. CHANGELOG only; the dated [1.3.0] section is unchanged (verified byte-identical, sha256 798a2ee0…, 9169 lines). --------- Co-authored-by: Vishnu J Co-authored-by: Karel Bourgois Co-authored-by: Adam Spiers Co-authored-by: Adam Spiers Co-authored-by: swedeinasia-flow Co-authored-by: Jim Wordelman --- CHANGELOG.md | 182 +++++++++++++++ .../audit_dependencies_readiness.go | 33 +-- cmd/bd/close.go | 2 +- cmd/bd/create.go | 2 +- cmd/bd/create_dispatch_test.go | 43 ++++ cmd/bd/create_input.go | 2 +- cmd/bd/create_input_test.go | 30 +++ cmd/bd/doctor/gitignore.go | 2 + cmd/bd/doctor/gitignore_test.go | 33 +++ cmd/bd/gate.go | 52 +++-- cmd/bd/gate_proxied_server.go | 17 +- cmd/bd/gate_routing_test.go | 77 +++++++ cmd/bd/gate_test.go | 41 +++- cmd/bd/list.go | 14 +- cmd/bd/list_footer_counts_test.go | 85 ++++++- cmd/bd/list_helpers_test.go | 4 +- cmd/bd/list_input.go | 23 ++ cmd/bd/list_output.go | 20 +- cmd/bd/list_output_test.go | 102 +++++++++ cmd/bd/list_proxied_server.go | 6 +- cmd/bd/list_show_filter_modes.go | 12 +- cmd/bd/list_tree.go | 58 +++-- cmd/bd/list_wisp_type_reachability_test.go | 130 +++++++++++ cmd/bd/markdown_test.go | 32 +++ cmd/bd/setup/claude.go | 39 +++- cmd/bd/setup/claude_test.go | 204 +++++++++++++++++ default.nix | 7 +- go.mod | 8 +- go.sum | 16 +- internal/storage/dolt/queries_test.go | 62 +++++ internal/storage/issueops/helpers.go | 2 +- internal/storage/issueops/helpers_test.go | 53 ++++- internal/storage/issueops/ready_work_test.go | 26 +++ internal/storage/sqlbuild/ready.go | 10 +- internal/storage/sqlbuild/sqlbuild_test.go | 16 +- internal/storage/uow/dolt_sql_provider.go | 94 +++++++- .../storage/uow/dolt_sql_provider_test.go | 216 ++++++++++++++++++ internal/workapi/list.go | 14 +- internal/workapi/list_pinned_test.go | 6 +- internal/workapi/list_ready_status_test.go | 123 ++++++++++ internal/workapi/ready.go | 4 +- internal/workapi/sort.go | 13 +- .../workapi/testdata/list_filter_golden.json | 4 +- issueops/reader.go | 7 +- issueops/reader_ready_scope.go | 17 +- 45 files changed, 1807 insertions(+), 136 deletions(-) create mode 100644 cmd/bd/gate_routing_test.go create mode 100644 cmd/bd/list_wisp_type_reachability_test.go create mode 100644 internal/storage/uow/dolt_sql_provider_test.go create mode 100644 internal/workapi/list_ready_status_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a4489a31..51e2e2ddd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -57,6 +57,91 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 setting one in a file whose top level is not a mapping. A script that tolerated the old exit 0 now has to handle the failure. +- **`bd ready` and `bd list --ready` now return issues whose custom status is in + the `active` category** ([#5918](https://github.com/gastownhall/beads/pull/5918), + [#5831](https://github.com/gastownhall/beads/issues/5831)). The ready query + treated its `open` pin as an exact match, so a workspace that defined, say, + `triaged:active` never saw those issues in ready work — even though + `bd statuses`, the `ready_issues` view, and claim eligibility all already + counted that category as ready. **The ready set grows on upgrade** for any + workspace using custom active statuses: counts and dashboards will read higher, + and `bd ready` can hand out issues it used to withhold. Custom + `wip`/`done`/`frozen`/unspecified statuses stay out, and `in_progress` stays + out, both unchanged. + +- **`bd list --wisp-type ` with no plane admitted is now a refusal instead of + an empty listing** ([#6098](https://github.com/gastownhall/beads/pull/6098)). + `--wisp-type` narrows an admitted plane rather than admitting one, so alone it + could not match any row and exited 0 with an empty result for every input. It + now fails with a hint naming `--include-ephemeral`. A script that ran it and + read the empty output as "no such wisps" now sees an error — which is the + point: the old success answered a question it had not asked. The refusal is + CLI-only; the equivalent API request stays lawful and still composes to an + empty page. + +- **The `--ready` footer names the status pin that actually applied** + ([#5924](https://github.com/gastownhall/beads/pull/5924)). With no `--status` + (or `--status all`) the line is unchanged: `open only — --ready excludes + in_progress`. With an explicit `--status` it now reads ` only` + instead of reusing the default-open sentence, which is no longer true once the + selector is honored (see **Fixed**). Anything scraping that parenthetical for + a `--status`-plus-`--ready` run sees new text — though that combination also + returned the wrong rows before this release. Note that the default sentence now + understates the set, since custom active statuses join it (above); the footer + and `--ready` help text are corrected upstream on main + ([#6187](https://github.com/gastownhall/beads/issues/6187), + [#6188](https://github.com/gastownhall/beads/issues/6188)). + +- **`bd doctor` reports a new missing gitignore pattern on every workspace + created before this release, until `bd doctor --fix` is run** + ([#5978](https://github.com/gastownhall/beads/pull/5978)). + `.beads/dolt-server-config.yaml` joined the required `.beads/.gitignore` + patterns (see **Fixed**), so a workspace that was clean a moment ago now + warns. `bd doctor --fix` appends the rule and silences it; workspaces + initialised after this release get it from the template and never warn. + +- **Opening a store against an unreachable Dolt server now takes up to ~40s to + fail instead of failing at once — on proxied-server workspaces, and under + `bd serve`** ([#6003](https://github.com/gastownhall/beads/pull/6003)). The + bootstrap ping is retried for up to 30 seconds, each attempt capped at 10s, so + a server still coming up is waited out rather than turned into a hard failure + (see **Fixed**). The cost is that a genuinely dead or misconfigured endpoint no + longer errors immediately: an interactive command against a stopped server, and + any script or CI step that counted on a fast failure, now waits out the retry + budget first. Durable rejections — bad credentials, unknown database, a + hostname that does not resolve — are still reported immediately without + retrying. + + Only the paths that build a unit-of-work provider take that ping, so this is a + minority of topologies: every command in a proxied-server workspace, local or + external, and `bd serve` in a server, external-server or shared-server + workspace, which builds its own provider through the same funnel. **Embedded — + the default — is unaffected entirely**; `bd serve` refuses it outright, and no + embedded command can slow down here. An ordinary CLI command in a server or + shared-server workspace is also unaffected: it opens a Dolt store, which does + not take this ping. Only that mode's `bd serve` startup does. + + The 30s figure is the retry budget, not the wall-clock bound. It stops + *scheduling* attempts at 30s but cannot cancel one already in flight, and each + attempt carries its own 10s cap, so ~40s is the honest worst case for a single + open. A provider open pings twice — once for the schema-init connection, once + for the database pool — each with a fresh budget, so a backend that satisfies + the first ping and the schema init and is then unreachable for the second can + push one command to ~80s. That ordering is narrow, but it is reachable. + +- **A backup sync that would leave the backup's manifest ahead of its chunk + files now fails at that point instead of corrupting the backup quietly** + ([#5931](https://github.com/gastownhall/beads/pull/5931), + [#4070](https://github.com/gastownhall/beads/issues/4070)). The Dolt bump + brings upstream's defense, which rejects a manifest update referencing a + missing table file. Previously that update was accepted and only some later + sync failed, confusingly, with `table file not found` — and stayed broken. A + sync that used to appear to succeed can now report a failure; that failure is + the corruption being caught at the moment it happens. This is a mitigation, + not a root-cause fix: backup file-and-manifest publish is still non-atomic, + and **backups already in the manifest-ahead state are not repaired** by + upgrading — they surface as a failing sync and have to be re-seeded. + ### Fixed - **`bd dolt start` no longer puts a second sql-server over a proxied @@ -96,6 +181,103 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 Two consequences of the nested write — the flat-key migration and two new failure exits — are under **Changed** above. +- **`bd list --status --ready` honors `--status`** + ([#5924](https://github.com/gastownhall/beads/pull/5924), + [#5832](https://github.com/gastownhall/beads/issues/5832)). Combining the two + silently dropped `--status` and returned the unfiltered ready set, because the + ready path pinned `open` in two places regardless of what was asked for, so a + custom status or `in_progress` never reached the ready query. An explicit + `--status` is now the ready-work status selector, matching how the other list + filters already compose with `--ready`. `--ready` with no `--status` still + defaults to open, and `--status all` remains the no-filter spelling. The footer + wording moves with it — see **Changed**. + +- **`bd list --wisp-type` can reach rows at all** + ([#6098](https://github.com/gastownhall/beads/pull/6098)). `bd list` now + registers `--include-ephemeral`, the plane knob `--wisp-type` needs, off by + default so the default listing is byte-identical. `bd list + --include-ephemeral --wisp-type heartbeat` is the working spelling. Before + this, the only way to admit the wisp plane from `bd list` was `--type + `, which also lifts the infra-type exclusions and changes template + treatment — three decisions for a caller who wanted one classification. Bare + `--wisp-type` is now refused rather than empty; see **Changed**. + +- **`bd doctor` knows about the generated `.beads/dolt-server-config.yaml`** + ([#5978](https://github.com/gastownhall/beads/pull/5978)). + `doltserver.Start()` writes that file when the resolved dolt binary supports + `auto_gc_behavior.archive_level`, but the hyphenated name falls outside the + `dolt-server.` prefix its five sibling server-state entries share, so neither + the gitignore template nor the required-pattern list covered it. It holds an + absolute `cfg_dir` and a per-machine port, so committing it breaks every other + clone — the same reason the template already gives for `redirect`. It shows up + as untracked in every consumer repo whose dolt supports `archive_level`. It is + now in both lists: the template covers new workspaces, and the required + patterns are what `bd doctor --fix` appends to an existing + `.beads/.gitignore` — see **Changed** for the warning that implies. + +- **The missing-prefix error stops recommending a config key that does nothing** + ([#5936](https://github.com/gastownhall/beads/pull/5936), + [#5916](https://github.com/gastownhall/beads/issues/5916)). When the database's + `issue_prefix` config row was absent, the error advised using the + `issue-prefix` key in `config.yaml` — but that gate only reads the database's + config table, so following the advice produced a tracked config field that + changed nothing and misled the next reader. The message now points at + `bd init --prefix` or `bd bootstrap`, which is what writes the row, and says + plainly which store the check reads. Error text only; resolution behaviour is + unchanged. + +- **`bd create --file` says that it creates one issue per `##` heading** + ([#5921](https://github.com/gastownhall/beads/pull/5921)). The usage string was + "Create multiple issues from markdown file", which named neither the per-H2 + semantics nor `--body-file`. Someone wanting "one issue, description loaded + from a file" reached for the shortest flag and got a silent bulk import — a + briefing file with `## Problem` / `## Context` / `## Acceptance Criteria` + became three issues. The usage now names the behaviour, and the existing + rejection of a positional title combined with `--file` carries a `--body-file` + hint. `--file` alone on a section-headed file is unchanged: bd cannot tell that + apart from a genuine batch file by content. + +- **The list truncation hint no longer leaves a padded blank line under the + shell prompt** ([#5927](https://github.com/gastownhall/beads/pull/5927), + [#5685](https://github.com/gastownhall/beads/issues/5685)). The hint passed its + surrounding newlines through the warning renderer, which treats them as a + block, pads the blank lines out to the widest line's width, and drops the + trailing newline — so the next prompt printed on top of a run of spaces. The + newlines are kept outside the renderer now, leaving one coloured line with a + real trailing newline. + +- **`bd setup claude` and `bd init` stop rewriting `.claude/settings.json` on + every run** ([#5944](https://github.com/gastownhall/beads/pull/5944), + [#5693](https://github.com/gastownhall/beads/issues/5693)). The settings map + was marshaled without a trailing newline and written unconditionally, so each + run stripped the newline Claude Code's own writes leave behind and rewrote the + file even when no hook had changed. Because `bd init` git-adds that file, it + surfaced as a staged diff with no semantic content. The write now keeps the + trailing newline and is skipped when the bytes match what is already on disk, + at all four call sites — the settings write and the `settings.local.json` + migration on install, and both again on remove. + +- **`bd gate check` resolves bead gates whose target lives in a prefix-routed + rig** ([#5859](https://github.com/gastownhall/beads/pull/5859)). After a local + miss, the evaluator follows the target bead ID through `routes.jsonl` and reads + the owning store without writing to it; previously such a gate stayed pending + until someone resolved it manually. This covers explicit gate checks in + embedded, server, and proxied-server command paths. The legacy + `:` await value remains accepted for compatibility, a malformed + cross-rig value is rejected, and when prefix routing is unavailable the original + local not-found error is preserved. + +- **A transient connection drop while opening a Dolt store is retried instead of + failing the command** + ([#6003](https://github.com/gastownhall/beads/pull/6003)). The bootstrap ping + in the unit-of-work provider was unretried, so a server still finishing startup + — or one that reset the connection mid-handshake — became a hard failure. + Connection-level shapes (`driver.ErrBadConn`, `mysql.ErrInvalidConn`, EOF, any + `*net.OpError`) are now retried for up to 30s, each attempt capped at 10s so a + server that accepts TCP and then stalls cannot block for the caller's whole + context. Durable rejections and an unresolvable hostname still fail without + retrying. See **Changed** for the latency this adds against a dead endpoint. + ## [1.3.0] - 2026-09-15 The first tested release off `main` since the 1.1 line. [1.2.2] was a recovery diff --git a/backend/conformance/audit_dependencies_readiness.go b/backend/conformance/audit_dependencies_readiness.go index f4e322271..aefded74e 100644 --- a/backend/conformance/audit_dependencies_readiness.go +++ b/backend/conformance/audit_dependencies_readiness.go @@ -385,27 +385,28 @@ func testAuditReadyTypeAndPinnedExclusions(t *testing.T, f Factory) { // Statuses, the legacy open/in_progress default when both are empty, and a // custom (non-built-in) status flowing through the same IN clause. // -// THIS IS THE ONLY OBSERVER OF THE OR-SET ARM, AND IT COVERS EVERY SEAM THE ARM -// IS REACHABLE FROM. No role can ask for it — publicops.ReadyRequest carries no -// status field of any kind, which RunReadyCounterCountsOnlyTheOpenRowsItsListingLists -// says from the other side — and no in-tree caller sets it either. Every builder -// that renders a ready query sends the SINGULAR status: workapi.BuildReadyFilter -// and BuildReadyCountFilter send StatusOpen, and ReadyFilterFromIssueFilter sends -// StatusOpen while dropping IssueFilter.Statuses, which BuildListFilter has -// already resolved to open under --ready (issueops/reader_ready_scope.go states -// that override and why the projection has nothing to drop). What is left is the -// storage seam this file runs against, which backend.DoltStorage publishes to -// out-of-tree backends — so Statuses is a public filter field whose only reachable -// consumer class is exactly the one RunAll is the proof obligation for. +// THE OR-SET ARM IS REACHABLE FROM `bd list --status a,b --ready`. +// ReadyFilterFromIssueFilter copies IssueFilter.Statuses onto the ready-work +// filter instead of dropping it, and BuildListFilter no longer always +// resolves --ready to open: an explicit selector is the intersection +// (GH#5832; issueops/reader_ready_scope.go states the honor path). +// publicops.ReadyRequest still carries no status field of any kind, which +// RunReadyCounterCountsOnlyTheOpenRowsItsListingLists says from the other +// side, so `bd ready` — workapi.BuildReadyFilter and BuildReadyCountFilter — +// still send the SINGULAR StatusOpen. The list --ready path is the in-tree +// caller that sets Statuses. This file remains the storage-seam observer +// for the OR-set arm against backend.DoltStorage, which publishes the field +// to out-of-tree backends; RunAll is still the proof obligation for that +// seam. // // The unit-of-work provider's ready union renders this arm from the same shared // builder (sqlbuild.BuildReadyWorkWhere), applied to the wisps table as well as // issues where the classic stack projects the wisp plane onto a types.IssueFilter // instead. Its copy is deliberately unpinned rather than overlooked: that provider -// is not a storage.DoltStorage, so no external caller reaches it, and no in-tree -// caller sets Statuses at all. Promoting a status set onto ReadyRequest is what -// would make it reachable, and would move this case to reader_contract.go beside -// RunReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions, where all three +// is not a storage.DoltStorage, so no external caller reaches it. A status set on +// ReadyRequest would additionally make `bd ready` vote on this case, and would +// move it to reader_contract.go beside +// RunReaderReadySetOwnsItsStatusPinnedAndTemplateDecisions, where the ready-role // wirings would vote on it. func testAuditReadyMultiStatusFilter(t *testing.T, f Factory) { s := f(t) diff --git a/cmd/bd/close.go b/cmd/bd/close.go index d87223c3d..dee1ddc3e 100644 --- a/cmd/bd/close.go +++ b/cmd/bd/close.go @@ -568,7 +568,7 @@ func checkGateSatisfaction(issue *types.Issue) error { case issue.AwaitType == "timer": resolved, escalated, reason, err = checkTimer(issue, time.Now()) case issue.AwaitType == "bead": - resolved, reason = checkBeadGate(rootCtx, store, issue.AwaitID) + resolved, reason = checkBeadGate(rootCtx, routedBeadGateGetter{localStore: store}, issue.AwaitID) if resolved { return nil } diff --git a/cmd/bd/create.go b/cmd/bd/create.go index 3d97cea34..58a421ece 100644 --- a/cmd/bd/create.go +++ b/cmd/bd/create.go @@ -915,7 +915,7 @@ func createDepsAcceptedTypeList() string { } func init() { - createCmd.Flags().StringP("file", "f", "", "Create multiple issues from markdown file") + createCmd.Flags().StringP("file", "f", "", "Create one issue per ## heading (### fields attach to that issue); for title plus description-from-file, use --body-file") createCmd.Flags().String("graph", "", "Create a graph of issues with dependencies from JSON plan file") createCmd.Flags().String("title", "", "Issue title (alternative to positional argument)") createCmd.Flags().Bool("silent", false, "Output only the issue ID (for scripting)") diff --git a/cmd/bd/create_dispatch_test.go b/cmd/bd/create_dispatch_test.go index 3f64487a2..0ee433a8a 100644 --- a/cmd/bd/create_dispatch_test.go +++ b/cmd/bd/create_dispatch_test.go @@ -171,3 +171,46 @@ func TestCreateSingleIssueAllowEmptyDescriptionRoundTrip(t *testing.T) { t.Fatalf("expected issue %q to be created via --allow-empty-description opt-in", title) } } + +// TestCreateFileDispatchRejectsPositionalTitle is a regression test for +// gastownhall/beads#4643: --file is a batch-import route, so a positional +// title has nowhere to go once createIssuesFromMarkdown takes over. This +// asserts gatherCreateInput's existing guard still rejects the combination +// before any issue is written, rather than silently discarding the title. +func TestCreateFileDispatchRejectsPositionalTitle(t *testing.T) { + saveAndRestoreGlobals(t) + ensureCleanGlobalState(t) + + readonlyMode = false + t.Cleanup(func() { readonlyMode = false }) + + fileFlag := createCmd.Flags().Lookup("file") + t.Cleanup(func() { + _ = fileFlag.Value.Set("") + fileFlag.Changed = false + }) + + tmpDir := t.TempDir() + mdPath := filepath.Join(tmpDir, "single.md") + if err := os.WriteFile(mdPath, []byte("## My Bead\n\nBody text.\n"), 0644); err != nil { + t.Fatalf("write markdown fixture: %v", err) + } + + if err := createCmd.Flags().Set("file", mdPath); err != nil { + t.Fatalf("set --file: %v", err) + } + + var err error + stderr := captureStderr(t, func() { + err = createCmd.RunE(createCmd, []string{"my new bead"}) + }) + if err == nil { + t.Fatal("expected --file dispatch with a positional title to be rejected") + } + if !strings.Contains(stderr, "cannot specify both title and --file flag") { + t.Fatalf("expected title/--file conflict message on stderr, got: %v (err: %v)", stderr, err) + } + if !strings.Contains(stderr, "--body-file") { + t.Fatalf("expected title/--file conflict to hint at --body-file, got: %v (err: %v)", stderr, err) + } +} diff --git a/cmd/bd/create_input.go b/cmd/bd/create_input.go index 6513e50fa..aecf5c685 100644 --- a/cmd/bd/create_input.go +++ b/cmd/bd/create_input.go @@ -97,7 +97,7 @@ func gatherCreateInput(cmd *cobra.Command, args []string) (createInput, error) { } if in.markdownFile != "" { if len(args) > 0 { - return in, HandleError("cannot specify both title and --file flag") + return in, HandleError("cannot specify both title and --file flag; for a single issue's description from a file, use --body-file") } if in.dryRun { return in, HandleError("--dry-run is not supported with --file flag") diff --git a/cmd/bd/create_input_test.go b/cmd/bd/create_input_test.go index 0574e8300..9fb796be1 100644 --- a/cmd/bd/create_input_test.go +++ b/cmd/bd/create_input_test.go @@ -40,6 +40,36 @@ func TestCreateCommandRegistersEmptyDescriptionOptIn(t *testing.T) { } } +func TestCreateFileFlagHelpDoesNotClaimSingleIssueGH4643(t *testing.T) { + f := createCmd.Flags().Lookup("file") + if f == nil { + t.Fatal("expected create to register --file") + } + if !strings.Contains(f.Usage, "one issue per") { + t.Fatalf("expected --file help text to name its per-heading batch behavior, got: %q", f.Usage) + } + if !strings.Contains(f.Usage, "--body-file") { + t.Fatalf("expected --file help text to point single-issue callers at --body-file, got: %q", f.Usage) + } +} + +func TestGatherCreateInputRejectsTitleWithFile(t *testing.T) { + cmd := newCreateFlagsCommand(t, "--file", "issues.md") + var err error + stderr := captureStderr(t, func() { + _, err = gatherCreateInput(cmd, []string{"Some title"}) + }) + if err == nil { + t.Fatal("expected gatherCreateInput to reject a positional title with --file") + } + if !strings.Contains(stderr, "cannot specify both title and --file flag") { + t.Fatalf("expected title/--file conflict on stderr, got stderr=%q err=%v", stderr, err) + } + if !strings.Contains(stderr, "--body-file") { + t.Fatalf("expected title/--file conflict to hint at --body-file, got stderr=%q err=%v", stderr, err) + } +} + func TestGatherCreateInputRejectsEmptyBodyFile(t *testing.T) { filePath := filepath.Join(t.TempDir(), "empty.md") if err := os.WriteFile(filePath, nil, 0644); err != nil { diff --git a/cmd/bd/doctor/gitignore.go b/cmd/bd/doctor/gitignore.go index 9bf418ab1..1a3c0b54f 100644 --- a/cmd/bd/doctor/gitignore.go +++ b/cmd/bd/doctor/gitignore.go @@ -65,6 +65,7 @@ dolt-server.log dolt-server.lock dolt-server.port dolt-server.activity +dolt-server-config.yaml # Debug-mode pprof artifacts (written when dolt.debug: true in config.yaml) dolt-pprof/ @@ -129,6 +130,7 @@ var requiredPatterns = []string{ "dolt-server.lock", "dolt-server.port", "dolt-server.activity", + "dolt-server-config.yaml", "daemon.*", "*.lock", "*.corrupt.backup/", diff --git a/cmd/bd/doctor/gitignore_test.go b/cmd/bd/doctor/gitignore_test.go index dc4ccd70c..d776a4c8d 100644 --- a/cmd/bd/doctor/gitignore_test.go +++ b/cmd/bd/doctor/gitignore_test.go @@ -1522,6 +1522,39 @@ func TestRequiredPatterns_ContainsSyncStatePatterns(t *testing.T) { } } +// TestGitignore_ContainsDoltServerConfig verifies that the generated Dolt +// server config is ignored like the other dolt-server.* runtime state files +// it sits beside. doltserver.Start() writes .beads/dolt-server-config.yaml +// when the resolved dolt binary supports auto_gc_behavior.archive_level, and +// it holds an absolute cfg_dir plus a per-machine port, so committing it +// would break every other clone. +// +// The hyphenated name falls outside the "dolt-server." prefix shared by the +// other five entries, so it was missed by both lists. It must be in +// requiredPatterns too, otherwise bd doctor --fix cannot heal an existing +// .beads/.gitignore. +func TestGitignore_ContainsDoltServerConfig(t *testing.T) { + // Keep this in sync with doltserver.doltServerConfigFileName. It is not + // imported here because cmd/bd/doctor must not depend on the server + // package for a string constant. + const pattern = "dolt-server-config.yaml" + + if !containsGitignorePattern(GitignoreTemplate, pattern) { + t.Errorf("GitignoreTemplate should contain %q", pattern) + } + + found := false + for _, p := range requiredPatterns { + if p == pattern { + found = true + break + } + } + if !found { + t.Errorf("requiredPatterns should include %q", pattern) + } +} + // TestCheckLastTouchedNotTracked_NoFile verifies that check passes when no last-touched file exists func TestCheckLastTouchedNotTracked_NoFile(t *testing.T) { tmpDir := t.TempDir() diff --git a/cmd/bd/gate.go b/cmd/bd/gate.go index 20950544e..1246d544e 100644 --- a/cmd/bd/gate.go +++ b/cmd/bd/gate.go @@ -12,6 +12,7 @@ import ( "github.com/spf13/cobra" "github.com/steveyegge/beads/internal/metrics" + "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/ui" ) @@ -33,9 +34,9 @@ Gate types: gh:pr - Waits for PR merge (Phase 3) bead - Waits for another bead to close (Phase 4) -For bead gates, await_id is a bead ID in this rig's database (e.g., "bd-abc123"). -The historical cross-rig form : can no longer be evaluated -(multi-rig routing removed) and stays pending until resolved manually. +For bead gates, await_id may be a bead ID in this rig's database (e.g., +"bd-abc123") or the historical cross-rig form :. Cross-rig +targets resolve through the bead ID's prefix route in routes.jsonl. Examples: bd gate list # Show all open gates @@ -669,7 +670,7 @@ Examples: } } - results := evaluateGates(ctx, filteredGates, time.Now(), store, persistAwaitID) + results := evaluateGates(ctx, filteredGates, time.Now(), routedBeadGateGetter{localStore: store}, persistAwaitID) resolvedCount, escalatedCount, errorCount := applyGateCheckResults( results, dryRun, escalateFlag, @@ -1182,27 +1183,48 @@ type issueGetter interface { GetIssue(ctx context.Context, id string) (*types.Issue, error) } +// routedBeadGateGetter gives direct-mode gate checks the same local -> prefix +// route -> contributor fallback used by other read commands. Routed stores are +// opened read-only and closed before the result is returned. +type routedBeadGateGetter struct { + localStore storage.DoltStorage +} + +func (g routedBeadGateGetter) GetIssue(ctx context.Context, id string) (*types.Issue, error) { + if g.localStore == nil { + return nil, fmt.Errorf("no local store available") + } + result, err := getIssueWithRouting(ctx, g.localStore, id) + if err != nil { + return nil, err + } + defer result.Close() + return result.Issue, nil +} + // checkBeadGate checks if a bead gate is satisfied. // Returns (satisfied, reason). // -// A plain await_id (no colon) names a bead in THIS rig's database: the gate -// resolves once that bead closes — the common case, an agent idle-waiting on -// local work (wy-hgms2; the old unconditional cross-rig refusal left every -// local bead gate permanently pending and its waiters asleep). -// -// The historical cross-rig form : cannot be evaluated since -// multi-rig routing was removed; it stays pending with a descriptive message. +// A plain await_id names a bead in this rig. The historical +// : form uses the bead ID as the routed lookup key; the rig +// component is retained for compatibility while routes.jsonl remains keyed by +// bead prefix. The supplied getter owns local-versus-routed lookup policy. func checkBeadGate(ctx context.Context, st issueGetter, awaitID string) (bool, string) { if awaitID == "" { return false, "bead gate has no await_id" } + targetID := awaitID if strings.Contains(awaitID, ":") { - return false, fmt.Sprintf("cross-rig bead gate %q cannot be checked (multi-rig routing removed)", awaitID) + parts := strings.SplitN(awaitID, ":", 2) + if parts[0] == "" || parts[1] == "" { + return false, fmt.Sprintf("invalid cross-rig bead gate %q: expected :", awaitID) + } + targetID = parts[1] } if st == nil { return false, fmt.Sprintf("bead gate %q: no local store available", awaitID) } - issue, err := st.GetIssue(ctx, awaitID) + issue, err := st.GetIssue(ctx, targetID) if err != nil { return false, fmt.Sprintf("bead gate %q: %v", awaitID, err) } @@ -1210,9 +1232,9 @@ func checkBeadGate(ctx context.Context, st issueGetter, awaitID string) (bool, s return false, fmt.Sprintf("bead gate %q: bead not found", awaitID) } if issue.Status == types.StatusClosed { - return true, fmt.Sprintf("bead %s closed", awaitID) + return true, fmt.Sprintf("bead %s closed", targetID) } - return false, fmt.Sprintf("bead %s is %s", awaitID, issue.Status) + return false, fmt.Sprintf("bead %s is %s", targetID, issue.Status) } // closeGate closes a gate issue with the given reason diff --git a/cmd/bd/gate_proxied_server.go b/cmd/bd/gate_proxied_server.go index d3bf918a6..d8bc1b34f 100644 --- a/cmd/bd/gate_proxied_server.go +++ b/cmd/bd/gate_proxied_server.go @@ -39,8 +39,21 @@ func (proxiedFreshReadGetter) GetIssue(ctx context.Context, id string) (*types.I if err != nil { return nil, err } - defer uw.Close(ctx) - return uw.IssueUseCase().GetIssue(ctx, id) + issue, localErr := uw.IssueUseCase().GetIssue(ctx, id) + uw.Close(ctx) + if localErr == nil { + return issue, nil + } + if !gateProxiedNotFound(localErr) { + return nil, localErr + } + + result, routeErr := resolveViaPrefixRouting(ctx, id) + if routeErr != nil { + return nil, localErr + } + defer result.Close() + return result.Issue, nil } func runGateCheckProxiedServer(cmd *cobra.Command, ctx context.Context) error { diff --git a/cmd/bd/gate_routing_test.go b/cmd/bd/gate_routing_test.go new file mode 100644 index 000000000..2b93fdf2e --- /dev/null +++ b/cmd/bd/gate_routing_test.go @@ -0,0 +1,77 @@ +//go:build cgo + +package main + +import ( + "context" + "os" + "path/filepath" + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +// TestCheckBeadGateCrossRigPrefixRoute proves the full evaluator seam: the +// historical rig:id value is reduced to the target bead ID, the current +// routes.jsonl prefix router opens the foreign store read-only, and target +// lifecycle state determines whether the gate resolves. +// +// NOTE: This test uses os.Chdir and cannot run in parallel with other tests. +func TestCheckBeadGateCrossRigPrefixRoute(t *testing.T) { + ctx := context.Background() + townRoot := t.TempDir() + townBeadsDir := filepath.Join(townRoot, ".beads") + rigBeadsDir := filepath.Join(townRoot, "rig", ".beads") + if err := os.MkdirAll(townBeadsDir, 0o755); err != nil { + t.Fatalf("create town beads dir: %v", err) + } + if err := os.MkdirAll(rigBeadsDir, 0o755); err != nil { + t.Fatalf("create rig beads dir: %v", err) + } + + townDBPath := filepath.Join(townBeadsDir, "dolt") + townStore := newTestStoreIsolatedDB(t, townDBPath, "hq") + rigStore := newTestStoreIsolatedDB(t, filepath.Join(rigBeadsDir, "dolt"), "gt") + for _, issue := range []*types.Issue{ + {ID: "gt-closed", Title: "Closed routed target", Status: types.StatusClosed, Priority: 2, IssueType: types.TypeTask}, + {ID: "gt-open", Title: "Open routed target", Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask}, + } { + if err := rigStore.CreateIssue(ctx, issue, "test"); err != nil { + t.Fatalf("create routed issue %s: %v", issue.ID, err) + } + } + if err := rigStore.Close(); err != nil { + t.Fatalf("close rig store: %v", err) + } + + routesPath := filepath.Join(townBeadsDir, "routes.jsonl") + if err := os.WriteFile(routesPath, []byte(`{"prefix":"gt-","path":"rig"}`), 0o644); err != nil { + t.Fatalf("write routes.jsonl: %v", err) + } + + oldDBPath := dbPath + dbPath = townDBPath + t.Cleanup(func() { dbPath = oldDBPath }) + oldWD, err := os.Getwd() + if err != nil { + t.Fatalf("get working directory: %v", err) + } + if err := os.Chdir(townRoot); err != nil { + t.Fatalf("change to town root: %v", err) + } + t.Cleanup(func() { _ = os.Chdir(oldWD) }) + + getter := routedBeadGateGetter{localStore: townStore} + resolved, reason := checkBeadGate(ctx, getter, "rig:gt-closed") + if !resolved { + t.Fatalf("closed cross-rig target did not resolve gate: %s", reason) + } + + resolved, reason = checkBeadGate(ctx, getter, "rig:gt-open") + if resolved { + t.Fatalf("open cross-rig target unexpectedly resolved gate: %s", reason) + } + if !gateTestContainsIgnoreCase(reason, "open") { + t.Fatalf("pending reason %q does not report target status", reason) + } +} diff --git a/cmd/bd/gate_test.go b/cmd/bd/gate_test.go index 73815c9d4..85baf386e 100644 --- a/cmd/bd/gate_test.go +++ b/cmd/bd/gate_test.go @@ -3,10 +3,12 @@ package main import ( "bytes" "context" + "database/sql" "encoding/json" "errors" "io" "os" + "path/filepath" "slices" "sync" "testing" @@ -154,28 +156,53 @@ func TestShouldCheckGate(t *testing.T) { type fakeBeadGateGetter struct { issues map[string]*types.Issue err error + gotID string } func (f *fakeBeadGateGetter) GetIssue(_ context.Context, id string) (*types.Issue, error) { + f.gotID = id if f.err != nil { return nil, f.err } return f.issues[id], nil } -func TestCheckBeadGate_CrossRigStaysPending(t *testing.T) { +func TestCheckBeadGate_CrossRigUsesBeadIDForRoutedLookup(t *testing.T) { ctx := context.Background() + st := &fakeBeadGateGetter{issues: map[string]*types.Issue{ + "gt-abc": {ID: "gt-abc", Status: types.StatusClosed}, + }} - // The cross-rig : form cannot be evaluated since multi-rig - // routing was removed; it must stay pending with the explanatory message, - // never consult the store, and never resolve. + satisfied, reason := checkBeadGate(ctx, st, "gastown:gt-abc") + if !satisfied { + t.Fatalf("expected closed routed bead to satisfy gate, got reason %q", reason) + } + if st.gotID != "gt-abc" { + t.Fatalf("routed lookup ID = %q, want %q", st.gotID, "gt-abc") + } +} + +func TestProxiedFreshReadGetterPreservesLocalNotFoundWhenRoutingUnavailable(t *testing.T) { + withStubbedProxiedLookup(t, nil) + + oldDBPath := dbPath + dbPath = filepath.Join(t.TempDir(), ".beads", "dolt") + t.Cleanup(func() { dbPath = oldDBPath }) + + _, err := (proxiedFreshReadGetter{}).GetIssue(context.Background(), "bd-missing") + if !errors.Is(err, sql.ErrNoRows) { + t.Fatalf("GetIssue error = %v, want original local not-found", err) + } +} + +func TestCheckBeadGate_InvalidCrossRigFormat(t *testing.T) { + ctx := context.Background() tests := []struct { name string awaitID string }{ {name: "missing rig", awaitID: ":gt-abc"}, {name: "missing bead", awaitID: "my-project:"}, - {name: "well-formed cross-rig", awaitID: "nonexistent:some-id"}, } for _, tt := range tests { @@ -184,8 +211,8 @@ func TestCheckBeadGate_CrossRigStaysPending(t *testing.T) { if satisfied { t.Errorf("expected not satisfied for %q", tt.awaitID) } - if !gateTestContainsIgnoreCase(reason, "multi-rig routing removed") { - t.Errorf("reason %q does not contain %q", reason, "multi-rig routing removed") + if !gateTestContainsIgnoreCase(reason, "expected :") { + t.Errorf("reason %q does not describe the expected format", reason) } }) } diff --git a/cmd/bd/list.go b/cmd/bd/list.go index c5e70ef59..0254c9274 100644 --- a/cmd/bd/list.go +++ b/cmd/bd/list.go @@ -240,7 +240,7 @@ func runListCore(cmd *cobra.Command, _ []string) error { } if in.watchMode { - if err := watchIssues(ctx, activeStore, filter, in.ReadyFlag, in.ParentID, in.SortBy, in.Reverse, in.effectiveLimit); err != nil { + if err := watchIssues(ctx, activeStore, filter, in.ReadyFlag, in.ParentID, in.SortBy, in.Reverse, in.effectiveLimit, in.Status); err != nil { if capErr := handleMaxRowsError(err); capErr != nil { return capErr } @@ -315,7 +315,7 @@ func runListCore(cmd *cobra.Command, _ []string) error { return HandleError("loading dependencies for --deps: %v", depErr) } // Hierarchical --parent walks use an unlimited per-level query, so the tree is never page-truncated. - displayPrettyListWithDepsMode(treeIssues, false, allDeps, in.depsMode, false, in.ReadyFlag) + displayPrettyListWithDepsMode(treeIssues, false, allDeps, in.depsMode, false, in.ReadyFlag, in.Status) printSkipLabelsFooter(in.SkipLabels) return nil } @@ -324,7 +324,7 @@ func runListCore(cmd *cobra.Command, _ []string) error { if depErr != nil && in.depsMode != "" { return HandleError("loading dependencies for --deps: %v", depErr) } - displayPrettyListWithDepsMode(issues, false, allDeps, in.depsMode, truncated, in.ReadyFlag) + displayPrettyListWithDepsMode(issues, false, allDeps, in.depsMode, truncated, in.ReadyFlag, in.Status) printTruncationHint(truncated, in.effectiveLimit) printSkipLabelsFooter(in.SkipLabels) return nil @@ -470,6 +470,14 @@ func init() { // Infra type filtering: exclude agent/role/message by default listCmd.Flags().Bool("include-infra", false, "Include infrastructure beads (agent/role/message) in output") + // Ephemeral plane: the wisps table is suppressed by default. This is the + // PLANE knob on its own — ListRequest.IncludeEphemeral — as distinct from + // --include-infra, which admits the plane AND lifts the infra-type + // exclusions above. Without it the plane is reachable from the CLI only + // through --include-infra's wider bundle, which leaves --wisp-type below + // with no narrow way to match anything. + listCmd.Flags().Bool("include-ephemeral", false, "Include ephemeral wisp-plane rows in output (normally hidden)") + // Explicit type exclusion listCmd.Flags().StringSlice("exclude-type", nil, "Exclude issue types from results (comma-separated or repeatable, e.g., --exclude-type=convoy,epic)") diff --git a/cmd/bd/list_footer_counts_test.go b/cmd/bd/list_footer_counts_test.go index af5707ff9..0e9fa765a 100644 --- a/cmd/bd/list_footer_counts_test.go +++ b/cmd/bd/list_footer_counts_test.go @@ -58,6 +58,7 @@ func TestListFooterLineCountsAreJustified(t *testing.T) { name string total, open, inProgress int truncated, readyFiltered bool + statusSelector string facts footerFacts }{ { @@ -90,11 +91,19 @@ func TestListFooterLineCountsAreJustified(t *testing.T) { total: 0, open: 0, inProgress: 0, facts: footerFacts{"total": 0, "open": 0, "in_progress": 0}, }, + { + // Explicit --status under --ready is a different pin. The footer + // names that selector and must not reuse the default-open sentence. + name: "ready-filtered, explicit in_progress", + total: 3, open: 0, inProgress: 3, readyFiltered: true, + statusSelector: "in_progress", + facts: footerFacts{"total": 3}, + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - line := listFooterLine(tt.total, tt.open, tt.inProgress, tt.truncated, tt.readyFiltered) + line := listFooterLine(tt.total, tt.open, tt.inProgress, tt.truncated, tt.readyFiltered, tt.statusSelector) assertEveryNumberIsJustified(t, line, tt.facts) // The headline count always describes the rows actually rendered. @@ -110,7 +119,7 @@ func TestListFooterLineCountsAreJustified(t *testing.T) { // reader cannot distinguish "none exist" from "none survived the filter". func TestListFooterLineReadyOmitsVacuousInProgressCount(t *testing.T) { // 40 in-progress issues match the same query; --ready removed them all. - line := listFooterLine(6, 6, 0, false, true) + line := listFooterLine(6, 6, 0, false, true, "") if strings.Contains(line, "in progress)") { t.Errorf("--ready summary asserts an in-progress count that the filter forced to zero: %q", line) @@ -126,7 +135,7 @@ func TestListFooterLineReadyOmitsVacuousInProgressCount(t *testing.T) { // Without --ready the breakdown is a genuine finding and must survive: this is // the half of the behaviour the fix must not regress. func TestListFooterLineUnfilteredKeepsBreakdown(t *testing.T) { - line := listFooterLine(9, 6, 3, false, false) + line := listFooterLine(9, 6, 3, false, false, "") for _, want := range []string{"Total: 9 issues", "6 open", "3 in progress"} { if !strings.Contains(line, want) { t.Errorf("unfiltered summary lost %q, got: %q", want, line) @@ -157,7 +166,7 @@ func TestDisplayWatchedIssueListReadyFooterDisclosesFilter(t *testing.T) { // for; the footer is what is under test, and this keeps the file free of // the cgo-tagged stub so it still compiles under CGO_ENABLED=0. out := captureStdout(t, func() error { - displayWatchedIssueList(context.Background(), nil, issues, false, true) + displayWatchedIssueList(context.Background(), nil, issues, false, true, "") return nil }) @@ -177,7 +186,7 @@ func TestDisplayPrettyListWithDepsReadyFooterDisclosesFilter(t *testing.T) { } out := captureStdout(t, func() error { - displayPrettyListWithDeps(issues, false, nil, false, true) + displayPrettyListWithDeps(issues, false, nil, false, true, "") return nil }) if strings.Contains(out, "in progress)") { @@ -190,7 +199,7 @@ func TestDisplayPrettyListWithDepsReadyFooterDisclosesFilter(t *testing.T) { // The other half: without --ready the same wrapper must keep the breakdown, // so the fix cannot be "never print counts". plain := captureStdout(t, func() error { - displayPrettyListWithDeps(issues, false, nil, false, false) + displayPrettyListWithDeps(issues, false, nil, false, false, "") return nil }) if !strings.Contains(plain, "in progress)") { @@ -201,7 +210,7 @@ func TestDisplayPrettyListWithDepsReadyFooterDisclosesFilter(t *testing.T) { // Truncation and readiness are independent scopes and both must be disclosed // when both apply — neither may silently mask the other. func TestListFooterLineTruncationDisclosedAlongsideReady(t *testing.T) { - line := listFooterLine(5, 5, 0, true, true) + line := listFooterLine(5, 5, 0, true, true, "") if !strings.Contains(line, "truncated by --limit") { t.Errorf("truncated page must say so even under --ready, got: %q", line) } @@ -212,3 +221,65 @@ func TestListFooterLineTruncationDisclosedAlongsideReady(t *testing.T) { t.Errorf("a truncated page must never be labelled Total: %q", line) } } + +// GH#5832: an explicit --status under --ready is a different pin. The default +// "open only — --ready excludes in_progress" sentence must not be reused. +func TestListFooterLineReadyExplicitStatusNamesSelector(t *testing.T) { + line := listFooterLine(3, 0, 3, false, true, "in_progress") + if strings.Contains(line, "open only") || strings.Contains(line, "excludes in_progress") { + t.Errorf("explicit --status must not reuse the default-open sentence: %q", line) + } + if !strings.Contains(line, "in_progress only") { + t.Errorf("explicit --status footer must name the selected status, got: %q", line) + } + if strings.Contains(line, "in progress)") { + t.Errorf("--ready summary must still omit the vacuous in-progress count: %q", line) + } +} + +// --pretty/--tree/--deps share displayPrettyListWithDepsMode; --watch uses +// displayWatchedIssueList. Both must thread the explicit selector so the +// default-open sentence cannot leak through a wrapper. +func TestDisplayPrettyListWithDepsReadyExplicitStatusFooter(t *testing.T) { + issues := []*types.Issue{ + {ID: "bd-1", Title: "A", Status: types.StatusInProgress, Priority: 1, IssueType: types.TypeTask}, + } + + out := captureStdout(t, func() error { + displayPrettyListWithDeps(issues, false, nil, false, true, "in_progress") + return nil + }) + if strings.Contains(out, "open only") || strings.Contains(out, "excludes in_progress") { + t.Errorf("--status in_progress --ready --pretty reused the default-open sentence: %q", out) + } + if !strings.Contains(out, "in_progress only") { + t.Errorf("--status in_progress --ready --pretty must name the selected status, got: %q", out) + } + + depsOut := captureStdout(t, func() error { + displayPrettyListWithDepsMode(issues, false, nil, "scheduling", false, true, "in_progress") + return nil + }) + if strings.Contains(depsOut, "open only") || strings.Contains(depsOut, "excludes in_progress") { + t.Errorf("--status in_progress --ready --deps reused the default-open sentence: %q", depsOut) + } + if !strings.Contains(depsOut, "in_progress only") { + t.Errorf("--status in_progress --ready --deps must name the selected status, got: %q", depsOut) + } +} + +func TestDisplayWatchedIssueListReadyExplicitStatusFooter(t *testing.T) { + issues := []*types.Issue{ + {ID: "bd-1", Title: "A", Status: types.StatusInProgress, Priority: 1, IssueType: types.TypeTask}, + } + out := captureStdout(t, func() error { + displayWatchedIssueList(context.Background(), nil, issues, false, true, "in_progress") + return nil + }) + if strings.Contains(out, "open only") || strings.Contains(out, "excludes in_progress") { + t.Errorf("--status in_progress --ready --watch reused the default-open sentence: %q", out) + } + if !strings.Contains(out, "in_progress only") { + t.Errorf("--status in_progress --ready --watch must name the selected status, got: %q", out) + } +} diff --git a/cmd/bd/list_helpers_test.go b/cmd/bd/list_helpers_test.go index 64d0a30ef..e205f0323 100644 --- a/cmd/bd/list_helpers_test.go +++ b/cmd/bd/list_helpers_test.go @@ -317,7 +317,7 @@ func TestListDisplayPrettyList_TruncatedSummary(t *testing.T) { } out := captureStdout(t, func() error { - displayPrettyListWithDepsMode(issues, false, nil, "", true, false) + displayPrettyListWithDepsMode(issues, false, nil, "", true, false, "") return nil }) if !strings.Contains(out, "Showing 2 issues") { @@ -343,7 +343,7 @@ func TestDisplayWatchedIssueList_UsesDependencyHierarchy(t *testing.T) { } out := captureStdout(t, func() error { - displayWatchedIssueList(context.Background(), store, []*types.Issue{child, parent}, false, false) + displayWatchedIssueList(context.Background(), store, []*types.Issue{child, parent}, false, false, "") return nil }) diff --git a/cmd/bd/list_input.go b/cmd/bd/list_input.go index 58c039d78..7d12e7c63 100644 --- a/cmd/bd/list_input.go +++ b/cmd/bd/list_input.go @@ -134,6 +134,7 @@ func gatherListInput(cmd *cobra.Command) (listInput, error) { in.IncludeTemplates, _ = cmd.Flags().GetBool("include-templates") in.IncludeGates, _ = cmd.Flags().GetBool("include-gates") in.IncludeInfra, _ = cmd.Flags().GetBool("include-infra") + in.IncludeEphemeral, _ = cmd.Flags().GetBool("include-ephemeral") in.ExcludeTypes, _ = cmd.Flags().GetStringSlice("exclude-type") in.ParentID, _ = cmd.Flags().GetString("parent") @@ -158,6 +159,28 @@ func gatherListInput(cmd *cobra.Command) (listInput, error) { return in, HandleError("invalid wisp-type %q (must be %s)", s, types.ValidWispTypeNames()) } in.WispType = &wt + + // wisp_type is a PREDICATE, not a plane selector (see + // issueops.ListRequest.WispType): it narrows whatever the rest of the + // request admitted. On a default listing that is the durable rows, + // which carry no classification — so this request cannot match a row + // for ANY input, and would report an empty listing rather than the + // unread plane that actually holds the answer. + // + // The request stays LAWFUL at the API layer, where composing to empty + // is the pinned contract. Refusing it belongs HERE, at the CLI, where + // the only thing a human can have meant is the combination that + // answers with rows. + // + // An explicit --type is left alone: an infra type routes to the plane + // by itself, so the request may be satisfiable and this cannot tell + // without the workspace's infra vocabulary, which this layer does not + // load. + if !in.IncludeEphemeral && !in.IncludeInfra && in.IssueType == "" { + return in, HandleErrorWithHint( + fmt.Sprintf("--wisp-type %s cannot match anything here: it filters the wisp_type column, and this listing admits only durable rows, which never carry one", s), + "add --include-ephemeral to admit the ephemeral plane (or --include-infra, which admits it as part of a wider bundle)") + } } in.DeferredFlag, _ = cmd.Flags().GetBool("deferred") diff --git a/cmd/bd/list_output.go b/cmd/bd/list_output.go index 48f5e01d9..7266f3a0a 100644 --- a/cmd/bd/list_output.go +++ b/cmd/bd/list_output.go @@ -5,6 +5,7 @@ import ( "fmt" "io" "os" + "strings" "text/template" "github.com/steveyegge/beads/internal/types" @@ -18,8 +19,23 @@ func printTruncationHint(truncated bool, effectiveLimit int) { if !truncated || effectiveLimit <= 0 || !ui.IsStderrTerminal() { return } - msg := fmt.Sprintf("\nShowing %d issues; more results matched but were hidden by --limit. Use --limit 0 for all, or --limit N to raise the cap.\n", effectiveLimit) - fmt.Fprint(os.Stderr, ui.RenderWarn(msg)) + fmt.Fprint(os.Stderr, formatTruncationHint(effectiveLimit)) +} + +func truncationHintText(effectiveLimit int) string { + return fmt.Sprintf("Showing %d issues; more results matched but were hidden by --limit. Use --limit 0 for all, or --limit N to raise the cap.", effectiveLimit) +} + +func formatTruncationHint(effectiveLimit int) string { + return composeTruncationHint(ui.RenderWarn, truncationHintText(effectiveLimit)) +} + +// composeTruncationHint keeps newlines outside the renderer. lipgloss pads +// blank lines to the widest-line width (alignTextHorizontal shortAmount = +// widestLine - lineWidth) and drops the trailing newline (GH#5685). +func composeTruncationHint(render func(string) string, text string) string { + rendered := strings.TrimRight(render(text), " \t\r\n") + return "\n" + rendered + "\n" } func outputDotFormat(out io.Writer, issues []*types.Issue, depsByIssueID map[string][]*types.Dependency) error { diff --git a/cmd/bd/list_output_test.go b/cmd/bd/list_output_test.go index 42acb561d..dc55af4c8 100644 --- a/cmd/bd/list_output_test.go +++ b/cmd/bd/list_output_test.go @@ -4,9 +4,11 @@ import ( "bytes" "errors" "io" + "regexp" "strings" "testing" + lipgloss "charm.land/lipgloss/v2" "github.com/steveyegge/beads/internal/types" ) @@ -167,3 +169,103 @@ func TestOutputFormattedListWithNoEdgesDoesNotWrite(t *testing.T) { t.Fatalf("zero-edge output made %d writes, want 0", writer.writes) } } + +func TestFormatTruncationHintExactBytes(t *testing.T) { + t.Parallel() + + got := formatTruncationHint(2) + plain := stripANSIForTest(got) + const want = "\nShowing 2 issues; more results matched but were hidden by --limit. Use --limit 0 for all, or --limit N to raise the cap.\n" + if plain != want { + t.Fatalf("truncation hint bytes differ\ngot:\n%q\nwant:\n%q", plain, want) + } + if !strings.HasSuffix(got, "\n") { + t.Fatal("truncation hint missing trailing newline") + } + if strings.HasSuffix(got, "\n\n") { + t.Fatal("truncation hint ended with a double newline") + } + assertNoWhitespaceOnlyLine(t, plain) + + got10 := stripANSIForTest(formatTruncationHint(10)) + if !strings.Contains(got10, "Showing 10 issues;") { + t.Fatalf("limit interpolation missing: %q", got10) + } + if strings.HasPrefix(got10, "\n\n") { + t.Fatalf("leading double newline: %q", got10) + } +} + +func TestComposeTruncationHintRejectsLipglossBlankLinePadding(t *testing.T) { + t.Parallel() + + // Empty lipgloss Style (props==0) is identity — the colorless test + // default, and why RenderWarn("\n"+text+"\n") did not catch GH#5685. + // Foreground sets props so alignTextHorizontal pads blank lines to + // the widest-line width (not terminal width; no PTY required). + style := lipgloss.NewStyle().Foreground(lipgloss.Color("3")) + render := func(s string) string { return style.Render(s) } + text := truncationHintText(2) + + buggy := render("\n" + text + "\n") + got := composeTruncationHint(render, text) + + buggyPlain := stripANSIForTest(buggy) + gotPlain := stripANSIForTest(got) + + padWidth := whitespaceOnlyLineWidth(buggyPlain) + if padWidth == 0 { + t.Fatalf("fixture: styled Render of newline-wrapped text should pad a blank line; got %q", buggyPlain) + } + if padWidth != len(text) { + t.Fatalf("lipgloss pad width = %d, want hint length %d (widest-line width, not terminal width)\nbuggy=%q", padWidth, len(text), buggyPlain) + } + widest := widestNonBlankLineWidth(buggyPlain) + if padWidth != widest { + t.Fatalf("lipgloss pad width = %d, want widest-line width %d\nbuggy=%q", padWidth, widest, buggyPlain) + } + + assertNoWhitespaceOnlyLine(t, gotPlain) + if !strings.HasSuffix(got, "\n") { + t.Fatalf("composeTruncationHint missing trailing newline: %q", got) + } + if strings.HasSuffix(got, "\n\n") { + t.Fatal("composeTruncationHint ended with a double newline") + } + if got == buggy { + t.Fatalf("composeTruncationHint still uses newline-inside-Render bytes:\n%q", got) + } +} + +func stripANSIForTest(s string) string { + return regexp.MustCompile(`\x1b\[[0-9;]*m`).ReplaceAllString(s, "") +} + +func whitespaceOnlyLineWidth(s string) int { + width := 0 + for _, line := range strings.Split(s, "\n") { + if line != "" && strings.TrimSpace(line) == "" && len(line) > width { + width = len(line) + } + } + return width +} + +func widestNonBlankLineWidth(s string) int { + width := 0 + for _, line := range strings.Split(s, "\n") { + if strings.TrimSpace(line) != "" && len(line) > width { + width = len(line) + } + } + return width +} + +func assertNoWhitespaceOnlyLine(t *testing.T, s string) { + t.Helper() + for i, line := range strings.Split(s, "\n") { + if line != "" && strings.TrimSpace(line) == "" { + t.Fatalf("whitespace-only line at index %d (len=%d): %q", i, len(line), line) + } + } +} diff --git a/cmd/bd/list_proxied_server.go b/cmd/bd/list_proxied_server.go index be540dbd8..2a184e254 100644 --- a/cmd/bd/list_proxied_server.go +++ b/cmd/bd/list_proxied_server.go @@ -156,7 +156,7 @@ func runListProxiedWatch(_ *cobra.Command, ctx context.Context, in listInput) er if err != nil { return fmt.Errorf("initial query: %w", err) } - displayPrettyListWithDeps(issues, true, deps, hasMore, in.ReadyFlag) + displayPrettyListWithDeps(issues, true, deps, hasMore, in.ReadyFlag, in.Status) printTruncationHint(hasMore, in.effectiveLimit) lastSnapshot := issueSnapshot(issues) @@ -183,7 +183,7 @@ func runListProxiedWatch(_ *cobra.Command, ctx context.Context, in listInput) er snap := issueSnapshot(issues) if snap != lastSnapshot { lastSnapshot = snap - displayPrettyListWithDeps(issues, true, deps, hasMore, in.ReadyFlag) + displayPrettyListWithDeps(issues, true, deps, hasMore, in.ReadyFlag, in.Status) printTruncationHint(hasMore, in.effectiveLimit) fmt.Fprintf(os.Stderr, "\nWatching for changes... (Press Ctrl+C to exit)\n") } @@ -239,7 +239,7 @@ func renderProxiedListText(ctx context.Context, out io.Writer, issues []*types.I printTruncationHint(truncated, in.effectiveLimit) return nil } - displayPrettyListWithDepsMode(issues, false, depsByIssueID, in.depsMode, truncated, in.ReadyFlag) + displayPrettyListWithDepsMode(issues, false, depsByIssueID, in.depsMode, truncated, in.ReadyFlag, in.Status) printTruncationHint(truncated, in.effectiveLimit) printSkipLabelsFooter(in.SkipLabels) return nil diff --git a/cmd/bd/list_show_filter_modes.go b/cmd/bd/list_show_filter_modes.go index 7573269ee..562f922c4 100644 --- a/cmd/bd/list_show_filter_modes.go +++ b/cmd/bd/list_show_filter_modes.go @@ -156,7 +156,7 @@ func loadWatchedIssues(ctx context.Context, store storage.DoltStorage, filter ty // names its scope instead of asserting an in-progress count the ready filter // forced to zero. A watch loop is where a stale "0 in progress" is most likely // to be read as a live fact, so this is the path that most needs it. -func displayWatchedIssueList(ctx context.Context, store watchListDependencyStore, issues []*types.Issue, truncated, readyFiltered bool) { +func displayWatchedIssueList(ctx context.Context, store watchListDependencyStore, issues []*types.Issue, truncated, readyFiltered bool, statusSelector string) { var allDeps map[string][]*types.Dependency if store != nil { deps, err := store.GetAllDependencyRecords(ctx) @@ -164,7 +164,7 @@ func displayWatchedIssueList(ctx context.Context, store watchListDependencyStore allDeps = deps } } - displayPrettyListWithDeps(issues, true, allDeps, truncated, readyFiltered) + displayPrettyListWithDeps(issues, true, allDeps, truncated, readyFiltered, statusSelector) } // watchIssues returns an error only for the initial query — a failure there @@ -174,7 +174,7 @@ func displayWatchedIssueList(ctx context.Context, store watchListDependencyStore // runListCore route a MaxRows cap violation through handleMaxRowsError for // exit-code-2 semantics instead of watchIssues swallowing it and exiting 0 // (be-x42v.4 follow-up, review MUST-FIX 5). -func watchIssues(ctx context.Context, store storage.DoltStorage, filter types.IssueFilter, ready bool, parentID string, sortBy string, reverse bool, effectiveLimit int) error { +func watchIssues(ctx context.Context, store storage.DoltStorage, filter types.IssueFilter, ready bool, parentID string, sortBy string, reverse bool, effectiveLimit int, statusSelector string) error { // Initial display issues, err := loadWatchedIssues(ctx, store, filter, ready, parentID, sortBy, reverse) if err != nil { @@ -184,7 +184,7 @@ func watchIssues(ctx context.Context, store storage.DoltStorage, filter types.Is // below depends on; what is left is the cut and its verdict, and those are // the shared epilogue's on every other listing this command has. issues, truncated := workapi.FinishPage(issues, "", false, effectiveLimit, false) - displayWatchedIssueList(ctx, store, issues, truncated, ready) + displayWatchedIssueList(ctx, store, issues, truncated, ready, statusSelector) printTruncationHint(truncated, effectiveLimit) lastSnapshot := issueSnapshot(issues) @@ -214,7 +214,7 @@ func watchIssues(ctx context.Context, store storage.DoltStorage, filter types.Is snap := issueSnapshot(issues) if snap != lastSnapshot { lastSnapshot = snap - displayWatchedIssueList(ctx, store, issues, truncated, ready) + displayWatchedIssueList(ctx, store, issues, truncated, ready, statusSelector) printTruncationHint(truncated, effectiveLimit) fmt.Fprintf(os.Stderr, "\nWatching for changes... (Press Ctrl+C to exit)\n") } @@ -256,7 +256,7 @@ func runListProxiedHierarchicalParent(ctx context.Context, uw uow.UnitOfWork, in } // Hierarchical --parent walks use an unlimited per-level query; never page-truncated. - displayPrettyListWithDepsMode(treeIssues, false, depsByIssueID, in.depsMode, false, in.ReadyFlag) + displayPrettyListWithDepsMode(treeIssues, false, depsByIssueID, in.depsMode, false, in.ReadyFlag, in.Status) printSkipLabelsFooter(in.SkipLabels) return nil } diff --git a/cmd/bd/list_tree.go b/cmd/bd/list_tree.go index 2cd17c0af..9b1838727 100644 --- a/cmd/bd/list_tree.go +++ b/cmd/bd/list_tree.go @@ -147,26 +147,28 @@ func printPrettyTree(childrenMap map[string][]*types.Issue, parentID string, pre // There is no --ready arm behind this one: it is the plain tree, so the // summary keeps its status breakdown. func displayPrettyList(issues []*types.Issue, showHeader bool) { - displayPrettyListWithDeps(issues, showHeader, nil, false, false) + displayPrettyListWithDeps(issues, showHeader, nil, false, false, "") } // displayPrettyListWithDeps displays issues in tree format using dependency data. -// readyFiltered must be threaded from the caller's --ready state rather than -// defaulted here: the watch paths reach the summary through this wrapper, and a -// hardcoded false silently restores the vacuous "(N open, 0 in progress)" that -// listFooterLine exists to suppress. -func displayPrettyListWithDeps(issues []*types.Issue, showHeader bool, allDeps map[string][]*types.Dependency, truncated, readyFiltered bool) { - displayPrettyListWithDepsMode(issues, showHeader, allDeps, "", truncated, readyFiltered) +// readyFiltered and statusSelector must be threaded from the caller's --ready +// / --status state rather than defaulted here: the watch paths reach the +// summary through this wrapper, and a hardcoded false silently restores the +// vacuous "(N open, 0 in progress)" that listFooterLine exists to suppress. +func displayPrettyListWithDeps(issues []*types.Issue, showHeader bool, allDeps map[string][]*types.Dependency, truncated, readyFiltered bool, statusSelector string) { + displayPrettyListWithDepsMode(issues, showHeader, allDeps, "", truncated, readyFiltered, statusSelector) } // listFooterLine renders the one-line summary under a text listing. // // The status breakdown is only meaningful when the query could have returned -// more than one status. Under --ready it cannot: BuildListFilter pins -// filter.Status to open (internal/workapi/list.go), so every row is open by -// construction and the breakdown degenerates to a tautology — "(N open, 0 in -// progress)" is what it prints for ANY database, including one with a thousand -// in-progress issues matching the same label. +// more than one status. Under --ready the query is status-pinned: the default +// (no --status, or --status all) is still open, so "(N open, 0 in progress)" +// is a tautology for ANY database, including one with a thousand in-progress +// issues matching the same label. An explicit --status is the intersection +// (GH#5832), and the same tautology applies to whatever selector was asked +// for — the footer must name that selector rather than reuse the default-open +// sentence. // // Printed next to a real count that number reads as a finding rather than an // artifact of the flag: "0 in progress" answers the question "is anything in @@ -175,13 +177,14 @@ func displayPrettyListWithDeps(issues []*types.Issue, showHeader bool, allDeps m // construction, say what was excluded instead of asserting a count for it. This // is the same principle as the truncation arm below, which refuses to label a // cut-off page "Total" (GH#5362): a count is only honest alongside its scope. -func listFooterLine(total, open, inProgress int, truncated, readyFiltered bool) string { +func listFooterLine(total, open, inProgress int, truncated, readyFiltered bool, statusSelector string) string { if readyFiltered { // No status breakdown: --ready makes it vacuous. Name the scope instead. + scope := readyFooterScope(statusSelector) if truncated { - return fmt.Sprintf("Showing %d ready issues (open only — --ready excludes in_progress); more match (truncated by --limit). Use --limit 0 for all.", total) + return fmt.Sprintf("Showing %d ready issues (%s); more match (truncated by --limit). Use --limit 0 for all.", total, scope) } - return fmt.Sprintf("Ready: %d issues with no active blockers (open only — --ready excludes in_progress)", total) + return fmt.Sprintf("Ready: %d issues with no active blockers (%s)", total, scope) } if truncated { return fmt.Sprintf("Showing %d issues (%d open, %d in progress); more match (truncated by --limit). Use --limit 0 for all.", @@ -190,14 +193,31 @@ func listFooterLine(total, open, inProgress int, truncated, readyFiltered bool) return fmt.Sprintf("Total: %d issues (%d open, %d in progress)", total, open, inProgress) } +// readyFooterScope names the status pin a --ready listing actually used. +// Empty / "all" still take the open default; an explicit selector is the +// intersection and must not reuse "excludes in_progress" (GH#5832). +func readyFooterScope(statusSelector string) string { + var parts []string + for _, part := range strings.Split(statusSelector, ",") { + part = strings.TrimSpace(part) + if part != "" { + parts = append(parts, part) + } + } + if len(parts) == 0 || (len(parts) == 1 && parts[0] == "all") { + return "open only — --ready excludes in_progress" + } + return strings.Join(parts, ",") + " only" +} + // displayPrettyListWithDepsMode displays issues in tree format. When depsMode is // "scheduling" or "all", the tree also annotates each node's dependency edges and // orders siblings by their scheduling dependencies (see orderSiblingsByDeps). An // empty depsMode is the plain parent-child tree. truncated means the page was cut // by --limit; the summary then says "Showing N" instead of "Total: N" (GH#5362). -// readyFiltered means --ready was in force, which pins the query to open issues — -// see listFooterLine for why the summary must say so. -func displayPrettyListWithDepsMode(issues []*types.Issue, showHeader bool, allDeps map[string][]*types.Dependency, depsMode string, truncated, readyFiltered bool) { +// readyFiltered means --ready was in force; statusSelector is the --status value +// so the summary names the pin that actually applied — see listFooterLine. +func displayPrettyListWithDepsMode(issues []*types.Issue, showHeader bool, allDeps map[string][]*types.Dependency, depsMode string, truncated, readyFiltered bool, statusSelector string) { if showHeader { // Clear screen and show header fmt.Print("\033[2J\033[H") @@ -243,7 +263,7 @@ func displayPrettyListWithDepsMode(issues []*types.Issue, showHeader bool, allDe inProgressCount++ } } - fmt.Println(listFooterLine(len(issues), openCount, inProgressCount, truncated, readyFiltered)) + fmt.Println(listFooterLine(len(issues), openCount, inProgressCount, truncated, readyFiltered, statusSelector)) fmt.Println() fmt.Println("Status: ○ open ◐ in_progress ● blocked ✓ closed ❄ deferred") fmt.Println("Priority: P0–P4 (label only; not a status icon)") diff --git a/cmd/bd/list_wisp_type_reachability_test.go b/cmd/bd/list_wisp_type_reachability_test.go new file mode 100644 index 000000000..20111859c --- /dev/null +++ b/cmd/bd/list_wisp_type_reachability_test.go @@ -0,0 +1,130 @@ +package main + +import ( + "strings" + "testing" + + "github.com/spf13/cobra" +) + +// newWispTypeInputCmd carries only the flags the --wisp-type reachability +// check reads. gatherListInput reads many others, but GetString / GetBool on +// an unregistered flag return zero values, so this stays minimal — the same +// shape as newListInputTestCmd. +func newWispTypeInputCmd() *cobra.Command { + cmd := &cobra.Command{Use: "list"} + f := cmd.Flags() + f.String("wisp-type", "", "") + f.Bool("include-ephemeral", false, "") + f.Bool("include-infra", false, "") + f.StringP("type", "t", "", "") + return cmd +} + +// TestListRegistersIncludeEphemeral pins the flag the guard below points at. +// +// It is the load-bearing half of this change: ListRequest.IncludeEphemeral is +// the plane knob its own doc names as the combination that answers with rows, +// and `bd list` did not register a flag for it. Without this the advice the +// guard prints names a flag that does not exist, which is worse than the +// silence it replaces. +func TestListRegistersIncludeEphemeral(t *testing.T) { + if listCmd.Flags().Lookup("include-ephemeral") == nil { + t.Fatal("bd list must register --include-ephemeral: it is the only narrow way to admit the wisp plane, and --wisp-type's error message tells callers to use it") + } +} + +// TestGatherListInput_WispTypeWithoutAPlaneIsRefused covers the request that +// cannot match a row for ANY input: wisp_type narrows whatever the rest of the +// request admitted, and a default listing admits only durable rows, which +// carry no classification. +// +// The API layer keeps composing this to an empty page — that is its pinned +// contract, and this does not touch it. The refusal is here, where the only +// thing a human can have meant is the combination that returns rows. +func TestGatherListInput_WispTypeWithoutAPlaneIsRefused(t *testing.T) { + defer func(prev bool) { jsonOutput = prev }(jsonOutput) + jsonOutput = false + + cmd := newWispTypeInputCmd() + if err := cmd.Flags().Set("wisp-type", "heartbeat"); err != nil { + t.Fatalf("set wisp-type: %v", err) + } + var err error + stderr := captureStderr(t, func() { _, err = gatherListInput(cmd) }) + if err == nil { + t.Fatal("--wisp-type with no plane knob cannot match any row and must be refused, got nil error") + } + // The message has to carry the way out, or it is just a different silence. + if !strings.Contains(stderr, "--include-ephemeral") { + t.Errorf("the refusal must name the flag that makes the request answerable; got:\n%s", stderr) + } +} + +// TestGatherListInput_WispTypeWithAPlaneIsAccepted is the discriminating half: +// a guard that refused everything would pass the test above on its own. Each +// case here admits the plane by a different route, and all must survive. +func TestGatherListInput_WispTypeWithAPlaneIsAccepted(t *testing.T) { + defer func(prev bool) { jsonOutput = prev }(jsonOutput) + jsonOutput = false + + for _, tc := range []struct { + name string + flag string + val string + }{ + {"the narrow plane knob", "include-ephemeral", "true"}, + {"the wider bundle that also admits it", "include-infra", "true"}, + // An explicit type is left alone: an infra type routes to the plane by + // itself, and this layer cannot tell which types those are without the + // workspace's infra vocabulary. Refusing here would reject a lawful, + // answerable request. + {"an explicit type, which may itself be infra", "type", "agent"}, + } { + t.Run(tc.name, func(t *testing.T) { + cmd := newWispTypeInputCmd() + if err := cmd.Flags().Set("wisp-type", "heartbeat"); err != nil { + t.Fatalf("set wisp-type: %v", err) + } + if err := cmd.Flags().Set(tc.flag, tc.val); err != nil { + t.Fatalf("set %s: %v", tc.flag, err) + } + in, err := gatherListInput(cmd) + if err != nil { + t.Fatalf("--wisp-type with --%s must be accepted, got: %v", tc.flag, err) + } + if in.WispType == nil || string(*in.WispType) != "heartbeat" { + t.Errorf("WispType must survive onto the request, got %v", in.WispType) + } + }) + } +} + +// TestGatherListInput_IncludeEphemeralReachesTheRequest confirms the new flag +// is wired through rather than merely registered — a flag that parses but is +// never read would leave the plane just as unreachable. +func TestGatherListInput_IncludeEphemeralReachesTheRequest(t *testing.T) { + defer func(prev bool) { jsonOutput = prev }(jsonOutput) + jsonOutput = false + + cmd := newWispTypeInputCmd() + if err := cmd.Flags().Set("include-ephemeral", "true"); err != nil { + t.Fatalf("set include-ephemeral: %v", err) + } + in, err := gatherListInput(cmd) + if err != nil { + t.Fatalf("gatherListInput: %v", err) + } + if !in.IncludeEphemeral { + t.Error("--include-ephemeral must set ListRequest.IncludeEphemeral, got false") + } + + // And the default stays off, so the durable-only listing is unchanged. + bare, err := gatherListInput(newWispTypeInputCmd()) + if err != nil { + t.Fatalf("gatherListInput (bare): %v", err) + } + if bare.IncludeEphemeral { + t.Error("IncludeEphemeral must default to false; the default listing is durable-only") + } +} diff --git a/cmd/bd/markdown_test.go b/cmd/bd/markdown_test.go index 1dca3462e..e91ef89d7 100644 --- a/cmd/bd/markdown_test.go +++ b/cmd/bd/markdown_test.go @@ -154,6 +154,38 @@ Just a title and description. }, }, }, + { + name: "GH-4643 H2-per-section briefing file parses as three issues", + content: `## Problem +Something is broken. + +## Context +Extra background. + +## Acceptance Criteria +It should work. +`, + expected: []*IssueTemplate{ + { + Title: "Problem", + Description: "Something is broken.", + Priority: 2, + IssueType: "task", + }, + { + Title: "Context", + Description: "Extra background.", + Priority: 2, + IssueType: "task", + }, + { + Title: "Acceptance Criteria", + Description: "It should work.", + Priority: 2, + IssueType: "task", + }, + }, + }, } for _, tt := range tests { diff --git a/cmd/bd/setup/claude.go b/cmd/bd/setup/claude.go index dac9cdd6e..fb10326d4 100644 --- a/cmd/bd/setup/claude.go +++ b/cmd/bd/setup/claude.go @@ -1,6 +1,7 @@ package setup import ( + "bytes" "encoding/json" "errors" "fmt" @@ -70,6 +71,28 @@ func globalSettingsPath(home string) string { return filepath.Join(home, ".claude", "settings.json") } +// marshalSettings renders a Claude settings map as two-space-indented JSON +// with the trailing newline json.MarshalIndent omits, so the file stays +// POSIX-clean and byte-stable across runs. +func marshalSettings(settings map[string]interface{}) ([]byte, error) { + data, err := json.MarshalIndent(settings, "", " ") + if err != nil { + return nil, err + } + return append(data, '\n'), nil +} + +// writeSettingsIfChanged writes data to path only when it differs from what is +// already on disk. GH#5693: bd init / bd setup claude marshaled and wrote +// settings.json on every run, so a no-op run still churned the file's mtime +// and — because MarshalIndent emits no trailing newline — silently stripped it. +func writeSettingsIfChanged(env claudeEnv, path string, data []byte) error { + if existing, err := env.readFile(path); err == nil && bytes.Equal(existing, data) { + return nil + } + return env.writeFile(path, data) +} + func claudeAgentsEnv(env claudeEnv) agentsEnv { ae, _ := claudeAgentsEnvRedirect(env) return ae @@ -277,13 +300,13 @@ func installClaude(env claudeEnv, global bool, stealth bool) error { } } - data, err := json.MarshalIndent(settings, "", " ") + data, err := marshalSettings(settings) if err != nil { _, _ = fmt.Fprintf(env.stderr, "Error: marshal settings: %v\n", err) return err } - if err := env.writeFile(settingsPath, data); err != nil { + if err := writeSettingsIfChanged(env, settingsPath, data); err != nil { _, _ = fmt.Fprintf(env.stderr, "Error: write settings: %v\n", err) return err } @@ -300,8 +323,8 @@ func installClaude(env claudeEnv, global bool, stealth bool) error { removeHookCommand(legacyHooks, "SessionStart", v) removeHookCommand(legacyHooks, "PreCompact", v) } - if migrated, marshalErr := json.MarshalIndent(legacySettings, "", " "); marshalErr == nil { - if writeErr := env.writeFile(legacyPath, migrated); writeErr == nil { + if migrated, marshalErr := marshalSettings(legacySettings); marshalErr == nil { + if writeErr := writeSettingsIfChanged(env, legacyPath, migrated); writeErr == nil { _, _ = fmt.Fprintf(env.stdout, "✓ Migrated hooks from %s\n", legacyPath) } } @@ -473,13 +496,13 @@ func removeClaude(env claudeEnv, global bool) error { removeHookCommand(hooks, "PreCompact", v) } - data, err = json.MarshalIndent(settings, "", " ") + data, err = marshalSettings(settings) if err != nil { _, _ = fmt.Fprintf(env.stderr, "Error: marshal settings: %v\n", err) return err } - if err := env.writeFile(settingsPath, data); err != nil { + if err := writeSettingsIfChanged(env, settingsPath, data); err != nil { _, _ = fmt.Fprintf(env.stderr, "Error: write settings: %v\n", err) return err } @@ -497,8 +520,8 @@ func removeClaude(env claudeEnv, global bool) error { removeHookCommand(legacyHooks, "SessionStart", v) removeHookCommand(legacyHooks, "PreCompact", v) } - if migrated, marshalErr := json.MarshalIndent(legacySettings, "", " "); marshalErr == nil { - _ = env.writeFile(legacyPath, migrated) + if migrated, marshalErr := marshalSettings(legacySettings); marshalErr == nil { + _ = writeSettingsIfChanged(env, legacyPath, migrated) } } } diff --git a/cmd/bd/setup/claude_test.go b/cmd/bd/setup/claude_test.go index 131919a84..4d1e178e1 100644 --- a/cmd/bd/setup/claude_test.go +++ b/cmd/bd/setup/claude_test.go @@ -1655,3 +1655,207 @@ func TestRemoveClaudeRedirectCleansStaleClaudeBlock(t *testing.T) { t.Fatalf("AGENTS.md should still contain its beads block after remove:\n%s", agentsData) } } + +// countingClaudeEnv wraps env.writeFile with a per-path write counter so tests +// can assert that an unchanged settings file is not rewritten. +func countingClaudeEnv(env claudeEnv, counts map[string]int) claudeEnv { + inner := env.writeFile + env.writeFile = func(path string, data []byte) error { + counts[path]++ + return inner(path, data) + } + return env +} + +func assertTrailingNewline(t *testing.T, path string) { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + if len(data) == 0 || data[len(data)-1] != '\n' { + t.Errorf("%s should end with a trailing newline, got %q", path, tailOf(data)) + } +} + +func tailOf(data []byte) string { + if len(data) > 24 { + return string(data[len(data)-24:]) + } + return string(data) +} + +// GH#5693: bd setup claude must leave settings.json POSIX-clean (trailing +// newline) instead of stripping the newline json.MarshalIndent omits. +func TestInstallClaudeSettingsEndsWithTrailingNewline(t *testing.T) { + t.Run("project", func(t *testing.T) { + env, _, _ := newClaudeTestEnv(t) + if err := installClaude(env, false, false); err != nil { + t.Fatalf("installClaude: %v", err) + } + assertTrailingNewline(t, projectSettingsPath(env.projectDir)) + }) + t.Run("global", func(t *testing.T) { + env, _, _ := newClaudeTestEnv(t) + if err := installClaude(env, true, false); err != nil { + t.Fatalf("installClaude: %v", err) + } + assertTrailingNewline(t, globalSettingsPath(env.homeDir)) + }) +} + +// GH#5693: a second bd setup claude with nothing to change must not rewrite +// settings.json. +func TestInstallClaudeSkipsRewriteWhenSettingsUnchanged(t *testing.T) { + env, _, _ := newClaudeTestEnv(t) + counts := map[string]int{} + env = countingClaudeEnv(env, counts) + settingsPath := projectSettingsPath(env.projectDir) + + if err := installClaude(env, false, false); err != nil { + t.Fatalf("first installClaude: %v", err) + } + if counts[settingsPath] != 1 { + t.Fatalf("first install should write settings once, got %d", counts[settingsPath]) + } + before, err := os.ReadFile(settingsPath) + if err != nil { + t.Fatalf("read settings: %v", err) + } + + if err := installClaude(env, false, false); err != nil { + t.Fatalf("second installClaude: %v", err) + } + if got := counts[settingsPath]; got != 1 { + t.Errorf("second install rewrote unchanged settings: %d total writes, want 1", got) + } + after, err := os.ReadFile(settingsPath) + if err != nil { + t.Fatalf("re-read settings: %v", err) + } + if !bytes.Equal(before, after) { + t.Errorf("second install changed settings content:\nbefore=%q\nafter=%q", before, after) + } +} + +// GH#5693: the same two properties must hold for the removeClaude write path, +// which marshals and writes settings.json through a separate call site. +func TestRemoveClaudeSettingsNewlineAndSkipsUnchangedRewrite(t *testing.T) { + env, _, _ := newClaudeTestEnv(t) + counts := map[string]int{} + env = countingClaudeEnv(env, counts) + settingsPath := projectSettingsPath(env.projectDir) + + if err := installClaude(env, false, false); err != nil { + t.Fatalf("installClaude: %v", err) + } + writesAfterInstall := counts[settingsPath] + + if err := removeClaude(env, false); err != nil { + t.Fatalf("first removeClaude: %v", err) + } + assertTrailingNewline(t, settingsPath) + writesAfterRemove := counts[settingsPath] + if writesAfterRemove != writesAfterInstall+1 { + t.Fatalf("first remove should write settings once, got %d writes (install left %d)", + writesAfterRemove-writesAfterInstall, writesAfterInstall) + } + + if err := removeClaude(env, false); err != nil { + t.Fatalf("second removeClaude: %v", err) + } + if got := counts[settingsPath]; got != writesAfterRemove { + t.Errorf("second remove rewrote unchanged settings: %d writes, want %d", got, writesAfterRemove) + } +} + +// GH#5693: the legacy settings.local.json migration writes are the two other +// marshal-and-write call sites in this file; they strip the newline too. +func TestClaudeLegacySettingsWritesEndWithNewline(t *testing.T) { + legacyWithBeadsHook := func() map[string]interface{} { + return map[string]interface{}{ + "hooks": map[string]interface{}{ + "SessionStart": []interface{}{ + map[string]interface{}{ + "matcher": "", + "hooks": []interface{}{ + map[string]interface{}{"type": "command", "command": "bd prime"}, + }, + }, + }, + }, + } + } + + // The second install writes settings.local.json zero times, but not via + // writeSettingsIfChanged: installClaude's migration block is guarded by + // hasBeadsHooks(legacyPath), which is false once the first install has + // migrated the hooks away, so the block is skipped outright. That guard is + // also why the skip inside writeSettingsIfChanged is unreachable on this + // call site — whenever the guard passes, removeHookCommand strips a hook + // that hasBeadsHooks just matched, so the marshaled bytes always differ. + // What this subtest pins is therefore the user-visible property (a repeat + // bd setup claude leaves settings.local.json alone) plus the newline, not + // the no-op-skip helper. + t.Run("install migration then no further legacy writes", func(t *testing.T) { + env, _, _ := newClaudeTestEnv(t) + counts := map[string]int{} + env = countingClaudeEnv(env, counts) + legacyPath := legacyProjectSettingsPath(env.projectDir) + writeSettings(t, legacyPath, legacyWithBeadsHook()) + + if err := installClaude(env, false, false); err != nil { + t.Fatalf("first installClaude: %v", err) + } + if hasBeadsHooks(legacyPath) { + t.Fatal("legacy beads hooks should have been migrated away") + } + assertTrailingNewline(t, legacyPath) + writesAfterFirst := counts[legacyPath] + if writesAfterFirst != 1 { + t.Fatalf("first install should write legacy settings once, got %d", writesAfterFirst) + } + before, err := os.ReadFile(legacyPath) + if err != nil { + t.Fatalf("read legacy settings: %v", err) + } + + if err := installClaude(env, false, false); err != nil { + t.Fatalf("second installClaude: %v", err) + } + if got := counts[legacyPath]; got != writesAfterFirst { + t.Errorf("second install rewrote migrated legacy settings: %d writes, want %d", got, writesAfterFirst) + } + after, err := os.ReadFile(legacyPath) + if err != nil { + t.Fatalf("re-read legacy settings: %v", err) + } + if !bytes.Equal(before, after) { + t.Errorf("second install changed legacy settings content:\nbefore=%q\nafter=%q", before, after) + } + }) + + t.Run("remove cleanup", func(t *testing.T) { + env, _, _ := newClaudeTestEnv(t) + counts := map[string]int{} + env = countingClaudeEnv(env, counts) + legacyPath := legacyProjectSettingsPath(env.projectDir) + writeSettings(t, legacyPath, legacyWithBeadsHook()) + + if err := removeClaude(env, false); err != nil { + t.Fatalf("first removeClaude: %v", err) + } + assertTrailingNewline(t, legacyPath) + writesAfterFirst := counts[legacyPath] + if writesAfterFirst != 1 { + t.Fatalf("first remove should write legacy settings once, got %d", writesAfterFirst) + } + + if err := removeClaude(env, false); err != nil { + t.Fatalf("second removeClaude: %v", err) + } + if got := counts[legacyPath]; got != writesAfterFirst { + t.Errorf("second remove rewrote unchanged legacy settings: %d writes, want %d", got, writesAfterFirst) + } + }) +} diff --git a/default.nix b/default.nix index cfe3c35bd..5ec1275c6 100644 --- a/default.nix +++ b/default.nix @@ -19,7 +19,12 @@ buildGoModule { # proxyVendor avoids vendor/modules.txt consistency checks when the vendored # tree lags go.mod/go.sum. proxyVendor = true; - vendorHash = "sha256-eakIdTpJcGYHOWNjW5Pq9lllVOm9WzKWYNq5FKSQj8w="; + # Recomputed for this branch: neither release/1.3.0's value nor #5931's is + # right here, because the branch already carried an independent dependency + # bump (d594092eb), so the #5931 cherry-pick lands on a vendor set that + # existed on neither side. Value taken from the `got:` hash the nix-build lane + # reported on this branch. + vendorHash = "sha256-DQdauEx5g48Xbxrz4wGLx4vkrQoYQX3FUx+7N/Y6YV4="; # Match go.mod to the selected Nix Go toolchain. buildGoModule also builds # vendored dependencies in the Nix sandbox, where toolchain downloads are not diff --git a/go.mod b/go.mod index dbe9a93f7..85f9a81ad 100644 --- a/go.mod +++ b/go.mod @@ -146,16 +146,16 @@ require ( github.com/docker/go-connections v0.6.0 // indirect github.com/docker/go-units v0.5.0 // indirect github.com/dolthub/aws-sdk-go-ini-parser v0.0.0-20250305001723-2821c37f6c12 // indirect - github.com/dolthub/dolt/go v0.40.5-0.20260715172757-a6690826d767 - github.com/dolthub/eventsapi_schema v0.0.0-20260310172945-37a9265ade69 // indirect + github.com/dolthub/dolt/go v0.40.5-0.20260806213044-796d07497741 + github.com/dolthub/eventsapi_schema v0.0.0-20260715220557-d9b4a1c6b4d4 // indirect github.com/dolthub/flatbuffers/v23 v23.3.3-dh.2 // indirect github.com/dolthub/fslock v0.0.5 // indirect github.com/dolthub/go-icu-regex v0.0.0-20260610153742-72563bc7ca83 // indirect - github.com/dolthub/go-mysql-server v0.20.1-0.20260713210757-6d01d00bbbf3 + github.com/dolthub/go-mysql-server v0.20.1-0.20260805191915-e5eafe0da809 github.com/dolthub/gozstd v0.0.0-20240423170813-23a2903bca63 // indirect github.com/dolthub/ishell v0.0.0-20260414231531-5f031e3e9037 // indirect github.com/dolthub/jsonpath v0.0.2-0.20240227200619-19675ab05c71 // indirect - github.com/dolthub/vitess v0.0.0-20260624214226-81d034e0fde8 // indirect + github.com/dolthub/vitess v0.0.0-20260728212736-0542037326d7 // indirect github.com/dustin/go-humanize v1.0.1 // indirect github.com/ebitengine/purego v0.10.0 // indirect github.com/edsrzf/mmap-go v1.2.0 // indirect diff --git a/go.sum b/go.sum index d72be7f0a..9196fa792 100644 --- a/go.sum +++ b/go.sum @@ -435,30 +435,30 @@ github.com/dolthub/aws-sdk-go-ini-parser v0.0.0-20250305001723-2821c37f6c12 h1:I github.com/dolthub/aws-sdk-go-ini-parser v0.0.0-20250305001723-2821c37f6c12/go.mod h1:rN7X8BHwkjPcfMQQ2QTAq/xM3leUSGLfb+1Js7Y6TVo= github.com/dolthub/dolt-mcp v0.3.4 h1:AyG5cw+fNWXDHXujtQnqUPZrpWtPg6FN6yYtjv1pP44= github.com/dolthub/dolt-mcp v0.3.4/go.mod h1:bCZ7KHvDYs+M0e+ySgmGiNvLhcwsN7bbf5YCyillLrk= -github.com/dolthub/dolt/go v0.40.5-0.20260715172757-a6690826d767 h1:RSO4YZ5xuZ2EnHv813q2q+QpEqDW1ugjRepPXyDvb7s= -github.com/dolthub/dolt/go v0.40.5-0.20260715172757-a6690826d767/go.mod h1:DXkKITEIpR85Ta86AyjAXYeTULr/tRylidxxy95eraI= +github.com/dolthub/dolt/go v0.40.5-0.20260806213044-796d07497741 h1:z+Disx4RJn0dxaYSzcp8pbiXddi3dpIoUJaMtVa/iBE= +github.com/dolthub/dolt/go v0.40.5-0.20260806213044-796d07497741/go.mod h1:houDkPceGRjDBdaDsfym5OQUB4465mBUnPvSxMOPrPo= github.com/dolthub/driver/v2 v2.2.0 h1:Qm9BhvvKNmeTDF6kZdDkdLe62CF1W6GrCA3ZDRx/tlk= github.com/dolthub/driver/v2 v2.2.0/go.mod h1:Ngki9vM88/sqin1RKfUt/9Ex8DmoePtHgY7d4+YoolU= github.com/dolthub/eventkit v0.0.0-20260611184414-99f5693e696a h1:qNecLF0XIN1BD0RfS/UmlN8ObC3G7bFUJtEyqJeBea4= github.com/dolthub/eventkit v0.0.0-20260611184414-99f5693e696a/go.mod h1:eHUkm69YMO2/v7bzwwYCEkd+3Xu7+KUpP7jFWAbpDlM= -github.com/dolthub/eventsapi_schema v0.0.0-20260310172945-37a9265ade69 h1:JShhbqMw26nKx3pqqu/cFxOpzBkN+4elVhzuUfgDw2k= -github.com/dolthub/eventsapi_schema v0.0.0-20260310172945-37a9265ade69/go.mod h1:SSLraQS/jGLYFgff3vuZ+JbVUct6vyEeMzjLBqWqoyM= +github.com/dolthub/eventsapi_schema v0.0.0-20260715220557-d9b4a1c6b4d4 h1:0mg9QEFdkkBwJMxvz1tCjHYmfG2iIC6aShj1InDq9/M= +github.com/dolthub/eventsapi_schema v0.0.0-20260715220557-d9b4a1c6b4d4/go.mod h1:SSLraQS/jGLYFgff3vuZ+JbVUct6vyEeMzjLBqWqoyM= github.com/dolthub/flatbuffers/v23 v23.3.3-dh.2 h1:u3PMzfF8RkKd3lB9pZ2bfn0qEG+1Gms9599cr0REMww= github.com/dolthub/flatbuffers/v23 v23.3.3-dh.2/go.mod h1:mIEZOHnFx4ZMQeawhw9rhsj+0zwQj7adVsnBX7t+eKY= github.com/dolthub/fslock v0.0.5 h1:QoXhBhgY1oumHE26qyE7tgmXUT8qjJwxsIzo54O/B/k= github.com/dolthub/fslock v0.0.5/go.mod h1:sdofYYqE0D79zNZyB4/kmlnsQOVap1C2yByjGKSirEM= github.com/dolthub/go-icu-regex v0.0.0-20260610153742-72563bc7ca83 h1:FEMjCGEroDnY/BXyAffVZxUpXhP2GpoUJyyq5KaLn8c= github.com/dolthub/go-icu-regex v0.0.0-20260610153742-72563bc7ca83/go.mod h1:F3cnm+vMRK1HaU6+rNqQrOCyR03HHhR1GWG2gnPOqaE= -github.com/dolthub/go-mysql-server v0.20.1-0.20260713210757-6d01d00bbbf3 h1:HdPnbDFCkxyEU1zDHVPcHq8aWFkar3fIz9I9Wz5cC0E= -github.com/dolthub/go-mysql-server v0.20.1-0.20260713210757-6d01d00bbbf3/go.mod h1:AnD2jKQZCf09sM2JhV/cTEtsoEhPNg6pVWjRCBox4Zw= +github.com/dolthub/go-mysql-server v0.20.1-0.20260805191915-e5eafe0da809 h1:c+S3O08LgGS+czU/CuN+/pOKMri5F6kxajoWrTyrE1s= +github.com/dolthub/go-mysql-server v0.20.1-0.20260805191915-e5eafe0da809/go.mod h1:9GLxpglhedopzqBGpW0PGrw7UlwG5kBIOepE2QfTQ1s= github.com/dolthub/gozstd v0.0.0-20240423170813-23a2903bca63 h1:OAsXLAPL4du6tfbBgK0xXHZkOlos63RdKYS3Sgw/dfI= github.com/dolthub/gozstd v0.0.0-20240423170813-23a2903bca63/go.mod h1:lV7lUeuDhH5thVGDCKXbatwKy2KW80L4rMT46n+Y2/Q= github.com/dolthub/ishell v0.0.0-20260414231531-5f031e3e9037 h1:oIW9HwuWrhxv+4HZxA+QQSKHLqWFyXZ2FmNjUYwkdiM= github.com/dolthub/ishell v0.0.0-20260414231531-5f031e3e9037/go.mod h1:ehexgi1mPxRTk0Mok/pADALuHbvATulTh6gzr7NzZto= github.com/dolthub/jsonpath v0.0.2-0.20240227200619-19675ab05c71 h1:bMGS25NWAGTEtT5tOBsCuCrlYnLRKpbJVJkDbrTRhwQ= github.com/dolthub/jsonpath v0.0.2-0.20240227200619-19675ab05c71/go.mod h1:2/2zjLQ/JOOSbbSboojeg+cAwcRV0fDLzIiWch/lhqI= -github.com/dolthub/vitess v0.0.0-20260624214226-81d034e0fde8 h1:zmKLyRCTiNZnizO++sNXP1QW2bxLE2Y+WJAdu/hcu6s= -github.com/dolthub/vitess v0.0.0-20260624214226-81d034e0fde8/go.mod h1:5SVEJgAhw5nnQUFnGgKI1Svqes1Mw+zKUsalyxmOuG0= +github.com/dolthub/vitess v0.0.0-20260728212736-0542037326d7 h1:gvOpHisi0CbG3hixdGulNQdXy3V7+AWxe37kGBgTmRI= +github.com/dolthub/vitess v0.0.0-20260728212736-0542037326d7/go.mod h1:5SVEJgAhw5nnQUFnGgKI1Svqes1Mw+zKUsalyxmOuG0= github.com/dprotaso/go-yit v0.0.0-20191028211022-135eb7262960/go.mod h1:9HQzr9D/0PGwMEbC3d5AB7oi67+h4TsQqItC1GVYG58= github.com/dprotaso/go-yit v0.0.0-20220510233725-9ba8df137936 h1:PRxIJD8XjimM5aTknUK9w6DHLDox2r2M3DI4i2pnd3w= github.com/dprotaso/go-yit v0.0.0-20220510233725-9ba8df137936/go.mod h1:ttYvX5qlB+mlV1okblJqcSMtR4c52UKxDiX9GRBS8+Q= diff --git a/internal/storage/dolt/queries_test.go b/internal/storage/dolt/queries_test.go index 3a87635aa..cb668239c 100644 --- a/internal/storage/dolt/queries_test.go +++ b/internal/storage/dolt/queries_test.go @@ -174,6 +174,68 @@ func TestGetReadyWork_StatusFilter(t *testing.T) { } } +// TestGetReadyWork_CustomActiveCategory pins GH#5831: `bd ready` pins +// StatusOpen, and that pin is the active category — built-in open plus any +// custom status whose category is active. WIP/done/frozen/unspecified +// customs stay out; in_progress stays out. +func TestGetReadyWork_CustomActiveCategory(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + + ctx, cancel := testContext(t) + defer cancel() + + if err := store.SetConfig(ctx, "status.custom", "triaged:active,polishing:wip,shipped:done,parked:frozen,legacy"); err != nil { + t.Fatalf("SetConfig status.custom: %v", err) + } + + mk := func(id string, status types.Status) { + t.Helper() + iss := &types.Issue{ + ID: id, + Title: id, + Status: types.StatusOpen, + Priority: 2, + IssueType: types.TypeTask, + } + if err := store.CreateIssue(ctx, iss, "tester"); err != nil { + t.Fatalf("CreateIssue %s: %v", id, err) + } + if status != types.StatusOpen { + if err := store.UpdateIssue(ctx, id, map[string]interface{}{"status": string(status)}, "tester"); err != nil { + t.Fatalf("UpdateIssue %s -> %s: %v", id, status, err) + } + } + } + mk("rw-ca-open", types.StatusOpen) + mk("rw-ca-active", types.Status("triaged")) + mk("rw-ca-wip", types.Status("polishing")) + mk("rw-ca-done", types.Status("shipped")) + mk("rw-ca-frozen", types.Status("parked")) + mk("rw-ca-legacy", types.Status("legacy")) + mk("rw-ca-inprog", types.StatusInProgress) + + work, err := store.GetReadyWork(ctx, types.WorkFilter{Status: types.StatusOpen}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + got := map[string]bool{} + for _, w := range work { + got[w.ID] = true + } + if !got["rw-ca-open"] { + t.Errorf("built-in open missing from ready set: %v", issueIDs(work)) + } + if !got["rw-ca-active"] { + t.Errorf("custom active-category status missing from ready set: %v", issueIDs(work)) + } + for _, id := range []string{"rw-ca-wip", "rw-ca-done", "rw-ca-frozen", "rw-ca-legacy", "rw-ca-inprog"} { + if got[id] { + t.Errorf("%s should not appear in the StatusOpen ready set: %v", id, issueIDs(work)) + } + } +} + func TestGetReadyWork_PriorityFilter(t *testing.T) { store, cleanup := setupTestStore(t) defer cleanup() diff --git a/internal/storage/issueops/helpers.go b/internal/storage/issueops/helpers.go index 32b3bece9..f305a3658 100644 --- a/internal/storage/issueops/helpers.go +++ b/internal/storage/issueops/helpers.go @@ -524,7 +524,7 @@ func ReadConfigPrefix(ctx context.Context, tx DBTX) (string, error) { underscoreYamlPrefix := strings.TrimSpace(config.GetString("issue_prefix")) debug.Logf("Debug: missing config.issue_prefix in database (err=%v, db value=%q, yaml issue-prefix=%q, yaml issue_prefix=%q)\n", err, configPrefix, yamlPrefix, underscoreYamlPrefix) - return "", fmt.Errorf("%w: issue_prefix config is missing (run 'bd init --prefix ' for a new project, or 'bd bootstrap' to clone an existing remote; if using config.yaml, use key 'issue-prefix', not 'issue_prefix')", storage.ErrNotInitialized) + return "", fmt.Errorf("%w: issue_prefix config is missing (run 'bd init --prefix ' for a new project, or 'bd bootstrap' to clone an existing remote; this check reads the database's config table, so adding 'issue-prefix' to config.yaml does not satisfy it)", storage.ErrNotInitialized) } else if err != nil { return "", fmt.Errorf("failed to get config: %w", err) } diff --git a/internal/storage/issueops/helpers_test.go b/internal/storage/issueops/helpers_test.go index 567e4d8c4..0479106d1 100644 --- a/internal/storage/issueops/helpers_test.go +++ b/internal/storage/issueops/helpers_test.go @@ -7,6 +7,7 @@ import ( "testing" "github.com/DATA-DOG/go-sqlmock" + "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" ) @@ -289,7 +290,7 @@ func TestReadConfigPrefix(t *testing.T) { } }) - t.Run("missing config row has actionable key naming hint", func(t *testing.T) { + t.Run("missing config row names the commands that initialize the prefix", func(t *testing.T) { t.Parallel() db, mock, err := sqlmock.New() @@ -319,14 +320,58 @@ func TestReadConfigPrefix(t *testing.T) { msg := err.Error() if !containsAll(msg, "issue_prefix config is missing", - "issue-prefix", - "not 'issue_prefix'", + "bd init --prefix", + "bd bootstrap", ) { - t.Fatalf("ReadConfigPrefix error missing key naming guidance: %s", msg) + t.Fatalf("ReadConfigPrefix error missing actionable recovery commands: %s", msg) } }) } +// TestReadConfigPrefixDoesNotRecommendInertYAMLKey pins GH#5916: the message +// used to tell an operator to "use key 'issue-prefix', not 'issue_prefix'" in +// config.yaml, but this gate only ever reads the database's config table. The +// YAML key is SET here and the gate still fails, so the recommendation sent +// operators to commit a tracked field that changes nothing and misleads the +// next reader. +// +// Not parallel: it initializes and mutates the process-global viper config. +// BEADS_TEST_IGNORE_REPO_CONFIG keeps the checkout's own tracked config.yaml +// out of the picture, so only the value injected below is in effect. +func TestReadConfigPrefixDoesNotRecommendInertYAMLKey(t *testing.T) { + t.Setenv("BEADS_TEST_IGNORE_REPO_CONFIG", "1") + if err := config.Initialize(); err != nil { + t.Fatalf("config.Initialize: %v", err) + } + defer config.ResetForTesting() + config.Set("issue-prefix", "skills") + + if got := config.GetString("issue-prefix"); got != "skills" { + t.Fatalf("precondition: config.GetString(issue-prefix) = %q, want %q — the assertions below would be vacuous", got, "skills") + } + + _, mock, tx := beginMockTx(t) + mock.ExpectQuery("SELECT value FROM config WHERE `key` = \\?"). + WithArgs("issue_prefix"). + WillReturnRows(sqlmock.NewRows([]string{"value"})) + + _, err := ReadConfigPrefix(context.Background(), tx) + if err == nil { + t.Fatal("ReadConfigPrefix() returned nil error, want error: the YAML key must not satisfy this gate") + } + if !errors.Is(err, storage.ErrNotInitialized) { + t.Fatalf("ReadConfigPrefix error = %v, want ErrNotInitialized", err) + } + + msg := err.Error() + if strings.Contains(msg, "use key 'issue-prefix'") { + t.Fatalf("ReadConfigPrefix recommends a config.yaml key that does not satisfy it: %s", msg) + } + if !containsAll(msg, "bd init --prefix", "bd bootstrap") { + t.Fatalf("ReadConfigPrefix error missing actionable recovery commands: %s", msg) + } +} + func containsAll(s string, parts ...string) bool { for _, part := range parts { if !strings.Contains(s, part) { diff --git a/internal/storage/issueops/ready_work_test.go b/internal/storage/issueops/ready_work_test.go index 3ae5393c0..552a359b1 100644 --- a/internal/storage/issueops/ready_work_test.go +++ b/internal/storage/issueops/ready_work_test.go @@ -244,6 +244,32 @@ func TestGetChildrenOfDeferredParentsInTx_IgnoresMissingWispDependenciesTable(t } } +func TestGetReadyWorkInTxOpenIncludesCustomActive(t *testing.T) { + t.Parallel() + + _, mock, tx := beginMockTx(t) + mock.ExpectQuery(deferredParentProbeRegex("issues")).WillReturnError(sql.ErrNoRows) + mock.ExpectQuery(deferredParentProbeRegex("wisps")).WillReturnError(sql.ErrNoRows) + mock.ExpectQuery(`SELECT id FROM issues\s+WHERE \(status = \? OR status IN \(SELECT name FROM custom_statuses WHERE category = 'active'\)\)`). + WillReturnRows(sqlmock.NewRows([]string{"id"})) + mock.ExpectQuery(`SELECT 1 FROM wisps LIMIT 1`).WillReturnError(sql.ErrNoRows) + + got, err := GetReadyWorkInTx( + context.Background(), + tx, + types.WorkFilter{Status: types.StatusOpen}, + ) + if err != nil { + t.Fatalf("GetReadyWorkInTx: %v", err) + } + if len(got) != 0 { + t.Fatalf("expected no rows, got %d", len(got)) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } +} + func TestGetReadyWorkInTxStatusesSingleQuery(t *testing.T) { t.Parallel() diff --git a/internal/storage/sqlbuild/ready.go b/internal/storage/sqlbuild/ready.go index f599467e3..b111765ca 100644 --- a/internal/storage/sqlbuild/ready.go +++ b/internal/storage/sqlbuild/ready.go @@ -95,7 +95,15 @@ func BuildReadyWorkWhere(filter types.WorkFilter, tables FilterTables, in ReadyW var args []any switch { case filter.Status != "": - statusClause = "status = ?" + // Singular StatusOpen is the `bd ready` pin and means the ready- + // eligible set: built-in open plus category-active custom statuses, + // matching the ready_issues view since migration 0025 (GH#5831). + // Any other singular status stays exact. + if filter.Status == types.StatusOpen { + statusClause = "(status = ? OR status IN (SELECT name FROM custom_statuses WHERE category = 'active'))" + } else { + statusClause = "status = ?" + } args = append(args, string(filter.Status)) case len(filter.Statuses) > 0: ph, statusArgs := InPlaceholders(filter.Statuses) diff --git a/internal/storage/sqlbuild/sqlbuild_test.go b/internal/storage/sqlbuild/sqlbuild_test.go index 5f3300ed7..5565b4544 100644 --- a/internal/storage/sqlbuild/sqlbuild_test.go +++ b/internal/storage/sqlbuild/sqlbuild_test.go @@ -396,10 +396,24 @@ func TestBuildReadyWorkWhereStatusFilter(t *testing.T) { { name: "SingularStatusWinsOverStatuses", filter: types.WorkFilter{Status: "open", Statuses: []types.Status{"blocked", "pinned"}}, - wantClause: "status = ?", + wantClause: "(status = ? OR status IN (SELECT name FROM custom_statuses WHERE category = 'active'))", rejectClause: "status IN (?", wantLeadingArgs: []any{"open"}, }, + { + name: "OpenIncludesCustomActiveCategory", + filter: types.WorkFilter{Status: types.StatusOpen}, + wantClause: "status IN (SELECT name FROM custom_statuses WHERE category = 'active')", + rejectClause: "status IN ('open', 'in_progress')", + wantLeadingArgs: []any{"open"}, + }, + { + name: "NonOpenSingularStatusStaysExact", + filter: types.WorkFilter{Status: types.StatusInProgress}, + wantClause: "status = ?", + rejectClause: "custom_statuses", + wantLeadingArgs: []any{"in_progress"}, + }, { name: "EmptyFilterLegacyDefault", filter: types.WorkFilter{}, diff --git a/internal/storage/uow/dolt_sql_provider.go b/internal/storage/uow/dolt_sql_provider.go index dcad55170..9650495c4 100644 --- a/internal/storage/uow/dolt_sql_provider.go +++ b/internal/storage/uow/dolt_sql_provider.go @@ -3,15 +3,18 @@ package uow import ( "context" "database/sql" + "database/sql/driver" "errors" "fmt" + "io" + "net" "os" "strings" "sync/atomic" "time" "github.com/cenkalti/backoff/v4" - _ "github.com/go-sql-driver/mysql" + "github.com/go-sql-driver/mysql" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" @@ -559,6 +562,91 @@ func buildDSN(ep proxy.Endpoint, database, user, password, tlsConfigName string) }.String() } +// pinger is the subset of *sql.DB that pingWithRetry needs, narrowed so +// tests can script a sequence of transient failures without a real +// connection. +type pinger interface { + PingContext(ctx context.Context) error +} + +// pingAttemptTimeout bounds a single ping attempt. The DSN sets a dial +// Timeout (5s, internal/storage/dbproxy/util/dsn.go) but no ReadTimeout, so a +// server that accepts the TCP connection and then stalls mid-handshake is +// otherwise bounded only by the caller's context. Kept above the dial timeout +// so a genuine dial timeout still surfaces as itself instead of being masked +// by this cap. +const pingAttemptTimeout = 10 * time.Second + +// isTransientPingError reports whether err is a connection-level failure worth +// retrying — the shapes a Dolt server produces when it drops or stalls a +// connection mid-handshake or mid-ping — as opposed to a durable rejection +// (bad credentials, unknown database) that retrying cannot fix. +// +// Any *net.OpError counts. Before the handshake completes there is no +// application-level rejection that can reach us as one, so at boot an OpError +// is always the connection itself: refused while the server restarts, reset, +// or broken pipe. MySQL's own rejections arrive as *mysql.MySQLError and stay +// permanent. A name that does not resolve is the exception — it will not start +// resolving, so retrying only spends the whole budget before reporting the +// same misconfiguration. +// +// context.DeadlineExceeded counts only because pingWithRetry applies a +// per-attempt deadline and checks the caller's context *before* consulting +// this function, so a DeadlineExceeded that reaches here can only be that +// per-attempt cap. Do not call this without that guard: classifying the +// caller's own expiry as transient would retry a context that is already dead, +// which is the regression #6000 introduced from the other direction. +// +// Deliberately narrower than internal/storage/dolt's isRetryableError, which +// covers this same ground for query-time retries in server mode. That one +// matches on error text and additionally admits migration-lock, Dolt +// read-only, and MySQL-1105 shapes — server states that describe a *statement* +// being refused, which a boot ping cannot be in because nothing has executed on +// the connection yet. Matching sentinels and types with errors.Is/errors.As +// keeps this side typed, and keeps a bootstrap ping from inheriting retry +// semantics that only make sense mid-transaction. +func isTransientPingError(err error) bool { + var dnsErr *net.DNSError + if errors.As(err, &dnsErr) && !dnsErr.IsTemporary && !dnsErr.IsTimeout { + return false + } + var opErr *net.OpError + return errors.Is(err, driver.ErrBadConn) || + errors.Is(err, mysql.ErrInvalidConn) || + errors.Is(err, io.EOF) || + errors.Is(err, io.ErrUnexpectedEOF) || + errors.Is(err, context.DeadlineExceeded) || + errors.As(err, &opErr) +} + +// pingWithRetry pings until the server answers, the error proves durable, or +// bo's MaxElapsedTime is spent. Each attempt gets its own deadline: +// backoff.Retry bounds the gap between attempts but cannot interrupt one +// already in flight, so without this cap a server that accepts TCP and then +// stalls blocks for the caller's entire context. +func pingWithRetry(ctx context.Context, p pinger, bo *backoff.ExponentialBackOff, attemptTimeout time.Duration) error { + return backoff.Retry(func() error { + attemptCtx, cancel := context.WithTimeout(ctx, attemptTimeout) + defer cancel() + + err := p.PingContext(attemptCtx) + if err == nil { + return nil + } + // The caller's context is spent, so nothing further can succeed and + // its Canceled/DeadlineExceeded must not be read as a transient + // per-attempt timeout. This check is what lets isTransientPingError + // treat DeadlineExceeded as retryable at all. + if ctx.Err() != nil { + return backoff.Permanent(err) + } + if isTransientPingError(err) { + return err + } + return backoff.Permanent(err) + }, backoff.WithContext(bo, ctx)) +} + // assertSessionDatabaseOnPool runs assertSessionDatabase on a connection pinned // out of pool. A pool hands out a different connection per call, so the // assertion has to name the connection it is asserting about. @@ -576,7 +664,9 @@ func openDB(ctx context.Context, dsn string) (*sql.DB, error) { if err != nil { return nil, fmt.Errorf("uow: open db: %w", err) } - if err := conn.PingContext(ctx); err != nil { + bo := backoff.NewExponentialBackOff() + bo.MaxElapsedTime = 30 * time.Second + if err := pingWithRetry(ctx, conn, bo, pingAttemptTimeout); err != nil { return nil, errors.Join(fmt.Errorf("uow: ping db: %w", err), conn.Close()) } return conn, nil diff --git a/internal/storage/uow/dolt_sql_provider_test.go b/internal/storage/uow/dolt_sql_provider_test.go new file mode 100644 index 000000000..4a6f84ed1 --- /dev/null +++ b/internal/storage/uow/dolt_sql_provider_test.go @@ -0,0 +1,216 @@ +package uow + +import ( + "context" + "errors" + "net" + "sync/atomic" + "testing" + "time" + + "github.com/cenkalti/backoff/v4" + "github.com/go-sql-driver/mysql" +) + +// fakePinger is a pinger test double that returns a scripted sequence of +// responses and fails the test if called more times than scripted — that +// over-call case is exactly the retry-budget bug this test suite guards +// against. +type fakePinger struct { + t *testing.T + errs []error + calls int +} + +func (p *fakePinger) PingContext(context.Context) error { + p.t.Helper() + if p.calls >= len(p.errs) { + p.t.Fatalf("PingContext called more times than expected (call #%d, only %d responses scripted)", p.calls+1, len(p.errs)) + } + err := p.errs[p.calls] + p.calls++ + return err +} + +// blockingPinger never answers on its own: it stalls until the context it was +// handed is cancelled, then reports that context's error. This is the shape of +// a Dolt server that accepts the TCP connection and then stalls without ever +// completing the MySQL handshake — the DSN sets a dial Timeout but no +// ReadTimeout, so nothing below pingWithRetry bounds it. +type blockingPinger struct { + calls atomic.Int32 + release chan struct{} +} + +func newBlockingPinger(t *testing.T) *blockingPinger { + t.Helper() + p := &blockingPinger{release: make(chan struct{})} + // Unblock any attempt still parked when the test ends, so a failure here + // does not leak a goroutine into the rest of the package's tests. + t.Cleanup(func() { close(p.release) }) + return p +} + +func (p *blockingPinger) PingContext(ctx context.Context) error { + p.calls.Add(1) + select { + case <-ctx.Done(): + return ctx.Err() + case <-p.release: + return errors.New("blockingPinger released at test cleanup") + } +} + +func testPingBackOff() *backoff.ExponentialBackOff { + bo := backoff.NewExponentialBackOff() + bo.InitialInterval = time.Millisecond + bo.MaxElapsedTime = time.Second + return bo +} + +// testPingAttemptTimeout is generous: the subtests that use it script their +// ping responses and never actually stall, so the per-attempt cap must not be +// what ends them. +const testPingAttemptTimeout = time.Second + +func TestPingWithRetryRecoversFromATransientBadConn(t *testing.T) { + t.Run("transient error retries then recovers", func(t *testing.T) { + p := &fakePinger{t: t, errs: []error{mysql.ErrInvalidConn, mysql.ErrInvalidConn, nil}} + + err := pingWithRetry(context.Background(), p, testPingBackOff(), testPingAttemptTimeout) + + if err != nil { + t.Fatalf("pingWithRetry() error = %v, want nil", err) + } + if p.calls != 3 { + t.Fatalf("PingContext called %d times, want exactly 3", p.calls) + } + }) + + t.Run("non-transient error fails without retrying", func(t *testing.T) { + wantErr := errors.New("Access denied") + p := &fakePinger{t: t, errs: []error{wantErr}} + + err := pingWithRetry(context.Background(), p, testPingBackOff(), testPingAttemptTimeout) + + if !errors.Is(err, wantErr) { + t.Fatalf("pingWithRetry() error = %v, want %v", err, wantErr) + } + if p.calls != 1 { + t.Fatalf("PingContext called %d times, want exactly 1 (non-transient errors must not retry)", p.calls) + } + }) + + t.Run("already-cancelled context fails promptly", func(t *testing.T) { + p := &fakePinger{t: t, errs: []error{mysql.ErrInvalidConn}} + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + err := pingWithRetry(ctx, p, testPingBackOff(), testPingAttemptTimeout) + + if err == nil { + t.Fatal("pingWithRetry() error = nil, want non-nil for an already-cancelled context") + } + if p.calls != 1 { + t.Fatalf("PingContext called %d times, want exactly 1 (cancelled context must not retry)", p.calls) + } + }) + + t.Run("connection-level net error retries then recovers", func(t *testing.T) { + // A boot ping cannot fail at the application level before the + // handshake completes, so any *net.OpError here is connection-level: + // connection refused (server restarting), reset, or broken pipe. + refused := &net.OpError{Op: "dial", Net: "tcp", Err: errors.New("connect: connection refused")} + p := &fakePinger{t: t, errs: []error{refused, nil}} + + err := pingWithRetry(context.Background(), p, testPingBackOff(), testPingAttemptTimeout) + + if err != nil { + t.Fatalf("pingWithRetry() error = %v, want nil (a connection-level net error is transient at boot)", err) + } + if p.calls != 2 { + t.Fatalf("PingContext called %d times, want exactly 2", p.calls) + } + }) + + t.Run("unresolvable host fails without retrying", func(t *testing.T) { + // The narrowing that pays for classifying *net.OpError broadly: a name + // that does not resolve will not start resolving, so retrying only + // spends the budget before reporting the same misconfiguration. + dnsErr := &net.OpError{Op: "dial", Net: "tcp", Err: &net.DNSError{ + Err: "no such host", Name: "no-such-dolt-host.invalid", IsNotFound: true, + }} + p := &fakePinger{t: t, errs: []error{dnsErr}} + + err := pingWithRetry(context.Background(), p, testPingBackOff(), testPingAttemptTimeout) + + if !errors.Is(err, dnsErr) { + t.Fatalf("pingWithRetry() error = %v, want %v", err, dnsErr) + } + if p.calls != 1 { + t.Fatalf("PingContext called %d times, want exactly 1 (a permanent DNS failure must not retry)", p.calls) + } + }) +} + +// TestPingWithRetryBoundsEachAttempt covers the half of the retry budget that +// MaxElapsedTime cannot reach: backoff.Retry bounds the gap *between* +// attempts, but it cannot interrupt an attempt already in flight. +func TestPingWithRetryBoundsEachAttempt(t *testing.T) { + t.Run("a hung ping is capped per attempt and retried", func(t *testing.T) { + p := newBlockingPinger(t) + bo := backoff.NewExponentialBackOff() + bo.InitialInterval = time.Millisecond + bo.MaxElapsedTime = 200 * time.Millisecond + + // Run off-goroutine: before the per-attempt cap exists this call never + // returns at all, and the assertion we want is "it returned", not a + // 10-minute package timeout. + done := make(chan error, 1) + go func() { + done <- pingWithRetry(context.Background(), p, bo, 20*time.Millisecond) + }() + + select { + case err := <-done: + if err == nil { + t.Fatal("pingWithRetry() error = nil, want non-nil once the retry budget is exhausted") + } + if got := p.calls.Load(); got < 2 { + t.Fatalf("PingContext called %d times, want >= 2 (each hung attempt must be capped and retried)", got) + } + case <-time.After(5 * time.Second): + t.Fatal("pingWithRetry() never returned: a hung ping is not bounded per attempt, so it blocks for the caller's whole context") + } + }) + + t.Run("the caller's own deadline fails without retrying", func(t *testing.T) { + // The regression guard for the trap #6000 fell into: once a + // per-attempt deadline exists, context.DeadlineExceeded becomes + // retryable — but only when it came from *that* deadline. The + // caller's own expiry must still stop immediately. + p := newBlockingPinger(t) + bo := backoff.NewExponentialBackOff() + bo.InitialInterval = time.Millisecond + bo.MaxElapsedTime = 10 * time.Second + + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond) + defer cancel() + + start := time.Now() + // Per-attempt cap far beyond the caller's deadline, so the only thing + // that can end this attempt is the caller's context. + err := pingWithRetry(ctx, p, bo, time.Hour) + elapsed := time.Since(start) + + if err == nil { + t.Fatal("pingWithRetry() error = nil, want non-nil once the caller's context expires") + } + if got := p.calls.Load(); got != 1 { + t.Fatalf("PingContext called %d times, want exactly 1 (the caller's own deadline must not be retried as transient)", got) + } + if elapsed > 5*time.Second { + t.Fatalf("pingWithRetry() took %v after a 20ms caller deadline, want it to stop promptly", elapsed) + } + }) +} diff --git a/internal/workapi/list.go b/internal/workapi/list.go index 761d822d2..1798824c1 100644 --- a/internal/workapi/list.go +++ b/internal/workapi/list.go @@ -225,7 +225,12 @@ func BuildListFilter(in issueops.ListRequest, cfg ListConfig) (types.IssueFilter statusParts := splitStatusSelector(in.Status) statusAll := len(statusParts) == 1 && statusParts[0] == "all" - if in.ReadyFlag { + // --ready defaults to open (the blocker-aware query's historical pin), + // but an explicit --status is an intersection, not an override. Dropping + // the selector used to answer `bd list --status X --ready` with the + // unfiltered ready set (GH#5832). `--status all` is the no-filter + // spelling, so the open default still applies there. + if in.ReadyFlag && (len(statusParts) == 0 || statusAll) { s := types.StatusOpen filter.Status = &s } else if len(statusParts) > 0 && !statusAll { @@ -465,9 +470,10 @@ func splitStatusSelector(status string) []string { // pinned-carrying beads: the pinned or hooked status, or — when the selector // actually applies to the query — the "all" selector, which promises every // status. The pinned default is not forced off for a filter that asks for -// those beads. allApplies is false under --ready, which forces status open -// and otherwise ignores the selector, so "all" must not lift the pinned -// default there. +// those beads. allApplies is false under --ready: `--status all` (and the +// omitted selector) still pin open, so "all" must not lift the pinned +// default there. Explicit selectors are honored under --ready (GH#5832); +// pinned and hooked still return true regardless of allApplies. func statusSelectsPinned(parts []string, allApplies bool) bool { for _, part := range parts { switch part { diff --git a/internal/workapi/list_pinned_test.go b/internal/workapi/list_pinned_test.go index 58d36280c..38e944092 100644 --- a/internal/workapi/list_pinned_test.go +++ b/internal/workapi/list_pinned_test.go @@ -66,11 +66,11 @@ func TestPinnedDefaultBySelector(t *testing.T) { } }) - // --ready forces status open and otherwise IGNORES the selector, so an - // "all" that the query never applies must not have a pinned side effect + // `--status all` is the no-filter spelling, so --ready still pins open. + // An "all" that the query never applies must not have a pinned side effect // either. Without this, `bd list --ready --status=all` would admit pinned // beads into a ready listing that is filtered to open. - t.Run("--ready ignores the selector, including for pinned", func(t *testing.T) { + t.Run("--ready --status=all does not admit pinned", func(t *testing.T) { got := pinnedDefault(t, issueops.ListRequest{Status: "all", ReadyFlag: true}) if got == nil || *got { t.Errorf("--ready --status=all: Pinned = %v, want false", got) diff --git a/internal/workapi/list_ready_status_test.go b/internal/workapi/list_ready_status_test.go new file mode 100644 index 000000000..ab30e4105 --- /dev/null +++ b/internal/workapi/list_ready_status_test.go @@ -0,0 +1,123 @@ +package workapi + +import ( + "reflect" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// TestReadyStatusAndReadyIntersect pins GH#5832: `bd list --status X --ready` +// is the intersection, not the unfiltered ready set. These are builder and +// projection tests so they stay in the CGO_ENABLED=0 lane. +func TestReadyStatusAndReadyIntersect(t *testing.T) { + t.Run("status X --ready keeps status X", func(t *testing.T) { + filter, err := BuildListFilter(issueops.ListRequest{ReadyFlag: true, Status: "in_progress"}, ListConfig{}) + if err != nil { + t.Fatalf("BuildListFilter: %v", err) + } + if filter.Status == nil || *filter.Status != types.StatusInProgress { + t.Fatalf("IssueFilter.Status = %v, want %q", filter.Status, types.StatusInProgress) + } + wf := ReadyFilterFromIssueFilter(filter) + if wf.Status != types.StatusInProgress { + t.Fatalf("WorkFilter.Status = %q, want %q", wf.Status, types.StatusInProgress) + } + if len(wf.Statuses) != 0 { + t.Fatalf("WorkFilter.Statuses = %v, want empty", wf.Statuses) + } + }) + + t.Run("custom status --ready keeps the custom status", func(t *testing.T) { + cfg := ListConfig{CustomStatuses: []types.CustomStatus{{Name: "unrefined", Category: types.CategoryWIP}}} + filter, err := BuildListFilter(issueops.ListRequest{ReadyFlag: true, Status: "unrefined"}, cfg) + if err != nil { + t.Fatalf("BuildListFilter: %v", err) + } + if filter.Status == nil || *filter.Status != types.Status("unrefined") { + t.Fatalf("IssueFilter.Status = %v, want %q", filter.Status, "unrefined") + } + wf := ReadyFilterFromIssueFilter(filter) + if wf.Status != types.Status("unrefined") { + t.Fatalf("WorkFilter.Status = %q, want %q", wf.Status, "unrefined") + } + }) + + t.Run("zero-match status is not rewritten to the open ready set", func(t *testing.T) { + plain, err := BuildListFilter(issueops.ListRequest{ReadyFlag: true}, ListConfig{}) + if err != nil { + t.Fatalf("BuildListFilter(ready): %v", err) + } + if got := ReadyFilterFromIssueFilter(plain).Status; got != types.StatusOpen { + t.Fatalf("plain --ready WorkFilter.Status = %q, want %q", got, types.StatusOpen) + } + + filtered, err := BuildListFilter(issueops.ListRequest{ReadyFlag: true, Status: "in_progress"}, ListConfig{}) + if err != nil { + t.Fatalf("BuildListFilter(ready+in_progress): %v", err) + } + wf := ReadyFilterFromIssueFilter(filtered) + if wf.Status != types.StatusInProgress { + t.Fatalf("WorkFilter.Status = %q, want %q (must not fall back to the unfiltered ready pin)", wf.Status, types.StatusInProgress) + } + if wf.Status == ReadyFilterFromIssueFilter(plain).Status { + t.Fatal("zero-match --status was rewritten to the unfiltered ready status") + } + }) + + t.Run("plain --ready still pins open", func(t *testing.T) { + filter, err := BuildListFilter(issueops.ListRequest{ReadyFlag: true}, ListConfig{}) + if err != nil { + t.Fatalf("BuildListFilter: %v", err) + } + if filter.Status == nil || *filter.Status != types.StatusOpen { + t.Fatalf("IssueFilter.Status = %v, want %q", filter.Status, types.StatusOpen) + } + if got := ReadyFilterFromIssueFilter(filter).Status; got != types.StatusOpen { + t.Fatalf("WorkFilter.Status = %q, want %q", got, types.StatusOpen) + } + }) + + t.Run("status all --ready keeps the open default", func(t *testing.T) { + filter, err := BuildListFilter(issueops.ListRequest{ReadyFlag: true, Status: "all"}, ListConfig{}) + if err != nil { + t.Fatalf("BuildListFilter: %v", err) + } + if filter.Status == nil || *filter.Status != types.StatusOpen { + t.Fatalf("IssueFilter.Status = %v, want %q", filter.Status, types.StatusOpen) + } + }) + + t.Run("multi-status --ready copies the OR set", func(t *testing.T) { + filter, err := BuildListFilter(issueops.ListRequest{ReadyFlag: true, Status: "open,in_progress"}, ListConfig{}) + if err != nil { + t.Fatalf("BuildListFilter: %v", err) + } + want := []types.Status{types.StatusOpen, types.StatusInProgress} + if !reflect.DeepEqual(filter.Statuses, want) { + t.Fatalf("IssueFilter.Statuses = %v, want %v", filter.Statuses, want) + } + if filter.Status != nil { + t.Fatalf("IssueFilter.Status = %v, want nil when Statuses is set", filter.Status) + } + wf := ReadyFilterFromIssueFilter(filter) + if wf.Status != "" { + t.Fatalf("WorkFilter.Status = %q, want empty so Statuses is not ignored", wf.Status) + } + if !reflect.DeepEqual(wf.Statuses, want) { + t.Fatalf("WorkFilter.Statuses = %v, want %v", wf.Statuses, want) + } + }) + + t.Run("invalid status --ready errors instead of dropping --status", func(t *testing.T) { + _, err := BuildListFilter(issueops.ListRequest{ReadyFlag: true, Status: "not-a-status"}, ListConfig{}) + if err == nil { + t.Fatal("BuildListFilter unexpectedly succeeded") + } + if !strings.Contains(err.Error(), "invalid status") || !strings.Contains(err.Error(), "not-a-status") { + t.Fatalf("unexpected error: %v", err) + } + }) +} diff --git a/internal/workapi/ready.go b/internal/workapi/ready.go index 073e5f475..729c4756f 100644 --- a/internal/workapi/ready.go +++ b/internal/workapi/ready.go @@ -41,7 +41,9 @@ func LimitOr(limit *int, fallback int) int { // own to shape, and running it through here would change it. func BuildReadyFilter(in issueops.ReadyRequest) (types.WorkFilter, error) { filter := types.WorkFilter{ - // Open only, not in_progress - the same set `bd list --ready` shows. + // Open only, not in_progress - the same set `bd list --ready` shows + // when --status is omitted (an explicit --status is an intersection; + // GH#5832). Status: types.StatusOpen, Type: utils.NormalizeIssueType(in.IssueType), Limit: LimitOr(in.Limit, DefaultReadyLimit), diff --git a/internal/workapi/sort.go b/internal/workapi/sort.go index ecc27ec41..cbaad3c5f 100644 --- a/internal/workapi/sort.go +++ b/internal/workapi/sort.go @@ -48,7 +48,6 @@ func CompareIssuesBy(a, b *types.Issue, sortBy string) int { // filter on. func ReadyFilterFromIssueFilter(filter types.IssueFilter) types.WorkFilter { wf := types.WorkFilter{ - Status: types.StatusOpen, Limit: filter.Limit, Offset: filter.Offset, Labels: filter.Labels, @@ -70,6 +69,18 @@ func ReadyFilterFromIssueFilter(filter types.IssueFilter) types.WorkFilter { // --brief` with full rows and nothing to say why. Lite: filter.Lite, } + // Status is copied the same way as assignee/priority/labels: an explicit + // selector on the list filter is the ready-work selector too. The open + // default is only the fallback when BuildListFilter did not write one + // (plain `--ready`, or `--ready --status all`). Hard-coding StatusOpen + // here used to drop `--status` even after the builder honored it (GH#5832). + if filter.Status != nil { + wf.Status = *filter.Status + } else if len(filter.Statuses) > 0 { + wf.Statuses = append([]types.Status(nil), filter.Statuses...) + } else { + wf.Status = types.StatusOpen + } if filter.IssueType != nil { wf.Type = string(*filter.IssueType) } diff --git a/internal/workapi/testdata/list_filter_golden.json b/internal/workapi/testdata/list_filter_golden.json index d7fa4f5f2..cd03e29a9 100644 --- a/internal/workapi/testdata/list_filter_golden.json +++ b/internal/workapi/testdata/list_filter_golden.json @@ -1165,7 +1165,7 @@ } }, { - "name": "ready_flag_overrides_status", + "name": "ready_flag_honors_status", "params": { "ReadyFlag": true, "Status": "closed", @@ -1173,7 +1173,7 @@ }, "config": {}, "filter": { - "Status": "open", + "Status": "closed", "Statuses": null, "Priority": null, "IssueType": null, diff --git a/issueops/reader.go b/issueops/reader.go index 80db1527d..a5b54cf78 100644 --- a/issueops/reader.go +++ b/issueops/reader.go @@ -340,7 +340,9 @@ type ListRequest struct { // narrower filter vocabulary than this request can describe, and only part // of the request reaches it. // - // WHAT IT CARRIES: IssueType, all five label forms, Assignee, NoAssignee, + // WHAT IT CARRIES: Status/Statuses (an explicit --status is the + // intersection; AllFlag and `--status all` still take the open default), + // IssueType, all five label forms, Assignee, NoAssignee, // the exact Priority, ParentID, MolType, WispType, MetadataFields, // HasMetadataKey, the type exclusions (ExcludeTypes, and with them // IncludeGates and IncludeInfra), IncludeEphemeral — the ready query has an @@ -348,8 +350,7 @@ type ListRequest struct { // IncludeInfra's plane half crosses with it — Limit, Offset and the MaxRows // cap with its attribution. SortBy and Reverse // still apply, because the display order is applied to the page after the - // query rather than inside it. Status and AllFlag are resolved to "open" - // and have no further effect: ready work is open work. + // query rather than inside it. // // WHAT IT REFUSES: every other filter here is one the ready query cannot // carry, so combining it with ReadyFlag returns ErrValidation naming the diff --git a/issueops/reader_ready_scope.go b/issueops/reader_ready_scope.go index bfcda1b3d..bf3ee326c 100644 --- a/issueops/reader_ready_scope.go +++ b/issueops/reader_ready_scope.go @@ -29,7 +29,7 @@ type readyScopeField struct { // Four groups are deliberately ABSENT, because listing them here would refuse // requests that are answered correctly today: // -// - What the projection carries: IssueType, all five label forms, Assignee, +// - What the projection carries: Status/Statuses (GH#5832), IssueType, all five label forms, Assignee, // NoAssignee, the exact Priority, ParentID, MolType, WispType, // MetadataFields, HasMetadataKey, ExcludeTypes (and with it IncludeGates // and IncludeInfra's type suppression), IncludeEphemeral (and with it @@ -46,12 +46,15 @@ type readyScopeField struct { // Refusing that would be refusing a correct answer over its cost, and the // promise is already stated where a caller reads it (ListRequest.ReadyFlag). // -// - Status and AllFlag. The builder resolves both to "open" under ReadyFlag -// before the projection ever runs, so IssueFilter.Statuses is never -// populated on this path and there is nothing for the projection to drop. -// Ready work is open work; that override is pinned by the builder's golden -// file (internal/workapi/testdata/list_filter_golden.json, -// ready_flag_overrides_status). +// - AllFlag, and `--status all`. The builder resolves both to "open" under +// ReadyFlag before the projection ever runs (`all` is the no-filter +// spelling). An explicit --status is carried: BuildListFilter writes it +// onto IssueFilter and ReadyFilterFromIssueFilter copies Status/Statuses +// onto the ready-work filter, so `bd list --status X --ready` is the +// intersection rather than the unfiltered ready set (GH#5832). Status is +// therefore ABSENT from this drop set because it is honored, not because +// it is overridden. The default-open pin is the golden +// `ready_flag` case; the honor path is `ready_flag_honors_status`. // // - NoPinnedFlag. Pinned is dropped by the projection, but the ready-work // WHERE clause excludes pinned rows unconditionally From e1e1158dd99e2514292afaab41100ff6aa804203 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 21 Sep 2026 02:49:16 -0700 Subject: [PATCH 07/45] release/1.3.0: stop the smart gate auto-migrating a data-behind clone (#6575) (#6659) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(schema): stop the smart gate auto-migrating a data-behind clone (#6575) Release-branch counterpart of the main-line fix. The four source files are byte-identical to the main commit; only the CHANGELOG placement differs (a 1.3.1 "Fixed" bullet instead of Unreleased). The smart migrate gate's first-mover verdict was computed from schema facts alone: equal content hashes against the cached remote ref, equal max version, and at-or-above the convergence floor. None of those say anything about where the clone's branch sits relative to the ref the hashes came from, so a clone that was level with the remote on SCHEMA but behind it in unpulled DATA commits satisfied every precondition and was classified a safe first-mover. It then migrated in place, minting local-only schema commits on a HEAD missing the remote's data commits. When the pending batch moves a tracked table onto the dolt_ignore plane (0062's events), every subsequent `bd dolt pull` refuses to merge (#6368) and the clone can no longer fetch the very commits it was already behind on. v1.3.0's upgrade audience is largely 1.1-era databases, which is exactly the cohort reaching this state — this is being field-hit now, which is why rc.2 is held for it. The decisive fact was already wired: routeSmartGate's remote-ahead branch consults adopt.IsStrictAncestor (ahead == 0 && behind >= 1) via routeAdoptFastForward, and both production call sites supply it. The equal-version path simply never read it. Read it there too, before granting the automatic migrate, and stop with a new data-behind fallback reason naming the one-command remedy. The predicate does the narrowing for free: a level clone (ahead 0, behind 0) and a clone with unpushed local commits (ahead > 0) both read false and still auto-migrate, so only a strictly-behind clone is refused. An inconclusive ancestry read degrades to the blunt #4515 block rather than assuming level, matching the file's contract that no fallback is less safe than the blunt gate. Below-floor outcomes are untouched — the check sits inside the floor arm. Patch-line discipline: no new schema migration, no new feature surface. The stop reuses the existing blunt RemoteMigrateGateError with an empty Decision, so there is no change to the JSON or agent contract — no new Error()/userBody()/ AgentDirective/Options arms. Both shipped escape hatches still work unchanged: --force/BD_ALLOW_REMOTE_MIGRATE=1 are consulted before the smart gate, and BD_SMART_GATE=0 still yields the blunt block. Both stay in the release notes. S0 capability matrix: no cell flips, so no same-commit matrix change is needed. Verified two independent ways. No matrix fixture configures a Dolt remote — the direct `dolt push` cell asserts "No remote is configured" and `dolt remote list` asserts an empty listing — so checkRemoteMigrateGate returns at the hasRemote check before the smart gate, making routeSmartGate (the only function whose outcome changes) unreachable in every cell. Independently, matrix fixtures are created by the binary under test, so schema is at latest and the gate exits earlier still at len(pending) == 0. The one migrate probe, the `migrate schema` row, therefore never reaches the changed code. Regression test builds a genuinely data-behind clone against a real Dolt server rather than stubbing the fact the bug is about, and adapts to this branch's own LatestVersion (which lacks 0067). Verified failing at d487b9796 and passing here. Fixes #6575 Co-Authored-By: Claude Opus 5 * fix(schema): give the data-behind stop a working remedy, and widen it Review follow-up on the #6575 fix. The stop was right; what it told the operator to do was not reachable, and it protected the rarer half of the cohort. F1 (blocker): the refusal prescribed `bd dolt pull`, and `bd dolt pull` was refused by it. Measured on a real data-behind embedded clone (ahead=0, behind=1, clean working set) at the previous head: the pull exited 1 printing "Run `bd dolt pull` first, then retry"; `bd sync` and `bd init --reinit-local` exited 1 the same way; `bd bootstrap` reported "Database already exists … Nothing to do." and exited 0; `bd migrate schema --force` then `bd dolt push` left ahead=1 behind=1 with the push rejected non-fast-forward. The only thing that moved the clone was BD_ALLOW_REMOTE_MIGRATE=1 — performing the migration the stop exists to prevent. An embedded workspace has no external `dolt` binary either, so that was a fence with no gate. It is the #4566 deadlock one refusal over, so it takes #4566's shape of fix: a lenient open for the command whose whole purpose is to clear the refused precondition. Narrower than #4566's, though — the remote-sync open tolerates ONE gate reason (RemoteMigrateGateError.IsDataBehind) and nothing else. A pull cannot resolve a fork skew, a below-floor database, a shared-store consent decision or an opted-out blunt block, so waving it through those would buy nothing and hide them; the dirty-table and dependency-re-key refusals stay fatal on this open too. Broadening the existing LenientOpen classification instead would have granted all of that in one line. Verified end to end on both shapes, exit codes real: `bd dolt pull` 0 (0,1 -> 0,0), retrying the refused write auto-migrates as a true first-mover, `bd dolt push` 0. On a diverged clone the pull merges: 0 when it can, 1 with "merge conflicts in config require operator resolution; merge aborted and working set restored" when it cannot, and `bd dolt pull --strategy theirs` then 0 — which is why the diverged guidance names it. F2 (high): LocalIsStrictAncestorOf is ahead == 0 && behind >= 1, so a clone with any local commit of its own read "not behind" and still auto-migrated. Measured at the previous head with ahead=1, behind=1: "safe first-mover", and the migration ran. bd auto-commits every write, so behind-AND-ahead is the ORDINARY multi-machine state and the shape the fix protected was the narrower one. It reaches the identical wedge — the migration lands on a history missing the remote's commits either way — so the predicate is now behind >= 1 whatever ahead is, read from LocalAheadBehind (the counts LocalIsStrictAncestorOf was already computing and discarding). A level clone and a purely-ahead clone have nothing to pull and still auto-migrate, verified through the CLI. The remedy differs by shape even though the command does not, so the two stay separate verdicts all the way out to the message, the agent directive and the JSON: one is told its pull fast-forwards, the other that it merges and how to resolve conflicts. An adopter that wires only the older callback keeps the narrower (pre-widening) reading rather than losing the stop. F3 (medium): with data-behind outranking fallbackReasonSharedStore, a shared store's refusal rendered with neither "co-resident" nor "#5920" anywhere — Options()' sharedRisk only ever decorated the adopt option. The #5920 consequence and the `bd migrate schema` consent step the retry needs there now travel with this reason, on every surface. F4 (medium): --json emitted only fallback_reason: "data-behind" while observed/expected/options stayed the blunt default, whose option 1 is ["bd migrate --force", "bd dolt push"] — the bug followed by a push that is guaranteed rejected while the clone is still behind — and whose option 2 no-ops. The pull-first remedy lived only in prose --json never emitted. observed/expected now describe this stop, options is the single pull-first option, and data_behind_shape says which pull to expect. decision stays absent and fallback_reason unchanged, so the key agents match on does not move. The blunt body is REPLACED for this reason rather than decorated: both of its options were measured wrong here, and leaving them under a note saying "pull first" is what sent the last round back into the wedge. Error() is deliberately untouched — the headline stays the blunt #4259 line, and the body's first sentence accounts for it. Both escape hatches are unchanged and re-measured: `bd migrate --force` and BD_ALLOW_REMOTE_MIGRATE=1 are still consulted before the smart gate (exit 0), and BD_SMART_GATE=0 still yields the blunt gate unconditionally — including for `bd dolt pull`, which is what opting out means. The two corrections from the first round are preserved: the "proceeds on its own" promise is still gated on !e.Shared, and the unparseable-BD_SMART_GATE carve-out still lets data-behind outrank an env complaint (BD_SMART_GATE=maybe re-measured). Tests cover both shapes at three levels. The schema routing table pins the widened predicate, the shape split, the error degradation, the IsStrictAncestor fallback, and every remedy surface including IsDataBehind's narrowness; neutralizing the widening fails three of its subtests. The real-Dolt regression test now runs both shapes over the same two-clone file:// fixture, measuring ahead/behind rather than stubbing it and asserting that the strict-ancestor relation is false exactly where the diverged shape needs the widening. A new embedded test builds a genuinely data-behind embedded store, proves OpenForRemoteSync gets through the refusal without migrating, and proves BD_SMART_GATE=0 on the same fixture does NOT. No S0 matrix cells flip. No matrix fixture configures a Dolt remote, so routeSmartGate stays unreachable; the only `dolt pull` probes are the proxied-mode typed refusals, which are classified in historyCapabilityMatrix before any store opens and are untouched. This is the release/1.3.0 counterpart of the main-line change, code-identical to it. Two differences, both from the base: the CHANGELOG entry is the expanded form of the existing #6575 bullet inside `## [1.3.1] - 2026-09-16`'s `### Fixed` (same position the bullet already held, so the #6653 CHANGELOG non-conflict is preserved), and the dolt.Config literal in cmd/bd/main.go has no PoolReadTimeoutFallback field on this base. The shipped `## [1.3.0]` section is byte-identical: sha256 798a2ee0b0f0b7a3f0fc017ae01c919807740348a34f5ea8719b82a008533d00, 9169 lines, and the only CHANGELOG hunk is above its header. This branch's own matrix (cmd/bd/topology_capability_matrix_test.go, absent on main) was re-checked after the widening: no cell flips. No fixture configures a Dolt remote (the direct `dolt push` cell asserts "No remote is configured" and the direct `sync` cell asserts "status": "no-remote"), so routeSmartGate stays unreachable, and there is no `dolt pull` probe row at all. Refs #6575 Refs #6368 Refs #4566 Co-Authored-By: Claude Opus 5 * fix(schema): retarget the shared data-behind consent verb under --global Self-review catch on the previous commit's own new text. The #5920 consent verb the shared data-behind guidance names is TARGET-SCOPED: under --global the refused open aimed at beads_global, so the project-scoped `bd migrate schema` would consent the wrong database and leave the refusal in place. The existing shared-no-remote arm already handles this (printGlobalDatabaseConsentHint and the JSON retarget), and the new arm had to as well. The JSON retarget could not simply reuse the Decision test: this stop reaches the renderer through the default empty-Decision arm, keyed on the fallback reason, so `e.Decision == "shared-no-remote"` misses it. The human body now names the --global variant beside the project-scoped one, exactly as the shared-no-remote body does. The pull itself is target-agnostic and is deliberately NOT rewritten. Refs #6575 Co-Authored-By: Claude Opus 5 * fix(schema): give the data-behind read path the pull remedy too Round 2 fixed the write path and the agent payload, but left the two lenient embedded opens rendering the blunt #4259 coordination block. Those opens do not fail on this refusal - they warn and carry on - so the warning IS the whole contract for `bd list`, `bd ready`, `bd show` (openReadOnlyCommand) and `bd dolt commit` (openWorkingSetReconcile). Both printed `gateErr` (Error(), the one-line summary) followed by the unchanged sharedGuidance bullets, so on a data-behind clone the first thing bd said was: • designated migrator (only ONE machine): bd migrate --force && bd dolt push • every other clone (another already migrated): bd bootstrap Measured on a fast-forward clone at the previous head: that migrate lands v67 on a history the clone is still behind on, and the `bd dolt push` after it exits 1, rejected non-fast-forward - the #6575 state, reached by following bd's own instruction, from the same clone whose write path says in capitals not to run it. `bd bootstrap` no-ops against an existing workspace. `bd list` is the first thing a 1.1-era upgrader runs and what agents run constantly, so this was the most-read surface of the whole stop. Both arms now branch on gateErr.IsDataBehind() and print gateErr.UserMessage(), which is where the shape-branched pull-first body already lives - the same thing server mode's warnLenientOpenRefusal has printed all along. The intent framing is kept rather than replaced: the command in hand is still succeeding against the old schema, which the body (written for the fatal refusal) does not say. Refusals that are not the data-behind stop keep the coordination bullets, which are right for them; a second test pins that so the new body cannot swallow them. Verified by running, not asserting - `bd list` and `bd dolt commit` on real data-behind clones of both shapes, then `bd dolt pull` and a re-run, which migrates. Also fixes the agent payload contradicting itself. For data-behind, AgentDirective calls `bd dolt pull` the single option, Options annotates it `when: always, for this stop` and `risk: none - a pure fast-forward`, yet human_decision_required stayed hard-coded true, so an agent keying on that field stalled for approval of a step the same payload calls unconditional and riskless - and the cost of that stall is reaching for `bd migrate --force`. It is now false for exactly that arm. It stays true when DataDiverged (the pull merges and can need conflict resolution and a --strategy choice) and when Shared (the payload's second option needs an operator who can confirm every co-resident client is upgraded, #5920); getting either backwards would be worse than the contradiction. The `when` string is deliberately unchanged for both shapes - the pull is the only way forward either way; it is the risk, not the applicability, that diverging changes. Refs #6575 Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 --- CHANGELOG.md | 74 +++ cmd/bd/events_journal_construction_test.go | 1 + cmd/bd/main.go | 25 + cmd/bd/remote_migrate_gate.go | 76 ++- cmd/bd/remote_sync_open_test.go | 354 ++++++++++++ cmd/bd/store_factory.go | 10 + .../dolt/smart_gate_data_behind_test.go | 277 ++++++++++ .../dolt/smart_remote_migrate_gate_test.go | 3 + internal/storage/dolt/store.go | 49 +- internal/storage/embeddeddolt/cache.go | 18 +- .../embeddeddolt/remote_sync_gate_test.go | 381 +++++++++++++ internal/storage/embeddeddolt/store.go | 129 ++++- internal/storage/embeddeddolt/store_stub.go | 5 + .../storage/schema/remote_migrate_gate.go | 216 +++++++- .../schema/smart_remote_migrate_gate.go | 114 +++- .../schema/smart_remote_migrate_gate_test.go | 502 ++++++++++++++++++ .../storage/versioncontrolops/fastforward.go | 38 +- 17 files changed, 2243 insertions(+), 29 deletions(-) create mode 100644 cmd/bd/remote_sync_open_test.go create mode 100644 internal/storage/dolt/smart_gate_data_behind_test.go create mode 100644 internal/storage/embeddeddolt/remote_sync_gate_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 51e2e2ddd..5734dd278 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -164,6 +164,80 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 without starting either. The `--json` payload changes shape to carry that — see **Changed** above before parsing it. +- **The smart migrate gate no longer auto-migrates a clone whose data is behind + the remote** ([#6575](https://github.com/gastownhall/beads/issues/6575)). The + gate's first-mover verdict was computed from schema facts alone — equal + migration content hashes against the cached remote ref, at the same version, + at or above the convergence floor — and none of those say where the clone's + branch sits relative to that ref. A clone level with the remote on *schema* + but behind it in unpulled *data* commits satisfied every precondition, was + classified a safe first-mover, and migrated in place, minting local-only + schema commits on a HEAD missing those commits. When the pending batch moves + a tracked table onto the `dolt_ignore` plane, every later `bd dolt pull` then + refuses with `local changes would be stomped by merge: events` + ([#6368](https://github.com/gastownhall/beads/issues/6368)) while `dolt + status` reads clean — and the clone can no longer fetch the very commits it + was behind on. Upgrading from a 1.1-era database is the ordinary way in, so + this release's upgrade audience is the exposed one. The gate now reads the + branch position its callers already supply and stops with guidance naming the + remedy: run `bd dolt pull`, then re-run what you were blocked on, and the + migrate proceeds as a true first-mover. + + The stop applies whenever the clone has commits left to pull, whether or not + it also has commits of its own. bd auto-commits every write, so a clone that + is both behind and ahead is the ordinary multi-machine state, and it reaches + the same wedge; it is told its pull will *merge* rather than fast-forward, and + how to resolve conflicts if it reports them. A clone that is level with the + remote, or that only has unpushed local commits with nothing to pull, still + auto-migrates exactly as before. + + **`bd dolt pull` works from the refused state.** It opens the store too, so + on a refused clone it would otherwise hit the very refusal that prescribes + it — and an embedded workspace has no external `dolt` binary to fall back to, + which would leave `BD_ALLOW_REMOTE_MIGRATE=1` (performing the migration the + stop exists to prevent) as the only way forward. `bd dolt pull` now opens + leniently for this one refusal, the way `bd dolt commit` does for the + dirty-working-set refusal + ([#4566](https://github.com/gastownhall/beads/issues/4566)); every other gate + refusal still fails that open. This supersedes the [1.3.0] upgrade note's + statement that the pending-migration gate refuses `bd dolt pull` as well as + `bd dolt push`: for the data-behind stop it does not, by design. + + Commands that keep working through the stop say the same thing. `bd list`, + `bd ready` and `bd show` succeed against the old schema, and `bd dolt commit` + still commits the working set; on embedded storage all of them used to print + the blunt migrate-or-adopt coordination bullets while doing so — naming + `bd migrate --force && bd dolt push`, which on a data-behind clone applies + the migration this stop exists to prevent and then fails the push + non-fast-forward, and `bd bootstrap`, which no-ops against an existing + workspace. They now print the pull-first guidance, shape-branched the same + way the fatal refusal is, alongside the note that the command in hand + continued at the current schema. Refusals that are *not* the data-behind stop + keep the coordination bullets, which are right for them. + + `--json` callers get the same remedy the terminal does: `observed`, + `expected` and `options` describe the pull (a single `pull-first` option) + rather than the migrate-or-adopt decision that does not apply here, plus a + `data_behind_shape` of `fast-forward` or `diverged`. `fallback_reason` stays + `data-behind`. For the fast-forward shape on a non-shared store — one + unconditional option, no local commits, nothing discarded — + `human_decision_required` is now `false`, so an agent can run the pull + instead of stalling for approval of a step the same payload calls riskless. + It stays `true` for the diverged shape (the pull merges and can need conflict + resolution) and on a shared store (the follow-up consent step needs an + operator who can confirm every co-resident client is upgraded). On a shared + Dolt sql-server the guidance carries #5920's consequence — migrating promotes + the schema for every co-resident client, and clients still on an older bd + refuse the database until upgraded — and names the `bd migrate schema` + consent step the retry needs there. + + Both existing workarounds keep working unchanged: `BD_SMART_GATE=0` opts out + of the smart gate entirely (which means the blunt gate applies to `bd dolt + pull` too — that is what opting out is), and running `bd dolt pull` before the + first open after upgrading avoids the state to begin with. `bd migrate + --force` / `BD_ALLOW_REMOTE_MIGRATE=1` are still consulted before the gate and + still override it. + - **A dotted config key now round-trips: what `bd config set` writes, `bd config get` and bd's own readers find** ([#6578](https://github.com/gastownhall/beads/pull/6578), bd-zj95). Setting a diff --git a/cmd/bd/events_journal_construction_test.go b/cmd/bd/events_journal_construction_test.go index ac762dc30..bdfb78fbd 100644 --- a/cmd/bd/events_journal_construction_test.go +++ b/cmd/bd/events_journal_construction_test.go @@ -58,6 +58,7 @@ var storeConstructors = map[string]map[string]bool{ "OpenForReadOnlyCommand": true, "OpenForPreviewCommand": true, "OpenForWorkingSetReconcile": true, + "OpenForRemoteSync": true, }, "github.com/steveyegge/beads/internal/storage/uow": { "NewDoltServerUOWProvider": true, diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 7a52cffc3..1d2bbb6c1 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -324,6 +324,30 @@ func isWorkingSetReconcileCommand(cmd *cobra.Command) bool { return parent.Name() == "dolt" || parent.Name() == "vc" } +// isRemoteSyncCommand reports whether cmd is `bd dolt pull`: the one command +// the #6575 data-behind migrate-gate refusal tells the operator to run. +// +// It is the same deadlock isWorkingSetReconcileCommand breaks for #4566, one +// refusal over. The gate stops a data-behind clone from migrating and names +// `bd dolt pull` as the remedy — but the pull opens the store too, so it hit +// that refusal before it could clear its cause. On an embedded clone there is +// no external `dolt` binary to fall back to, which left the refused clone with +// exactly one exit: BD_ALLOW_REMOTE_MIGRATE=1, i.e. performing the migration +// the refusal exists to prevent. Opening via embeddeddolt.OpenForRemoteSync / +// dolt.Config.RemoteSyncOpen tolerates that ONE gate reason and nothing else. +// +// Deliberately just the pull, not `bd sync`: sync also pushes and can write +// issue rows, and a write against a stale schema is the hazard the gate is +// about. The pull only moves the commit graph, which is precisely the +// precondition the refusal is waiting on. +func isRemoteSyncCommand(cmd *cobra.Command) bool { + if cmd.Name() != "pull" { + return false + } + parent := cmd.Parent() + return parent != nil && parent.Name() == "dolt" +} + // isForcedMigrate reports whether cmd is `bd migrate` or `bd migrate schema` // invoked with --force: the operator confirming they are the single designated // migrator, so the remote-migrate gate (#4259) must not block this run's store @@ -1622,6 +1646,7 @@ var rootCmd = &cobra.Command{ DisableAutoStart: policy.disableAutoStart, BeadsDir: beadsDir, LenientOpen: isWorkingSetReconcileCommand(cmd), + RemoteSyncOpen: isRemoteSyncCommand(cmd), } // Load config to get database name and server connection settings. diff --git a/cmd/bd/remote_migrate_gate.go b/cmd/bd/remote_migrate_gate.go index 76076512b..eb0a2fb97 100644 --- a/cmd/bd/remote_migrate_gate.go +++ b/cmd/bd/remote_migrate_gate.go @@ -18,6 +18,49 @@ const ( sharedServersDocsURL = "https://github.com/gastownhall/beads/blob/main/docs/getting-started/upgrading.md#shared-servers" ) +// humanDecisionRequired reports whether this refusal is one an agent must stop +// and hand to a human, rather than act on. +// +// Every arm of this gate but one is a coordination or data-loss decision the +// process cannot make — which clone is the designated migrator, whether it is +// safe to discard unpushed work, whether every co-resident client is upgraded +// — so the answer is true by default, and was hard-coded true before the +// #6575 data-behind stop existed. +// +// The data-behind stop is the exception, and only in its narrowest shape. It +// carries a single option whose command is `bd dolt pull`, whose When is +// unconditional ("always, for this stop") and whose Risk is "none — a pure +// fast-forward"; AgentDirective says the same in prose. Leaving the field true +// there made this payload contradict itself: an agent keying on it stopped to +// ask a human to approve a step the rest of the payload calls unconditional +// and riskless — and the cost of that stall is the operator reaching for +// `bd migrate --force` instead, which is the #6575 wedge. +// +// Both narrowing conditions are deliberate, and getting either backwards would +// be worse than leaving the field true: +// +// - DataDiverged: the clone has commits of its own, so the pull MERGES. +// Options() already says so — its Risk for this shape is conflicts "that +// must be resolved before it completes", with a `--strategy ours|theirs` +// choice behind them. A merge that can require conflict resolution, and a +// strategy that decides whose rows win, is exactly a human decision. +// AgentDirective agrees ("surface a conflict outcome to the operator +// rather than forcing past it"). Stays true. +// +// - Shared: the pull itself is still safe, but on a shared store the payload +// carries a SECOND option, migrate-shared-after-pulling, whose When +// requires the operator to confirm every co-resident bd client is upgraded +// (#5920) — unobservable from this process. This field describes the +// payload, not its first option, so one option needing a human makes the +// answer true. AgentDirective agrees ("Do NOT auto-run it"). Stays true. +// +// What is NOT narrowed on: the When string, which is "always, for this stop" +// in both shapes and correctly so — the pull is the only way forward either +// way. It is the Risk, not the applicability, that the diverged shape changes. +func humanDecisionRequired(e *schema.RemoteMigrateGateError) bool { + return !(e.IsDataBehind() && !e.DataDiverged && !e.Shared) +} + // handleRemoteMigrateGateJSON renders the #4259 remote-migrate gate error as a // structured JSON error block for agent consumption. // @@ -44,7 +87,13 @@ func handleRemoteMigrateGateJSON(e *schema.RemoteMigrateGateError) { if globalFlag { sharedConsent = schema.SharedConsentCommandGlobal } - retargetShared := globalFlag && e.Decision == "shared-no-remote" + // The #6575 data-behind stop on a SHARED store carries the same consent + // verb in its second option (migrate-shared-after-pulling), so it needs + // the same retarget: under --global the project-scoped verb would + // consent the wrong database and leave the refusal in place. It reaches + // here through the default arm (empty Decision), so the Decision test + // alone would miss it. + retargetShared := globalFlag && (e.Decision == "shared-no-remote" || (e.IsDataBehind() && e.Shared)) opts := make([]map[string]interface{}, 0, len(e.Options())) for _, o := range e.Options() { @@ -76,7 +125,7 @@ func handleRemoteMigrateGateJSON(e *schema.RemoteMigrateGateError) { "latest_version": e.LatestVersion, "pending": e.Pending, "severity": "blocking", - "human_decision_required": true, + "human_decision_required": humanDecisionRequired(e), "observed": fmt.Sprintf("%d pending schema migration(s) and a configured remote", e.Pending), "expected": "exactly one designated clone migrates and publishes; every other clone adopts the result", "options": opts, @@ -122,6 +171,29 @@ func handleRemoteMigrateGateJSON(e *schema.RemoteMigrateGateError) { if e.FallbackReason != "" { gate["fallback_reason"] = e.FallbackReason } + // gastownhall/beads#6575: the data-behind stop is a blunt stop with + // a specific, verified, one-command remedy, and the blunt + // observed/expected pair above describes a decision that does not + // apply to it. Leaving them in place pointed an agent at + // `bd migrate --force` (the bug) followed by a `bd dolt push` that + // is guaranteed to be rejected non-fast-forward while the clone is + // still behind. e.Options() already returns the pull-first option + // for this reason, so options/hint are correct above; these two + // fields are what was still lying. decision stays absent — the + // stop has no Decision, and fallback_reason is the key an agent + // matches on. + if e.IsDataBehind() { + observed := "this clone is level with the remote on schema but is BEHIND it in commits it has not pulled, so the pending migration would land on a history missing them (#6575, #6368)" + if e.DataDiverged { + observed += "; it also has commits of its own, so pulling merges rather than fast-forwards" + gate["data_behind_shape"] = "diverged" + } else { + observed += "; it has no commits of its own, so pulling is a pure fast-forward" + gate["data_behind_shape"] = "fast-forward" + } + gate["observed"] = observed + gate["expected"] = "run `" + schema.DataBehindRemedyCommand + "` first; the migration is only allowed once this clone has nothing left to pull" + } } m["remote_migrate_gate"] = gate } diff --git a/cmd/bd/remote_sync_open_test.go b/cmd/bd/remote_sync_open_test.go new file mode 100644 index 000000000..2059b682d --- /dev/null +++ b/cmd/bd/remote_sync_open_test.go @@ -0,0 +1,354 @@ +package main + +import ( + "bytes" + "encoding/json" + "io" + "os" + "strings" + "testing" + + "github.com/spf13/cobra" + "github.com/steveyegge/beads/internal/storage/schema" +) + +// TestIsRemoteSyncCommand pins which command gets the #6575 narrow lenient +// open. It is exactly `bd dolt pull` — the command the data-behind refusal +// prescribes, which before this fix was blocked by the refusal that prescribed +// it (the #4566 deadlock shape, one refusal over). +// +// The negative cases matter as much as the positive one: `bd sync` also pushes +// and can write issue rows, and a write against a stale schema is the hazard +// the gate exists for, so it is deliberately NOT enrolled. A bare `pull` with +// no `dolt` parent must not match either, or any future `bd pull` would +// inherit an exemption nobody reviewed. +func TestIsRemoteSyncCommand(t *testing.T) { + build := func(parent, child string) *cobra.Command { + c := &cobra.Command{Use: child} + if parent != "" { + p := &cobra.Command{Use: parent} + p.AddCommand(c) + } + return c + } + cases := []struct { + parent, child string + want bool + }{ + {parent: "dolt", child: "pull", want: true}, + {parent: "dolt", child: "push", want: false}, + {parent: "dolt", child: "commit", want: false}, + {parent: "vc", child: "pull", want: false}, + {parent: "", child: "pull", want: false}, + {parent: "", child: "sync", want: false}, + } + for _, c := range cases { + name := c.parent + " " + c.child + t.Run(strings.TrimSpace(name), func(t *testing.T) { + if got := isRemoteSyncCommand(build(c.parent, c.child)); got != c.want { + t.Errorf("isRemoteSyncCommand(%q) = %v, want %v", name, got, c.want) + } + }) + } +} + +// TestIsRemoteSyncCommandMatchesTheRealPullCommand guards the classification +// against the real command tree rather than a hand-built stand-in, so renaming +// or re-parenting `bd dolt pull` cannot silently drop the exemption and +// re-wedge the refused cohort. +func TestIsRemoteSyncCommandMatchesTheRealPullCommand(t *testing.T) { + if !isRemoteSyncCommand(doltPullCmd) { + t.Error("isRemoteSyncCommand(doltPullCmd) = false; the data-behind refusal's own remedy would be blocked again (#6575)") + } + if isRemoteSyncCommand(doltPushCmd) { + t.Error("isRemoteSyncCommand(doltPushCmd) = true; only the pull may open past the refusal") + } + if isRemoteSyncCommand(doltCommitCmd) { + t.Error("isRemoteSyncCommand(doltCommitCmd) = true; the commit has its own #4566 classification") + } +} + +// TestHandleRemoteMigrateGateJSON_DataBehind covers the review's F4: the +// agent-facing block for the #6575 stop used to carry only an opaque +// `fallback_reason: "data-behind"` while observed/expected/options stayed the +// blunt migrate-or-adopt default — so an agent reading the documented contract +// surfaced `bd migrate --force` (the bug) followed by a `bd dolt push` that is +// guaranteed to be rejected non-fast-forward while the clone is still behind, +// plus a `bd bootstrap` that no-ops. The prose remedy existed only in text +// --json never emitted. +func TestHandleRemoteMigrateGateJSON_DataBehind(t *testing.T) { + capture := func(t *testing.T, gate *schema.RemoteMigrateGateError) map[string]interface{} { + t.Helper() + origStderr := os.Stderr + r, w, pipeErr := os.Pipe() + if pipeErr != nil { + t.Fatal(pipeErr) + } + os.Stderr = w + defer func() { os.Stderr = origStderr }() + handleRemoteMigrateGateJSON(gate) + _ = w.Close() + var buf bytes.Buffer + if _, err := io.Copy(&buf, r); err != nil { + t.Fatal(err) + } + _ = r.Close() + var parsed map[string]interface{} + if err := json.Unmarshal(buf.Bytes(), &parsed); err != nil { + t.Fatalf("json.Unmarshal stderr: %v\nstderr was: %s", err, buf.String()) + } + return parsed + } + + for _, tc := range []struct { + name string + diverged bool + wantShape string + }{ + {name: "fast-forward shape", diverged: false, wantShape: "fast-forward"}, + {name: "diverged shape", diverged: true, wantShape: "diverged"}, + } { + t.Run(tc.name, func(t *testing.T) { + gate := &schema.RemoteMigrateGateError{ + CurrentVersion: 66, LatestVersion: 67, Pending: 1, + FallbackReason: "data-behind", + DataDiverged: tc.diverged, + } + parsed := capture(t, gate) + obj, ok := parsed["remote_migrate_gate"].(map[string]interface{}) + if !ok { + t.Fatalf("remote_migrate_gate missing or wrong type: %T", parsed["remote_migrate_gate"]) + } + // The machine-readable reason an agent matches on is unchanged. + if got, _ := obj["fallback_reason"].(string); got != "data-behind" { + t.Errorf("fallback_reason = %v, want \"data-behind\"", obj["fallback_reason"]) + } + if _, ok := obj["decision"]; ok { + t.Errorf("this is a blunt stop and must not carry a decision key, got %v", obj["decision"]) + } + if got, _ := obj["data_behind_shape"].(string); got != tc.wantShape { + t.Errorf("data_behind_shape = %v, want %q", obj["data_behind_shape"], tc.wantShape) + } + // observed/expected must describe THIS stop, not the + // designated-migrator decision that does not apply to it. + observed, _ := obj["observed"].(string) + if !strings.Contains(observed, "BEHIND") { + t.Errorf("observed = %q, want the behind-the-remote framing", observed) + } + expected, _ := obj["expected"].(string) + if !strings.Contains(expected, schema.DataBehindRemedyCommand) { + t.Errorf("expected = %q, want the pull-first remedy", expected) + } + if strings.Contains(expected, "designated clone migrates") { + t.Errorf("expected = %q, still the blunt migrate-or-adopt framing", expected) + } + // options must be the single measured remedy, and must not hand an + // agent either of the two dead ends. + rawOpts, ok := obj["options"].([]interface{}) + if !ok || len(rawOpts) != 1 { + t.Fatalf("options = %v, want exactly one (the pull)", obj["options"]) + } + o, _ := rawOpts[0].(map[string]interface{}) + if id, _ := o["id"].(string); id != "pull-first" { + t.Errorf("option id = %v, want \"pull-first\"", o["id"]) + } + cmds, _ := o["commands"].([]interface{}) + if len(cmds) != 1 || cmds[0] != schema.DataBehindRemedyCommand { + t.Errorf("option commands = %v, want [%q]", cmds, schema.DataBehindRemedyCommand) + } + if o["when"] == nil || o["risk"] == nil { + t.Errorf("option missing when/risk: %v", o) + } + // The top-level hint stays the non-runnable directive, and must + // name the remedy rather than the escape hatch. + hint, _ := parsed["hint"].(string) + if hint != gate.AgentDirective() { + t.Errorf("hint = %q, want the agent directive", hint) + } + if !strings.Contains(hint, schema.DataBehindRemedyCommand) { + t.Errorf("hint = %q, want it to name the remedy", hint) + } + if hint == gate.EscapeHint() { + t.Errorf("hint must not be the runnable escape command %q", gate.EscapeHint()) + } + // Nothing anywhere in the block may offer the migrate/push pair or + // the no-op bootstrap for this state. + blob, err := json.Marshal(parsed) + if err != nil { + t.Fatal(err) + } + for _, banned := range []string{"bd bootstrap", "bd migrate --force"} { + if strings.Contains(string(blob), banned) { + t.Errorf("the data-behind block offers %q, which was measured not to work from this state:\n%s", banned, blob) + } + } + }) + } + + // A shared, data-behind refusal carries the #5920 consent verb in its + // second option, and that verb is target-scoped: under --global the + // project-scoped `bd migrate schema` would consent the WRONG database and + // leave the refusal in place. This stop reaches the renderer through the + // default (empty-Decision) arm, so the retarget cannot be keyed on + // Decision alone the way shared-no-remote's is. + t.Run("shared under --global retargets the consent verb", func(t *testing.T) { + origGlobal := globalFlag + globalFlag = true + defer func() { globalFlag = origGlobal }() + + gate := &schema.RemoteMigrateGateError{ + CurrentVersion: 66, LatestVersion: 67, Pending: 1, + FallbackReason: "data-behind", + Shared: true, + } + parsed := capture(t, gate) + obj, ok := parsed["remote_migrate_gate"].(map[string]interface{}) + if !ok { + t.Fatalf("remote_migrate_gate missing or wrong type: %T", parsed["remote_migrate_gate"]) + } + rawOpts, ok := obj["options"].([]interface{}) + if !ok || len(rawOpts) != 2 { + t.Fatalf("options = %v, want the pull plus the shared consent step", obj["options"]) + } + consent, _ := rawOpts[1].(map[string]interface{}) + cmds, _ := consent["commands"].([]interface{}) + if len(cmds) != 1 || cmds[0] != schema.SharedConsentCommandGlobal { + t.Errorf("consent option commands = %v, want [%q] under --global", cmds, schema.SharedConsentCommandGlobal) + } + // The pull is target-agnostic and must NOT be rewritten. + pull, _ := rawOpts[0].(map[string]interface{}) + pullCmds, _ := pull["commands"].([]interface{}) + if len(pullCmds) != 1 || pullCmds[0] != schema.DataBehindRemedyCommand { + t.Errorf("pull option commands = %v, want [%q] unchanged", pullCmds, schema.DataBehindRemedyCommand) + } + }) +} + +// TestHandleRemoteMigrateGateJSON_HumanDecisionRequired pins the one arm of +// this payload where human_decision_required is false, and — more importantly +// — the three where it must stay true. +// +// The field was hard-coded true, which made the data-behind payload contradict +// itself: the single option it carries is annotated `when: "always, for this +// stop"` and `risk: "none — a pure fast-forward"`, and AgentDirective says the +// same, yet an agent keying on this field stopped to ask a human to approve an +// unconditional, riskless step. The cost of that stall is the operator +// reaching for `bd migrate --force` instead — the #6575 wedge. +// +// The two narrowings are the substance of the finding: a diverged pull MERGES +// and can need conflict resolution (and a `--strategy ours|theirs` choice), and +// a shared store carries a SECOND option whose precondition is operator +// confirmation that every co-resident client is upgraded (#5920), which this +// process cannot observe. Either one flipping to false would be worse than the +// contradiction it replaced. +func TestHandleRemoteMigrateGateJSON_HumanDecisionRequired(t *testing.T) { + capture := func(t *testing.T, gate *schema.RemoteMigrateGateError) map[string]interface{} { + t.Helper() + origStderr := os.Stderr + r, w, pipeErr := os.Pipe() + if pipeErr != nil { + t.Fatal(pipeErr) + } + os.Stderr = w + defer func() { os.Stderr = origStderr }() + handleRemoteMigrateGateJSON(gate) + _ = w.Close() + var buf bytes.Buffer + if _, err := io.Copy(&buf, r); err != nil { + t.Fatal(err) + } + _ = r.Close() + var parsed map[string]interface{} + if err := json.Unmarshal(buf.Bytes(), &parsed); err != nil { + t.Fatalf("json.Unmarshal stderr: %v\nstderr was: %s", err, buf.String()) + } + return parsed + } + + for _, tc := range []struct { + name string + gate *schema.RemoteMigrateGateError + want bool + why string + }{ + { + name: "data-behind, fast-forward, not shared", + gate: &schema.RemoteMigrateGateError{CurrentVersion: 66, LatestVersion: 67, Pending: 1, FallbackReason: "data-behind"}, + want: false, + why: "the single option is an unconditional, riskless, non-destructive pull", + }, + { + name: "data-behind, diverged", + gate: &schema.RemoteMigrateGateError{CurrentVersion: 66, LatestVersion: 67, Pending: 1, FallbackReason: "data-behind", DataDiverged: true}, + want: true, + why: "the pull merges and can need conflict resolution / a strategy choice", + }, + { + name: "data-behind, shared", + gate: &schema.RemoteMigrateGateError{CurrentVersion: 66, LatestVersion: 67, Pending: 1, FallbackReason: "data-behind", Shared: true}, + want: true, + why: "the second option needs operator confirmation that every co-resident client is upgraded (#5920)", + }, + { + name: "data-behind, diverged and shared", + gate: &schema.RemoteMigrateGateError{CurrentVersion: 66, LatestVersion: 67, Pending: 1, FallbackReason: "data-behind", DataDiverged: true, Shared: true}, + want: true, + why: "both narrowings apply", + }, + { + name: "blunt remote-backed stop", + gate: &schema.RemoteMigrateGateError{CurrentVersion: 66, LatestVersion: 67, Pending: 1}, + want: true, + why: "only ONE clone may migrate a shared remote — a coordination decision", + }, + { + name: "adopt", + gate: &schema.RemoteMigrateGateError{CurrentVersion: 66, LatestVersion: 67, Pending: 1, Decision: "adopt"}, + want: true, + why: "adoption is destructive", + }, + { + name: "fork-skew", + gate: &schema.RemoteMigrateGateError{CurrentVersion: 66, LatestVersion: 67, Pending: 1, Decision: "fork-skew"}, + want: true, + why: "picking a canonical clone discards the others' unpushed work", + }, + { + name: "shared-no-remote", + gate: &schema.RemoteMigrateGateError{CurrentVersion: 66, LatestVersion: 67, Pending: 1, Decision: "shared-no-remote", Shared: true}, + want: true, + why: "other clients' binary versions are not observable from this process", + }, + } { + t.Run(tc.name, func(t *testing.T) { + parsed := capture(t, tc.gate) + obj, ok := parsed["remote_migrate_gate"].(map[string]interface{}) + if !ok { + t.Fatalf("remote_migrate_gate missing or wrong type: %T", parsed["remote_migrate_gate"]) + } + got, ok := obj["human_decision_required"].(bool) + if !ok { + t.Fatalf("human_decision_required missing or not a bool: %T", obj["human_decision_required"]) + } + if got != tc.want { + t.Errorf("human_decision_required = %v, want %v — %s", got, tc.want, tc.why) + } + // The contradiction is only resolved if the rest of the payload + // still says what it said: a false here has to come with the + // unconditional, riskless single option it is claiming. + if !tc.want { + rawOpts, ok := obj["options"].([]interface{}) + if !ok || len(rawOpts) != 1 { + t.Fatalf("human_decision_required=false with options = %v; it may only be false for the single-option pull", obj["options"]) + } + o, _ := rawOpts[0].(map[string]interface{}) + if risk, _ := o["risk"].(string); !strings.HasPrefix(risk, "none") { + t.Errorf("human_decision_required=false but option risk = %q; want a risk-free option", risk) + } + if when, _ := o["when"].(string); !strings.HasPrefix(when, "always") { + t.Errorf("human_decision_required=false but option when = %q; want an unconditional option", when) + } + } + }) + } +} diff --git a/cmd/bd/store_factory.go b/cmd/bd/store_factory.go index c6cedd9bd..00f1afb32 100644 --- a/cmd/bd/store_factory.go +++ b/cmd/bd/store_factory.go @@ -109,6 +109,16 @@ func newDoltStore(ctx context.Context, cfg *dolt.Config) (s storage.DoltStorage, // branch is the embedded half of one policy, not the whole of it. return embeddeddolt.OpenForWorkingSetReconcile(ctx, cfg.BeadsDir, cfg.Database, "main") } + if cfg.RemoteSyncOpen { + // `bd dolt pull` must not be bricked by the #6575 data-behind gate + // refusal, because that refusal's entire remedy IS this pull: the gate + // stops a clone that is behind the remote from migrating and tells the + // operator to pull first, and an embedded clone has no external dolt + // binary to do it with. Same deadlock as #4566, and this open tolerates + // only that one gate reason (see openRemoteSync). The server arm above + // honors the same cfg.RemoteSyncOpen inside dolt.New. + return embeddeddolt.OpenForRemoteSync(ctx, cfg.BeadsDir, cfg.Database, "main") + } return embeddeddolt.Open(ctx, cfg.BeadsDir, cfg.Database, "main") } diff --git a/internal/storage/dolt/smart_gate_data_behind_test.go b/internal/storage/dolt/smart_gate_data_behind_test.go new file mode 100644 index 000000000..54d0395d3 --- /dev/null +++ b/internal/storage/dolt/smart_gate_data_behind_test.go @@ -0,0 +1,277 @@ +package dolt + +import ( + "context" + "database/sql" + "errors" + "fmt" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/doltutil" + "github.com/steveyegge/beads/internal/storage/schema" + "github.com/steveyegge/beads/internal/storage/versioncontrolops" +) + +// TestDoltNew_SmartRemoteMigrateGate_DataBehindBlocks_RealDolt is the +// regression test for gastownhall/beads#6575: the smart gate's first-mover +// verdict used to be computed from schema facts alone, so a clone level with +// the remote on SCHEMA but behind it in DATA commits was classified a safe +// first-mover and auto-migrated in place — minting local-only schema commits +// on a HEAD missing the remote's data commits, which is one route into the +// #6368 events pull refusal. +// +// The fixture builds a genuinely data-behind clone against a real Dolt server +// — the behind-ness is MEASURED here (the same ahead/behind primitive the gate +// consults), never stubbed, because a test that fakes the fact the bug is about +// proves nothing: +// +// - "source" regresses its schema cursor one migration below latest (commit +// C0) and publishes C0 to a file:// origin. +// - "laggingClone" is a genuine server-side DOLT_CLONE of origin at C0. +// - "source" then makes a pure DATA commit C1 (a config row — no schema +// change at all, so the two stay at the SAME schema version) and pushes. +// - laggingClone runs CALL DOLT_FETCH('origin') only, so its cached +// remotes/origin/main advances to C1 while its own branch HEAD stays at C0. +// +// laggingClone is now schema-level with the remote (equal content hashes, +// equal max version — the smart gate's first-mover precondition) and yet one +// data commit behind it. The gate must refuse with the pull-first fallback +// reason rather than auto-migrate. +// +// BOTH shapes of that state are covered, because the review of the first fix +// measured that the narrower one was the rarer one (F2). The subtests differ +// only in whether the clone also makes a local commit of its own before the +// gate runs: +// +// - ahead=0, behind=1 — a strict ancestor; `bd dolt pull` fast-forwards. +// - ahead=1, behind=1 — diverged; `bd dolt pull` merges. Since bd +// auto-commits every write, this is the ordinary multi-machine state, and +// it reaches the identical wedge. At the first fix's head this shape +// printed "safe first-mover" and migrated. +// +// The second half of each subtest then proves the remedy loop self-heals and +// that the fix does not over-refuse: once the clone has nothing left to pull, +// the very same gate call auto-migrates it as a true first-mover. +func TestDoltNew_SmartRemoteMigrateGate_DataBehindBlocks_RealDolt(t *testing.T) { + for _, tc := range []struct { + name string + // localCommits is how many commits of its own the lagging clone makes + // after fetching: 0 is the strict-ancestor shape, 1 the diverged one. + localCommits int + wantDiverged bool + // wantRemedy is the substring the guidance must carry for this shape, + // so a clone is never told its pull will fast-forward when it merges. + wantRemedy string + }{ + {name: "strict ancestor", localCommits: 0, wantRemedy: "pure fast-forward"}, + {name: "diverged", localCommits: 1, wantDiverged: true, wantRemedy: "MERGES"}, + } { + t.Run(tc.name, func(t *testing.T) { + runDataBehindGateCase(t, tc.localCommits, tc.wantDiverged, tc.wantRemedy) + }) + } +} + +func runDataBehindGateCase(t *testing.T, localCommits int, wantDiverged bool, wantRemedy string) { + skipIfNoDolt(t) + t.Setenv(schema.SmartGateEnv, "1") + t.Setenv(schema.AllowRemoteMigrateEnv, "0") + + ctx, cancel := testContext(t) + defer cancel() + + latest := schema.LatestVersion() + floor := schema.LastNonDeterministicMigration + if latest-1 < floor { + t.Skipf("latest=%d too close to floor=%d to build the fixture", latest, floor) + } + pBehind := latest - 1 + + tmpDir := t.TempDir() + sourceDB := uniqueTestDBName(t) + + source, err := New(ctx, &Config{ + Path: tmpDir, + CommitterName: "test", + CommitterEmail: "test@example.com", + Database: sourceDB, + CreateIfMissing: true, + MaxOpenConns: 1, // single session so working-set regressions are visible to the gate reads + }) + if err != nil { + t.Fatalf("New (source): %v", err) + } + sdb := source.db + defer func() { + dropCtx, dropCancel := context.WithTimeout(context.Background(), 5*testTimeout) + defer dropCancel() + _, _ = sdb.ExecContext(dropCtx, fmt.Sprintf("DROP DATABASE IF EXISTS `%s`", sourceDB)) + source.Close() + }() + + mustExec := func(stage string, db *sql.DB, q string, args ...any) { + t.Helper() + if _, err := db.ExecContext(ctx, q, args...); err != nil { + t.Fatalf("%s: %v", stage, err) + } + } + + remoteURL := "file://" + filepath.Join(tmpDir, "data-behind-remote") + + // --- source: regress to pBehind (commit C0) and publish it. --- + mustExec("regress to pBehind", sdb, "DELETE FROM schema_migrations WHERE version = ?", latest) + mustExec("stage C0", sdb, "CALL DOLT_ADD('-A')") + mustExec("commit C0", sdb, "CALL DOLT_COMMIT('--allow-empty', '-m', 'test: regress to pBehind (C0)')") + mustExec("add remote", sdb, "CALL DOLT_REMOTE('add', 'origin', ?)", remoteURL) + mustExec("push C0", sdb, "CALL DOLT_PUSH('origin', 'main')") + + // --- laggingClone: a genuine server-side clone of origin at C0. --- + laggingDB := uniqueTestDBName(t) + mustExec("clone lagging peer", sdb, "CALL DOLT_CLONE(?, ?)", remoteURL, laggingDB) + defer func() { + dropCtx, dropCancel := context.WithTimeout(context.Background(), 5*testTimeout) + defer dropCancel() + _, _ = sdb.ExecContext(dropCtx, fmt.Sprintf("DROP DATABASE IF EXISTS `%s`", laggingDB)) + }() + + laggingConn, err := sql.Open("mysql", doltutil.ServerDSN{ + Host: "127.0.0.1", Port: testServerPort, User: "root", Database: laggingDB, + }.String()) + if err != nil { + t.Fatalf("open laggingClone connection: %v", err) + } + defer laggingConn.Close() + laggingConn.SetMaxOpenConns(1) + + // --- source: a pure DATA commit C1 (no schema change) and publish. The + // schema cursor stays at pBehind on BOTH sides, so the clone lands on the + // gate's equal-version first-mover path while being a data commit behind. --- + mustExec("source data edit", sdb, "REPLACE INTO config (`key`, value) VALUES ('data-behind-marker', 'C1')") + mustExec("commit C1", sdb, "CALL DOLT_COMMIT('-Am', 'test: pure data commit (C1)')") + mustExec("push C1", sdb, "CALL DOLT_PUSH('origin', 'main')") + + // --- laggingClone: fetch only (no merge) so its cached ref advances but + // its own branch HEAD stays at C0. --- + mustExec("fetch (no merge)", laggingConn, "CALL DOLT_FETCH('origin')") + + // --- and, for the diverged shape, commits of its own on top of C0. bd + // auto-commits every write, so this is what an ordinary local `bd create` + // leaves behind. --- + for i := 0; i < localCommits; i++ { + mustExec("clone data edit", laggingConn, + "REPLACE INTO config (`key`, value) VALUES ('local-marker', ?)", fmt.Sprintf("L%d", i)) + mustExec("clone local commit", laggingConn, "CALL DOLT_COMMIT('-Am', 'test: local-only data commit')") + } + + // Fixture sanity: the clone's schema cursor equals the remote's (the + // first-mover precondition), not below it. + var laggingCurrent int + if err := laggingConn.QueryRowContext(ctx, + "SELECT COALESCE(MAX(version), 0) FROM schema_migrations").Scan(&laggingCurrent); err != nil { + t.Fatalf("read laggingClone current version: %v", err) + } + if laggingCurrent != pBehind { + t.Fatalf("laggingClone current version = %d, want %d", laggingCurrent, pBehind) + } + var remoteCurrent int + if err := laggingConn.QueryRowContext(ctx, + "SELECT COALESCE(MAX(version), 0) FROM schema_migrations AS OF 'remotes/origin/main'").Scan(&remoteCurrent); err != nil { + t.Fatalf("read cached remote current version: %v", err) + } + if remoteCurrent != laggingCurrent { + t.Fatalf("cached remote version = %d, want %d (equal-version first-mover path)", remoteCurrent, laggingCurrent) + } + + // The load-bearing measurement: the clone IS behind its own cached remote + // ref, in exactly the shape this subtest claims, read through the same + // primitive the gate consults. + ahead, behind, err := versioncontrolops.LocalAheadBehind(ctx, laggingConn, "remotes/origin/main") + if err != nil { + t.Fatalf("LocalAheadBehind: %v", err) + } + if behind < 1 || ahead != localCommits { + t.Fatalf("fixture position = ahead %d, behind %d; want ahead %d, behind >= 1", ahead, behind, localCommits) + } + // The strict-ancestor relation is the NARROWER fact the first fix used, and + // the diverged shape is precisely where it reads false while the clone is + // still behind — pin that, so this subtest cannot silently stop covering + // the widening it exists for. + strictAncestor, err := versioncontrolops.LocalIsStrictAncestorOf(ctx, laggingConn, "remotes/origin/main") + if err != nil { + t.Fatalf("LocalIsStrictAncestorOf: %v", err) + } + if want := localCommits == 0; strictAncestor != want { + t.Fatalf("LocalIsStrictAncestorOf = %v, want %v for ahead=%d", strictAncestor, want, ahead) + } + // And it is behind for data reasons only — the working set is clean, so the + // refusal below cannot be attributed to local dirt. + clean, err := versioncontrolops.WorkingSetClean(ctx, laggingConn) + if err != nil { + t.Fatalf("WorkingSetClean: %v", err) + } + if !clean { + t.Fatalf("fixture working set is dirty; the data-behind refusal must not depend on dirt") + } + + // #6575: the gate must refuse rather than auto-migrate this clone. + err = schema.CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt( + ctx, laggingConn, "origin", nil, realFastForwardAdopter()) + var gateErr *schema.RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("data-behind clone (ahead=%d): expected the gate to refuse, got %v (#6575: schema parity is not proof of first-mover status)", ahead, err) + } + if gateErr.Decision != "" { + t.Errorf("Decision = %q, want \"\" (the blunt #4515 stop)", gateErr.Decision) + } + if !gateErr.IsDataBehind() { + t.Errorf("FallbackReason = %q, want %q", gateErr.FallbackReason, "data-behind") + } + if gateErr.DataDiverged != wantDiverged { + t.Errorf("DataDiverged = %v, want %v", gateErr.DataDiverged, wantDiverged) + } + msg := gateErr.UserMessage() + // The remedy has to be there, and has to be the one measured to work from + // this state: the refusal's own open classification lets `bd dolt pull` + // through (see embeddeddolt's TestEmbeddedOpenForRemoteSync_DataBehind). + if !strings.Contains(msg, schema.DataBehindRemedyCommand) { + t.Errorf("UserMessage does not carry the pull-first remedy:\n%s", msg) + } + if !strings.Contains(msg, wantRemedy) { + t.Errorf("UserMessage does not describe this shape's pull (%q):\n%s", wantRemedy, msg) + } + + // No data motion: the refusal must not have moved the clone's HEAD. + aheadAfter, behindAfter, err := versioncontrolops.LocalAheadBehind(ctx, laggingConn, "remotes/origin/main") + if err != nil { + t.Fatalf("compare laggingClone HEAD to origin/main after refusal: %v", err) + } + if aheadAfter != ahead || behindAfter != behind { + t.Errorf("after refusal: ahead=%d behind=%d, want ahead=%d behind=%d (no data motion)", + aheadAfter, behindAfter, ahead, behind) + } + + // --- remedy loop: `bd dolt pull` (a fast-forward for the strict-ancestor + // shape, a merge for the diverged one) leaves the clone with nothing left to + // pull, and the SAME gate call then auto-migrates it as a true first-mover. + // This is what keeps the fix from over-refusing. --- + if localCommits == 0 { + mustExec("pull (fast-forward)", laggingConn, "CALL DOLT_MERGE('--ff-only', 'remotes/origin/main')") + } else { + mustExec("pull (merge)", laggingConn, "CALL DOLT_MERGE('remotes/origin/main')") + } + + _, behindAfterPull, err := versioncontrolops.LocalAheadBehind(ctx, laggingConn, "remotes/origin/main") + if err != nil { + t.Fatalf("LocalAheadBehind after pull: %v", err) + } + if behindAfterPull != 0 { + t.Fatalf("clone is still behind by %d after the pull; fixture broken", behindAfterPull) + } + + if err := schema.CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt( + ctx, laggingConn, "origin", nil, realFastForwardAdopter()); err != nil { + t.Fatalf("the first-mover auto-migrate must be allowed once there is nothing left to pull, got %v", err) + } +} diff --git a/internal/storage/dolt/smart_remote_migrate_gate_test.go b/internal/storage/dolt/smart_remote_migrate_gate_test.go index e41ee90cf..a862155e4 100644 --- a/internal/storage/dolt/smart_remote_migrate_gate_test.go +++ b/internal/storage/dolt/smart_remote_migrate_gate_test.go @@ -141,6 +141,9 @@ func realFastForwardAdopter() *schema.FastForwardAdopter { WorkingSetClean: func(ctx context.Context, db schema.DBConn) (bool, error) { return versioncontrolops.WorkingSetClean(ctx, db) }, + AheadBehind: func(ctx context.Context, db schema.DBConn, ref string) (int, int, error) { + return versioncontrolops.LocalAheadBehind(ctx, db, ref) + }, FastForward: func(ctx context.Context, db schema.DBConn, ref string) error { return versioncontrolops.FastForwardAdopt(ctx, db, ref) }, diff --git a/internal/storage/dolt/store.go b/internal/storage/dolt/store.go index 966acedcb..bc9c17968 100644 --- a/internal/storage/dolt/store.go +++ b/internal/storage/dolt/store.go @@ -382,6 +382,15 @@ type Config struct { // open of a clean database converges normally. LenientOpen bool + // RemoteSyncOpen is LenientOpen's narrow sibling for the #6575 + // data-behind gate refusal: it tolerates ONLY that refusal (and only that + // refusal — not the dirty-table guard, not any other gate reason), because + // the refusal's documented remedy is `bd dolt pull`, which opens the store + // and so hit the refusal that prescribed it. Set for the remote-sync + // commands that can clear the refused precondition. Honored in embedded + // (openRemoteSync) and server mode alike. + RemoteSyncOpen bool + // Server connection options ServerSocket string // Unix domain socket path (overrides Host/Port when set) ServerHost string // Server host (default: 127.0.0.1) @@ -2041,7 +2050,9 @@ func newServerMode(ctx context.Context, cfg *Config) (*DoltStore, error) { if !cfg.ReadOnly && !cfg.Gateway { applied, err := store.initSchema(ctx, dbFacts.bootstrapHeal) if err != nil { - if !cfg.LenientOpen || !warnLenientOpenRefusal(err) { + tolerated := (cfg.LenientOpen && warnLenientOpenRefusal(err)) || + (cfg.RemoteSyncOpen && warnRemoteSyncOpenRefusal(err)) + if !tolerated { return nil, fmt.Errorf("failed to initialize schema: %w", err) } // A tolerated refusal still reports what the aborted pass @@ -2118,6 +2129,35 @@ func warnLenientOpenRefusal(err error) bool { return false } +// warnRemoteSyncOpenRefusal reports whether a remote-sync open +// (Config.RemoteSyncOpen) may continue past err instead of failing, warning on +// stderr when it may. +// +// It is warnLenientOpenRefusal's narrow sibling and tolerates exactly one +// refusal: the #6575 data-behind remote-migrate gate stop, whose documented +// remedy is `bd dolt pull` — a command that opens the store and therefore hit +// the very refusal that prescribed it. That is the #4566 deadlock shape, and +// #4566's own fix (a lenient open for `bd dolt commit`) is the precedent. +// +// Everything else — every other gate reason, the dirty-table guard, and any +// other migration failure — still fails the open here, exactly as on a strict +// open. A pull cannot resolve a fork skew, a below-floor database or a +// shared-store consent decision, so letting it through those refusals would +// buy nothing and hide them. +func warnRemoteSyncOpenRefusal(err error) bool { + var gateErr *schema.RemoteMigrateGateError + if errors.As(err, &gateErr) && gateErr.IsDataBehind() { + fmt.Fprintf(os.Stderr, + "Warning: %s"+ + " Remote-sync command: continuing on schema v%d without migrating, so this\n"+ + " pull can bring in the commits this clone is behind on. Re-run the command\n"+ + " you were blocked on once it completes.\n", + gateErr.UserMessage(), gateErr.CurrentVersion) + return true + } + return false +} + var ( cleanServerCircuitState = CleanStaleCircuitBreakerFiles newServerCircuitBreaker = maybeNewCircuitBreaker @@ -2913,6 +2953,13 @@ func (s *DoltStore) initSchema(ctx context.Context, bootstrapHeal *schema.FreshB IsStrictAncestor: func(ctx context.Context, db schema.DBConn, ref string) (bool, error) { return versioncontrolops.LocalIsStrictAncestorOf(ctx, db, ref) }, + // The raw counts the equal-version data-behind check needs + // (gastownhall/beads#6575): behind >= 1 whatever ahead is, plus + // which shape it is so the refusal names the pull the operator + // will actually get. + AheadBehind: func(ctx context.Context, db schema.DBConn, ref string) (int, int, error) { + return versioncontrolops.LocalAheadBehind(ctx, db, ref) + }, WorkingSetClean: func(ctx context.Context, db schema.DBConn) (bool, error) { return versioncontrolops.WorkingSetClean(ctx, db) }, diff --git a/internal/storage/embeddeddolt/cache.go b/internal/storage/embeddeddolt/cache.go index 1e4e50f6c..72d9166e7 100644 --- a/internal/storage/embeddeddolt/cache.go +++ b/internal/storage/embeddeddolt/cache.go @@ -54,6 +54,19 @@ func OpenForWorkingSetReconcile(ctx context.Context, beadsDir, database, branch return openCached(ctx, beadsDir, database, branch, openWorkingSetReconcile) } +// OpenForRemoteSync opens like Open, except that the #6575 data-behind +// remote-migrate gate refusal — and only that one — skips the pending +// migrations with a stderr warning instead of failing the open: that refusal's +// entire remedy is `bd dolt pull`, which opens the store itself and so hit the +// refusal that prescribed it, leaving the refused clone with no in-band exit at +// all (the same deadlock #4566 broke for `bd dolt commit`). Every other gate +// refusal, the dirty-table guard and the dependency re-key still fail this open +// exactly as they fail a strict one. The returned store is otherwise a normal +// writable store. +func OpenForRemoteSync(ctx context.Context, beadsDir, database, branch string) (*EmbeddedDoltStore, error) { + return openCached(ctx, beadsDir, database, branch, openRemoteSync) +} + func openCached(ctx context.Context, beadsDir, database, branch string, intent openIntent) (*EmbeddedDoltStore, error) { key, err := cacheKey(beadsDir) if err != nil { @@ -65,8 +78,9 @@ func openCached(ctx context.Context, beadsDir, database, branch string, intent o // Cache hit: the requested intent is ignored - the store keeps // whatever intent it was opened with on the slow path below. This is // safe today because intent is derived once per process from the - // command classification (isReadOnlyCommand / isWorkingSetReconcileCommand - // in cmd/bd/main.go), so a single process never opens the same data + // command classification (isReadOnlyCommand / + // isWorkingSetReconcileCommand / isRemoteSyncCommand in + // cmd/bd/main.go), so a single process never opens the same data // directory under two different intents, and autoMigrateOnVersionBump // (cmd/bd/version_tracking.go) always opens its own openStrict store // and closes it before the main command's open runs, so it never diff --git a/internal/storage/embeddeddolt/remote_sync_gate_test.go b/internal/storage/embeddeddolt/remote_sync_gate_test.go new file mode 100644 index 000000000..adb140be6 --- /dev/null +++ b/internal/storage/embeddeddolt/remote_sync_gate_test.go @@ -0,0 +1,381 @@ +//go:build cgo + +package embeddeddolt_test + +import ( + "database/sql" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/embeddeddolt" + "github.com/steveyegge/beads/internal/storage/schema" +) + +// buildDataBehindEmbedded leaves the embedded database at beadsDir in the +// #6575 state: level with its own cached remotes/origin/main on SCHEMA (both +// at latest-1, so the smart gate takes its equal-version first-mover path) and +// BEHIND it by one pure DATA commit. extraLocalCommits > 0 additionally gives +// it commits of its own, producing the diverged shape (ahead >= 1, behind >= 1). +// +// The behind-ness is produced the way the field cohort reaches it — a real +// file:// remote that has commits this history does not — rather than by +// stubbing the ancestry fact. A single database can build it because pushing +// C1 and then rewinding local HEAD to C0 leaves the remote-tracking ref ahead +// of the branch, which is exactly what a fetch-without-merge produces on a +// clone; no second engine is needed. +func buildDataBehindEmbedded(t *testing.T, beadsDir string, extraLocalCommits int) { + t.Helper() + ctx := t.Context() + dataDir := filepath.Join(beadsDir, "embeddeddolt") + + store, err := embeddeddolt.Open(ctx, beadsDir, "testdb", "main") + if err != nil { + t.Fatalf("Open (create): %v", err) + } + store.Close() + + db, cleanup, err := embeddeddolt.OpenSQL(ctx, dataDir, "testdb", "main") + if err != nil { + t.Fatalf("OpenSQL: %v", err) + } + defer func() { _ = cleanup() }() + + mustExec := func(stage, q string, args ...any) { + t.Helper() + if _, err := db.ExecContext(ctx, q, args...); err != nil { + t.Fatalf("%s: %v", stage, err) + } + } + + // C0: regress the schema cursor one migration below latest — the "old + // binary" state the 1.3.0 upgrade cohort is in — and publish it. + mustExec("regress schema cursor", "DELETE FROM schema_migrations WHERE version = ?", schema.LatestVersion()) + mustExec("stage C0", "CALL DOLT_ADD('-A')") + mustExec("commit C0", "CALL DOLT_COMMIT('--allow-empty', '-m', 'test: regress schema cursor (C0)')") + + var c0 string + if err := db.QueryRowContext(ctx, "SELECT commit_hash FROM dolt_log LIMIT 1").Scan(&c0); err != nil { + t.Fatalf("read C0 hash: %v", err) + } + + mustExec("add remote", "CALL DOLT_REMOTE('add', 'origin', ?)", + "file://"+filepath.Join(t.TempDir(), "remote")) + mustExec("push C0", "CALL DOLT_PUSH('origin', 'main')") + + // C1: a pure DATA commit, published. No schema change, so both sides stay + // at latest-1 and the gate lands on its equal-version path. + mustExec("data edit", "REPLACE INTO config (`key`, value) VALUES ('data-behind-marker', 'C1')") + mustExec("commit C1", "CALL DOLT_COMMIT('-Am', 'test: pure data commit (C1)')") + mustExec("push C1", "CALL DOLT_PUSH('origin', 'main')") + + // Rewind the branch to C0. remotes/origin/main stays at C1, so this + // history is now behind its cached remote ref by one data commit. + mustExec("rewind to C0", "CALL DOLT_RESET('--hard', ?)", c0) + + for i := 0; i < extraLocalCommits; i++ { + mustExec("local data edit", "REPLACE INTO config (`key`, value) VALUES ('local-marker', ?)", i) + mustExec("local commit", "CALL DOLT_COMMIT('-Am', 'test: local-only data commit')") + } + + assertAheadBehind(t, db, extraLocalCommits, 1) +} + +// assertAheadBehind pins the fixture's measured position against +// remotes/origin/main, so a refusal below can never be attributed to a fixture +// that failed to produce the state it claims. +func assertAheadBehind(t *testing.T, db *sql.DB, wantAhead, wantBehind int) { + t.Helper() + var ahead, behind int + if err := db.QueryRowContext(t.Context(), ` + SELECT + (SELECT COUNT(*) FROM dolt_log WHERE commit_hash NOT IN + (SELECT commit_hash FROM dolt_log AS OF 'remotes/origin/main')) AS ahead, + (SELECT COUNT(*) FROM dolt_log AS OF 'remotes/origin/main' WHERE commit_hash NOT IN + (SELECT commit_hash FROM dolt_log)) AS behind + `).Scan(&ahead, &behind); err != nil { + t.Fatalf("measure ahead/behind: %v", err) + } + if ahead != wantAhead || behind != wantBehind { + t.Fatalf("fixture position = ahead %d, behind %d; want ahead %d, behind %d", + ahead, behind, wantAhead, wantBehind) + } +} + +// pendingMigrationStillPending fails unless the binary's latest migration is +// still unapplied — i.e. the open under test did NOT migrate. +func pendingMigrationStillPending(t *testing.T, beadsDir string) { + t.Helper() + db, cleanup, err := embeddeddolt.OpenSQL(t.Context(), filepath.Join(beadsDir, "embeddeddolt"), "testdb", "main") + if err != nil { + t.Fatalf("OpenSQL (verify): %v", err) + } + defer func() { _ = cleanup() }() + var count int + if err := db.QueryRowContext(t.Context(), + "SELECT COUNT(*) FROM schema_migrations WHERE version = ?", schema.LatestVersion()).Scan(&count); err != nil { + t.Fatalf("read schema_migrations: %v", err) + } + if count != 0 { + t.Fatalf("the open applied the pending migration (version %d recorded); a data-behind clone must not migrate", + schema.LatestVersion()) + } +} + +// TestEmbeddedOpenForRemoteSync_DataBehind is the F1 regression test for +// gastownhall/beads#6575: the refusal must not block its own prescribed +// remedy. +// +// The data-behind stop tells the operator to run `bd dolt pull`. That command +// opens the store, and a strict open re-hits the refusal that prescribed it, so +// before this the refused clone had no in-band exit at all — on embedded there +// is no external `dolt` binary either, leaving only +// BD_ALLOW_REMOTE_MIGRATE=1, i.e. performing the migration the refusal exists +// to prevent. This is the same deadlock #4566 broke for `bd dolt commit`, and +// the remote-sync open is its narrow twin. +// +// Both shapes are covered: strict-ancestor (ahead 0) and diverged (ahead 1). +func TestEmbeddedOpenForRemoteSync_DataBehind(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt tests") + } + + for _, tc := range []struct { + name string + extraLocalCommits int + wantDiverged bool + }{ + {name: "strict ancestor (ahead 0, behind 1)", extraLocalCommits: 0}, + {name: "diverged (ahead 1, behind 1)", extraLocalCommits: 1, wantDiverged: true}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Setenv(schema.AllowRemoteMigrateEnv, "0") + t.Setenv(schema.SmartGateEnv, "1") + schema.SetForceAllowRemoteMigrate(false) + + ctx := t.Context() + beadsDir := filepath.Join(t.TempDir(), ".beads") + buildDataBehindEmbedded(t, beadsDir, tc.extraLocalCommits) + + // A strict open still refuses, and refuses for the data-behind + // reason (not some other fixture artifact). + gated, gateErr := embeddeddolt.Open(ctx, beadsDir, "testdb", "main") + if gateErr == nil { + gated.Close() + t.Fatal("Open of a data-behind, remote-backed DB = nil error, want the #6575 refusal") + } + var rmErr *schema.RemoteMigrateGateError + if !errors.As(gateErr, &rmErr) { + t.Fatalf("Open error = %T (%v), want *schema.RemoteMigrateGateError", gateErr, gateErr) + } + if !rmErr.IsDataBehind() { + t.Fatalf("FallbackReason = %q, want the data-behind stop", rmErr.FallbackReason) + } + if rmErr.DataDiverged != tc.wantDiverged { + t.Errorf("DataDiverged = %v, want %v", rmErr.DataDiverged, tc.wantDiverged) + } + + // ...and the remote-sync open — what `bd dolt pull` uses — gets + // through it, without migrating. + syncStore, syncErr := embeddeddolt.OpenForRemoteSync(ctx, beadsDir, "testdb", "main") + if syncErr != nil { + t.Fatalf("OpenForRemoteSync of a data-behind DB = %v; the refusal must not block the pull it prescribes (#6575)", syncErr) + } + syncStore.Close() + pendingMigrationStillPending(t, beadsDir) + }) + } +} + +// TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal pins the narrowness of +// the exemption above, which is the whole reason it is keyed on the gate's +// REASON rather than being another blanket lenient open: a pull cannot resolve +// a fork skew, a below-floor database or an opted-out blunt block, so it must +// not be waved through them. Here BD_SMART_GATE=0 produces the blunt +// opted-out stop on the very same fixture that the data-behind case above +// opens successfully. +func TestEmbeddedOpenForRemoteSync_OtherRefusalsStayFatal(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt tests") + } + t.Setenv(schema.AllowRemoteMigrateEnv, "0") + schema.SetForceAllowRemoteMigrate(false) + + ctx := t.Context() + beadsDir := filepath.Join(t.TempDir(), ".beads") + // Build the fixture with the smart gate ON (the fixture helper's own + // Open must not be the thing that is opted out), then opt out for the + // probe below. + t.Setenv(schema.SmartGateEnv, "1") + buildDataBehindEmbedded(t, beadsDir, 0) + t.Setenv(schema.SmartGateEnv, "0") + + store, err := embeddeddolt.OpenForRemoteSync(ctx, beadsDir, "testdb", "main") + if err == nil { + store.Close() + t.Fatal("OpenForRemoteSync = nil error with BD_SMART_GATE=0; the exemption must apply to the data-behind reason only") + } + var rmErr *schema.RemoteMigrateGateError + if !errors.As(err, &rmErr) { + t.Fatalf("error = %T (%v), want *schema.RemoteMigrateGateError", err, err) + } + if rmErr.IsDataBehind() { + t.Fatalf("FallbackReason = %q; with the smart gate off this must be the blunt opted-out stop", rmErr.FallbackReason) + } +} + +// TestEmbeddedLenientOpenWarning_DataBehind is the READ-path half of +// gastownhall/beads#6575, and the half the write-path fix missed. +// +// The lenient embedded opens do not fail on this refusal — they warn and carry +// on — so the warning IS the entire user-facing contract for `bd list`, +// `bd ready`, `bd show` (openReadOnlyCommand) and `bd dolt commit` +// (openWorkingSetReconcile). Those are the first commands a 1.1-era upgrader +// runs and the ones agents run constantly, and they printed the blunt #4259 +// coordination bullets: `bd migrate --force && bd dolt push`, which on a +// data-behind clone applies the migration this stop exists to prevent and then +// fails the push non-fast-forward, and `bd bootstrap`, which no-ops against an +// existing workspace. A fence whose own signage pointed into the wedge. +// +// The assertions are deliberately two-sided — the pull must be present AND the +// wedge commands must be absent — because the previous round satisfied +// "mentions the remedy somewhere" while still printing the bullets underneath +// it. +func TestEmbeddedLenientOpenWarning_DataBehind(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt tests") + } + + shapes := []struct { + name string + extraLocalCommits int + wantShapeText string + notShapeText string + }{ + { + name: "fast-forward (ahead 0, behind 1)", + wantShapeText: "pure fast-forward", + notShapeText: "MERGES rather than", + }, + { + name: "diverged (ahead 1, behind 1)", + extraLocalCommits: 1, + wantShapeText: "MERGES rather than", + notShapeText: "pure fast-forward", + }, + } + intents := []struct { + name string + open func(t *testing.T, beadsDir string) (*embeddeddolt.EmbeddedDoltStore, error) + modeText string + }{ + { + name: "openReadOnlyCommand (bd list)", + open: func(t *testing.T, beadsDir string) (*embeddeddolt.EmbeddedDoltStore, error) { + return embeddeddolt.OpenForReadOnlyCommand(t.Context(), beadsDir, "testdb", "main") + }, + modeText: "Read-only command: continuing on schema v", + }, + { + name: "openWorkingSetReconcile (bd dolt commit)", + open: func(t *testing.T, beadsDir string) (*embeddeddolt.EmbeddedDoltStore, error) { + return embeddeddolt.OpenForWorkingSetReconcile(t.Context(), beadsDir, "testdb", "main") + }, + modeText: "Working-set reconcile command: continuing on schema v", + }, + } + + for _, shape := range shapes { + for _, intent := range intents { + t.Run(shape.name+"/"+intent.name, func(t *testing.T) { + t.Setenv(schema.AllowRemoteMigrateEnv, "0") + t.Setenv(schema.SmartGateEnv, "1") + schema.SetForceAllowRemoteMigrate(false) + + beadsDir := filepath.Join(t.TempDir(), ".beads") + buildDataBehindEmbedded(t, beadsDir, shape.extraLocalCommits) + + var store *embeddeddolt.EmbeddedDoltStore + var openErr error + warning := captureStderr(t, func() { + store, openErr = intent.open(t, beadsDir) + }) + if openErr != nil { + t.Fatalf("lenient open of a data-behind DB = %v; it must warn and continue", openErr) + } + store.Close() + pendingMigrationStillPending(t, beadsDir) + + for _, want := range []string{ + schema.DataBehindRemedyCommand, + "This clone is BEHIND the remote", + "Do NOT use `bd migrate --force` to get past this", + shape.wantShapeText, + intent.modeText, + } { + if !strings.Contains(warning, want) { + t.Errorf("warning is missing %q; got:\n%s", want, warning) + } + } + if strings.Contains(warning, shape.notShapeText) { + t.Errorf("warning describes the wrong pull shape (%q); got:\n%s", shape.notShapeText, warning) + } + + // The blunt #4259 coordination bullets, which ARE the #6575 + // wedge when this is the refusal. + for _, unwanted := range []string{ + "bd migrate --force && bd dolt push", + "designated migrator (only ONE machine)", + "bd bootstrap", + } { + if strings.Contains(warning, unwanted) { + t.Errorf("warning still prescribes %q on a data-behind clone; got:\n%s", unwanted, warning) + } + } + }) + } + } +} + +// TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance is the other +// side of that fence. The data-behind body REPLACES the blunt coordination +// bullets rather than decorating them, so it must not swallow them for the +// refusals they are right for. BD_SMART_GATE=0 turns the same fixture into the +// blunt opted-out stop, where migrate-or-adopt genuinely is the decision. +func TestEmbeddedLenientOpenWarning_OtherRefusalsKeepBluntGuidance(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt tests") + } + t.Setenv(schema.AllowRemoteMigrateEnv, "0") + schema.SetForceAllowRemoteMigrate(false) + + beadsDir := filepath.Join(t.TempDir(), ".beads") + t.Setenv(schema.SmartGateEnv, "1") + buildDataBehindEmbedded(t, beadsDir, 0) + t.Setenv(schema.SmartGateEnv, "0") + + var store *embeddeddolt.EmbeddedDoltStore + var openErr error + warning := captureStderr(t, func() { + store, openErr = embeddeddolt.OpenForReadOnlyCommand(t.Context(), beadsDir, "testdb", "main") + }) + if openErr != nil { + t.Fatalf("lenient open with the smart gate off = %v; it must warn and continue", openErr) + } + store.Close() + + for _, want := range []string{ + "designated migrator (only ONE machine): bd migrate --force && bd dolt push", + "every other clone (another already migrated): bd bootstrap", + "Read-only command: continuing on schema v", + } { + if !strings.Contains(warning, want) { + t.Errorf("blunt warning is missing %q; got:\n%s", want, warning) + } + } + if strings.Contains(warning, "This clone is BEHIND the remote") { + t.Errorf("the data-behind body leaked into a non-data-behind refusal; got:\n%s", warning) + } +} diff --git a/internal/storage/embeddeddolt/store.go b/internal/storage/embeddeddolt/store.go index 57c663146..58c96b785 100644 --- a/internal/storage/embeddeddolt/store.go +++ b/internal/storage/embeddeddolt/store.go @@ -66,9 +66,11 @@ type EmbeddedDoltStore struct { } // openIntent classifies why a store is being opened. openStrict fails the -// open on any pending-migration refusal; the other two intents relax both -// the #4259 remote-migrate gate refusal and the #4566 dirty-table refusal, -// each with its own warning text (see initSchema). +// open on any pending-migration refusal; openReadOnlyCommand and +// openWorkingSetReconcile relax both the #4259 remote-migrate gate refusal and +// the #4566 dirty-table refusal, each with its own warning text (see +// initSchema); openRemoteSync relaxes exactly one gate refusal and nothing +// else. type openIntent int const ( @@ -86,8 +88,52 @@ const ( // touch, so failing the open here would deadlock the documented recovery // (#4566). Used by OpenForWorkingSetReconcile. openWorkingSetReconcile + // openRemoteSync is the same shape of deadlock break as + // openWorkingSetReconcile, for the #6575 data-behind gate refusal: that + // refusal's entire remedy is `bd dolt pull`, which itself opens the store + // and so hit the refusal that prescribed it — a fence with no gate. Unlike + // the two intents above it relaxes exactly ONE refusal and nothing else: + // only a gate error whose reason is data-behind + // (RemoteMigrateGateError.IsDataBehind). Every other gate refusal, the + // #4566 dirty-table guard, and the #5268 dependency re-key still fail this + // open exactly as they do a strict one — the pull is being let through its + // own precondition, not granted a general exemption. Used by + // OpenForRemoteSync. + openRemoteSync ) +// toleratesGateRefusal reports whether this open's intent may warn and +// continue past a remote-migrate gate refusal (#4259/#5920/#6575) instead of +// failing the open. +// +// openRemoteSync is deliberately conditional on the REASON rather than being +// another blanket exemption: it exists only so `bd dolt pull` can execute the +// #6575 data-behind refusal's own remedy, and widening it to every gate +// refusal would quietly let a pull through a fork-skew or shared-store stop +// that the pull cannot help with. +func (s *EmbeddedDoltStore) toleratesGateRefusal(gateErr *schema.RemoteMigrateGateError) bool { + switch s.intent { + case openReadOnlyCommand, openWorkingSetReconcile: + return true + case openRemoteSync: + return gateErr.IsDataBehind() + default: // openStrict + return false + } +} + +// toleratesMigrationRefusal reports whether this open's intent may warn and +// continue past a MigrateUp refusal that is not the gate: the #4566 +// dirty-table guard and the #5268 dependency re-key conflict. +// +// openRemoteSync is absent on purpose. It is not listed as "not openStrict" +// because that phrasing is what would have silently enrolled it: the remote- +// sync exemption was granted for one gate reason, and a dirty working set or a +// re-key conflict is a different refusal with a different recovery. +func (s *EmbeddedDoltStore) toleratesMigrationRefusal() bool { + return s.intent == openReadOnlyCommand || s.intent == openWorkingSetReconcile +} + // errClosed is returned when a method is called after Close. var errClosed = errors.New("embeddeddolt: store is closed") @@ -369,6 +415,13 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error { IsStrictAncestor: func(ctx context.Context, db schema.DBConn, ref string) (bool, error) { return versioncontrolops.LocalIsStrictAncestorOf(ctx, db, ref) }, + // The raw counts the equal-version data-behind check needs + // (gastownhall/beads#6575): behind >= 1 whatever ahead is, plus + // which shape it is so the refusal names the pull the operator + // will actually get. + AheadBehind: func(ctx context.Context, db schema.DBConn, ref string) (int, int, error) { + return versioncontrolops.LocalAheadBehind(ctx, db, ref) + }, WorkingSetClean: func(ctx context.Context, db schema.DBConn) (bool, error) { return versioncontrolops.WorkingSetClean(ctx, db) }, @@ -388,7 +441,7 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error { } if err := schema.CheckRemoteMigrateGateWithAdopt(ctx, conn, adopt); err != nil { var gateErr *schema.RemoteMigrateGateError - if s.intent != openStrict && errors.As(err, &gateErr) { + if errors.As(err, &gateErr) && s.toleratesGateRefusal(gateErr) { // The gate exists to stop in-place migration on a remote-backed, // already-initialized database (#4259), not to block reads or a // working-set commit. Warn and continue on the current schema; @@ -402,20 +455,76 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error { " • every other clone (another already migrated): bd bootstrap\n" + " • several machines: only ONE migrates; sync each other clone and run\n" + " bd dolt pull after the migrator pushes, before upgrading it\n" + // #6575: sharedGuidance is the blunt migrate-or-adopt block, and + // on the data-behind stop every bullet in it is measurably wrong + // — `bd migrate --force` applies the migration this stop exists + // to prevent, the `bd dolt push` after it is rejected + // non-fast-forward while the clone is still behind, and + // `bd bootstrap` no-ops against an existing workspace. The one + // command that moves the clone forward is `bd dolt pull`, and + // gateErr's own UserMessage is where that body lives, already + // branched for the fast-forward and diverged pulls. Printing + // %v (Error(), the one-line summary) and appending the bullets + // dropped it: a 1.1-era upgrader's first `bd list` and their + // `bd dolt commit` both land in the two arms below, so the first + // thing bd said to a data-behind clone was the wedge. Server + // mode's warnLenientOpenRefusal (dolt/store.go) has rendered + // UserMessage all along; these arms are the ones that lagged. + // + // The intent framing stays: the command in hand is still + // SUCCEEDING against the old schema, which UserMessage — written + // for the fatal refusal — does not say. So the mode line is + // appended to the data-behind body rather than replacing it. + body := "Warning: " + gateErr.Error() + "\n" + if gateErr.IsDataBehind() { + body = "Warning: " + gateErr.UserMessage() + } switch s.intent { - case openWorkingSetReconcile: + case openRemoteSync: + // This is the refusal's own prescribed remedy running. It gets + // the short confirmation rather than the full data-behind body: + // toleratesGateRefusal admits only the data-behind stop here, so + // the operator has necessarily just been handed that body by the + // command this pull is unblocking, and re-printing "pull first" + // at the moment they are finally doing it is noise. What it must + // never do is name `bd migrate --force`. fmt.Fprintf(os.Stderr, "Warning: %[1]v\n"+ + " Remote-sync command: continuing on schema v%[2]d without migrating, so\n"+ + " this pull can bring in the commits this clone is behind on. Re-run the\n"+ + " command you were blocked on once it completes.\n", + gateErr, gateErr.CurrentVersion) + case openWorkingSetReconcile: + if gateErr.IsDataBehind() { + fmt.Fprintf(os.Stderr, + "%[1]s"+ + " Working-set reconcile command: continuing on schema v%[2]d without\n"+ + " migrating; the commit applies to the working set at the current\n"+ + " schema. It does not resolve the schema — the pull above is what does.\n", + body, gateErr.CurrentVersion) + break + } + fmt.Fprintf(os.Stderr, + "%[1]s"+ " Working-set reconcile command: continuing on schema v%[2]d without\n"+ " migrating; the commit applies to the working set at the current\n"+ " schema."+sharedGuidance, - gateErr, gateErr.CurrentVersion) + body, gateErr.CurrentVersion) default: // openReadOnlyCommand + if gateErr.IsDataBehind() { + fmt.Fprintf(os.Stderr, + "%[1]s"+ + " Read-only command: continuing on schema v%[2]d without migrating, so\n"+ + " this read succeeds against the old schema. Writes stay blocked until\n"+ + " this clone has pulled.\n", + body, gateErr.CurrentVersion) + break + } fmt.Fprintf(os.Stderr, - "Warning: %[1]v\n"+ + "%[1]s"+ " Read-only command: continuing on schema v%[2]d without migrating.\n"+ " Writes are blocked until the schema is reconciled."+sharedGuidance, - gateErr, gateErr.CurrentVersion) + body, gateErr.CurrentVersion) } return nil } @@ -426,7 +535,7 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error { // controls; schema.MigrateUpWithLock requires a sql-server session lock. if _, err := schema.MigrateUp(ctx, conn); err != nil { var dirtyErr *schema.DirtyTablesError - if s.intent != openStrict && errors.As(err, &dirtyErr) { + if s.toleratesMigrationRefusal() && errors.As(err, &dirtyErr) { // The guard exists to keep dirty user data from being entangled // with a migration, but its documented recovery - committing the // working set - also opens the store and would otherwise hit @@ -451,7 +560,7 @@ func (s *EmbeddedDoltStore) initSchema(ctx context.Context) error { return nil } var rekeyErr *schema.DependencyRekeyConflictError - if s.intent != openStrict && errors.As(err, &rekeyErr) { + if s.toleratesMigrationRefusal() && errors.As(err, &rekeyErr) { // Same principle for the dependency re-key's refusal (#5268): it is // a convergence repair, not a precondition for reading, and before // that pass existed these clones opened fine. Bricking 'bd list' and diff --git a/internal/storage/embeddeddolt/store_stub.go b/internal/storage/embeddeddolt/store_stub.go index 87b046245..db5efd66f 100644 --- a/internal/storage/embeddeddolt/store_stub.go +++ b/internal/storage/embeddeddolt/store_stub.go @@ -40,3 +40,8 @@ func OpenForReadOnlyCommand(_ context.Context, _, _, _ string) (*EmbeddedDoltSto func OpenForWorkingSetReconcile(_ context.Context, _, _, _ string) (*EmbeddedDoltStore, error) { return nil, errNoCGO } + +// OpenForRemoteSync returns an error when CGO is not enabled. +func OpenForRemoteSync(_ context.Context, _, _, _ string) (*EmbeddedDoltStore, error) { + return nil, errNoCGO +} diff --git a/internal/storage/schema/remote_migrate_gate.go b/internal/storage/schema/remote_migrate_gate.go index 15848928b..5a881d7fe 100644 --- a/internal/storage/schema/remote_migrate_gate.go +++ b/internal/storage/schema/remote_migrate_gate.go @@ -118,6 +118,15 @@ type RemoteMigrateGateError struct { // (the gate staying enabled by default). Set only when // FallbackReason == fallbackReasonUnparseableEnv. UnrecognizedSmartGateEnv string + + // DataDiverged distinguishes the two shapes of the data-behind stop + // (gastownhall/beads#6575). Both are "this clone is missing commits the + // remote has", and both take the same remedy COMMAND — `bd dolt pull` — + // but false means the clone has no commits of its own so the pull is a + // pure fast-forward, and true means it has both, so the pull MERGES and + // can report conflicts to resolve first. Set only when + // FallbackReason == fallbackReasonDataBehind; meaningless otherwise. + DataDiverged bool } const ( @@ -161,8 +170,37 @@ const ( // auto-executing it would promote the schema for every co-resident client. // Structural, not a routing outcome — it outranks the other reasons. fallbackReasonSharedStore = "shared-store" + // fallbackReasonDataBehind (gastownhall/beads#6575): schema parity with + // the cached remote ref held, but this clone's local HEAD is a strict + // ancestor of that ref — it is behind the remote in data commits it has + // not pulled, so it is not the first-mover the smart gate may + // auto-resolve. Unlike the reasons above, this one has a one-command + // remedy the operator can run right now (`bd dolt pull`), after which the + // migrate is a true first-mover. + fallbackReasonDataBehind = "data-behind" ) +// DataBehindRemedyCommand is the one command that moves a data-behind clone +// forward, and the single source for every surface that names it: the human +// body and note, the agent directive, the JSON option, and the store-open +// classification that has to let this command through the very refusal that +// prescribes it (gastownhall/beads#6575 F1 — a refusal whose documented +// recovery is blocked by the refusal is a fence with no gate; the #4566 +// precedent for `bd dolt commit` is the same shape). +const DataBehindRemedyCommand = "bd dolt pull" + +// IsDataBehind reports whether this refusal is the #6575 data-behind stop: +// the clone is level with the cached remote ref on schema but missing commits +// the ref has, so the pending migration must not be applied until it pulls. +// +// It is exported because the recovery for this particular stop is itself a +// store-opening command (DataBehindRemedyCommand), so the mode-specific store +// opens have to recognize this reason — and ONLY this reason — to open +// leniently for it. Every other refusal stays fatal on those opens. +func (e *RemoteMigrateGateError) IsDataBehind() bool { + return e != nil && e.Decision == "" && e.FallbackReason == fallbackReasonDataBehind +} + func (e *RemoteMigrateGateError) Error() string { unit := "migrations" if e.Pending == 1 { @@ -229,15 +267,97 @@ func (e *RemoteMigrateGateError) fallbackReasonNote() string { why = SmartGateEnv + "=" + e.UnrecognizedSmartGateEnv + " is set but was not recognized (only boolean values enable/disable it), so it stayed enabled by default but still could not resolve this stop" case fallbackReasonSharedStore: why = "this store is shared (dolt sql-server); the first-mover auto-migrate is disabled here because it would lock out co-resident clients (#5920)" + case fallbackReasonDataBehind: + why = "this clone is behind the remote in data commits it has not pulled, so it is not the first-mover it looks like on schema alone — migrating in place would diverge from those commits, and can leave every later `bd dolt pull` refusing to merge (#6575, #6368). Run `" + DataBehindRemedyCommand + "` first, then retry: this clone is then the true first-mover it looked like" + // Only promise the retry resolves itself where it actually does. On a + // shared store the first-mover auto-migrate is suppressed regardless + // (fallbackReasonSharedStore, #5920), so pulling clears THIS stop but + // the retry still needs explicit consent — saying otherwise would send + // the operator in a loop. + if !e.Shared { + why += " and the migrate proceeds on its own" + } else { + // #5920's consequence has to travel with this reason too. Before + // this arm existed a shared, data-behind store reported + // fallbackReasonSharedStore, whose note names the co-resident + // lockout; data-behind now outranks it, so without this clause the + // operator loses the warning that promoting the schema on a shared + // server locks out every co-resident client still on an older bd. + why += " — but on this shared store the migrate still needs explicit consent (`" + SharedConsentCommand + "`), because it promotes the schema for every co-resident bd client at once and clients still on an older binary will refuse this database until they are upgraded (#5920)" + } default: return "" } return "\n Smart gate (#4516): " + why + ".\n" } +// dataBehindBody is the guidance block for the #6575 data-behind stop. It +// REPLACES the blunt migrate-or-adopt body rather than decorating it, because +// in this state both of that body's options are wrong and were measured wrong: +// `bd migrate --force` is exactly the migration this stop exists to prevent +// (and the `bd dolt push` that follows it is rejected as a non-fast-forward, +// since the clone is still behind), and `bd bootstrap` against an existing +// workspace reports "Database already exists … Nothing to do." and exits 0. +// The one command that moves this clone forward is the pull. +func (e *RemoteMigrateGateError) dataBehindBody() string { + body := "\n" + + " This clone is BEHIND the remote: the remote has commits this clone has not\n" + + " pulled. On schema alone it looks like a first-mover, which is why the line\n" + + " above reads the way it does — but applying the pending migration here would\n" + + " put new schema commits on a history that is missing those commits. Once a\n" + + " migration moves a tracked table onto Dolt's ignore plane (0062's `events`),\n" + + " every later `" + DataBehindRemedyCommand + "` refuses to merge and this clone can no longer\n" + + " fetch the commits it was already behind on (#6575, #6368).\n" + + "\n" + + " Pull first — that is the whole remedy:\n" + + " " + DataBehindRemedyCommand + "\n" + if e.DataDiverged { + body += "" + + " This clone also has commits of its own, so the pull MERGES rather than\n" + + " fast-forwards. If it reports conflicts, resolve them and let the pull\n" + + " finish before retrying (`" + DataBehindRemedyCommand + " --strategy ours|theirs` decides them\n" + + " in bulk on embedded storage; `bd conflicts resolve` on a server store).\n" + } else { + body += "" + + " This clone has no commits of its own, so the pull is a pure fast-forward:\n" + + " nothing local is merged or discarded.\n" + } + body += "" + + "\n" + + " Then re-run what you just ran.\n" + if e.Shared { + // #5920: on a shared store the first-mover auto-migrate stays + // suppressed after the pull, so the retry needs the consent verb. + // Saying "it proceeds on its own" here would loop the operator. + body += "" + + " This database is served to co-resident bd clients, so the migration itself\n" + + " still needs explicit consent afterwards — it promotes the schema for EVERY\n" + + " client at once, and clients still on an older bd will refuse this database\n" + + " until they are upgraded (#5920):\n" + + " " + SharedConsentCommand + "\n" + + " (" + SharedConsentCommandGlobal + " for the shared global database)\n" + } else { + body += "" + + " The migration then runs on its own: with nothing left to pull, this clone\n" + + " is the first-mover it looked like.\n" + } + body += "" + + "\n" + + " Do NOT use `bd migrate --force` to get past this. It applies the migration\n" + + " while this clone is still behind — the exact state this stop exists to\n" + + " prevent — and the `bd dolt push` that would follow is rejected as a\n" + + " non-fast-forward, because the clone is still missing the remote's commits.\n" + return body +} + // userBody returns the decision-specific guidance block. The default (blunt // #4515) body is byte-identical to before the smart gate existed. func (e *RemoteMigrateGateError) userBody() string { + // The #6575 stop is a blunt stop (empty Decision) with a remedy the blunt + // body contradicts, so it is keyed on the fallback reason instead. + if e.IsDataBehind() { + return e.dataBehindBody() + } switch e.Decision { case gateDecisionAdopt: return "\n" + @@ -332,6 +452,35 @@ func (e *RemoteMigrateGateError) EscapeHint() string { // to the operator and let them choose, per the AgentDiagnostic contract ("Go // observes and reports, the agent decides and acts"). func (e *RemoteMigrateGateError) AgentDirective() string { + // #6575: the data-behind stop is the one refusal in this type with a + // single, safe, non-destructive, auto-runnable remedy — pulling commits + // this clone is already behind on takes nothing away and needs no + // cross-clone coordination. Saying so is the point: the default directive + // below sends the agent to remote_migrate_gate.options, and if those + // options were the blunt migrate-or-adopt pair the agent would surface + // the migration this stop exists to prevent (whose follow-up push is + // rejected non-fast-forward anyway) and a bootstrap that no-ops. + if e.IsDataBehind() { + d := "Pull, do NOT migrate: this clone is level with the remote on schema but is missing commits the remote " + + "has, so the pending migration would land on a history without them and can wedge every later " + + "`" + DataBehindRemedyCommand + "` (#6575, #6368). The remedy is `" + DataBehindRemedyCommand + "`, which is in " + + "remote_migrate_gate.options as the single option." + if e.DataDiverged { + d += " This clone also has commits of its own, so that pull MERGES rather than fast-forwards and may " + + "report conflicts to resolve before it completes — surface a conflict outcome to the operator " + + "rather than forcing past it." + } else { + d += " This clone has no commits of its own, so that pull is a pure fast-forward and discards nothing." + } + if e.Shared { + d += " After the pull the migration still needs the operator's explicit consent (" + SharedConsentCommand + + "): this database is shared with co-resident bd clients and migrating promotes the schema for all of " + + "them at once (#5920). Do NOT auto-run it." + } else { + d += " After the pull, re-running the original command migrates on its own; nothing else is needed." + } + return d + } switch e.Decision { case gateDecisionAdopt: return "Adopt, do NOT migrate: the remote is already migrated, so migrating here would fork it (#4259). " + @@ -392,6 +541,39 @@ func (e *RemoteMigrateGateError) Options() []GateOption { Commands: []string{"bd bootstrap"}, Risk: "re-clones and replaces the local database; push or export unpushed work first or it is lost" + sharedRisk, } + // #6575: exactly ONE option, and it is the pull. The blunt pair below is + // actively wrong in this state — `bd migrate --force` is the bug, its + // `bd dolt push` is rejected non-fast-forward while the clone is still + // behind, and `bd bootstrap` no-ops against an existing workspace — so + // offering them here would hand an agent two dead ends and no exit. + // Unlike every other option in this type it carries no precondition the + // operator has to confirm: pulling commits this clone is already behind on + // is not a coordination decision. + if e.IsDataBehind() { + risk := "none — a pure fast-forward; this clone has no commits of its own, so nothing local is merged or discarded" + when := "always, for this stop: this clone is behind the remote and must pull before it may migrate" + if e.DataDiverged { + risk = "this clone also has commits of its own, so the pull MERGES; it can report conflicts that must be resolved before it completes (bd dolt pull --strategy ours|theirs on embedded storage, bd conflicts resolve on a server store). Nothing is discarded without that resolution" + } + pull := GateOption{ + ID: "pull-first", + When: when, + Commands: []string{DataBehindRemedyCommand}, + Risk: risk, + } + if e.Shared { + // The pull itself is still safe; what the shared store changes is + // what happens AFTER it, so the consent step becomes a second + // option rather than a footnote on the first (#5920). + return []GateOption{pull, { + ID: "migrate-shared-after-pulling", + When: "the pull above has completed AND every co-resident bd client of this server is upgraded to this binary (confirmed with the operator)", + Commands: []string{SharedConsentCommand}, + Risk: "co-resident clients still on an older bd will refuse this database until upgraded; running it before the pull completes is the wedge this stop prevents", + }} + } + return []GateOption{pull} + } switch e.Decision { case gateDecisionAdopt: // Remote is confirmed ahead: migrate is not a valid path, only adopt. @@ -584,6 +766,10 @@ func checkRemoteMigrateGate(ctx context.Context, db DBConn, remoteName string, e // follow-up). fallbackReason := "" unrecognizedSmartGateEnv := "" + // dataDiverged records which shape of the #6575 data-behind stop fired, so + // the guidance names the pull the operator will actually get (a merge, not + // a fast-forward). Meaningless unless fallbackReason is data-behind. + dataDiverged := false if SmartGateEnabled() { decision, skew, ref, atLatest := routeSmartGate(ctx, db, current, latest, remoteName, adopt) switch decision { @@ -669,6 +855,25 @@ func checkRemoteMigrateGate(ctx context.Context, db DBConn, remoteName string, e Decision: gateDecisionForkSkew, SkewVersions: skew, } + case smartDataBehind, smartDataDiverged: + // gastownhall/beads#6575: schema parity held but this clone is + // missing commits the cached remote ref has. Stop, and carry the + // pull-first remedy on every surface (body, note, agent directive, + // options) — the blunt migrate-or-adopt guidance is wrong here, + // not merely incomplete. + // + // This reason is reported even on a shared store, where + // fallbackReasonSharedStore would otherwise outrank it: that one + // says the gate WOULD have resolved and was overruled, which is + // not true here. The gate refuses this state on its own merits, + // and the pull-first remedy is a precondition the operator has to + // satisfy either way; #5920's co-resident consequence is folded + // into the data-behind guidance instead of being lost with the + // reason it outranks. + fallbackReason = fallbackReasonDataBehind + // The two verdicts share the reason and the remedy command; they + // differ in whether that command fast-forwards or merges. + dataDiverged = decision == smartDataDiverged case smartBelowFloor: fallbackReason = fallbackReasonBelowFloor case smartUndetermined: @@ -679,9 +884,15 @@ func checkRemoteMigrateGate(ctx context.Context, db DBConn, remoteName string, e // fact for the operator than the technical routing outcome, so it // takes priority as the surfaced reason. Not over shared-store: that // one is structural (the gate WOULD have resolved, and was overruled), - // so fixing the env value would not change the outcome. + // so fixing the env value would not change the outcome. Not over + // data-behind either, for the same reason (gastownhall/beads#6575): + // that stop stands whatever BD_SMART_GATE says, and it is the one + // reason carrying a remedy the operator can act on right now, so + // burying it behind an env-value complaint would cost them the fix. envState, envValue := smartGateEnvValue() - if envState == smartGateEnvUnparseable && fallbackReason != fallbackReasonSharedStore { + if envState == smartGateEnvUnparseable && + fallbackReason != fallbackReasonSharedStore && + fallbackReason != fallbackReasonDataBehind { fallbackReason = fallbackReasonUnparseableEnv unrecognizedSmartGateEnv = envValue } @@ -697,6 +908,7 @@ func checkRemoteMigrateGate(ctx context.Context, db DBConn, remoteName string, e FallbackReason: fallbackReason, Shared: shared, UnrecognizedSmartGateEnv: unrecognizedSmartGateEnv, + DataDiverged: dataDiverged, } } diff --git a/internal/storage/schema/smart_remote_migrate_gate.go b/internal/storage/schema/smart_remote_migrate_gate.go index 46cca35e7..ab26d88a2 100644 --- a/internal/storage/schema/smart_remote_migrate_gate.go +++ b/internal/storage/schema/smart_remote_migrate_gate.go @@ -12,9 +12,10 @@ import ( // gate would fire, it consults the remote's cached schema state and // auto-resolves the one provably-safe case (first-mover migrate), while still // stopping — with sharper guidance — on the cases that genuinely need a human. -// Every fallback path (unreadable remote state, below the convergence floor) -// degrades to the blunt #4515 block, so the default is never less safe than -// the blunt gate; it only resolves cases the blunt gate cannot distinguish. +// Every fallback path (unreadable remote state, below the convergence floor, +// a clone behind the remote in data commits) degrades to the blunt #4515 +// block, so the default is never less safe than the blunt gate; it only +// resolves cases the blunt gate cannot distinguish. // // Set BD_SMART_GATE=0 (or any boolean false) to opt out and keep the blunt // #4515 behavior unconditionally: any remote-backed database with pending @@ -128,6 +129,28 @@ const ( // a TOCTOU miss, or the merge itself refusing) falls through to the // plain smartAdopt directive instead, never forcing the write. smartAdoptFastForward + // smartDataBehind (gastownhall/beads#6575): schema parity with the cached + // remote ref holds — the first-mover precondition — but local HEAD is + // BEHIND that ref with no local commits of its own (ahead == 0, + // behind >= 1), so this clone is missing DATA commits it has not pulled. + // Migrating in place here mints local-only schema commits on a HEAD that + // lacks those commits; when the pending batch moves a tracked table onto + // the dolt_ignore plane (0062's events), every subsequent `bd dolt pull` + // then refuses (#6368) and the clone can no longer fetch the very commits + // it was behind on. Stop, and direct the operator to pull first — a pure + // fast-forward in this shape — after which the same open auto-migrates as + // a true first-mover. + smartDataBehind + // smartDataDiverged is smartDataBehind's other shape: behind >= 1 AND + // ahead >= 1, i.e. this clone has unpushed local commits as well as + // unpulled remote ones. It reaches the identical wedge — the pending + // migration lands on a HEAD missing the remote's commits either way — so + // it stops for the same reason. It is a separate verdict only because the + // remedy behaves differently: `bd dolt pull` MERGES here rather than + // fast-forwarding, and can report conflicts the operator has to resolve + // before it completes. bd auto-commits every write, so this is the + // ORDINARY multi-machine shape, not an exotic one. + smartDataDiverged ) // FastForwardAdopter injects the driver-side fast-forward primitives that @@ -149,6 +172,20 @@ type FastForwardAdopter struct { // IsStrictAncestor reports whether local HEAD is a strict ancestor of // ref (versioncontrolops.LocalIsStrictAncestorOf). IsStrictAncestor func(ctx context.Context, db DBConn, ref string) (bool, error) + // AheadBehind reports the raw commit counts behind IsStrictAncestor: + // how many commits local HEAD has that ref lacks, and how many ref has + // that local lacks (versioncontrolops.LocalAheadBehind). The + // data-behind stop (gastownhall/beads#6575) needs behind >= 1 whatever + // ahead is, which IsStrictAncestor's ahead == 0 && behind >= 1 cannot + // express, and needs to tell the two shapes apart so the remedy it + // prints matches the pull the operator will actually get (fast-forward + // vs merge). + // + // A nil AheadBehind is safe: localDataBehind falls back to + // IsStrictAncestor, which reports only the ahead == 0 shape — the + // narrower, pre-#6575-widening behavior — so an injection site that + // wires only the older callback keeps working. + AheadBehind func(ctx context.Context, db DBConn, ref string) (ahead, behind int, err error) // WorkingSetClean reports whether the working set has no uncommitted // changes, dolt-ignored wisp tables excepted // (versioncontrolops.WorkingSetClean). @@ -194,6 +231,54 @@ func routeAdoptFastForward(ctx context.Context, db DBConn, ref string, adopt *Fa return true } +// localDataBehind reports whether local HEAD is missing commits that ref has +// (behind >= 1) and, when it is, whether the clone ALSO has commits of its own +// that ref lacks (ahead >= 1 — the diverged shape). It is the equal-version +// path's only ancestry fact (gastownhall/beads#6575). +// +// The predicate is behind >= 1 regardless of ahead, not the narrower strict- +// ancestor relation the remote-ahead path uses. Both shapes reach the same +// wedge: the migration lands on a HEAD missing the remote's commits either +// way, and the ignore-plane flip then refuses every later `bd dolt pull`. +// Since bd auto-commits every write, "has local commits AND is behind" is the +// ordinary multi-machine state, so restricting the stop to ahead == 0 would +// leave the larger half of the affected cohort unprotected. A level clone +// (behind == 0) and a purely-ahead clone (behind == 0, ahead >= 1) are still +// permitted — they are genuine first-movers with nothing to pull. +// +// adopt may be nil, or wire neither callback (the shared unit-of-work +// provider's injection site) — then there is no ancestry fact to read and +// routing stays exactly as it was before this check existed. With only +// IsStrictAncestor wired, the fact narrows to the ahead == 0 shape; the +// diverged shape then reads "not behind", which is the pre-widening +// behavior rather than a new hazard. +// +// Unlike routeAdoptFastForward, a query error is NOT folded into "false": +// here false means "permit the more automatic action", so an inconclusive +// read has to be reported to the caller and degrade to the blunt block. +func localDataBehind(ctx context.Context, db DBConn, ref string, adopt *FastForwardAdopter) (behind bool, diverged bool, err error) { + if adopt == nil { + return false, false, nil + } + if adopt.AheadBehind != nil { + ahead, behindCount, err := adopt.AheadBehind(ctx, db, ref) + if err != nil { + return false, false, err + } + return behindCount >= 1, behindCount >= 1 && ahead >= 1, nil + } + if adopt.IsStrictAncestor == nil { + return false, false, nil + } + ancestor, err := adopt.IsStrictAncestor(ctx, db, ref) + if err != nil { + return false, false, err + } + // A strict ancestor is by definition ahead == 0, so this fallback can + // only ever report the non-diverged shape. + return ancestor, false, nil +} + // canAutoFastForward reports whether adopt is actually able to EXECUTE the // fast-forward once the read-side preconditions (routeAdoptFastForward) are // satisfied: a FastForward callback must be wired, and the store must not be @@ -319,6 +404,29 @@ func routeSmartGate(ctx context.Context, db DBConn, current, latest int, remoteN // remote == local on every shared version and at the same max version: a first-mover. if current >= LastNonDeterministicMigration { + // Schema parity is not proof of first-mover status + // (gastownhall/beads#6575). It says nothing about where this clone's + // branch sits relative to the ref the hashes just came from: a clone + // that is level on schema and behind on DATA satisfies every + // precondition above. Read the ancestry fact the caller already wired + // for the remote-ahead path before granting the automatic migrate. + behind, diverged, err := localDataBehind(ctx, db, ref, adopt) + if err != nil { + // Inconclusive ancestry read on a ref whose content hashes just + // resolved. Degrade to the blunt block rather than assume level: + // the gate's contract is that every fallback is at least as safe + // as #4515. + return smartUndetermined, nil, "", false + } + if behind { + // Same stop, same remedy command; the two verdicts differ only in + // what that command DOES (fast-forward vs merge), which the + // operator-facing text has to get right. + if diverged { + return smartDataDiverged, nil, ref, false + } + return smartDataBehind, nil, ref, false + } return smartAutoMigrate, nil, ref, false } return smartBelowFloor, nil, ref, false diff --git a/internal/storage/schema/smart_remote_migrate_gate_test.go b/internal/storage/schema/smart_remote_migrate_gate_test.go index 98a3e8c41..55375c8c1 100644 --- a/internal/storage/schema/smart_remote_migrate_gate_test.go +++ b/internal/storage/schema/smart_remote_migrate_gate_test.go @@ -416,6 +416,17 @@ type fakeAdopter struct { ffCalls int readOnly bool + + // aheadResult/behindResult/aheadBehindErr back the AheadBehind callback, + // counted by aheadBehindCalls. Wired into the adopter only when + // withAheadBehind is true, so the tests that leave it off keep exercising + // the IsStrictAncestor-only fallback an injection site that never wired + // the newer callback would take (gastownhall/beads#6575). + withAheadBehind bool + aheadResult int + behindResult int + aheadBehindErr error + aheadBehindCalls int } func (f *fakeAdopter) adopter() *FastForwardAdopter { @@ -436,6 +447,12 @@ func (f *fakeAdopter) adopter() *FastForwardAdopter { return f.ffErr } } + if f.withAheadBehind { + a.AheadBehind = func(_ context.Context, _ DBConn, _ string) (int, int, error) { + f.aheadBehindCalls++ + return f.aheadResult, f.behindResult, f.aheadBehindErr + } + } return a } @@ -800,6 +817,491 @@ func TestSmartGateRoutingFastForward(t *testing.T) { }) } +// TestSmartGateRoutingDataBehind covers the equal-version path's ancestry +// check (gastownhall/beads#6575): schema parity with the cached remote ref is +// not proof of first-mover status, because a clone can be level on schema and +// behind on data commits. These subtests pin the ROUTING around canned +// ancestry results; the genuinely data-behind clone — two real clones, a +// file:// remote, a real DOLT_FETCH, the ancestry fact measured rather than +// stubbed — lives in internal/storage/dolt +// (TestDoltNew_SmartRemoteMigrateGate_DataBehindBlocks_RealDolt). +func TestSmartGateRoutingDataBehind(t *testing.T) { + floor := LastNonDeterministicMigration + + t.Run("data-behind clone: blunt block naming the pull-first remedy", func(t *testing.T) { + t.Setenv(SmartGateEnv, "1") + t.Setenv(AllowRemoteMigrateEnv, "0") + db, mock, _ := sqlmock.New() + defer db.Close() + expectSmartFiringGate(mock, floor) + hashes := map[int]string{floor - 1: "h1", floor: "h2"} + expectSmartRemoteRead(mock, hashes, hashes) + + fa := &fakeAdopter{ancestorResult: true, cleanResult: true} + err := CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt(context.Background(), db, "", nil, fa.adopter()) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("a data-behind clone must not auto-migrate, got %v", err) + } + if gateErr.Decision != "" { + t.Errorf("Decision = %q, want \"\" (the blunt #4515 stop — no new JSON/agent contract)", gateErr.Decision) + } + if gateErr.FallbackReason != fallbackReasonDataBehind { + t.Errorf("FallbackReason = %q, want %q", gateErr.FallbackReason, fallbackReasonDataBehind) + } + if !strings.Contains(gateErr.UserMessage(), "Run `bd dolt pull` first") { + t.Errorf("UserMessage should name the pull-first remedy:\n%s", gateErr.UserMessage()) + } + if fa.ancestorCalls != 1 { + t.Errorf("IsStrictAncestor calls = %d, want 1", fa.ancestorCalls) + } + // Deliberately NOT part of the equal-version precondition: a dirty + // working set on a level clone is not the #6575 wedge state, and + // refusing it would broaden the gate beyond the field-hit bug. + if fa.cleanCalls != 0 { + t.Errorf("WorkingSetClean calls = %d, want 0 (the equal-version path must not require a clean working set)", fa.cleanCalls) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + t.Run("level clone: still the safe first-mover auto-migrate", func(t *testing.T) { + t.Setenv(SmartGateEnv, "1") + t.Setenv(AllowRemoteMigrateEnv, "0") + db, mock, _ := sqlmock.New() + defer db.Close() + expectSmartFiringGate(mock, floor) + hashes := map[int]string{floor: "h1"} + expectSmartRemoteRead(mock, hashes, hashes) + + // ancestorResult false covers BOTH non-behind shapes the predicate + // collapses: HEAD == ref (level) and ahead > 0 (unpushed local + // commits). Neither is refused. + fa := &fakeAdopter{ancestorResult: false, cleanResult: true} + if err := CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt(context.Background(), db, "", nil, fa.adopter()); err != nil { + t.Fatalf("a clone that is not behind must still auto-migrate, got %v", err) + } + if fa.ancestorCalls != 1 { + t.Errorf("IsStrictAncestor calls = %d, want 1", fa.ancestorCalls) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + t.Run("ancestry read error: blunt block, never the automatic migrate", func(t *testing.T) { + t.Setenv(SmartGateEnv, "1") + t.Setenv(AllowRemoteMigrateEnv, "0") + db, mock, _ := sqlmock.New() + defer db.Close() + expectSmartFiringGate(mock, floor) + hashes := map[int]string{floor: "h1"} + expectSmartRemoteRead(mock, hashes, hashes) + + fa := &fakeAdopter{ancestorErr: errors.New("dolt_log AS OF: branch not found")} + err := CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt(context.Background(), db, "", nil, fa.adopter()) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("an inconclusive ancestry read must fall back to the blunt block, got %v", err) + } + if gateErr.FallbackReason != fallbackReasonUnreadableState { + t.Errorf("FallbackReason = %q, want %q", gateErr.FallbackReason, fallbackReasonUnreadableState) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + t.Run("no adopter wired: routing is unchanged", func(t *testing.T) { + t.Setenv(SmartGateEnv, "1") + t.Setenv(AllowRemoteMigrateEnv, "0") + db, mock, _ := sqlmock.New() + defer db.Close() + expectSmartFiringGate(mock, floor) + hashes := map[int]string{floor: "h1"} + expectSmartRemoteRead(mock, hashes, hashes) + + // The unit-of-work provider's injection site passes nil (and is + // shared, where the first-mover arm is suppressed anyway). With no + // ancestry fact to read, the verdict stays exactly what it was. + if err := CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt(context.Background(), db, "", nil, nil); err != nil { + t.Fatalf("a nil adopter must not change the first-mover verdict, got %v", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + t.Run("below the convergence floor: outcome and reads unchanged", func(t *testing.T) { + t.Setenv(SmartGateEnv, "1") + t.Setenv(AllowRemoteMigrateEnv, "0") + db, mock, _ := sqlmock.New() + defer db.Close() + current := floor - 1 + expectSmartFiringGate(mock, current) + hashes := map[int]string{current: "h1"} + expectSmartRemoteRead(mock, hashes, hashes) + + fa := &fakeAdopter{ancestorResult: true, cleanResult: true} + err := CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt(context.Background(), db, "", nil, fa.adopter()) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("expected gate error, got %v", err) + } + if gateErr.FallbackReason != fallbackReasonBelowFloor { + t.Errorf("FallbackReason = %q, want %q (below-floor outcomes must stay byte-identical)", gateErr.FallbackReason, fallbackReasonBelowFloor) + } + if fa.ancestorCalls != 0 { + t.Errorf("IsStrictAncestor calls = %d, want 0 (the ancestry read must not fire below the floor)", fa.ancestorCalls) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + t.Run("shared store: data-behind outranks the shared-store suppression reason", func(t *testing.T) { + t.Setenv(SmartGateEnv, "1") + t.Setenv(AllowRemoteMigrateEnv, "0") + SetForceAllowRemoteMigrate(false) + SetSharedMigrateConsent(false) + db, mock, _ := sqlmock.New() + defer db.Close() + expectSmartFiringGate(mock, floor) + hashes := map[int]string{floor: "h1"} + expectSmartRemoteRead(mock, hashes, hashes) + + // Both facts hold, and either one blocks. The shared-store reason + // says "the gate WOULD have resolved and was overruled", which is not + // true here — and the pull-first remedy is a precondition this + // operator has to satisfy either way. + fa := &fakeAdopter{ancestorResult: true, cleanResult: true} + err := CheckSharedStoreMigrateGate(context.Background(), db, "", nil, fa.adopter()) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("expected gate error, got %v", err) + } + if gateErr.FallbackReason != fallbackReasonDataBehind { + t.Errorf("FallbackReason = %q, want %q", gateErr.FallbackReason, fallbackReasonDataBehind) + } + if !gateErr.Shared { + t.Error("Shared should still be set on a shared-store refusal") + } + // But the note must NOT promise the retry resolves itself here: on a + // shared store the first-mover auto-migrate stays suppressed (#5920), + // so pulling clears this stop and the retry still needs consent. + msg := gateErr.UserMessage() + if !strings.Contains(msg, "Run `bd dolt pull` first") { + t.Errorf("shared store should still get the pull-first remedy:\n%s", msg) + } + if strings.Contains(msg, "proceeds on its own") { + t.Errorf("shared store must not be promised an automatic retry (the first-mover arm is suppressed here):\n%s", msg) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + t.Run("non-shared store IS promised the automatic retry", func(t *testing.T) { + t.Setenv(SmartGateEnv, "1") + t.Setenv(AllowRemoteMigrateEnv, "0") + db, mock, _ := sqlmock.New() + defer db.Close() + expectSmartFiringGate(mock, floor) + hashes := map[int]string{floor: "h1"} + expectSmartRemoteRead(mock, hashes, hashes) + + fa := &fakeAdopter{ancestorResult: true, cleanResult: true} + err := CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt(context.Background(), db, "", nil, fa.adopter()) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("expected gate error, got %v", err) + } + if msg := gateErr.UserMessage(); !strings.Contains(msg, "proceeds on its own") { + t.Errorf("an ordinary clone's retry DOES auto-resolve and should say so:\n%s", msg) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) + + // An unparseable BD_SMART_GATE value normally outranks the routing reason, + // but it must not bury data-behind: that stop stands whatever the env says, + // and it is the only reason carrying a remedy the operator can act on now. + t.Run("unparseable BD_SMART_GATE does not bury the pull-first remedy", func(t *testing.T) { + t.Setenv(SmartGateEnv, "yes") + t.Setenv(AllowRemoteMigrateEnv, "0") + db, mock, _ := sqlmock.New() + defer db.Close() + expectSmartFiringGate(mock, floor) + hashes := map[int]string{floor: "h1"} + expectSmartRemoteRead(mock, hashes, hashes) + + fa := &fakeAdopter{ancestorResult: true, cleanResult: true} + err := CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt(context.Background(), db, "", nil, fa.adopter()) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("expected gate error, got %v", err) + } + if gateErr.FallbackReason != fallbackReasonDataBehind { + t.Errorf("FallbackReason = %q, want %q (the env complaint must not outrank it)", gateErr.FallbackReason, fallbackReasonDataBehind) + } + if msg := gateErr.UserMessage(); !strings.Contains(msg, "Run `bd dolt pull` first") { + t.Errorf("UserMessage lost the pull-first remedy:\n%s", msg) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations: %v", err) + } + }) +} + +// TestSmartGateRoutingDataBehindShapes covers the widened data-behind +// predicate (gastownhall/beads#6575 review finding F2): the stop is keyed on +// behind >= 1 whatever ahead is, not on the strict-ancestor relation. +// +// The narrower form protected only a clone with zero local commits, and bd +// auto-commits every write — so "has local commits AND is behind" is the +// ORDINARY multi-machine state, and it reaches the identical wedge (the +// pending migration lands on a history missing the remote's commits either +// way). What differs is the remedy's behavior, not the remedy: the pull +// fast-forwards in one shape and merges in the other, so the two verdicts stay +// distinguishable all the way out to the message and the JSON. +func TestSmartGateRoutingDataBehindShapes(t *testing.T) { + floor := LastNonDeterministicMigration + + route := func(t *testing.T, fa *fakeAdopter, shared bool) error { + t.Helper() + t.Setenv(SmartGateEnv, "1") + t.Setenv(AllowRemoteMigrateEnv, "0") + SetForceAllowRemoteMigrate(false) + SetSharedMigrateConsent(false) + db, mock, _ := sqlmock.New() + defer db.Close() + expectSmartFiringGate(mock, floor) + hashes := map[int]string{floor: "h1"} + expectSmartRemoteRead(mock, hashes, hashes) + var err error + if shared { + err = CheckSharedStoreMigrateGate(context.Background(), db, "", nil, fa.adopter()) + } else { + err = CheckRemoteMigrateGateForRemoteWithRemoteCheckAndAdopt(context.Background(), db, "", nil, fa.adopter()) + } + if mockErr := mock.ExpectationsWereMet(); mockErr != nil { + t.Fatalf("unmet expectations: %v", mockErr) + } + return err + } + + t.Run("behind with no local commits: the fast-forward shape", func(t *testing.T) { + fa := &fakeAdopter{withAheadBehind: true, aheadResult: 0, behindResult: 1} + err := route(t, fa, false) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("a data-behind clone must not auto-migrate, got %v", err) + } + if gateErr.FallbackReason != fallbackReasonDataBehind { + t.Fatalf("FallbackReason = %q, want %q", gateErr.FallbackReason, fallbackReasonDataBehind) + } + if gateErr.DataDiverged { + t.Error("DataDiverged = true, want false (ahead == 0 is a pure fast-forward)") + } + if msg := gateErr.UserMessage(); !strings.Contains(msg, "pure fast-forward") { + t.Errorf("message should say the pull fast-forwards:\n%s", msg) + } + // The raw counts are what the widened predicate reads; the older + // boolean callback must not be consulted when they are available. + if fa.aheadBehindCalls != 1 { + t.Errorf("AheadBehind calls = %d, want 1", fa.aheadBehindCalls) + } + if fa.ancestorCalls != 0 { + t.Errorf("IsStrictAncestor calls = %d, want 0 once AheadBehind is wired", fa.ancestorCalls) + } + }) + + t.Run("behind WITH local commits: the diverged shape still stops", func(t *testing.T) { + // The F2 case: before the widening this routed to smartAutoMigrate and + // the migration ran, measured on a real clone at ahead=1, behind=1. + fa := &fakeAdopter{withAheadBehind: true, aheadResult: 1, behindResult: 1} + err := route(t, fa, false) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("a diverged, behind clone reaches the same wedge and must not auto-migrate, got %v", err) + } + if gateErr.FallbackReason != fallbackReasonDataBehind { + t.Fatalf("FallbackReason = %q, want %q", gateErr.FallbackReason, fallbackReasonDataBehind) + } + if !gateErr.DataDiverged { + t.Error("DataDiverged = false, want true (ahead >= 1 means the pull merges)") + } + msg := gateErr.UserMessage() + if !strings.Contains(msg, "MERGES") { + t.Errorf("a diverged clone must be told its pull merges rather than fast-forwards:\n%s", msg) + } + if strings.Contains(msg, "pure fast-forward") { + t.Errorf("a diverged clone must NOT be promised a fast-forward:\n%s", msg) + } + }) + + t.Run("purely ahead: still the safe first-mover auto-migrate", func(t *testing.T) { + // Nothing to pull, so nothing to wedge. Refusing here would be the + // over-refusal a patch line must not ship. + fa := &fakeAdopter{withAheadBehind: true, aheadResult: 3, behindResult: 0} + if err := route(t, fa, false); err != nil { + t.Fatalf("a clone with only unpushed local commits must still auto-migrate, got %v", err) + } + }) + + t.Run("level: still the safe first-mover auto-migrate", func(t *testing.T) { + fa := &fakeAdopter{withAheadBehind: true, aheadResult: 0, behindResult: 0} + if err := route(t, fa, false); err != nil { + t.Fatalf("a level clone must still auto-migrate, got %v", err) + } + }) + + t.Run("ahead/behind read error: blunt block, never the automatic migrate", func(t *testing.T) { + fa := &fakeAdopter{withAheadBehind: true, aheadBehindErr: errors.New("dolt_log AS OF: branch not found")} + err := route(t, fa, false) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("an inconclusive read must fall back to the blunt block, got %v", err) + } + if gateErr.FallbackReason != fallbackReasonUnreadableState { + t.Errorf("FallbackReason = %q, want %q", gateErr.FallbackReason, fallbackReasonUnreadableState) + } + }) + + t.Run("no AheadBehind wired: falls back to the strict-ancestor fact", func(t *testing.T) { + // An injection site that wired only the older callback keeps the + // pre-widening (narrower) behavior rather than losing the stop + // altogether: a strict ancestor is by definition ahead == 0, so the + // shape it reports is never "diverged". + fa := &fakeAdopter{ancestorResult: true, cleanResult: true} + err := route(t, fa, false) + var gateErr *RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("expected the data-behind stop via the fallback, got %v", err) + } + if gateErr.FallbackReason != fallbackReasonDataBehind { + t.Errorf("FallbackReason = %q, want %q", gateErr.FallbackReason, fallbackReasonDataBehind) + } + if gateErr.DataDiverged { + t.Error("DataDiverged = true; the IsStrictAncestor fallback can only report ahead == 0") + } + if fa.aheadBehindCalls != 0 { + t.Errorf("AheadBehind calls = %d, want 0 (not wired)", fa.aheadBehindCalls) + } + if fa.ancestorCalls != 1 { + t.Errorf("IsStrictAncestor calls = %d, want 1", fa.ancestorCalls) + } + }) +} + +// TestDataBehindRemedySurfaces pins the review's F1/F3/F4 findings on the +// message and agent contract: every surface has to name the ONE remedy that +// was measured to work, the shared-store consequence has to survive the +// precedence choice that made data-behind outrank shared-store, and the blunt +// migrate-or-adopt options must not be offered — in this state +// `bd migrate --force` is the bug (and its follow-up `bd dolt push` is +// rejected non-fast-forward while the clone is still behind) while +// `bd bootstrap` no-ops against an existing workspace. +func TestDataBehindRemedySurfaces(t *testing.T) { + base := func(shared, diverged bool) *RemoteMigrateGateError { + return &RemoteMigrateGateError{ + CurrentVersion: 66, LatestVersion: 67, Pending: 1, + FallbackReason: fallbackReasonDataBehind, + Shared: shared, + DataDiverged: diverged, + } + } + + t.Run("only the pull is offered", func(t *testing.T) { + for _, diverged := range []bool{false, true} { + e := base(false, diverged) + opts := e.Options() + if len(opts) != 1 || opts[0].ID != "pull-first" { + t.Fatalf("diverged=%v: Options() = %+v, want exactly one pull-first option", diverged, opts) + } + if len(opts[0].Commands) != 1 || opts[0].Commands[0] != DataBehindRemedyCommand { + t.Errorf("diverged=%v: option commands = %v, want [%q]", diverged, opts[0].Commands, DataBehindRemedyCommand) + } + for _, o := range opts { + for _, c := range o.Commands { + if c == "bd migrate --force" || c == "bd bootstrap" { + t.Errorf("diverged=%v: option %q offers %q, which was measured not to work in this state", diverged, o.ID, c) + } + } + } + if d := e.AgentDirective(); !strings.Contains(d, DataBehindRemedyCommand) { + t.Errorf("diverged=%v: AgentDirective must name the remedy:\n%s", diverged, d) + } + if body := e.userBody(); strings.Contains(body, "bd bootstrap") { + t.Errorf("diverged=%v: the body must not send the operator to a no-op bootstrap:\n%s", diverged, body) + } + } + }) + + t.Run("shared store keeps the #5920 lockout warning", func(t *testing.T) { + // F3: before this, Shared+data-behind rendered with neither + // "co-resident" nor "#5920" anywhere, because data-behind outranks + // fallbackReasonSharedStore (whose note carried them) and Options()' + // sharedRisk only ever decorated the adopt option. + msg := base(true, false).UserMessage() + for _, want := range []string{"co-resident", "#5920", SharedConsentCommand} { + if !strings.Contains(msg, want) { + t.Errorf("shared data-behind message is missing %q:\n%s", want, msg) + } + } + if strings.Contains(msg, "proceeds on its own") { + t.Errorf("a shared store must not be promised an automatic retry:\n%s", msg) + } + opts := base(true, false).Options() + if len(opts) != 2 || opts[0].ID != "pull-first" || opts[1].ID != "migrate-shared-after-pulling" { + t.Fatalf("shared Options() = %+v, want pull-first then the consent step", opts) + } + if d := base(true, false).AgentDirective(); !strings.Contains(d, "#5920") { + t.Errorf("shared AgentDirective must carry the co-resident consequence:\n%s", d) + } + }) + + t.Run("non-shared store IS promised the automatic retry", func(t *testing.T) { + msg := base(false, false).UserMessage() + if !strings.Contains(msg, "proceeds on its own") { + t.Errorf("an ordinary clone's retry DOES auto-resolve and should say so:\n%s", msg) + } + if strings.Contains(msg, SharedConsentCommand) { + t.Errorf("a non-shared clone must not be sent to the shared consent verb:\n%s", msg) + } + }) + + t.Run("IsDataBehind is exactly this stop", func(t *testing.T) { + // The store opens key their one narrow exemption on this predicate, so + // it must not widen to any other refusal. + if !base(false, false).IsDataBehind() { + t.Error("IsDataBehind() = false for the data-behind stop") + } + for _, reason := range []string{ + fallbackReasonUnreadableState, fallbackReasonBelowFloor, + fallbackReasonOptedOut, fallbackReasonUnparseableEnv, fallbackReasonSharedStore, "", + } { + e := base(false, false) + e.FallbackReason = reason + if e.IsDataBehind() { + t.Errorf("IsDataBehind() = true for FallbackReason %q", reason) + } + } + // A tailored decision is never this stop either, whatever the reason + // field happens to hold. + e := base(false, false) + e.Decision = gateDecisionForkSkew + if e.IsDataBehind() { + t.Error("IsDataBehind() = true for a fork-skew decision") + } + if (*RemoteMigrateGateError)(nil).IsDataBehind() { + t.Error("IsDataBehind() = true on a nil receiver") + } + }) +} + // TestSmartGateEnabled pins the default-on contract: unset and unparseable // values keep the smart gate active; only an explicit boolean false opts out. func TestSmartGateEnabled(t *testing.T) { diff --git a/internal/storage/versioncontrolops/fastforward.go b/internal/storage/versioncontrolops/fastforward.go index 3387517f5..317da1d04 100644 --- a/internal/storage/versioncontrolops/fastforward.go +++ b/internal/storage/versioncontrolops/fastforward.go @@ -15,17 +15,23 @@ import ( // elsewhere calls these yet — they are wired into the smart migrate gate in // a later change. -// LocalIsStrictAncestorOf reports whether local HEAD is a STRICT ancestor of -// ref in the Dolt commit graph: local has zero commits that ref lacks -// (ahead == 0) and at least one commit that local lacks (behind >= 1). A -// local HEAD equal to ref (ahead == 0, behind == 0) is NOT a strict -// ancestor, and returns false. +// LocalAheadBehind reports how far local HEAD has diverged from ref in the +// Dolt commit graph: ahead is the number of commits local has that ref lacks, +// behind the number ref has that local lacks. (0, 0) means the two are level. +// +// It is the raw fact behind LocalIsStrictAncestorOf, exposed separately +// because the two callers need different slices of it: a fast-forward adopt +// needs "ahead == 0 && behind >= 1", while the data-behind migrate stop +// (gastownhall/beads#6575) needs "behind >= 1" whatever ahead is — a clone +// with local commits of its own AND unpulled remote commits reaches the same +// wedge, and telling the operator which shape they are in changes whether +// their `bd dolt pull` fast-forwards or merges. // // ref must already be present locally (e.g. a cached remote-tracking ref // such as "origin/main" after a fetch); this performs no fetch of its own. -func LocalIsStrictAncestorOf(ctx context.Context, db DBConn, ref string) (bool, error) { +func LocalAheadBehind(ctx context.Context, db DBConn, ref string) (ahead, behind int, err error) { if err := issueops.ValidateRef(ref); err != nil { - return false, fmt.Errorf("invalid ref: %w", err) + return 0, 0, fmt.Errorf("invalid ref: %w", err) } // Dolt's AS OF requires a literal ref, not a bind parameter; ref was @@ -41,11 +47,25 @@ func LocalIsStrictAncestorOf(ctx context.Context, db DBConn, ref string) (bool, (SELECT commit_hash FROM dolt_log)) AS behind `, ref, ref) - var ahead, behind int if err := db.QueryRowContext(ctx, query).Scan(&ahead, &behind); err != nil { - return false, fmt.Errorf("compare local HEAD to %s: %w", ref, err) + return 0, 0, fmt.Errorf("compare local HEAD to %s: %w", ref, err) } + return ahead, behind, nil +} +// LocalIsStrictAncestorOf reports whether local HEAD is a STRICT ancestor of +// ref in the Dolt commit graph: local has zero commits that ref lacks +// (ahead == 0) and at least one commit that local lacks (behind >= 1). A +// local HEAD equal to ref (ahead == 0, behind == 0) is NOT a strict +// ancestor, and returns false. +// +// ref must already be present locally (e.g. a cached remote-tracking ref +// such as "origin/main" after a fetch); this performs no fetch of its own. +func LocalIsStrictAncestorOf(ctx context.Context, db DBConn, ref string) (bool, error) { + ahead, behind, err := LocalAheadBehind(ctx, db, ref) + if err != nil { + return false, err + } return ahead == 0 && behind >= 1, nil } From 1e754a015645c1630321606d86b7909d5d7ca4d5 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 23 Sep 2026 21:59:27 -0700 Subject: [PATCH 08/45] docs(engdocs): proxied-server surfaces an embedder reads, as observed at 1.3.1 (#6652) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs(engdocs): write down the proxied-server surfaces an embedder reads, as observed at 1.3.1 An embedder — an external tool linking bd's Go packages, or one running beside bd against the same workspace — ends up depending on surfaces bd never published: proxy.pid's field names, the db-proxy-child argv, the shape of `bd dolt status --json`, whether a Go symbol is exported or reachable only by type assertion. Eleven of those were asked about; none of them is a contract, and the answer to "will these hold" is no promise. This writes them down anyway, as observed behaviour with the commit they were read at, so that an embedder can build against what the code does instead of guessing. Framing is deliberate and the document says so twice: descriptive, not a compatibility contract; every surface may change in any release, including a patch release. Four are moving in 1.3.1 itself and are given as before/after — `bd dolt status --json`'s reshape (#6580) has landed and is described in its new shape; the `bd migrate schema` refusal, the explicit sidecar `idle_timeout`, and clean-backend-exit handling plus backend-probing adoption (#6651) are decided but had not landed at d487b9796, so each states current behaviour as current. A closing section ranks which surfaces are likelier to move, and a final one lists the claims that would need a running proxy to confirm and were inferred from source instead. Four of the eleven pointers the request carried were off, and are corrected here rather than copied: - The loopback root/no-password fact is not in the proxy's connection handler at all. Only the byte-pump half lives there; the credentials come from newManagedProxiedServerUOWProvider passing "root"/"" in cmd/bd/uow_factory.go, over a generated sql-server config that binds 127.0.0.1 and declares no users. - SetEventsJournalEnabled is not on the Storage interface and is not named in beads_cgo.go. It is storage.EventsJournalConfigurer, an optional interface in internal/storage/storage.go, which is not re-exported publicly — so the assertion has to be written against a locally declared method set. - The shared-store gate's 30s default constant lives in internal/storage/uow/dolt_sql_provider.go, not in the provider that applies it. - The argv pointer stopped at --config and missed --dolt-bin, --database and the four --external-* flags. While verifying #8 a routing fact turned up that an embedder needs more than the interface question: neither copy of OpenBestAvailable has a proxied arm, and IsDoltServerMode and IsDoltProxiedServerMode are mutually exclusive, so on a proxied workspace it falls through to an embedded open over .beads/dolt — the same directory the proxy's sql-server is serving. Documented, with the consequence of that open marked as untested here. Filed under engdocs/ beside EXTENDING.md, which is already the entry point for "not user-facing; for code that embeds bd", linked from it, and registered in DOC_INVENTORY.md's Active Docs and Reference Freshness tables. The Last reviewed / Freshness source markers match check-doc-freshness.sh's format but the doc is deliberately NOT added to that script's enforced list: putting a 90-day tripwire on a release branch is a maintainer's call. scripts/check-doc-flags.sh and scripts/check-doc-freshness.sh both pass. Co-Authored-By: Claude Fable 5 * docs(engdocs): describe the data-port capability generically The prior wording named an integration shape rather than the capability. State what the byte pump plus the generated config actually let an embedder do. Co-Authored-By: Claude Fable 5 * docs(engdocs): correct five factual claims and drop four promise leaks Two adversarial reviews returned FIX_FIRST on the observed-surfaces doc. Corrections, each re-verified against source: - §8: the embedded fall-through does not open the directory the proxy is serving. embeddeddolt hardcodes /embeddeddolt (newStore, cacheKey, HasRepository); the proxy serves the proxied root, default /dolt. The hazard is a silently separate database — empty on a workspace born proxied, stale on one that was ever embedded, since no mode-migration verb removes that directory — not contention. - §10: backend_running carries no omitempty, so five keys are always present, not four. The doc's own external example contradicted its rule. - §3: neither readiness probe uses credentials. Both poll DatabaseServer.Dial, a bare TCP connect. root/"" belongs to bd's SQL open and to DoltServer's shutdown GC. - §2: the --external-* flags need both conditions — external backend AND a set value — and ExternalDoltConfig.Validate makes host/port and socket mutually exclusive, so all four never appear together. - §6: the proxied migrate payload is NOT the direct path's shape. Direct emits {status, applied, latest_version}; proxied drops applied and adds mode and note. - §4: Handle.Kill is SIGKILL on macOS too, and the stop epoch advances once per call rather than per record. Promise leaks removed: the invented "would break the same consumers twice" maintainer policy, flat future tense on an unlanded warning, "safe for an embedder to poll", and "least likely to move" as a stability signal. check-doc-flags.sh and check-doc-freshness.sh both pass. Co-Authored-By: Claude Fable 5 * docs(engdocs): drop a Markdown-unsafe escaped quote from the §8 rewrite Co-Authored-By: Claude Fable 5 * docs(engdocs): re-anchor to the rebase base and record the #6575 data-behind stop Maintainer review fixup for #6652. The doc landed marked `Read at: d487b9796`, but the branch merges into `e1e1158dd`, and five of the sources it names as its freshness basis drifted in between — including §7's sole source. By the doc's own rule that makes the marker wrong, not merely old. - Re-anchor `Read at:`/`Last reviewed:` and the four in-body read-commit references to `e1e1158dd`, and add the sources the re-read reached: `smart_remote_migrate_gate.go`, `internal/storage/embeddeddolt/`, `cmd/bd/remote_migrate_gate.go`, `cmd/bd/store_factory.go`, `internal/storage/dolt/store.go`. - §7: record #6575's data-behind stop — trigger, its reporting ahead of `shared-store`, the collapse to a single `pull-first` option, the exported `IsDataBehind()`/`DataBehindRemedyCommand` an embedder classifies on, and the lenient `OpenForRemoteSync` carve-out for the remedy. - §7: scope the `OpenForRemoteSync` carve-out to the CLI. Both that function and the `dolt.Config.RemoteSyncOpen` field live under `internal/`, and none of the public open paths takes an option that could request the lenient open, so an embedder cannot replicate the remedy in-process: it runs `bd dolt pull` out of process, outside proxied mode, and re-opens. - §7: scope that stop's reachability to the call sites that wire the ancestry adopter, rather than asserting a proxied workspace meets it. `localDataBehind` reads `FastForwardAdopter.AheadBehind` and reports "not behind" when the adopter is nil; of the two production callers of `CheckSharedStoreMigrateGate`, only `internal/storage/dolt`'s server-mode open wires it. `internal/storage/uow/dolt_sql_provider.go` — the proxied CLI path and `bd serve`'s own provider, per `cmd/bd/uow_factory.go` — passes nil, so a proxied, remote-backed clone that is data-behind at schema parity routes to `smartAutoMigrate` and gets the shared-store refusal, never the pull-first one. Embedded mode reaches the stop by the other route, `CheckRemoteMigrateGateWithAdopt` with a fully-wired adopter. The proxied `dolt pull` observation is re-framed to match: the pull is still the precondition and proxied mode still refuses the verb, but the gate never names it there. The underlying gap is base behaviour at this commit. - §7: the trigger is `behind >= 1` whatever `ahead` is, not the narrower strict-ancestor relation — `localDataBehind` states that restricting the stop to `ahead == 0` "would leave the larger half of the affected cohort unprotected". The `DataDiverged` bullet now names `ahead >= 1` as the shape, which is why that pull merges rather than fast-forwards. - §7's "adds two refusals" is deliberately kept. That count is the shared form's delta over the remote-backed flow and mirrors the source comment on `CheckSharedStoreMigrateGate`; the data-behind stop lives in `checkRemoteMigrateGate` underneath, so raising it to three would introduce a new error. The section now says so in place. - §8: four join sites, not three — name `openReadOnly` and scope the create-rather-than-fail claim to the read-write path. - §3: carry the "not verified by running" marker inline beside the MySQL client claim, as §11 already does. - §6: scope the migrate-verb consent to the arm that reads it. The headline and the consent bullet said the verb migrates on a proxied workspace unconditionally, contradicting §7's own already-correct wording. `sharedMigrateConsent` has exactly one non-test read site, `sharedNoRemoteGate`, reached only from `checkRemoteMigrateGate`'s `if !hasRemote` branch; the comment there scopes it deliberately, because with a remote configured #4259's cross-clone fork risk still demands the stronger designated-migrator confirmation. A remote-backed shared or proxied database consults only `--force` / `BD_ALLOW_REMOTE_MIGRATE` and otherwise refuses the pending main-lane migration at the store open. §7's `SharedConsentCommand` hedge is deliberately unchanged. - §6: qualify the `--force` parenthetical in that same consent bullet. `bd migrate schema --force` trips `isForcedMigrate`, but the bare `bd migrate --force` never gets that far on a proxied workspace: `validateProxyMaintenanceBeforeProvider` runs inside the proxied-mode block of the root `PersistentPreRunE`, ahead of `schema.SetForceAllowRemoteMigrate`, and its full-path lookup matches `migrate` and exits with `proxy.migrate.unsupported` before any store opens — the refusal §6's own second paragraph already states. `bd migrate schema` escapes only because its command path contains a space and returns nil first. The clause now scopes the bare verb instead of offering it as an unlock on the one workspace that refuses it. - §6: name the `migrate sync`/`migrate-personal`/`migrate-issues` rows and the history-capability gate that runs before the multi-word fall-through. - Update the "Four of the eleven" count and the `DOC_INVENTORY` freshness row to match. --------- Co-authored-by: Claude Fable 5 --- engdocs/DOC_INVENTORY.md | 3 + engdocs/EMBEDDING_OVER_PROXIED_SERVER.md | 861 +++++++++++++++++++++++ engdocs/EXTENDING.md | 9 + 3 files changed, 873 insertions(+) create mode 100644 engdocs/EMBEDDING_OVER_PROXIED_SERVER.md diff --git a/engdocs/DOC_INVENTORY.md b/engdocs/DOC_INVENTORY.md index 76d821f3e..d48da2c1d 100644 --- a/engdocs/DOC_INVENTORY.md +++ b/engdocs/DOC_INVENTORY.md @@ -38,6 +38,7 @@ freshness source. | `SERVE_RUNBOOK.md` | `Last reviewed:` marker tied to the operating-envelope constants in `internal/httpapi/server.go` and the log fields in its `event`/`request` emitters. | | `CI_CLEANUP_PLAN.md` | `Last reviewed:` marker tied to the CI audit, workflow files, wrapper scripts, `Makefile`, package manifests, and maintainer decision review. | | `design/otel/otel-data-model.md` | `Last reviewed:` marker tied to telemetry, Dolt storage, hooks, and AI call sites. | +| `EMBEDDING_OVER_PROXIED_SERVER.md` | `Last reviewed:` marker naming the commit it was read at, tied to `internal/storage/dbproxy/`, `internal/storage/uow/`, `internal/storage/schema/` gate and lock, `cmd/bd/dolt_proxied_lifecycle.go`, and the root `beads*.go` open path down into `internal/storage/dolt/` and `internal/storage/embeddeddolt/`. Re-read against those sources per release; the doc describes observed behaviour, so a stale marker makes it wrong rather than merely old. | Follow-up automation should replace marker-only checks with generated or `--check` blocks where a clean code source exists. @@ -81,7 +82,9 @@ Follow-up automation should replace marker-only checks with generated or | `DOLT-BACKEND.md` | Consolidated pointer | Old duplicate staged; stable path points to canonical `DOLT.md`. | | `DOLT.md` | Keep/canonical | Canonical Dolt backend guide. | | `ERROR_HANDLING.md` | Keep with freshness | Pattern guide with code-linked examples; marker added. | +| `EMBEDDING_OVER_PROXIED_SERVER.md` | Keep with freshness | Observed proxied-server surfaces an embedder reads (pid file, child argv, status JSON, optional interfaces). Descriptive, not a compatibility contract; marker tied to the dbproxy, uow, schema-gate and proxied-lifecycle sources it quotes. | | `EXCLUSIVE_LOCK.md` | Keep | Behaviour/design doc for lock protocol. | +| `EXTENDING.md` | Keep | Storage-API contracts callers must honor; entry point for embedder docs. | | `FAQ.md` | Revise | Opening wording now describes beads as Dolt-powered; stale pre-1.0 status removed. | | `FEDERATION.md` | Keep/canonical | Canonical federation (peer-to-peer sync) guide; moved from root `FEDERATION-SETUP.md`, which is now a stable pointer. Absorbed the duplicate federation section previously in `DOLT.md`. | | `GIT_INTEGRATION.md` | Keep | User-facing git/worktree/hook behaviour. | diff --git a/engdocs/EMBEDDING_OVER_PROXIED_SERVER.md b/engdocs/EMBEDDING_OVER_PROXIED_SERVER.md new file mode 100644 index 000000000..8ca5057ae --- /dev/null +++ b/engdocs/EMBEDDING_OVER_PROXIED_SERVER.md @@ -0,0 +1,861 @@ +# Embedding over a proxied-server workspace: observed behaviour at 1.3.1 + +Last reviewed: 2026-09-23 + +Read at: `e1e1158dd` (`v1.3.1-rc.1-2-ge1e1158dd`). Because this document +describes observed behaviour rather than a contract, a stale marker makes it +wrong, not merely old. Re-read it against the sources below before trusting it +against a later build. + +Freshness source: `internal/storage/dbproxy/pidfile/pidfile.go`, +`internal/storage/dbproxy/proxy/` (`endpoint.go`, `server.go`, `shutdown.go`, +`read_status.go`, `backend.go`), `internal/storage/dbproxy/server/doltserver.go`, +`internal/configfile/proxied_server_client_info.go`, +`internal/configfile/configfile.go` (the storage-mode predicates), +`internal/doltserver/physical_root.go`, +`internal/storage/uow/doltserver_provider.go`, +`internal/storage/uow/dolt_sql_provider.go`, +`internal/storage/schema/remote_migrate_gate.go`, +`internal/storage/schema/smart_remote_migrate_gate.go`, +`internal/storage/schema/lock.go`, `internal/storage/storage.go`, +`internal/storage/dolt/store.go` (the server-mode shared-gate call site), +`internal/storage/embeddeddolt/` (`store.go`, `cache.go`, `detection.go`), +`cmd/bd/db_proxy_child.go`, `cmd/bd/dolt.go`, +`cmd/bd/dolt_proxied_lifecycle.go`, `cmd/bd/migrate.go`, `cmd/bd/main.go`, +`cmd/bd/proxied_server.go`, `cmd/bd/proxy_capability.go`, +`cmd/bd/remote_migrate_gate.go`, `cmd/bd/store_factory.go`, +`cmd/bd/uow_factory.go`, `beads.go`, `beads_cgo.go`, `beads_nocgo.go`. + +## What this document is, and is not + +An embedder — an external tool that links bd's Go packages, or one that runs +beside `bd` against the same workspace — ends up leaning on surfaces bd never +published: a PID file's field names, a child process's argv, the shape of a +`--json` payload, whether a Go symbol is exported. This document writes those +down as **observed behaviour at 1.3.1**, so that an embedder can build against +what the code actually does instead of guessing. + +It is not a compatibility contract. Nothing here is a published interface with a +deprecation policy behind it. Every surface below is an implementation detail +that bd's own code reaches through, and each one may change in any release, +including a patch release. Five of the eleven are moving in 1.3.1 itself and say +so in place: §7 and §10 have landed and are described in their new shape, and +§5, §6 and §11 are decided for 1.3.1 but had not landed at the commit above, so +each is given as before/after with the current behaviour marked as current. Read +this as a snapshot with the commit it was taken at, and re-read it against the +source before an upgrade. + +Two things follow for anyone building on this: + +- **Prefer the front door.** Where `bd` offers a command or a `--json` payload + that answers your question, go through it. The control files under the proxied + root are bd's private lifecycle state; reading them means tracking bd's + lifecycle work release by release. +- **Fail loudly on a shape you do not recognise.** A missing JSON key, an + unexpected `schema` number, a new argv flag — treat these as "this bd is not + the one I was written against", not as a default to paper over. + +Everything below was checked by reading the source at `e1e1158dd`. Where a claim +would need a running proxy to confirm end to end, it says so. + +## The topology being described + +"Proxied-server" mode puts a small bd-owned TCP proxy between every bd client +and a `dolt sql-server`. The proxy is a separate detached process (`bd +db-proxy-child`), it starts on demand when a command needs the store, it owns +the lifetime of its `dolt` child, and it exits when idle. The workspace's +**proxied root** holds the proxy's control files; it defaults to `.beads/dolt` +and is resolved by `doltserver.ResolveProxiedServerRootPath` from, in order: +`BEADS_PROXIED_SERVER_ROOT_PATH`, the sidecar's `root_path`, then the default. + +The control files in that root, as enumerated by `proxy.ControlFilePaths`: + +| File | Written by | Holds | +|---|---|---| +| `proxy.pid` | the proxy | the published endpoint record (§1) | +| `proxy.lock` | the proxy | held for the proxy's whole lifetime | +| `proxy.log` | the proxy | proxy trace log | +| `proxy.secret` | the proxy | 64 hex chars authenticating the control port | +| `proxy.spawn` | the spawning parent | in-progress-start marker | +| `proxy.stop-epoch` | `bd dolt stop` | monotonic token that dooms in-flight starts | +| `proxy-child.pid` | the proxy | the `dolt sql-server` record | +| `proxy-child.lock` | the proxy | held while the backend runs | +| `.stale-` | either | quarantined records, swept after 30 days | + +Two topologies share this machinery: **managed-local**, where bd spawns the +`dolt sql-server` itself, and **external**, where the proxy fronts a +`dolt sql-server` somebody else runs. They differ in several of the surfaces +below, and the difference is usually the part an embedder gets wrong. + +--- + +## 1. The `proxy.pid` record + +`internal/storage/dbproxy/pidfile/pidfile.go` — one `PidFile` type serves both +records. JSON keys, in declaration order: + +```json +{ + "pid": 251973, + "port": 33825, + "upstream_id": "…", + "schema": 2, + "kind": "db-proxy", + "birth": "…", + "root_id": "…", + "control_port": 41277 +} +``` + +`pid` and `port` are always present. Every other field carries `omitempty`, so a +zero value is absent from the file rather than written as `0` or `""`. + +- `port` is the **data port** — the loopback MySQL-protocol endpoint bd clients + connect to. +- `control_port` is a separate loopback listener that answers an authenticated + identity challenge using the secret in `proxy.secret`. It is not a data port. +- `kind` is `"db-proxy"` in `proxy.pid` and `"dolt-backend"` in + `proxy-child.pid` (`pidfile.KindProxy` / `pidfile.KindDoltBackend`). +- `schema` is `2` today (`pidfile.SchemaV2`). `PidFile.ValidateV2` accepts + `schema >= 2`, so a later bd may raise it without bd's own readers rejecting + the record; a reader of your own should do the same rather than equality-match. +- `birth` is a platform-specific process-identity token, and `root_id` is the + SHA-256 of the symlink-resolved absolute proxied root. Treat both as opaque. + +`ValidateV2` rejects a record with `schema < 2`, `pid <= 0`, a `port` outside +1–65535, a `control_port` that is neither 0 nor 1–65535, a `kind` other than the +expected one, or an empty `birth`. + +**What an embedder can observe, and the trap.** Reading `port` out of +`proxy.pid` is not the same as knowing a proxy is live on it. Before bd adopts a +record (`proxy.readAndDial`) it also verifies `birth` against the live PID, +completes the authenticated challenge on `control_port` and cross-checks every +field of the reply against the file and the workspace, and TCP-probes the data +port. A record left behind by a killed proxy passes a naive parse and points at +a port that may since have been recycled by an unrelated process. If you read +this file at all, verify liveness yourself before connecting — or use `bd dolt +status --json` (§10), which performs that whole check for you. + +## 2. The `db-proxy-child` argv + +`proxy.forkExecChild` in `internal/storage/dbproxy/proxy/endpoint.go` builds the +argv; `cmd/bd/db_proxy_child.go` declares the flags. The command is hidden and +is not meant to be invoked by hand. + +Always present, in this order: + +``` +bd db-proxy-child --root --port --idle-timeout --backend --stop-epoch +``` + +Then, each only when its value is non-empty: + +``` +--config --logpath --dolt-bin --database +``` + +And, only when `--backend` is `external` — and then, like the group above, each +only when its own value is set (non-empty string, non-zero number or duration): + +``` +--external-host --external-port --external-socket-path --external-keep-alive +``` + +Both conditions matter, because neither alone predicts the argv. The group is +skipped wholesale on the other two backends, and inside it you never see all +four: `ExternalDoltConfig.Validate` rejects a socket set alongside a host or +port, and refuses a host without a port, so a socket topology carries only +`--external-socket-path` and a TCP one carries `--external-host` *and* +`--external-port` together. `--external-keep-alive` is the one flag whose +absence is a default rather than a gap — the child's flag help documents 30s — +so read a missing one as "unset", not as "off". + +`--root`, `--port` and `--backend` are the required flags. `--backend` takes one +of `local-server`, `external`, or `local-shared-server` — the last is declared +but returns "not yet implemented" from `newDatabaseServer`. + +Notes on three values an embedder is likely to read: + +- **`--root`** is always an absolute path. The provider resolves it with + `filepath.Abs` before the proxy is opened. +- **`--port 0`** is the normal case, not a literal port: the child binds an + OS-assigned port and publishes it in `proxy.pid`. A non-zero value is only + passed when the workspace pins one. +- **`--idle-timeout`** is a Go duration string. Negative means never, and the + parent normalises *any* negative value to `proxy.IdleTimeoutNever` (`-1`), so + the argv you will actually see for "never" is `--idle-timeout -1ns`. The + child's own flag help states `0 or negative = never shut down`, which matches + the watcher in §4. This is the third of three different encodings of the same + setting — see §5 before writing one. + +## 3. A loopback root / no-password client on the data port + +Two separate facts combine here, and they live in different files. + +**The proxy does not parse what it carries.** `proxyServer.handleConn` in +`internal/storage/dbproxy/proxy/server.go` dials the backend once per accepted +client connection and then runs two `io.Copy` goroutines, client→backend and +backend→client, in an errgroup. There is no MySQL protocol awareness, no +connection pooling and no multiplexing: the proxy is a byte pump, and *N* client +connections become *N* backend connections. When either direction finishes, or +the proxy's context is cancelled, both sides are closed. + +**The managed-local backend authenticates as `root` with no password.** In +`newManagedProxiedServerUOWProvider` (`cmd/bd/uow_factory.go`) bd passes the +literal `"root"` and `""` to the provider, commented "proxy is loopback-only, no +auth". The generated `dolt sql-server` config (`renderProxiedServerConfig` in +`cmd/bd/proxied_server.go`) binds `127.0.0.1` on a free port, sets log level +`info` and auto-GC archive level `0`, and declares no users section, so Dolt's +default `root` with an empty password applies. The same pair turns up once more +inside the proxy's own process tree, in `DoltServer`'s shutdown-GC connection +(`internal/storage/dbproxy/server/doltserver.go`). + +**The readiness probes prove nothing about credentials.** Both of them — +`DoltServer.waitReady` and the proxy's `waitForServerReady` +(`internal/storage/dbproxy/proxy/server.go`) — poll `DatabaseServer.Dial`, which +is a bare `net.Dialer` connect to the backend's host/port or socket, closed again +the moment it succeeds. No handshake is attempted and no credentials are +supplied, so "ready" here means only that something is listening. That is worth +knowing in both directions: it is why the probes cannot be cited as evidence of +the auth posture, and it is the same shallowness §11 turns on. + +**What an embedder can observe.** On a managed-local proxied workspace, a +plain MySQL client connecting to `127.0.0.1:` as `root` +with no password reaches the same database bd does, and the proxy passes the +handshake through untouched. That is what lets an embedder read and write the +workspace's live database without standing up a server of its own. +*Not verified by running:* this is the composition of three separately-read +facts — the credential pair, the generated config's loopback bind with no users +section, and the proxy's lack of protocol handling — not an observed connection. +The proxy dials the backend per accepted connection and closes the client on any +dial or handshake failure, so treat a successful connect as the thing to check +first, not as given. + +Three qualifications worth carrying: + +- It follows from the *generated config*, not from a decision to offer an + authentication posture. A workspace that supplies its own server config, or + the external topology (where credentials come from `ExternalDoltConfig` and + the password from that config's environment variable), does not behave this + way. +- The loopback bind is the whole access control. Anything that can open a TCP + connection to that port on the host is `root` on the database. +- The **control** port in the same record is not open in this sense: it requires + the `proxy.secret` value. + +## 4. Open connections, idle exit, and `bd dolt stop` + +**Any accepted connection defers idle exit.** `proxyServer.idleWatcher` +(`internal/storage/dbproxy/proxy/server.go`) returns immediately into a wait +when `idleTimeout <= 0` — that is the "never" case. Otherwise it ticks every +`idleTimeout/4`, with a 1s floor. A tick that sees `activeConns > 0` clears the +armed state; the first tick that sees zero *arms* a timestamp; a later tick then +exits once `timeout` has elapsed since arming. `activeConns` is incremented in +`handleConn`, so an accepted TCP connection counts even if it never sends a +byte. + +Two consequences an embedder should plan for: + +- Holding one idle connection open keeps the proxy and its backend alive + indefinitely. That falls out of how the watcher counts, so it is as much an + easy way to pin the tree up by accident as a deliberate one — and it is a + property of the watcher, not an offer. +- Exit is not punctual. Arming costs a tick, so the observed lifetime after the + last connection closes is longer than the configured timeout — up to roughly + two ticks longer. + +**`bd dolt stop` does not wait for clients.** On a proxied workspace (`cmd/bd/dolt.go`) +the command calls `proxy.Shutdown(rootDir)`, which: + +1. advances `proxy.stop-epoch`, which makes every start attempt already in + flight abort terminally rather than retry; +2. acquires `proxy.lock`, re-reads `proxy.pid`, validates it against the + workspace, and **kills the recorded process** — `procid.Handle.Kill` is + `SIGKILL` on Linux and macOS, `os.Kill` on Windows; +3. waits only for that *process* to be gone (`max(remaining of 5s, 2s)`), then + removes the record; +4. repeats steps 2 and 3 for `proxy-child.pid` — the stop epoch is advanced once + for the whole call, not per record. + +There is no drain phase, no notification to connected clients and no wait on +`activeConns` anywhere in that path. An embedder's in-flight query is severed +when the proxy dies. The proxy does install SIGTERM/SIGINT handlers for its own +graceful path, but `bd dolt stop` does not use them, and even that path +force-closes both sides of every connection on context cancellation rather than +draining. + +If `Shutdown` refuses because a record's identity cannot be verified, +`bd dolt stop --force` escalates to `proxy.ForceStopUnverified`; the refusal is +recognisable with `proxy.CanForceStopUnverified`. + +## 5. The sidecar's `idle_timeout` — changing in 1.3.1 + +One setting has three encodings, and they disagree about what `0` means. Getting +this wrong is how a workspace ends up with a proxy that never exits, or one that +exits under a long-running embedder. + +**(a) The CLI flags.** `bd init --proxied-server-idle-timeout` and the +`--idle-timeout` flag on the proxied-mode migrate verbs share a convention: +omitting the flag selects the built-in default, an explicit `0` means *never*, +and a negative value is refused outright ("must be 0 (never) or a positive +duration"). `resolveMigrateIdleTimeout` and `init.go` both map that explicit `0` +to `proxy.IdleTimeoutNever` before storing it. + +**(b) The sidecar file.** +`.beads/proxied_server_client_info.json`, typed by +`configfile.ProxiedServerClientInfo`: + +The full key set is `root_path`, `config_path`, `log_path`, `port`, +`idle_timeout`, `external` — and **every one of them carries `omitempty`**, so a +realistic managed-local sidecar with an OS-assigned port and an explicit 45s +timeout is just: + +```json +{ + "root_path": "/abs/path/.beads/dolt", + "idle_timeout": 45000000000 +} +``` + +`port: 0` (the OS-assigned case), a nil `external`, and unset paths are all +absent rather than written as `0`, `null` or `""`. Paths, when present, may be +relative to the `.beads` directory; the `Resolved*Path` helpers on the type are +what bd uses to join them. + +`idle_timeout` is a Go `time.Duration` with no custom marshaller, so it is **an +integer number of nanoseconds**, not a duration string: 45s is `45000000000`, +30s is `30000000000`, never is `-1`. Combined with `omitempty`, that means a +value of `0` is written as *no key at all*. + +**(c) The effective value.** `NewDoltServerUOWProvider` +(`internal/storage/uow/doltserver_provider.go`) applies +`if idleTimeout == 0 { idleTimeout = defaultProxyIdleTimeout }`, and +`defaultProxyIdleTimeout` is **30s** (`internal/storage/uow/dolt_sql_provider.go`). +The external provider does the same. So, at the sidecar layer: + +| sidecar `idle_timeout` | effective | +|---|---| +| key absent | 30s | +| `0` (cannot occur — `omitempty` drops it) | 30s | +| negative, e.g. `-1` | never | +| positive | as written | + +**What is changing in 1.3.1.** Writing the *effective* `idle_timeout` +explicitly, rather than letting `omitempty` drop it and relying on the +provider-side default, is approved for 1.3.1. It had not landed at +`e1e1158dd`, and there is no tracking issue for it at the time of writing. + +Before (observed at this commit): a workspace on the default timeout has **no** +`idle_timeout` key in its sidecar, and "default" is therefore indistinguishable +from "unset" to any reader. That also makes the effective value invisible in +`bd dolt status --json`, which only emits `idle_timeout` when the sidecar value +is `> 0` (§10) — so both "default 30s" and "never" render as an absent key. + +After: the key is expected to be present with the effective value. An embedder +should read an absent key as "ask this bd, do not assume", not as "no timeout". + +## 6. `bd migrate schema` on a proxied workspace — changing in 1.3.1 + +**Observed at this commit: the verb is permitted, and on a served database with +no remote it migrates. With a remote configured it is still permitted — it +reaches the store open — but the §7 gate refuses the migration there, because +the verb's own consent is not read on that arm.** + +`proxyMaintenanceRefusals` in `cmd/bd/proxy_capability.go` refuses the bare +`migrate` command with code `proxy.migrate.unsupported`, and its `init()` +expands child rows for `migrate hooks` and `migrate issues`. The same table +carries `migrate sync`, `migrate-personal` and `migrate-issues` as rows of their +own — the `migrate` neighbourhood is enumerated, not covered by a prefix rule. +`migrate schema` is in none of it. + +Two gates then stand between an unmatched path and the store open, and only the +second is a fall-through: `validateProxyMaintenanceBeforeProvider` consults +`LookupHistoryCapability` first and refuses anything classed `HistoryDirectOnly` +with `proxy.history.unsupported`; *after* that it returns `nil` for any +unmatched multi-word command path. `migrate schema` matches neither gate, so the +verb reaches the store open. A reader predicting some other multi-word verb from +this section should apply both gates in that order — the multi-word `return nil` +on its own over-predicts "permitted". + +The mechanics matter, because they are not "a command that applies a migration": + +- `cmd/bd/main.go` sets `schema.SetSharedMigrateConsent(...)` in the root + pre-run for this one verb (`isSchemaMigrateVerb` matches `bd migrate schema` + and deliberately nothing else in the `migrate` tree). That consent satisfies + the shared-store gate in §7 on **one arm only**. Its single non-test read site + is `sharedNoRemoteGate`, reached from that gate's `if !hasRemote` branch, and + the comment there scopes it deliberately: the verb "unlocks ONLY here: with a + remote configured, #4259's cross-clone fork risk still demands the stronger + designated-migrator confirmation". When the served database *does* have a + remote — on the proxied path that is read from its own `dolt_remotes` alone, + because `dolt_sql_provider.go` calls the gate as + `CheckSharedStoreMigrateGate(ctx, c, "", nil, nil)` and so supplies no on-disk + fallback probe — the gate takes the other arm, which consults only `--force` + (`bd migrate schema --force` trips `isForcedMigrate`; `bd migrate --force` + trips it too, but only off a proxied workspace, since the bare verb is + refused before the store open per the refusal above) and the + `BD_ALLOW_REMOTE_MIGRATE` environment override. + Without one of those, a main-lane pending migration is refused at the store + open, carrying the shared-store reason wherever the smart gate would otherwise + have auto-migrated. The public twin of this knob, + `beads.AllowSharedSchemaMigration`, documents the same scope. +- The **provider open then performs the migration**, before `RunE` runs. +- `RunE`'s proxied arm (`reportProxiedSchemaMigrate` in `cmd/bd/migrate.go`) + therefore only reports, and it reports a **different JSON shape** from the + direct path: + +```json +{ + "status": "current", + "latest_version": 66, + "mode": "proxied-server", + "note": "schema reconciled during provider open" +} +``` + +(`latest_version` is `schema.LatestVersion()`, shown here illustratively — read +the value, do not hard-code it.) The direct path emits `{"status", "applied", +"latest_version"}`; the proxied arm drops `applied` and adds `mode` and `note`, +so only `status` and `latest_version` are common to both. No applied count is +reported because by the time `RunE` runs the work belongs to an open that has +already returned — there is no number for it to have. An embedder decoding this +payload into one struct across both topologies should treat `applied` as +absent-not-zero on the proxied route, and should not infer "nothing was applied" +from its absence. + +**What an embedder can observe.** The DDL lands under co-resident clients that +already hold connections and prepared state, with no coordination of any kind — +the proxy is a byte pump (§3) and has no notion of who is connected. An embedder +whose process outlives a `bd migrate schema` should treat the schema as changed +beneath it and re-establish its connections; in practice, restart. + +**What is changing in 1.3.1.** Refusing `bd migrate schema` on a proxied +workspace is approved for 1.3.1: a flat typed refusal by default, with `--force` +emitting a **generic** warning to close co-resident library clients first. The +approved scope is a generic warning with no enumeration of connected clients, for +the reason above — the proxy does not track them, so there is nothing truthful to +list. This had not landed at `e1e1158dd`, and there is no tracking issue for it +at the time of writing. + +So an embedder should expect the verb to stop working on proxied workspaces +rather than build a workflow on it, and should not read the absence of a refusal +today as the absence of one tomorrow. + +## 7. The shared-store migration gate — main lane only, new stop in 1.3.1 + +`schema.CheckSharedStoreMigrateGate` +(`internal/storage/schema/remote_migrate_gate.go`) is the gate for a database +served to co-resident clients, which includes every proxied workspace. On top of +the remote-backed flow it adds two refusals: with no remote it refuses a silent +in-place migration unless the operator consented (the verb in §6, `--force`, or +the environment override), and with a remote it suppresses the two auto-execute +arms, because on a shared server neither can observe the other clients. + +Its decision facts are read from the **main lane only**. The shared arm delegates +to `checkRemoteMigrateGate`, which calls `schema.CurrentVersion` and +`schema.PendingVersions` — both of which resolve to `mainSource`. The +`dolt_ignore`d lane has its own cursor and its own accessors +(`CurrentIgnoredVersion`, `PendingIgnoredVersions`), and the gate consults +neither. + +`internal/storage/schema/lock.go` shows where that lands: the gate runs under +the database-scoped migration lock immediately before `MigrateUp`, and +`MigrateUp` reconciles **both** lanes. + +**What an embedder can observe.** A main-lane pending migration is what a +refusal is about. An open whose only pending work is on the ignored lane sees +the gate return early (`len(pending) == 0` — nothing to migrate) and then watches +`MigrateUp` apply that ignored-lane work anyway. So "the gate did not refuse" +does not mean "no schema changed under me". Two related carve-outs also let work +through: a fresh database (`current == 0`) always migrates, on the reasoning that +creating a database is consent for its schema, and a fresh-bootstrap heal +authority from the same logical open bypasses the gate entirely. + +**What changed in 1.3.1: a third refusal shape arrives from underneath.** #6575 +(landed at this commit) added `fallbackReasonDataBehind` to +`checkRemoteMigrateGate`. It is not a fourth item in the "two refusals" list +above — that count is the shared form's delta over the remote-backed flow, and it +is unchanged. This stop lives in the flow *both* forms delegate to, so the +shared gate inherits it rather than adding it. + +**Whether it can fire is a property of the call site, not of the gate.** The +stop's only input is an ancestry fact the *caller* injects: `localDataBehind` +(`internal/storage/schema/smart_remote_migrate_gate.go`) reads +`FastForwardAdopter.AheadBehind`, and reports "not behind" whenever the adopter +is nil or wires neither callback. Its own comment names "the shared unit-of-work +provider's injection site" as exactly that case, where "routing stays exactly as +it was before this check existed". At this commit there are two production +callers of `CheckSharedStoreMigrateGate`: + +- `internal/storage/dolt`'s store open — the **server-mode** open — wires + `AheadBehind` (added expressly for #6575), so the stop is live there. +- `internal/storage/uow/dolt_sql_provider.go` passes `nil`, and per + `cmd/bd/uow_factory.go` that funnel is "the proxied CLI path AND `bd serve`'s + own provider". A **proxied** workspace therefore reaches the shared gate with + no ancestry fact at all. + +So do not expect this refusal on a proxied workspace. A proxied, remote-backed +clone that is data-behind at schema parity reads as level, the smart gate routes +to `smartAutoMigrate`, and the shared arm then returns the ordinary +`fallbackReasonSharedStore` refusal — the one whose framing ("the gate would +have resolved and was overruled") the data-behind comment says is *not* true of +this state. Embedded mode reaches the stop by a different route: it calls +`CheckRemoteMigrateGateWithAdopt` with a fully-wired adopter +(`internal/storage/embeddeddolt/store.go`), which is why the `OpenForRemoteSync` +carve-out below has anything to carve. That the proxied injection site reads no +ancestry is base behaviour at this commit — #6575's protection is absent on the +path this section is about — and it is not this document's to fix. + +Where the stop *is* live, it fires when schema parity with the cached remote ref +holds but this clone's local HEAD is **missing commits the cached ref has** +(`behind >= 1`, whatever `ahead` is — a strict ancestor only when this clone has +no commits of its own). The clone is behind in data commits it has not pulled, +so it is not the first-mover the smart gate may auto-resolve. The wide predicate +is deliberate: because bd auto-commits every write, `localDataBehind` reasons +that "has local commits AND is behind" is the ordinary multi-machine state, so +stopping only at `ahead == 0` "would leave the larger half of the affected +cohort unprotected". Two details make it matter more than its size suggests to +an embedder classifying gate errors: + +- **It is reported on shared stores too.** The `smartDataBehind` / + `smartDataDiverged` arms set this reason ahead of `fallbackReasonSharedStore`, + which would ordinarily outrank it. The comment gives the reason: the + shared-store reason says the gate *would* have resolved and was overruled, + which is not true here — the gate refuses this state on its own merits. +- **Its options collapse.** `Options()` returns a single `pull-first` option + whose command is `DataBehindRemedyCommand` (`bd dolt pull`), instead of the + usual migrate-or-adopt pair; on a shared store a second + `migrate-shared-after-pulling` option follows it. The `DataDiverged` variant + is the same reason and the same command; it is the `ahead >= 1` shape — the + clone also has commits of its own — which is *why* its pull **merges** (and + can report conflicts) rather than fast-forwarding. + +**What an embedder can observe.** This is the one gate refusal with a +one-command operator remedy, and it is the one the gate exports API for: +`(*schema.RemoteMigrateGateError).IsDataBehind()` and the +`schema.DataBehindRemedyCommand` constant. Code that classifies gate errors +should test `IsDataBehind()` before treating a `RemoteMigrateGateError` as +fatal — the doc's own "fail loudly on a shape you do not recognise" rule cuts +the other way here, because this shape *is* recoverable and the exported +predicate exists precisely to say so. + +The remedy is itself a store-opening command, so bd carves a path for it: +`isRemoteSyncCommand` in `cmd/bd/main.go` matches `bd dolt pull` alone and sets +`RemoteSyncOpen`, which routes the embedded open to +`embeddeddolt.OpenForRemoteSync` (`cmd/bd/store_factory.go`) — a lenient open +for this gate reason and no other. That carve-out is CLI-only, and an embedder +cannot replicate it in-process: both `embeddeddolt.OpenForRemoteSync` and the +`dolt.Config.RemoteSyncOpen` field live under `internal/`, unimportable from +outside `github.com/steveyegge/beads`, and none of the public open paths +(`OpenBestAvailable`, `Open`, `OpenFromConfig`, `OpenGated`) takes an option +that could request the lenient open. The embedder's remedy is therefore +out-of-process: run `bd dolt pull` (and, per the first observation below, +outside proxied mode), then re-open. A normal in-process open will hit the same +refusal that prescribed the command. + +Two observations worth carrying, neither of them this document's to fix: + +- On a **proxied** workspace the pull is still the precondition, the workspace + still refuses the verb, and the gate never names it. + `proxyMaintenanceRefusals` in `cmd/bd/proxy_capability.go` holds + `"dolt pull"` with code `proxy.dolt_pull.unsupported`, so the one command a + data-behind clone needs is declined by that same workspace. And because the + proxied call site wires no ancestry adopter (above), the gate does not + prescribe it there either: the operator gets the shared-store refusal, with no + mention of the data they are missing. That is the worse of the two shapes — a + named remedy you must route around is at least diagnosable. An embedder that + surfaces gate remedies to users should expect to detect the data-behind state + itself on this path, and to pull outside proxied mode to clear it. +- The shared store's second option names `SharedConsentCommand` + (`bd migrate schema`), whose own declaration scopes it to "a shared database + that has **no** remote". Do not read that option as a confirmed unlock for the + remote-backed shared case; treat the pull as the part of the remedy this + document vouches for. + +## 8. `SetEventsJournalEnabled` on `OpenBestAvailable`'s `Storage` + +`OpenBestAvailable` is the public open path in package `beads` at the module +root. It exists in two build-tagged copies: `beads_cgo.go` (`//go:build cgo`), +which handles embedded Dolt, dolt-server mode and registered extension +backends; and `beads_nocgo.go` (`//go:build !cgo`), which handles the latter two +and returns an error for embedded Dolt. + +**First, a routing fact worth knowing before the interface question.** Neither +copy has a proxied-server arm. Both dispatch on `cfg.IsDoltServerMode()`, and +`configfile.IsDoltServerMode` and `IsDoltProxiedServerMode` are mutually +exclusive — proxied workspaces are explicitly exempt from the host-based +server-mode inference. So on a proxied-server workspace `OpenBestAvailable` falls +through to `embeddeddolt.Open`, and it does not connect through the proxy. + +The consequence is not contention over the directory the proxy is serving. It is +that the two paths look at **different directories entirely**. `embeddeddolt` +joins its data directory as `/embeddeddolt` and does so in four +non-test places — `newStore`, `openReadOnly`, the open cache's `cacheKey`, and +`HasRepository` — with no way for a caller to redirect it. The proxy's +`dolt sql-server`, meanwhile, serves the proxied root, which defaults to +`/dolt` +(`internal/doltserver/physical_root.go`; `Config.DatabasePath`'s fallback joins +the literal `dolt`, with a comment saying always to). + +So an embedder that calls `OpenBestAvailable` on a proxied workspace does not +fight the running server for its files — it silently opens **a separate database** +beside it. On the read-write path — the one `OpenBestAvailable` takes — +`newStore` `MkdirAll`s `.beads/embeddeddolt`, and `initSchema` issues +`CREATE DATABASE IF NOT EXISTS` and migrates, so the open creates and writes a +second database rather than failing. What it contains depends on the workspace's +history: empty on one initialised straight into proxied mode, and whatever +`.beads/embeddeddolt` still holds on one that was ever embedded, since none of +the mode-migration verbs removes that directory. The second case is the nastier +one — stale issues that look real. + +The read-only join site behaves the other way, which is worth knowing before +generalising the paragraph above: `openReadOnly` does **not** create. It +`os.Stat`s the same directory and returns +`embeddeddolt: no embedded database at …` when it is absent, so a caller +arriving through the read-only open on a workspace that was never embedded gets +an error instead of an empty database. + +Nothing in the open path refuses on that ground and no error comes back. The +store is genuine; it is just not the one the workspace's issues are in. An +embedder that wants the proxied endpoint should connect to the data port (§3) +rather than expect this function to find it. + +`SetEventsJournalEnabled(enabled bool)` is **not a method on the `Storage` +interface.** `beads.Storage` aliases `internal/beads.Storage`, which aliases +`internal/storage.Storage`; the method is declared on a separate optional +interface, `storage.EventsJournalConfigurer` +(`internal/storage/storage.go`). Its contract is per-instance: implementations +must not use process-global state, because one process can hold several stores +at once. + +The two concrete types `OpenBestAvailable` can return from its Dolt arms do +implement it: `dolt.DoltStore` and `embeddeddolt.EmbeddedDoltStore` each declare +the method. (bd's own proxied path forwards it through the unit-of-work provider, +but that provider is not something `OpenBestAvailable` returns — see the routing +fact above.) An embedder reaches it by type assertion: + +```go +store, err := beads.OpenBestAvailable(ctx, beadsDir) +// … +if c, ok := store.(interface{ SetEventsJournalEnabled(bool) }); ok { + c.SetEventsJournalEnabled(true) +} +``` + +**Two things an embedder should know.** `storage.EventsJournalConfigurer` is +internal and is not re-exported from the public `beads` package, so the +assertion has to be written against a locally declared single-method interface +as above; there is no exported name to assert against. And a store reached +through a registered extension backend +(`backends.Lookup`, dispatched ahead of every Dolt path in both copies of +`OpenBestAvailable`) need not implement the method at all — which is what the +`ok` is for. + +This is among the more likely surfaces here to move, because it is reached by +assertion rather than by a declared method: no compile error marks its +disappearance. No test at the module root asserts that +`OpenBestAvailable`'s result satisfies it, so nothing in CI fails if it stops +doing so. + +## 9. `ErrCommitIndeterminate` and `ErrCircuitOpen` + +Both are exported from the public `beads` package (`beads.go`) as re-exports of +the values they wrap: + +```go +var ( + ErrCircuitOpen = dolt.ErrCircuitOpen + ErrCommitIndeterminate = storage.ErrCommitIndeterminate +) +``` + +Because they are the same error values the internal packages return — not +copies with equal text — `errors.Is(err, beads.ErrCircuitOpen)` and +`errors.Is(err, beads.ErrCommitIndeterminate)` match errors raised internally +and wrapped on the way out. + +**What an embedder can observe.** `ErrCircuitOpen` identifies a read or write +rejected because the Dolt circuit breaker is open. `ErrCommitIndeterminate` +identifies a commit whose outcome is unknown — the distinction that lets a +caller decline to retry a write that may already have landed. Match with +`errors.Is`, not on message text — matching on these values is currently the only +discriminator a caller has for those two conditions, so there is nothing else to +write against. That is a statement about today's surface, not about its +longevity; the note at the top of this document still applies. + +## 10. `bd dolt status --json` on a proxied workspace — reshaped in 1.3.1 + +This payload **changed shape in 1.3.1** (#6580, landed at this commit; see the +1.3.1 CHANGELOG entry). Before, on a proxied workspace, `bd dolt status` read +the classic `dolt-server.pid` that proxied mode never writes, and reported the +always-false `{"running": false, "pid": 0, "port": 0}` against a healthy +backend. `pid` and `port` are gone, replaced by the `proxy_*` and `backend_*` +pairs. Direct-server, shared, embedded and externally-managed workspaces keep +the output they had. + +The proxied payload is `proxiedDoltStatus` in +`cmd/bd/dolt_proxied_lifecycle.go`: + +The full key set is `mode`, `root`, `running`, `proxy_pid`, `proxy_port`, +`backend_managed`, `backend_running`, `backend_pid`, `backend_port`, +`backend_endpoint`, `idle_timeout`. Five are always present: `mode`, `root`, +`running`, `backend_managed` and `backend_running`. None of the three `bool`s +carries `omitempty` — for a bool, `false` is the answer rather than the absence +of one, and `backend_running: false` in the external example below is that rule +in action. The other six (`proxy_pid`, `proxy_port`, `backend_pid`, +`backend_port`, `backend_endpoint`, `idle_timeout`) do carry `omitempty`, so +absent means zero — including `proxy_pid`/`proxy_port` when the proxy is down. + +A managed-local workspace with both processes up, whose sidecar carries an +explicit positive timeout: + +```json +{ + "mode": "proxied-server", + "root": "/abs/path/.beads/dolt", + "running": true, + "proxy_pid": 251973, + "proxy_port": 33825, + "backend_managed": true, + "backend_running": true, + "backend_pid": 251980, + "backend_port": 41163, + "idle_timeout": "45s" +} +``` + +The same workspace on an external topology, where bd supervises no dolt: + +```json +{ + "mode": "proxied-server", + "root": "/abs/path/.beads/dolt", + "running": true, + "proxy_pid": 251973, + "proxy_port": 33825, + "backend_managed": false, + "backend_running": false, + "backend_endpoint": "db.internal:3306" +} +``` + +Reading the fields: + +- **`running` describes the proxy**, not the dolt server: the endpoint every bd + command connects through. It is usually true where the old payload was always + false. +- **`backend_managed` is `false` on an external proxied topology**, where the + dolt server is not bd's process to report on and `backend_endpoint` names it + instead (`host:port`, or `unix:` for a socket). On that topology + `backend_running` says nothing about the database — distinguishing "the + backend is down" from "bd never had a backend to report" is exactly what the + flag is for. +- **`idle_timeout` here is a duration *string*** (`"45s"`), unlike the sidecar's + nanosecond integer in §5 — and it is emitted only when the sidecar's value is + `> 0`. Both "default 30s" and "never" therefore render as an absent key today, + which is why neither example above could show the default. That is the + observation §5's 1.3.1 change is about. + +The command has no lifecycle side effects, which is why polling it beats parsing +`proxy.pid`: `proxy.ReadStatus` reads the two records and verifies the recorded +processes without starting, stopping or adopting anything. A record that fails +any check reports nothing rather than a PID a caller might act on. + +## 11. A clean backend exit — changing in 1.3.1 + +**Observed at this commit: the proxy does not notice.** `DoltServer.Start` +(`internal/storage/dbproxy/server/doltserver.go`) feeds `cmd.Wait()` into an +`errgroup.WithContext`, and an errgroup cancels its context only on a **non-nil** +return. A `dolt sql-server` that stops cleanly returns 0 — its ordinary +shutdown path, not a crash — so: + +- the group context is never cancelled; +- `DoltServer.Running` returns `egCtx.Err() == nil`, which stays true; +- the proxy's 100ms backend health tick (`proxy/server.go`) never fires. + +The proxy therefore keeps `proxy.pid`, `proxy.lock`, its control port and its +listener, and goes on accepting clients — closing each one after the backend +dial in `handleConn` fails. Adoption does not catch it either: `readAndDial` +verifies the proxy's identity and TCP-probes the *proxy*, never the backend, so +an adopting client succeeds and then fails its first query. The workspace stays +wedged until `bd dolt stop`. + +This is #6651, which reports it reproduced 3/3 against v1.3.0. The trigger is a +signal delivered to the dolt child *alone* — a per-PID supervisor stop, or +`kill `. A `kill -9` of the child and a cgroup-wide SIGTERM are not +affected; the latter signals the proxy too. + +**What an embedder can observe.** Reading the code, this state is visible in +`bd dolt status --json` (§10) as `running: true` with `backend_running: false` — +`readBackendRecord` verifies the recorded backend PID's birth token, which a +departed process fails — while a liveness check that only adopts the proxy +reports healthy. That asymmetry is what makes §10 the better probe of the two. +*Not verified by running:* this document was written without starting a proxy, +so the `status` output in that specific wedged state is inferred from the source +rather than observed. + +**What is changing in 1.3.1.** Two fixes are approved under #6651: treating +**any** `cmd.Wait()` return as a backend exit, rather than relying on errgroup's +nil-means-keep-going semantics, and making adoption **probe the backend** rather +than only the proxy. Neither had landed at `e1e1158dd`. After they land, a clean +backend exit is expected to tear the proxy down like any other backend loss, and +adoption is expected to decline a proxy whose backend is gone — so an embedder +should not build recovery logic that depends on the zombie persisting, nor on +adoption succeeding while the backend is absent. + +--- + +## Which of these are most likely to move + +Not all eleven carry the same risk, and it is worth saying which is which. + +**Most likely to move, and already in motion:** §6 (`bd migrate schema` becomes +a refusal), §11 (clean-backend-exit handling and backend-probing adoption), §5 +(the sidecar starts writing its effective `idle_timeout`). All three are decided +for 1.3.1 and described above as before/after. + +**Likely to move:** §2, the `db-proxy-child` argv. It is a hidden internal +command whose flags exist to serve the parent that spawns it, and flags have +been added to it as topologies were added. §1's `proxy.pid` and the control-file +set are next: they are bd's private lifecycle state, and the lifecycle work that +added `root_id`, `control_port`, the spawn marker and the stop epoch is not +finished. §8 is likely for a different reason — an assertion-reached optional +interface can disappear without a compile error anywhere. + +**Less likely to move, for a reason:** §9's two error sentinels, because +`errors.Is` on them is the only way a caller can tell those two conditions +apart. §10's payload, because it was *just* reshaped deliberately. §3's loopback +byte-pump behaviour, because the proxy's whole design is to not understand the +protocol — though the credentials half of §3 is a property of a generated config +file, which is a much softer thing than the byte-pump half. + +**A caveat on that ranking.** It is a reading of the direction of current work, +not a schedule and not a promise. A surface in the third group can still change +in a patch release; it is where to spend your safety margin, not where to stop +checking. + +## Not verified by running + +This document was written by reading the source at `e1e1158dd`. No proxy was +started and no test suite was run while writing it. The claims that would need a +live proxy to confirm end to end, and are therefore inferred from the code: + +- The exact `bd dolt status --json` output in the §11 wedged state + (`running: true` with `backend_running: false`). +- The observed idle-exit latency in §4 being longer than the configured timeout + by up to roughly two watcher ticks. +- That a third-party MySQL client authenticating as `root` with no password + succeeds against the data port on a managed-local workspace (§3). The + credentials, the loopback bind and the absence of protocol handling in the + proxy are all read directly from the source; the end-to-end connection is not. +- That `OpenBestAvailable`'s embedded fall-through (§8) completes and hands back + a usable but separate store, rather than failing somewhere in the create and + migrate steps. The *routing* (it takes the embedded arm, because proxied mode + is exempt from `IsDoltServerMode`) and the *directory* (`embeddeddolt` joins + `/embeddeddolt`, the proxy serves `/dolt`) are read off the + source and are not in doubt; the open itself was not run against a live + proxied workspace. + +Everything else — field names, JSON keys and their types, argv and flag +spellings, exported Go symbols, default values, and which code path decides +what — was read off the files listed in the freshness source above. diff --git a/engdocs/EXTENDING.md b/engdocs/EXTENDING.md index 7f2222f13..4559d348f 100644 --- a/engdocs/EXTENDING.md +++ b/engdocs/EXTENDING.md @@ -4,6 +4,15 @@ This file documents contracts that callers of the storage API must honor. It is not user-facing; it is for code that embeds bd or talks to the storage layer directly. +An embedder running against a **proxied-server** workspace also ends up +reading surfaces that are not contracts at all — the proxy's PID file, its +child argv, `bd dolt status --json`, optional interfaces reached by type +assertion. Those are written down, as observed behaviour with the commit +they were read at, in +[EMBEDDING_OVER_PROXIED_SERVER.md](EMBEDDING_OVER_PROXIED_SERVER.md). +Nothing in that document is a compatibility promise; read it alongside the +contracts here, not as one of them. + ## Lite SELECT shape — `IssueFilter.Lite` `store.SearchIssues(ctx, query, filter)` accepts an `IssueFilter` value. From bb03a310a902adab77de312258b0a3d078e73f78 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Thu, 24 Sep 2026 03:46:42 -0700 Subject: [PATCH 09/45] fix(schema,uow): cover the proxied data-behind path and make its consent step runnable (post-merge #6659) (#6695) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(schema,uow): cover the proxied data-behind path and make its consent step runnable (post-merge #6659) Post-merge review of the landed range d0063ac..e1e1158 (PR #6659, the release/1.3.0 backport of #6658) raised two majors. Neither is a regression from that PR; both are gaps its surface left open. F-A — the uow/proxied path never reached the #6575 stop, and its blunt body prescribed the wedge. `doltSQLProvider.initSchema` passed a nil adopter to `CheckSharedStoreMigrateGate`, so `localDataBehind` short-circuited and routing could never emit `smartDataBehind`. A proxied store that was level on schema and behind in data reported `shared-store` instead, whose body is the designated-migrator recipe — `bd migrate --force` then `bd dolt push` — which in that state is exactly the #6368 wedge the stop exists to prevent. The refusal was always safe (the shared arm suppresses the auto-migrate either way); the guidance was not. Only `AheadBehind` is wired. It is the complete data-behind predicate, while leaving `IsStrictAncestor`/`WorkingSetClean` nil keeps `routeAdoptFastForward` false, so the remote-AHEAD arm on this path keeps the plain adopt directive it has today. `FastForward` stays nil: it is a write that would promote the schema for every co-resident client. The remedy also had to be topology-correct before it could be printed. `bd dolt pull` is refused at the proxied front door (`proxy.dolt_pull.unsupported`), as are `bd dolt push`, `bd migrate` and `bd conflicts` — so the body, agent directive and `options` now name the machine the pull runs on, and the option id becomes `pull-first-on-server-host`. The don't-force warning gains `BD_ALLOW_REMOTE_MIGRATE=1`, the one consent surface the proxy does not refuse. Read-only proxied opens print the pull-first block instead of the shared consent template. The proxied fact is the caller's, not the provider's: `newSQLServerUOWProvider` is also `bd serve`'s provider for SERVER-mode workspaces, where that pull works. A new `uow.WithProxiedServerMode()` is set only by `openProxiedServerUOWProvider`. F-B — the shared data-behind consent step named a verb that cannot unlock a remote-backed shared store. `sharedMigrateConsent` has one reader, `sharedNoRemoteGate`, reachable only on the `!hasRemote` branch; a data-behind stop has a remote by construction. The four surfaces now name `SharedConsentCommandForced` (`bd migrate schema --force`) with its `--global` twin, and the human/JSON `--global` retargets follow. Widening the verb consent to the remote-backed arm was rejected: it would drop the designated-migrator confirmation #4259 deliberately requires whenever a remote exists. This mirrors PR #6692, the main-line remediation for the same defect on #6658, so the two lines converge rather than diverge. Tests: the consent step is proven by EXECUTING its path (a text assertion on `Options()` is how the dead command shipped), with the bare verb as the counterfactual; the proxied route is driven through `initSchema` against a mocked behind-clone and asserts the body does not prescribe `bd migrate --force`; a server-mode open pins that it keeps the runnable-here remedy; a level clone pins that reading ancestry does not turn a genuine first-mover into a stop. Scope: the scorecard's two Required Changes only. Its non-gating items — F-C (`noticeSharedMigrateRefusal` lacks an `IsDataBehind()` branch), F-D (`warnRemoteSyncOpenRefusal` verbosity), F-E (`docs` not retargeted), F-F/F-G (stale comments) — are left for separate follow-ups. Co-Authored-By: Claude Opus 5 * fix(schema,cli): the proxied consent step is not refused — say so (post-merge #6659 iter 2) Post-merge review iteration 2 of PR #6695 found one major: the proxied data-behind guidance this PR added asserts that its own prescribed consent step, `bd migrate schema --force`, is refused at the proxied front door. It is not. The refusal table keys on the full command path. `proxyMaintenanceRefusals` registers the bare `migrate` verb and its hooks/issues/sync children, but has no `migrate schema` row, so the two-word path misses the lookup and `validateProxyMaintenanceBeforeProvider` falls through to nil. `bd migrate schema` then runs its proxied arm, which reports the migration the provider open already applied under that verb's consent. That makes the claim wrong in the unsafe direction: `forceOrEnvConsent` is read BEFORE the data-behind stop is routed, so an operator told "it is refused here anyway" who types it to confirm applies the migration to a data-behind shared store — the #6368 wedge, reachable from the exact topology this remediation was written for. The don't-force warning made it worse by presenting BD_ALLOW_REMOTE_MIGRATE=1 as the only consent surface this workspace can reach. Reworded to the truth on all five surfaces, and pinned so it cannot drift again: - dataBehindBody's proxied consent clause now says the step CAN be run here and that the precondition is temporal, not locational: run it only after the pull lands on the server host. - the proxied don't-force warning names `bd migrate schema --force` alongside BD_ALLOW_REMOTE_MIGRATE=1 as the two reachable consent surfaces, both honored before this stop is reached. - the consent option's When/Risk drop "bd migrate is refused in proxied-server mode" and say the pull-first half is load-bearing here because this one does NOT fail closed the way the pull does. - CHANGELOG drops "the one consent surface the proxy does not refuse". - the same false premise is corrected in the uow test comment. - TestProxyMaintenanceAllowsTheSharedConsentVerb resolves the command from SharedConsentCommandForced through the same cobra lookup the root pre-run uses, asserts validateProxyMaintenanceBeforeProvider returns nil for it, and keeps the bare `bd migrate` refusal (proxy.migrate.unsupported) as the counterfactual. No test called that function with this command before, which is why the false wording shipped green. Also the two non-gating consistency findings from the same iteration: - `--json` gate["expected"] carries the location qualifier when Proxied, so an expected-only reader is not handed a locally-refused command while options/hint carry it. Extracted to dataBehindExpectedField so the qualifier does not add a branch to handleRemoteMigrateGateJSON (cognitive stays 33, cyclomatic 18). - fallbackReasonNote's data-behind arm gates its remedy on Proxied; the note prints after the body, so unqualified it made the last paragraph prescribe what the paragraph above called refused. - smart_remote_migrate_gate.go's injection-site comment no longer says the shared UOW provider wires neither callback; it wires AheadBehind only. Tests: ./internal/storage/schema (full, ok), ./cmd/bd -run 'Migrate|Consent|Proxy|RemoteGate|DataBehind|SharedStore|RemoteSyncOpen' (ok), ./internal/storage/uow -run 'DataBehind|Proxied|MigrateGate' (ok). Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 --- CHANGELOG.md | 30 +- cmd/bd/main.go | 18 +- cmd/bd/proxy_capability_test.go | 70 +++++ cmd/bd/remote_migrate_gate.go | 44 ++- cmd/bd/remote_sync_open_test.go | 44 ++- cmd/bd/shared_migrate_consent_test.go | 163 +++++++++- cmd/bd/uow_factory.go | 7 + .../storage/schema/remote_migrate_gate.go | 179 ++++++++++- .../schema/smart_remote_migrate_gate.go | 8 +- .../schema/smart_remote_migrate_gate_test.go | 71 ++++- .../uow/data_behind_migrate_gate_test.go | 288 ++++++++++++++++++ internal/storage/uow/dolt_sql_provider.go | 81 ++++- 12 files changed, 959 insertions(+), 44 deletions(-) create mode 100644 internal/storage/uow/data_behind_migrate_gate_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 5734dd278..651478e16 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -228,8 +228,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 operator who can confirm every co-resident client is upgraded). On a shared Dolt sql-server the guidance carries #5920's consequence — migrating promotes the schema for every co-resident client, and clients still on an older bd - refuse the database until upgraded — and names the `bd migrate schema` - consent step the retry needs there. + refuse the database until upgraded — and names the consent step the retry + needs there: `bd migrate schema --force`. It is the forced form because this + stop always has a remote configured (behind-ness is read from the + remote-tracking ref), and the bare `bd migrate schema` consent is only read + for a shared database with *no* remote; the flag consents to migrating a + remote-backed shared store, and by then the pull has landed the commits the + clone was missing. + + **Proxied-server mode reaches the stop too, and is told where to run the + remedy.** The store-open gate on that path supplied no branch-position + callback, so a proxied clone that was level on schema and behind in data + could never be classified data-behind: it got the blunt shared-store refusal, + whose body is the designated-migrator recipe — `bd migrate --force` then + `bd dolt push` — which in this state is the wedge the stop exists to prevent. + It now routes to the same data-behind stop as every other topology. Because + `bd dolt pull` is refused at the proxied front door + (`proxy.dolt_pull.unsupported`), along with `bd dolt push`, the bare + `bd migrate` and `bd conflicts`, the guidance there names the machine the pull + has to run on rather than printing a command this binary rejects; the `--json` + option is `pull-first-on-server-host`, and `expected` carries the same + qualifier so a single-field reader is not handed a locally-refused command. + The shared-store consent step is the other way round: `bd migrate schema + --force` is *not* refused here — the refusal table keys on the command path + and has no `migrate schema` row — so the guidance says it can be run from this + workspace but only after the pull lands on the server host, and the warning + against forcing past the stop names it alongside `BD_ALLOW_REMOTE_MIGRATE=1` + as the two consent surfaces this topology can still reach. Read-only proxied + opens print the same pull-first block instead of the shared-consent template. Both existing workarounds keep working unchanged: `BD_SMART_GATE=0` opts out of the smart gate entirely (which means the blunt gate applies to `bd dolt diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 1d2bbb6c1..5742c07ef 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -367,15 +367,27 @@ func isForcedMigrate(cmd *cobra.Command) bool { // open targets `beads_global`, so the block's `bd migrate schema` would // migrate the PROJECT database and leave the refusal in place — the working // remedy is the same verb with the same flag. -func printGlobalDatabaseConsentHint(w io.Writer) { +// +// It takes the refusal because "the same verb" is not the same verb on every +// arm: the #6575 data-behind stop on a shared store is remote-backed by +// construction, and there the bare verb's consent is never read (see +// schema.SharedConsentCommandForced), so retargeting the bare form would hand +// the operator a global-scoped command that still cannot succeed. This mirrors +// the retarget in handleRemoteMigrateGateJSON. A nil error keeps the +// pre-existing bare-verb wording. +func printGlobalDatabaseConsentHint(w io.Writer, e *schema.RemoteMigrateGateError) { if !globalFlag { return } + consent := schema.SharedConsentCommandGlobal + if e != nil && e.IsDataBehind() && e.Shared { + consent = schema.SharedConsentCommandForcedGlobal + } fmt.Fprintf(w, "\n This command targeted the global database (--global), so run the\n"+ " migrate step with the same flag:\n"+ " %s\n", - schema.SharedConsentCommandGlobal) + consent) } // renderTypedOpenError prints the actionable block for the store-open failures @@ -405,7 +417,7 @@ func renderTypedOpenError(err error) bool { handleRemoteMigrateGateJSON(gateErr) } else { fmt.Fprint(os.Stderr, gateErr.UserMessage()) - printGlobalDatabaseConsentHint(os.Stderr) + printGlobalDatabaseConsentHint(os.Stderr, gateErr) } return true } diff --git a/cmd/bd/proxy_capability_test.go b/cmd/bd/proxy_capability_test.go index 80308e292..54656af26 100644 --- a/cmd/bd/proxy_capability_test.go +++ b/cmd/bd/proxy_capability_test.go @@ -9,8 +9,78 @@ import ( "testing" "github.com/spf13/cobra" + + "github.com/steveyegge/beads/internal/storage/schema" ) +// TestProxyMaintenanceAllowsTheSharedConsentVerb pins the capability fact the +// #6575 proxied data-behind guidance is written against: the consent command +// that guidance prescribes is NOT refused at the proxied front door. +// +// The refusal table keys on the full command path. It has rows for the bare +// `migrate` verb and for `migrate hooks` / `migrate issues` / `migrate sync`, +// but none for `migrate schema` — so that path falls through +// validateProxyMaintenanceBeforeProvider's multi-word arm to `bd migrate +// schema`'s own proxied arm, which reports the migration the provider open +// already applied under this verb's consent. +// +// An earlier revision of that guidance asserted the opposite on three runtime +// surfaces and nothing failed, because no test called this function with this +// command. The wording is not cosmetic: forceOrEnvConsent honors `--force` +// BEFORE the data-behind stop is routed, so an operator told "it is refused +// here anyway" who types it to confirm takes the exact wedge the stop exists +// to prevent. +// +// The command is resolved from the constant the gate prescribes rather than +// retyped here, so drift on either side has to face this test. +func TestProxyMaintenanceAllowsTheSharedConsentVerb(t *testing.T) { + oldJSON := jsonOutput + jsonOutput = true + t.Cleanup(func() { jsonOutput = oldJSON }) + + args := strings.Fields(strings.TrimPrefix(schema.SharedConsentCommandForced, "bd ")) + target, _, err := rootCmd.Find(args) + if err != nil || target == rootCmd { + t.Fatalf("the gate prescribes %q, which does not resolve to a subcommand: %v", + schema.SharedConsentCommandForced, err) + } + path := strings.TrimSpace(strings.TrimPrefix(target.CommandPath(), rootCmd.Name())) + if path != "migrate schema" { + t.Fatalf("prescribed consent command resolves to path %q, want %q", path, "migrate schema") + } + + var allowErr error + out := captureStdout(t, func() error { + allowErr = validateProxyMaintenanceBeforeProvider(target) + return nil + }) + if allowErr != nil { + t.Errorf("%q is refused before the provider (%v), but the #6575 proxied data-behind guidance tells the operator it can be run from this workspace", + path, allowErr) + } + if out != "" { + t.Errorf("a non-refusal still emitted a typed error: %s", out) + } + + // The counterfactual, so "only the bare `bd migrate` verb is refused here" + // is a claim and not a vacuous one. + bare, _, err := rootCmd.Find([]string{"migrate"}) + if err != nil || bare == rootCmd { + t.Fatalf("bd migrate does not resolve: %v", err) + } + var bareErr error + refusal := captureStdout(t, func() error { + bareErr = validateProxyMaintenanceBeforeProvider(bare) + return nil + }) + if bareErr == nil { + t.Fatal("bare `bd migrate` is no longer refused in proxied-server mode; the data-behind guidance names it as the one form that is") + } + if !strings.Contains(refusal, `"code": "proxy.migrate.unsupported"`) { + t.Errorf("bare migrate refusal = %q, want proxy.migrate.unsupported", refusal) + } +} + func TestProxyCapabilityMatrix(t *testing.T) { for _, cap := range []ProxyCapability{ProxyCapReadonly, ProxyCapMaxRows} { err := AssertProxyCapability(ProxyModeProxied, cap) diff --git a/cmd/bd/remote_migrate_gate.go b/cmd/bd/remote_migrate_gate.go index eb0a2fb97..e1f6d290a 100644 --- a/cmd/bd/remote_migrate_gate.go +++ b/cmd/bd/remote_migrate_gate.go @@ -61,6 +61,22 @@ func humanDecisionRequired(e *schema.RemoteMigrateGateError) bool { return !(e.IsDataBehind() && !e.DataDiverged && !e.Shared) } +// dataBehindExpectedField renders the data-behind stop's `expected` field. +// +// On a proxied workspace the remedy is the same command in a different PLACE, +// and `expected` is what a single-field agent reader keys on. The option id, +// its when/risk and the hint all carry that qualifier, so leaving it off here +// alone would hand such a reader a command its own front door refuses +// (proxy.dolt_pull.unsupported). Kept out of handleRemoteMigrateGateJSON so +// the qualifier does not add another branch to that already-deep switch. +func dataBehindExpectedField(proxied bool) string { + const tail = "; the migration is only allowed once this clone has nothing left to pull" + if proxied { + return "run `" + schema.DataBehindRemedyCommand + "` on the machine that hosts this database first — this workspace is in proxied-server mode, where it is refused (proxy.dolt_pull.unsupported)" + tail + } + return "run `" + schema.DataBehindRemedyCommand + "` first" + tail +} + // handleRemoteMigrateGateJSON renders the #4259 remote-migrate gate error as a // structured JSON error block for agent consumption. // @@ -87,12 +103,13 @@ func handleRemoteMigrateGateJSON(e *schema.RemoteMigrateGateError) { if globalFlag { sharedConsent = schema.SharedConsentCommandGlobal } - // The #6575 data-behind stop on a SHARED store carries the same consent - // verb in its second option (migrate-shared-after-pulling), so it needs - // the same retarget: under --global the project-scoped verb would - // consent the wrong database and leave the refusal in place. It reaches - // here through the default arm (empty Decision), so the Decision test - // alone would miss it. + // The #6575 data-behind stop on a SHARED store carries a consent verb in + // its second option (migrate-shared-after-pulling) too — the forced one, + // since that stop is always remote-backed — so it needs the same + // retarget: under --global the project-scoped verb would consent the + // wrong database and leave the refusal in place. It reaches here through + // the default arm (empty Decision), so the Decision test alone would + // miss it. retargetShared := globalFlag && (e.Decision == "shared-no-remote" || (e.IsDataBehind() && e.Shared)) opts := make([]map[string]interface{}, 0, len(e.Options())) @@ -105,9 +122,18 @@ func handleRemoteMigrateGateJSON(e *schema.RemoteMigrateGateError) { // "expected" would still hand it the wrong-target command. retargeted := make([]string, len(commands)) for i, c := range commands { - if c == schema.SharedConsentCommand { + switch c { + case schema.SharedConsentCommand: retargeted[i] = schema.SharedConsentCommandGlobal - } else { + case schema.SharedConsentCommandForced: + // The data-behind arm's consent step is the FORCED + // verb (that stop is remote-backed by construction, + // where the bare verb's consent is never read), so it + // needs its own global form — matching only the bare + // verb would leave this option pointing at the + // project database. + retargeted[i] = schema.SharedConsentCommandForcedGlobal + default: retargeted[i] = c } } @@ -192,7 +218,7 @@ func handleRemoteMigrateGateJSON(e *schema.RemoteMigrateGateError) { gate["data_behind_shape"] = "fast-forward" } gate["observed"] = observed - gate["expected"] = "run `" + schema.DataBehindRemedyCommand + "` first; the migration is only allowed once this clone has nothing left to pull" + gate["expected"] = dataBehindExpectedField(e.Proxied) } } m["remote_migrate_gate"] = gate diff --git a/cmd/bd/remote_sync_open_test.go b/cmd/bd/remote_sync_open_test.go index 2059b682d..e7201940a 100644 --- a/cmd/bd/remote_sync_open_test.go +++ b/cmd/bd/remote_sync_open_test.go @@ -101,18 +101,27 @@ func TestHandleRemoteMigrateGateJSON_DataBehind(t *testing.T) { } for _, tc := range []struct { - name string - diverged bool - wantShape string + name string + diverged bool + proxied bool + wantShape string + wantOptionID string }{ - {name: "fast-forward shape", diverged: false, wantShape: "fast-forward"}, - {name: "diverged shape", diverged: true, wantShape: "diverged"}, + {name: "fast-forward shape", diverged: false, wantShape: "fast-forward", wantOptionID: "pull-first"}, + {name: "diverged shape", diverged: true, wantShape: "diverged", wantOptionID: "pull-first"}, + // On a proxied workspace the remedy is the same command in a different + // PLACE, and observed/expected are what a single-field reader keys on. + // The qualifier reached the option id and the hint an iteration before + // it reached expected, which is the divergence this case exists to + // catch: all three are asserted together below. + {name: "proxied qualifies every remedy field", proxied: true, wantShape: "fast-forward", wantOptionID: "pull-first-on-server-host"}, } { t.Run(tc.name, func(t *testing.T) { gate := &schema.RemoteMigrateGateError{ CurrentVersion: 66, LatestVersion: 67, Pending: 1, FallbackReason: "data-behind", DataDiverged: tc.diverged, + Proxied: tc.proxied, } parsed := capture(t, gate) obj, ok := parsed["remote_migrate_gate"].(map[string]interface{}) @@ -149,8 +158,8 @@ func TestHandleRemoteMigrateGateJSON_DataBehind(t *testing.T) { t.Fatalf("options = %v, want exactly one (the pull)", obj["options"]) } o, _ := rawOpts[0].(map[string]interface{}) - if id, _ := o["id"].(string); id != "pull-first" { - t.Errorf("option id = %v, want \"pull-first\"", o["id"]) + if id, _ := o["id"].(string); id != tc.wantOptionID { + t.Errorf("option id = %v, want %q", o["id"], tc.wantOptionID) } cmds, _ := o["commands"].([]interface{}) if len(cmds) != 1 || cmds[0] != schema.DataBehindRemedyCommand { @@ -171,6 +180,21 @@ func TestHandleRemoteMigrateGateJSON_DataBehind(t *testing.T) { if hint == gate.EscapeHint() { t.Errorf("hint must not be the runnable escape command %q", gate.EscapeHint()) } + // The location qualifier is all-or-nothing across the three fields + // an agent may read on their own: an `expected`-only reader on this + // topology would otherwise be handed a command its own front door + // refuses, while `options` and `hint` said where to run it. + if tc.proxied { + for name, surface := range map[string]string{"expected": expected, "hint": hint} { + for _, want := range []string{"proxied-server mode", "proxy.dolt_pull.unsupported"} { + if !strings.Contains(surface, want) { + t.Errorf("proxied %s = %q, want it to carry %q the way options[0] does", name, surface, want) + } + } + } + } else if strings.Contains(expected, "proxied-server mode") { + t.Errorf("expected = %q carries the proxied qualifier on a direct workspace, where the pull is runnable here", expected) + } // Nothing anywhere in the block may offer the migrate/push pair or // the no-op bootstrap for this state. blob, err := json.Marshal(parsed) @@ -212,8 +236,10 @@ func TestHandleRemoteMigrateGateJSON_DataBehind(t *testing.T) { } consent, _ := rawOpts[1].(map[string]interface{}) cmds, _ := consent["commands"].([]interface{}) - if len(cmds) != 1 || cmds[0] != schema.SharedConsentCommandGlobal { - t.Errorf("consent option commands = %v, want [%q] under --global", cmds, schema.SharedConsentCommandGlobal) + // The forced form, not the bare one: this stop is remote-backed by + // construction and the bare verb's consent is never read there. + if len(cmds) != 1 || cmds[0] != schema.SharedConsentCommandForcedGlobal { + t.Errorf("consent option commands = %v, want [%q] under --global", cmds, schema.SharedConsentCommandForcedGlobal) } // The pull is target-agnostic and must NOT be rewritten. pull, _ := rawOpts[0].(map[string]interface{}) diff --git a/cmd/bd/shared_migrate_consent_test.go b/cmd/bd/shared_migrate_consent_test.go index ada15c28a..4181bd02e 100644 --- a/cmd/bd/shared_migrate_consent_test.go +++ b/cmd/bd/shared_migrate_consent_test.go @@ -2,12 +2,15 @@ package main import ( "bytes" + "context" "io" "os" "strings" "testing" + sqlmock "github.com/DATA-DOG/go-sqlmock" "github.com/spf13/cobra" + "github.com/spf13/pflag" "github.com/steveyegge/beads/internal/storage/schema" ) @@ -189,17 +192,173 @@ func TestPrintGlobalDatabaseConsentHint(t *testing.T) { orig := globalFlag defer func() { globalFlag = orig }() + sharedNoRemote := &schema.RemoteMigrateGateError{ + CurrentVersion: 65, LatestVersion: 66, Pending: 1, + Decision: "shared-no-remote", Shared: true, + } + globalFlag = false var off bytes.Buffer - printGlobalDatabaseConsentHint(&off) + printGlobalDatabaseConsentHint(&off, sharedNoRemote) if off.Len() != 0 { t.Errorf("no hint without --global, got:\n%s", off.String()) } globalFlag = true var on bytes.Buffer - printGlobalDatabaseConsentHint(&on) + printGlobalDatabaseConsentHint(&on, sharedNoRemote) if !strings.Contains(on.String(), schema.SharedConsentCommandGlobal) { t.Errorf("hint must name %q:\n%s", schema.SharedConsentCommandGlobal, on.String()) } + + // The data-behind stop on a shared store is the arm where "the same verb + // with the same flag" is not enough: it is remote-backed by construction, + // so the bare verb's consent is never read and a --global rewrite of it is + // still a command that cannot succeed. Mirrors the JSON retarget. + var behind bytes.Buffer + printGlobalDatabaseConsentHint(&behind, &schema.RemoteMigrateGateError{ + CurrentVersion: 66, LatestVersion: 67, Pending: 1, + FallbackReason: "data-behind", Shared: true, + }) + if !strings.Contains(behind.String(), schema.SharedConsentCommandForcedGlobal) { + t.Errorf("the shared data-behind hint must name %q:\n%s", + schema.SharedConsentCommandForcedGlobal, behind.String()) + } +} + +// expectRemoteBackedSharedProbe mocks the probe sequence a shared, +// remote-backed store makes before the gate decides: CurrentVersion, then +// PendingVersions (both a cursor probe plus a max-version read), then the +// remote count. With the smart router pinned off, no further query follows +// either outcome, so an unmet expectation means the gate took a path this test +// does not describe. +func expectRemoteBackedSharedProbe(mock sqlmock.Sqlmock) { + for i := 0; i < 2; i++ { + mock.ExpectQuery(`SELECT COUNT\(\*\) FROM information_schema\.tables`). + WithArgs("schema_migrations"). + WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) + mock.ExpectQuery(`SELECT COALESCE\(MAX\(version\), 0\) FROM schema_migrations`). + WillReturnRows(sqlmock.NewRows([]string{"version"}).AddRow(1)) + } + mock.ExpectQuery(`SELECT COUNT\(\*\) FROM dolt_remotes`). + WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) +} + +// runPrescribedConsent replays the root PersistentPreRunE for one command line +// and reports what the real shared gate does afterwards on a remote-backed +// store. It resolves the command through the same cobra lookup the pre-run +// uses and feeds it to the same two predicates, so a command string that is +// not actually the consenting verb cannot pass by looking like one. +func runPrescribedConsent(t *testing.T, cmdline string) error { + t.Helper() + t.Setenv(schema.SmartGateEnv, "0") + t.Setenv(schema.AllowRemoteMigrateEnv, "0") + + args := strings.Fields(strings.TrimPrefix(cmdline, "bd ")) + target, rest, err := rootCmd.Find(args) + if err != nil { + t.Fatalf("the gate prescribes %q, which does not resolve to a command: %v", cmdline, err) + } + if target == rootCmd { + t.Fatalf("the gate prescribes %q, which resolves to bare bd", cmdline) + } + if err := target.ParseFlags(rest); err != nil { + t.Fatalf("the gate prescribes %q, whose flags do not parse: %v", cmdline, err) + } + t.Cleanup(func() { + // These cobra commands are package-level singletons, so a parsed flag + // outlives the test that parsed it and would silently consent for the + // next one. + target.Flags().VisitAll(func(f *pflag.Flag) { + if !f.Changed { + return + } + _ = f.Value.Set(f.DefValue) + f.Changed = false + }) + schema.SetForceAllowRemoteMigrate(false) + schema.SetSharedMigrateConsent(false) + }) + + // The two lines the root pre-run runs, verbatim in effect. + schema.SetForceAllowRemoteMigrate(isForcedMigrate(target)) + schema.SetSharedMigrateConsent(isSchemaMigrateVerb(target) && !isPreviewCommand(target)) + + db, mock, err := sqlmock.New() + if err != nil { + t.Fatal(err) + } + defer db.Close() + expectRemoteBackedSharedProbe(mock) + + var gateErr error + _ = captureNoticeStderr(t, func() { + gateErr = schema.CheckSharedStoreMigrateGate(context.Background(), db, "", nil, nil) + }) + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet expectations for %q: %v", cmdline, err) + } + return gateErr +} + +// TestDataBehindSharedConsentCommandUnlocks is the executing twin of +// TestSharedRefusalPrescribesAWorkingVerb, for the #6575 data-behind stop on a +// shared store — the one arm where the structural pairing that test checks is +// not enough. +// +// That stop always has a remote configured (behind-ness is read from the +// remote-tracking ref), and on the remote-backed arm the bare verb's consent +// is never read: schema.sharedMigrateConsent has a single reader, inside +// sharedNoRemoteGate, which is reachable only when no remote is configured. +// So the option this refusal carries has to be checked by RUNNING its consent +// path. Asserting its wording is exactly what let a dead command ship: the +// pre-existing tests pinned the option's text and passed throughout. +// +// The command under test is taken from Options() rather than written out here, +// so this stays a test of whatever the gate actually prescribes. +func TestDataBehindSharedConsentCommandUnlocks(t *testing.T) { + behindShared := &schema.RemoteMigrateGateError{ + CurrentVersion: 66, LatestVersion: 67, Pending: 1, + FallbackReason: "data-behind", Shared: true, + } + + opts := behindShared.Options() + if len(opts) != 2 || opts[1].ID != "migrate-shared-after-pulling" || len(opts[1].Commands) != 1 { + t.Fatalf("Options() = %+v, want pull-first then a single-command consent step", opts) + } + prescribed := opts[1].Commands[0] + + t.Run("the prescribed command unlocks the post-pull retry", func(t *testing.T) { + if err := runPrescribedConsent(t, prescribed); err != nil { + t.Fatalf("the gate prescribes %q for this state, but running its consent path still refuses: %v", + prescribed, err) + } + }) + + // The counterfactual, and the regression pin: the command that shipped. + // If this ever passes, the consent read has moved and the option above can + // go back to the bare verb. + t.Run("the bare verb does not, which is why the option carries --force", func(t *testing.T) { + err := runPrescribedConsent(t, schema.SharedConsentCommand) + if err == nil { + t.Fatal("the bare verb now unlocks a remote-backed shared store; revisit SharedConsentCommandForced") + } + if !schema.IsRemoteMigrateGateError(err) { + t.Fatalf("want the gate refusal, got %T: %v", err, err) + } + }) + + // Every surface that names a consent command for this stop must name the + // one just executed — the finding was four surfaces agreeing on a dead + // command, so agreement alone is not the property worth pinning. + t.Run("the prose surfaces name the executed command", func(t *testing.T) { + for name, surface := range map[string]string{ + "UserMessage": behindShared.UserMessage(), + "AgentDirective": behindShared.AgentDirective(), + } { + if !strings.Contains(surface, prescribed) { + t.Errorf("%s does not name the prescribed consent command %q:\n%s", name, prescribed, surface) + } + } + }) } diff --git a/cmd/bd/uow_factory.go b/cmd/bd/uow_factory.go index a6bdac741..adbfb8aa7 100644 --- a/cmd/bd/uow_factory.go +++ b/cmd/bd/uow_factory.go @@ -197,6 +197,13 @@ func openProxiedServerUOWProvider(ctx context.Context, beadsDir, databaseOverrid if err != nil { return nil, err } + // This is the one entry point that is proxied-server mode by definition; + // the funnel below is shared with `bd serve`'s SERVER-mode provider, which + // must not be told it is proxied. The distinction reaches exactly one + // thing: a migration-gate refusal whose remedy is `bd dolt pull` has to + // say where that pull can be run, because proxied mode refuses it at the + // front door (proxy.dolt_pull.unsupported) and server mode does not. + opts = append(append([]uow.ProviderOption{}, opts...), uow.WithProxiedServerMode()) return newSQLServerUOWProvider(ctx, beadsDir, topology, opts...) } diff --git a/internal/storage/schema/remote_migrate_gate.go b/internal/storage/schema/remote_migrate_gate.go index 5a881d7fe..262edf8dd 100644 --- a/internal/storage/schema/remote_migrate_gate.go +++ b/internal/storage/schema/remote_migrate_gate.go @@ -127,6 +127,20 @@ type RemoteMigrateGateError struct { // can report conflicts to resolve first. Set only when // FallbackReason == fallbackReasonDataBehind; meaningless otherwise. DataDiverged bool + + // Proxied marks a refusal raised on a proxied-server-mode open — the unit + // of work provider, which reaches its database only through the dbproxy. + // It changes WHERE the data-behind remedy can be run, not what it is: in + // that mode `bd dolt pull` is refused at the front door + // (proxy.dolt_pull.unsupported, cmd/bd/proxy_capability.go), along with + // `bd dolt push`, `bd migrate` and `bd conflicts`. Printing the bare + // command there would hand the operator a remedy their own binary rejects, + // so the surfaces that name it say to run it where the database lives. + // + // Deliberately set by the provider after the gate returns rather than + // plumbed into the gate: the topology is the caller's fact, and every + // other injection site would have to pass a constant false. + Proxied bool } const ( @@ -268,7 +282,16 @@ func (e *RemoteMigrateGateError) fallbackReasonNote() string { case fallbackReasonSharedStore: why = "this store is shared (dolt sql-server); the first-mover auto-migrate is disabled here because it would lock out co-resident clients (#5920)" case fallbackReasonDataBehind: - why = "this clone is behind the remote in data commits it has not pulled, so it is not the first-mover it looks like on schema alone — migrating in place would diverge from those commits, and can leave every later `bd dolt pull` refusing to merge (#6575, #6368). Run `" + DataBehindRemedyCommand + "` first, then retry: this clone is then the true first-mover it looked like" + // UserMessage prints this note AFTER userBody, so on a proxied + // workspace an unqualified remedy makes the LAST paragraph tell the + // operator to type here the command the paragraph above just said the + // front door refuses. Gate it the same way dataBehindBody and + // Options() already do. + remedy := "Run `" + DataBehindRemedyCommand + "` first, then retry" + if e.Proxied { + remedy = "Run `" + DataBehindRemedyCommand + "` on the server host first (this workspace is in proxied-server mode, which refuses it — proxy.dolt_pull.unsupported), then retry here" + } + why = "this clone is behind the remote in data commits it has not pulled, so it is not the first-mover it looks like on schema alone — migrating in place would diverge from those commits, and can leave every later `bd dolt pull` refusing to merge (#6575, #6368). " + remedy + ": this clone is then the true first-mover it looked like" // Only promise the retry resolves itself where it actually does. On a // shared store the first-mover auto-migrate is suppressed regardless // (fallbackReasonSharedStore, #5920), so pulling clears THIS stop but @@ -283,7 +306,7 @@ func (e *RemoteMigrateGateError) fallbackReasonNote() string { // lockout; data-behind now outranks it, so without this clause the // operator loses the warning that promoting the schema on a shared // server locks out every co-resident client still on an older bd. - why += " — but on this shared store the migrate still needs explicit consent (`" + SharedConsentCommand + "`), because it promotes the schema for every co-resident bd client at once and clients still on an older binary will refuse this database until they are upgraded (#5920)" + why += " — but on this shared store the migrate still needs explicit consent (`" + SharedConsentCommandForced + "`), because it promotes the schema for every co-resident bd client at once and clients still on an older binary will refuse this database until they are upgraded (#5920)" } default: return "" @@ -322,20 +345,60 @@ func (e *RemoteMigrateGateError) dataBehindBody() string { " This clone has no commits of its own, so the pull is a pure fast-forward:\n" + " nothing local is merged or discarded.\n" } + if e.Proxied { + // The remedy is the same command; what changes is where it can be + // typed. This workspace reaches the database through the dbproxy, and + // the proxied front door refuses `bd dolt pull` outright — so leaving + // the line above unqualified would name a command this very binary + // rejects (gastownhall/beads#6575 follow-up). It sits after the shape + // clause because the conflict handling that clause describes moves to + // the same machine. + body += "" + + "\n" + + " That pull cannot be run from here: this workspace is in proxied-server mode,\n" + + " which refuses `" + DataBehindRemedyCommand + "` at the front door (proxy.dolt_pull.unsupported),\n" + + " along with `bd dolt push`, the bare `bd migrate` and `bd conflicts`. Run\n" + + " it — and any conflict resolution it needs — where the database lives: on\n" + + " the server host, from a workspace with direct access to that database.\n" + } body += "" + "\n" + " Then re-run what you just ran.\n" if e.Shared { // #5920: on a shared store the first-mover auto-migrate stays - // suppressed after the pull, so the retry needs the consent verb. + // suppressed after the pull, so the retry needs explicit consent. // Saying "it proceeds on its own" here would loop the operator. + // The command is the FORCED verb, not the bare one: this stop always + // has a remote (that is where behind-ness is read from), and the bare + // verb's consent is only read on the no-remote arm — see + // SharedConsentCommandForced. body += "" + " This database is served to co-resident bd clients, so the migration itself\n" + " still needs explicit consent afterwards — it promotes the schema for EVERY\n" + " client at once, and clients still on an older bd will refuse this database\n" + - " until they are upgraded (#5920):\n" + - " " + SharedConsentCommand + "\n" + - " (" + SharedConsentCommandGlobal + " for the shared global database)\n" + " until they are upgraded (#5920). Once the pull above has completed:\n" + + " " + SharedConsentCommandForced + "\n" + + " (" + SharedConsentCommandForcedGlobal + " for the shared global database)\n" + if e.Proxied { + // Unlike the pull, this consent step is NOT refused here, and + // saying otherwise would be wrong in the unsafe direction. The + // proxied refusal table keys on the command path + // (cmd/bd.proxyMaintenanceRefusals): it registers the bare + // `migrate` verb and its hooks/issues/sync children, but has no + // `migrate schema` row, so the two-word path falls through + // validateProxyMaintenanceBeforeProvider to a proxied arm that + // reports the migration the provider open just applied under this + // verb's consent. A reader told it is blocked here tries it to + // confirm — and forceOrEnvConsent is honored BEFORE this stop is + // ever routed, so the wedge lands on the one topology that can + // still take it. State the real precondition instead: not where, + // but when. + body += "" + + " Unlike the pull, that consent step CAN be run from here: the proxied front\n" + + " door refuses the bare `bd migrate` verb, not `" + SharedConsentCommandForced + "`,\n" + + " and the provider open applies the migration under it. That is exactly why\n" + + " it has to wait — run it only after the pull on the server host completes.\n" + } } else { body += "" + " The migration then runs on its own: with nothing left to pull, this clone\n" + @@ -347,6 +410,32 @@ func (e *RemoteMigrateGateError) dataBehindBody() string { " while this clone is still behind — the exact state this stop exists to\n" + " prevent — and the `bd dolt push` that would follow is rejected as a\n" + " non-fast-forward, because the clone is still missing the remote's commits.\n" + if e.Shared { + // The consent command this arm prescribes is itself a forced migrate, + // so the warning above has to say what makes that one different: the + // pull having already landed, not the flag. Named in full rather than + // as "the command above", which is ambiguous once the proxied clause + // below is also printed. + body += "" + + " The same holds for `" + SharedConsentCommandForced + "` until the pull has\n" + + " finished: what makes it safe is that there is nothing left to pull by then.\n" + } + if e.Proxied { + // Only the BARE `bd migrate` the warning above names is refused here. + // Two other surfaces carry the same forced consent and are both + // reachable from this workspace: the two-word `migrate schema` path, + // which no proxy refusal row matches, and the env hatch, which no + // command dispatch touches at all. forceOrEnvConsent reads both before + // this stop is routed, so a proxied warning that implies the topology + // already blocks the wedge would leave it open on the one topology + // that can still take it. + body += "" + + " Being in proxied-server mode does not put that out of reach: the front door\n" + + " refuses the bare `bd migrate` verb, not `" + SharedConsentCommandForced + "`, and\n" + + " " + AllowRemoteMigrateEnv + "=1 grants the same consent to ANY command, including\n" + + " the one you just ran. Both are honored before this stop is even reached, so\n" + + " do not reach for either to get past it.\n" + } return body } @@ -472,10 +561,17 @@ func (e *RemoteMigrateGateError) AgentDirective() string { } else { d += " This clone has no commits of its own, so that pull is a pure fast-forward and discards nothing." } + if e.Proxied { + d += " This workspace is in proxied-server mode, where `" + DataBehindRemedyCommand + "` is refused at the " + + "front door (proxy.dolt_pull.unsupported) — do NOT attempt it here. The pull has to be run where the " + + "database lives (the server host, from a workspace with direct access to it); surface that to the " + + "operator rather than retrying locally." + } if e.Shared { - d += " After the pull the migration still needs the operator's explicit consent (" + SharedConsentCommand + - "): this database is shared with co-resident bd clients and migrating promotes the schema for all of " + - "them at once (#5920). Do NOT auto-run it." + d += " After the pull the migration still needs the operator's explicit consent (" + SharedConsentCommandForced + + " — this store is remote-backed, so the bare verb does not unlock it; see SharedConsentCommandForced): " + + "this database is shared with co-resident bd clients and migrating promotes the schema for all of " + + "them at once (#5920). Do NOT auto-run it, and do NOT run it before the pull has completed." } else { d += " After the pull, re-running the original command migrates on its own; nothing else is needed." } @@ -555,8 +651,20 @@ func (e *RemoteMigrateGateError) Options() []GateOption { if e.DataDiverged { risk = "this clone also has commits of its own, so the pull MERGES; it can report conflicts that must be resolved before it completes (bd dolt pull --strategy ours|theirs on embedded storage, bd conflicts resolve on a server store). Nothing is discarded without that resolution" } + id := "pull-first" + if e.Proxied { + // An option an agent can read as runnable-here is the harm on this + // topology: the proxied front door refuses the command. Say WHERE + // in both the id and the precondition, so neither a Commands-only + // reader nor an id-only reader is misled. + id = "pull-first-on-server-host" + when = "always, for this stop, but NOT from this workspace: it is in proxied-server mode, where " + + DataBehindRemedyCommand + " is refused (proxy.dolt_pull.unsupported). Run it on the machine that " + + "hosts this database, from a workspace with direct access to it, then retry here" + risk += "; running it from this proxied workspace fails with proxy.dolt_pull.unsupported and changes nothing" + } pull := GateOption{ - ID: "pull-first", + ID: id, When: when, Commands: []string{DataBehindRemedyCommand}, Risk: risk, @@ -564,12 +672,29 @@ func (e *RemoteMigrateGateError) Options() []GateOption { if e.Shared { // The pull itself is still safe; what the shared store changes is // what happens AFTER it, so the consent step becomes a second - // option rather than a footnote on the first (#5920). + // option rather than a footnote on the first (#5920). It carries + // the FORCED verb because this stop is always remote-backed and + // the bare verb's consent is read only on the no-remote arm — an + // option whose command cannot succeed in the state that printed it + // is worse than no option (see SharedConsentCommandForced). + consentWhen := "the pull above has completed AND every co-resident bd client of this server is upgraded to this binary (confirmed with the operator)" + consentRisk := "co-resident clients still on an older bd will refuse this database until upgraded; running it before the pull completes is the wedge this stop prevents. Its --force consents to migrating a remote-backed shared store (the bare verb's consent is read only with no remote configured); it does not make migrating while behind safe" + if e.Proxied { + // Unlike the pull, this command is NOT refused here: the + // refusal table has no "migrate schema" row, so the two-word + // path falls through and the provider open applies the + // migration under its consent. The precondition is therefore + // temporal, not locational — and it is the only thing between + // an agent on this topology and the wedge, because forced + // consent is honored before this stop is routed at all. + consentWhen += ". Unlike the pull this one IS runnable from this proxied workspace — the front door refuses the bare `bd migrate`, not this two-word path — so the pull-first half of this precondition is load-bearing here, not a formality" + consentRisk += ". Running it from this proxied workspace does NOT fail closed the way the pull does: the consent is honored before this stop is routed, and the provider open applies the migration" + } return []GateOption{pull, { ID: "migrate-shared-after-pulling", - When: "the pull above has completed AND every co-resident bd client of this server is upgraded to this binary (confirmed with the operator)", - Commands: []string{SharedConsentCommand}, - Risk: "co-resident clients still on an older bd will refuse this database until upgraded; running it before the pull completes is the wedge this stop prevents", + When: consentWhen, + Commands: []string{SharedConsentCommandForced}, + Risk: consentRisk, }} } return []GateOption{pull} @@ -974,6 +1099,32 @@ const SharedConsentCommand = "bd migrate schema" // migrates the wrong one. const SharedConsentCommandGlobal = SharedConsentCommand + " --global" +// SharedConsentCommandForced is the consent command for a shared database that +// IS remote-backed. Every #6575 data-behind stop is in that state by +// construction — behind-ness is read from `remotes//`, so there +// is always a remote — and there the bare verb above is a dead end: +// sharedMigrateConsent has exactly one reader, sharedNoRemoteGate, which +// checkRemoteMigrateGate reaches only on the !hasRemote branch. With a remote +// configured the retry routes through forceOrEnvConsent instead, so typing the +// bare verb sets a consent nothing reads and the operator lands on the blunt +// shared-store refusal, whose body prescribes the `bd migrate --force` this +// stop's own body tells them not to use. +// +// The fix is deliberately the command, not the gate: widening +// sharedNoRemoteGate's verb consent to the remote-backed arm would drop the +// designated-migrator confirmation that #4259's cross-clone fork risk demands +// whenever a remote exists, which is the very thing sharedNoRemoteGate's +// comment says is intentional. `--force` is what forceOrEnvConsent honors, and +// the CLI accepts it on this verb (cmd/bd.isForcedMigrate covers both +// `bd migrate` and `bd migrate schema`), so this is the narrowest command that +// still names the schema migration AND actually unlocks the retry. +const SharedConsentCommandForced = SharedConsentCommand + " --force" + +// SharedConsentCommandForcedGlobal is SharedConsentCommandForced aimed at the +// shared global database, for the same reason SharedConsentCommandGlobal +// exists. +const SharedConsentCommandForcedGlobal = SharedConsentCommandGlobal + " --force" + // sharedNoRemoteGate decides the shared-store-without-a-remote arm // (gastownhall/beads#5920). There is no remote, so none of the #4259 // migrate-vs-adopt machinery applies: the only question is whether the diff --git a/internal/storage/schema/smart_remote_migrate_gate.go b/internal/storage/schema/smart_remote_migrate_gate.go index ab26d88a2..3b48b07cb 100644 --- a/internal/storage/schema/smart_remote_migrate_gate.go +++ b/internal/storage/schema/smart_remote_migrate_gate.go @@ -246,9 +246,11 @@ func routeAdoptFastForward(ctx context.Context, db DBConn, ref string, adopt *Fa // (behind == 0) and a purely-ahead clone (behind == 0, ahead >= 1) are still // permitted — they are genuine first-movers with nothing to pull. // -// adopt may be nil, or wire neither callback (the shared unit-of-work -// provider's injection site) — then there is no ancestry fact to read and -// routing stays exactly as it was before this check existed. With only +// adopt may be nil, or wire neither callback — then there is no ancestry fact +// to read and routing stays exactly as it was before this check existed. (The +// shared unit-of-work provider's injection site wires AheadBehind only; every +// other callback stays nil there deliberately, so no new auto-write path +// opens.) With only // IsStrictAncestor wired, the fact narrows to the ahead == 0 shape; the // diverged shape then reads "not behind", which is the pre-widening // behavior rather than a new hazard. diff --git a/internal/storage/schema/smart_remote_migrate_gate_test.go b/internal/storage/schema/smart_remote_migrate_gate_test.go index 55375c8c1..28093fc1c 100644 --- a/internal/storage/schema/smart_remote_migrate_gate_test.go +++ b/internal/storage/schema/smart_remote_migrate_gate_test.go @@ -1246,11 +1246,23 @@ func TestDataBehindRemedySurfaces(t *testing.T) { // fallbackReasonSharedStore (whose note carried them) and Options()' // sharedRisk only ever decorated the adopt option. msg := base(true, false).UserMessage() - for _, want := range []string{"co-resident", "#5920", SharedConsentCommand} { + for _, want := range []string{"co-resident", "#5920", SharedConsentCommandForced} { if !strings.Contains(msg, want) { t.Errorf("shared data-behind message is missing %q:\n%s", want, msg) } } + // This stop always has a remote (that is where behind-ness is read + // from), so the bare verb is a dead end here: its consent is read only + // by sharedNoRemoteGate, on the !hasRemote branch. Asserting the forced + // form above is not enough — every forced form CONTAINS the bare form + // as a substring, so a regression to the bare verb would still satisfy + // it. Remove the forms that do work, then require nothing prescribing + // the bare one is left. + scrubbed := strings.ReplaceAll(msg, SharedConsentCommandForcedGlobal, "") + scrubbed = strings.ReplaceAll(scrubbed, SharedConsentCommandForced, "") + if strings.Contains(scrubbed, SharedConsentCommand) { + t.Errorf("shared data-behind message prescribes the bare verb, which cannot succeed with a remote configured:\n%s", msg) + } if strings.Contains(msg, "proceeds on its own") { t.Errorf("a shared store must not be promised an automatic retry:\n%s", msg) } @@ -1300,6 +1312,63 @@ func TestDataBehindRemedySurfaces(t *testing.T) { t.Error("IsDataBehind() = true on a nil receiver") } }) + + // Proxied-server mode is the one topology where the remedy command is + // correct and unrunnable at the same time: `bd dolt pull` is refused at the + // front door (proxy.dolt_pull.unsupported), as are `bd dolt push`, + // `bd migrate` and `bd conflicts`. Naming it unqualified there hands the + // operator a command their own binary rejects, which is how they end up + // reaching for the one hatch the proxy does NOT refuse. + t.Run("proxied surfaces say where the pull can be run", func(t *testing.T) { + proxied := func(shared bool) *RemoteMigrateGateError { + e := base(shared, false) + e.Proxied = true + return e + } + + for _, shared := range []bool{false, true} { + msg := proxied(shared).UserMessage() + for _, want := range []string{ + "proxied-server mode", "proxy.dolt_pull.unsupported", "server host", + DataBehindRemedyCommand, AllowRemoteMigrateEnv + "=1", + } { + if !strings.Contains(msg, want) { + t.Errorf("shared=%v: proxied body missing %q:\n%s", shared, want, msg) + } + } + d := proxied(shared).AgentDirective() + for _, want := range []string{"proxy.dolt_pull.unsupported", "do NOT attempt it here"} { + if !strings.Contains(d, want) { + t.Errorf("shared=%v: proxied AgentDirective missing %q:\n%s", shared, want, d) + } + } + opts := proxied(shared).Options() + if opts[0].ID != "pull-first-on-server-host" { + t.Errorf("shared=%v: pull option id = %q, want the host-qualified id", shared, opts[0].ID) + } + // The command itself is unchanged — it is the right command — so an + // agent that reads only Commands must find the qualification in the + // precondition it is required to check. + if len(opts[0].Commands) != 1 || opts[0].Commands[0] != DataBehindRemedyCommand { + t.Errorf("shared=%v: pull option commands = %v, want [%q]", shared, opts[0].Commands, DataBehindRemedyCommand) + } + if !strings.Contains(opts[0].When, "proxy.dolt_pull.unsupported") { + t.Errorf("shared=%v: pull option precondition must name the refusal:\n%s", shared, opts[0].When) + } + } + + // Nothing above may leak into the ordinary embedded/server-mode stop, + // whose remedy IS runnable where it is printed. + direct := base(true, false).UserMessage() + for _, forbidden := range []string{"proxied-server mode", "server host"} { + if strings.Contains(direct, forbidden) { + t.Errorf("non-proxied body gained %q:\n%s", forbidden, direct) + } + } + if base(true, false).Options()[0].ID != "pull-first" { + t.Error("non-proxied pull option id changed") + } + }) } // TestSmartGateEnabled pins the default-on contract: unset and unparseable diff --git a/internal/storage/uow/data_behind_migrate_gate_test.go b/internal/storage/uow/data_behind_migrate_gate_test.go new file mode 100644 index 000000000..d137d0488 --- /dev/null +++ b/internal/storage/uow/data_behind_migrate_gate_test.go @@ -0,0 +1,288 @@ +package uow + +import ( + "context" + "errors" + "fmt" + "regexp" + "strings" + "testing" + + "github.com/DATA-DOG/go-sqlmock" + "github.com/steveyegge/beads/internal/storage/schema" +) + +// dataBehindCurrent is a schema cursor at or above the convergence floor +// (schema.LastNonDeterministicMigration) and below this binary's latest, so the +// smart gate reaches the equal-version first-mover arm with migrations still +// pending — the only arm where the #6575 ancestry check runs. +const dataBehindCurrent = 65 + +// expectSmartGateDataBehindRoute mocks the smart router's reads for a clone +// that is LEVEL with the cached remote ref on schema and BEHIND it in data: +// the local hashes, the active branch, the remote ref's hashes (identical, so +// no skew and the same max version), and finally the ahead/behind commit +// counts the #6575 check reads through the injected adopter. +// +// The last query is the one that did not happen before this fix: with a nil +// adopter localDataBehind short-circuits, so the route ended at +// smartAutoMigrate and the refusal reported shared-store instead. +func expectSmartGateDataBehindRoute(mock sqlmock.Sqlmock, ahead, behind int) { + hashes := func() *sqlmock.Rows { + return sqlmock.NewRows([]string{"version", "content_hash"}). + AddRow(dataBehindCurrent-1, "hash-a"). + AddRow(dataBehindCurrent, "hash-b") + } + ref := "remotes/origin/main" + + mock.ExpectQuery(regexp.QuoteMeta("SELECT version, content_hash FROM schema_migrations")). + WillReturnRows(hashes()) + mock.ExpectQuery(regexp.QuoteMeta("SELECT active_branch()")). + WillReturnRows(sqlmock.NewRows([]string{"branch"}).AddRow("main")) + mock.ExpectQuery(regexp.QuoteMeta(fmt.Sprintf("SHOW TABLES AS OF '%s' LIKE 'schema_migrations'", ref))). + WillReturnRows(sqlmock.NewRows([]string{"table"}).AddRow("schema_migrations")) + mock.ExpectQuery(regexp.QuoteMeta(fmt.Sprintf("SHOW COLUMNS FROM schema_migrations AS OF '%s' LIKE 'content_hash'", ref))). + WillReturnRows(sqlmock.NewRows([]string{"column"}).AddRow("content_hash")) + mock.ExpectQuery(regexp.QuoteMeta(fmt.Sprintf("SELECT version, content_hash FROM schema_migrations AS OF '%s'", ref))). + WillReturnRows(hashes()) + // versioncontrolops.LocalAheadBehind — reached only through a wired + // AheadBehind callback. + mock.ExpectQuery(`FROM dolt_log AS OF`). + WillReturnRows(sqlmock.NewRows([]string{"ahead", "behind"}).AddRow(ahead, behind)) +} + +// TestInitSchemaProxiedDataBehind covers the proxied/`bd serve` open path's +// half of gastownhall/beads#6575. +// +// The store-open gate on this path passed a nil adopter, so localDataBehind +// had no ancestry fact to read and the route could never emit smartDataBehind. +// A proxied clone that was level on schema and behind in data therefore +// reported the blunt shared-store refusal — whose body is the designated- +// migrator recipe, `bd migrate --force` then `bd dolt push`. In this state that +// recipe IS the #6368 wedge the stop exists to prevent, so the refusal was safe +// (the shared arm suppresses the auto-migrate either way) while the guidance it +// printed was not. +func TestInitSchemaProxiedDataBehind(t *testing.T) { + // newProxiedProvider returns a provider over a fresh mock with the two + // escape hatches pinned off, so the gate is reached on its own merits. + newProxiedProvider := func(t *testing.T) (*doltSQLProvider, sqlmock.Sqlmock) { + t.Helper() + t.Setenv(schema.AllowRemoteMigrateEnv, "0") + t.Setenv(schema.SmartGateEnv, "1") + t.Cleanup(func() { + schema.SetSharedMigrateConsent(false) + schema.SetForceAllowRemoteMigrate(false) + }) + db, mock, err := sqlmock.New() + if err != nil { + t.Fatalf("create sql mock: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + return &doltSQLProvider{ + defaultBranch: defaultBranch, + db: db, + serverEndpoint: "tcp:127.0.0.1:3306", + proxiedServerMode: true, + }, mock + } + + t.Run("routes to the data-behind stop instead of the blunt shared-store one", func(t *testing.T) { + p, mock := newProxiedProvider(t) + + lockName := expectBehindDatabaseThroughPreparation(mock, "beads", dataBehindCurrent) + expectSharedGateProbe(mock, dataBehindCurrent, 1) + expectSmartGateDataBehindRoute(mock, 0, 2) + // Nothing between the route and the lock release: the refusal must land + // before MigrateUp issues its first statement. + expectLockRelease(mock, lockName) + + err := p.initSchema(context.Background(), "beads") + var gateErr *schema.RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("initSchema() error = %T (%v), want *schema.RemoteMigrateGateError", err, err) + } + if !gateErr.IsDataBehind() { + t.Fatalf("FallbackReason = %q, Decision = %q — want the #6575 data-behind stop", + gateErr.FallbackReason, gateErr.Decision) + } + if !gateErr.Shared { + t.Error("Shared = false; this provider only ever serves a shared database") + } + if !gateErr.Proxied { + t.Error("Proxied = false; every refusal from this provider is a proxied-server-mode refusal") + } + + // The finding, stated as an assertion: the body an operator reads must + // not hand them the migrate-in-place recipe while they are still behind. + body := gateErr.UserMessage() + for _, forbidden := range []string{"bd migrate --force\n", "designated migrator"} { + if strings.Contains(body, forbidden) { + t.Errorf("proxied data-behind body still prescribes %q:\n%s", forbidden, body) + } + } + if !strings.Contains(body, schema.DataBehindRemedyCommand) { + t.Errorf("body must name the pull:\n%s", body) + } + // ...and because `bd dolt pull` is refused at the proxied front door, + // the body has to say where it can actually be run. + for _, want := range []string{"proxied-server mode", "proxy.dolt_pull.unsupported", "server host"} { + if !strings.Contains(body, want) { + t.Errorf("proxied body missing %q:\n%s", want, body) + } + } + // The env hatch is not the only consent surface that stays reachable + // here: the proxy refusal table keys on the command path and has no + // `migrate schema` row, so the forced schema verb falls through too + // (pinned in cmd/bd by TestProxyMaintenanceAllowsTheSharedConsentVerb). + // Both are read by forceOrEnvConsent before this stop is routed, so a + // warning that names only the refused bare `bd migrate --force` leaves + // the wedge reachable on exactly this topology. + for _, want := range []string{schema.AllowRemoteMigrateEnv + "=1", schema.SharedConsentCommandForced} { + if !strings.Contains(body, want) { + t.Errorf("proxied body must warn about the reachable consent surface %q:\n%s", want, body) + } + } + + opts := gateErr.Options() + if len(opts) != 2 || opts[0].ID != "pull-first-on-server-host" { + t.Fatalf("Options() = %+v, want the host-qualified pull plus the consent step", opts) + } + for _, o := range opts { + for _, c := range o.Commands { + if c == "bd migrate --force" || c == "bd bootstrap" { + t.Errorf("option %q offers %q, measured not to work in this state", o.ID, c) + } + } + } + + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } + }) + + // The counterfactual for the topology flag. This same provider is the one + // `bd serve` builds for a SERVER-mode workspace, where `bd dolt pull` is + // not refused — telling that operator to go find another machine would be + // the same class of wrong guidance in the other direction. + t.Run("a server-mode open keeps the runnable-here remedy", func(t *testing.T) { + p, mock := newProxiedProvider(t) + p.proxiedServerMode = false + + lockName := expectBehindDatabaseThroughPreparation(mock, "beads", dataBehindCurrent) + expectSharedGateProbe(mock, dataBehindCurrent, 1) + expectSmartGateDataBehindRoute(mock, 0, 2) + expectLockRelease(mock, lockName) + + err := p.initSchema(context.Background(), "beads") + var gateErr *schema.RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("initSchema() error = %T (%v), want *schema.RemoteMigrateGateError", err, err) + } + if !gateErr.IsDataBehind() { + t.Fatalf("FallbackReason = %q — the stop itself is topology-independent", gateErr.FallbackReason) + } + if gateErr.Proxied { + t.Error("Proxied = true for a server-mode open") + } + if body := gateErr.UserMessage(); strings.Contains(body, "server host") { + t.Errorf("server-mode body must keep the runnable-here pull:\n%s", body) + } + if id := gateErr.Options()[0].ID; id != "pull-first" { + t.Errorf("server-mode pull option id = %q, want %q", id, "pull-first") + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } + }) + + t.Run("a level clone is unaffected", func(t *testing.T) { + // The counterfactual for the wiring: same route, behind == 0. Reading + // the ancestry must not turn a genuine first-mover into a data-behind + // stop — it stays the shared-store refusal it was before this fix. + p, mock := newProxiedProvider(t) + + lockName := expectBehindDatabaseThroughPreparation(mock, "beads", dataBehindCurrent) + expectSharedGateProbe(mock, dataBehindCurrent, 1) + expectSmartGateDataBehindRoute(mock, 0, 0) + expectLockRelease(mock, lockName) + + err := p.initSchema(context.Background(), "beads") + var gateErr *schema.RemoteMigrateGateError + if !errors.As(err, &gateErr) { + t.Fatalf("initSchema() error = %T (%v), want *schema.RemoteMigrateGateError", err, err) + } + if gateErr.IsDataBehind() { + t.Error("a level clone must not reach the data-behind stop") + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } + }) + + t.Run("read-only open prints the pull-first block, not the consent template", func(t *testing.T) { + // readThroughRefusedMigration's template names the shared consent verb + // for every refusal. For this stop that verb is doubly wrong: the + // remedy is the pull, and a remote-backed shared store never reads the + // bare verb's consent at all (schema.SharedConsentCommandForced). + p, mock := newProxiedProvider(t) + p.readOnly = true + + lockName := expectBehindDatabaseThroughPreparation(mock, "beads", dataBehindCurrent) + expectSharedGateProbe(mock, dataBehindCurrent, 1) + expectSmartGateDataBehindRoute(mock, 1, 2) // diverged shape + expectLockRelease(mock, lockName) + mock.ExpectExec(regexp.QuoteMeta("USE `beads`")). + WillReturnResult(sqlmock.NewResult(0, 0)) + + var initErr error + stderr := captureStderr(t, func() { + initErr = p.initSchema(context.Background(), "beads") + }) + if initErr != nil { + t.Fatalf("initSchema() error = %v, want nil — reads keep working on the old schema", initErr) + } + for _, want := range []string{ + "Read-only command", "without migrating", + schema.DataBehindRemedyCommand, "MERGES", "server host", + } { + if !strings.Contains(stderr, want) { + t.Errorf("read-through warning missing %q:\n%s", want, stderr) + } + } + // Two-sided: the template's "run 'bd migrate schema'" line must be gone + // for this stop. Scrub the forms that DO work first — every forced form + // contains the bare verb as a substring. + scrubbed := strings.ReplaceAll(stderr, schema.SharedConsentCommandForcedGlobal, "") + scrubbed = strings.ReplaceAll(scrubbed, schema.SharedConsentCommandForced, "") + if strings.Contains(scrubbed, schema.SharedConsentCommand) { + t.Errorf("read-through still prescribes the bare consent verb:\n%s", stderr) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } + }) +} + +// TestDataBehindAdopterWiring pins the shape of the injected adopter, not just +// its effect. Only AheadBehind is wired: it is the complete #6575 predicate, +// while leaving IsStrictAncestor/WorkingSetClean nil keeps routeAdoptFastForward +// returning false, so the remote-AHEAD arm on this path keeps the plain adopt +// directive it had before. FastForward must stay nil — it is a WRITE that would +// promote the schema for every co-resident client of this server. +func TestDataBehindAdopterWiring(t *testing.T) { + for _, readOnly := range []bool{false, true} { + a := dataBehindAdopter(readOnly) + if a.AheadBehind == nil { + t.Fatal("AheadBehind must be wired; it is the whole point of the injection") + } + if a.FastForward != nil { + t.Error("FastForward must stay nil: a fast-forward here promotes the schema for every co-resident client") + } + if a.IsStrictAncestor != nil || a.WorkingSetClean != nil { + t.Error("IsStrictAncestor/WorkingSetClean must stay nil so the remote-ahead arm is unchanged") + } + if a.ReadOnly != readOnly { + t.Errorf("ReadOnly = %v, want %v", a.ReadOnly, readOnly) + } + } +} diff --git a/internal/storage/uow/dolt_sql_provider.go b/internal/storage/uow/dolt_sql_provider.go index 9650495c4..371dc53ff 100644 --- a/internal/storage/uow/dolt_sql_provider.go +++ b/internal/storage/uow/dolt_sql_provider.go @@ -22,6 +22,7 @@ import ( db "github.com/steveyegge/beads/internal/storage/domain/db" "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/storage/schema" + "github.com/steveyegge/beads/internal/storage/versioncontrolops" ) const ( @@ -52,6 +53,10 @@ type doltSQLProvider struct { // preview it still bootstraps and migrates normally — it only changes what // happens when the migration gate REFUSES; see providerOptions.readOnly. readOnly bool + // proxiedServerMode: the workspace that opened this provider is in + // proxied-server mode, so the commands the proxied front door refuses are + // off the table for its operator; see providerOptions.proxiedServerMode. + proxiedServerMode bool // eventsJournalEnabled activates the durable events journal for THIS // provider instance only. See SetEventsJournalEnabled. eventsJournalEnabled atomic.Bool @@ -122,6 +127,17 @@ type providerOptions struct { // working through the upgrade window. That is the same warn-and-continue // contract embeddeddolt gives its read-only-command intent. readOnly bool + // proxiedServerMode records that the WORKSPACE is in proxied-server mode, + // which this provider cannot infer: it is the single funnel for the + // proxied CLI and for `bd serve`'s own provider on a SERVER-mode + // workspace, and both front a Dolt sql-server through the same dbproxy. + // The distinction matters to exactly one thing — which commands the front + // door refuses. In proxied mode `bd dolt pull` is rejected before it opens + // anything (proxy.dolt_pull.unsupported), so the #6575 data-behind stop + // has to name where that pull can be run; on a server-mode workspace the + // same command works, and saying otherwise would be the same class of + // wrong guidance in the other direction. + proxiedServerMode bool } // WithPreview opens the provider for a non-mutating preview command. @@ -134,6 +150,13 @@ func WithReadOnly() ProviderOption { return func(o *providerOptions) { o.readOnly = true } } +// WithProxiedServerMode marks the open as belonging to a proxied-server-mode +// workspace. See providerOptions.proxiedServerMode: only the refusal guidance +// depends on it, and only the proxied CLI open may set it. +func WithProxiedServerMode() ProviderOption { + return func(o *providerOptions) { o.proxiedServerMode = true } +} + func applyProviderOptions(opts []ProviderOption) providerOptions { var resolved providerOptions for _, opt := range opts { @@ -264,10 +287,18 @@ func (p *doltSQLProvider) initSchemaAttempt(ctx context.Context, database string // database by the time it runs, so a fresh bootstrap reads version 0 // and passes. schema.WithMigrationGate(func(ctx context.Context, c *sql.Conn) error { - return schema.CheckSharedStoreMigrateGate(ctx, c, "", nil, nil) + return schema.CheckSharedStoreMigrateGate(ctx, c, "", nil, dataBehindAdopter(p.readOnly)) })); err != nil { var gateErr *schema.RemoteMigrateGateError if errors.As(err, &gateErr) { + // The gate reads the database, never the workspace, so it cannot + // know which commands this operator's front door will refuse. Only + // a proxied-server-mode workspace refuses `bd dolt pull`, and this + // provider also serves `bd serve` on a server-mode workspace, + // where that pull works — so the flag is the caller's, not a + // property of being behind a proxy (gastownhall/beads#6575 + // follow-up). + gateErr.Proxied = p.proxiedServerMode if p.readOnly { return readThroughRefusedMigration(ctx, conn, database, gateErr) } @@ -281,6 +312,39 @@ func (p *doltSQLProvider) initSchemaAttempt(ctx context.Context, database string return nil } +// dataBehindAdopter injects the one ancestry fact the smart gate's #6575 +// data-behind check needs on this path. Without it localDataBehind short- +// circuits on a nil adopter, so a proxied store that is level on schema and +// BEHIND the remote in data could never route to smartDataBehind: it reported +// the blunt shared-store refusal instead, whose body names `bd migrate --force` +// and `bd dolt push` — the designated-migrator recipe, which in this state is +// the #6368 wedge the stop exists to prevent. The refusal itself was always +// safe here (the shared arm suppresses the first-mover auto-migrate whatever +// the ancestry says); the guidance it printed was not. +// +// Only AheadBehind is wired, deliberately. It is the complete data-behind +// predicate (behind >= 1 whatever ahead is, plus which shape), so the stop is +// fully covered — while leaving IsStrictAncestor/WorkingSetClean nil keeps +// routeAdoptFastForward returning false, so the remote-AHEAD arm still routes +// to the same plain adopt directive it does today. Nothing but the data-behind +// state changes message. FastForward stays nil for a stronger reason: the +// fast-forward is a WRITE that would promote the schema for every co-resident +// client of this server, which is exactly what the shared gate refuses to do +// unattended (#5920), and CheckSharedStoreMigrateGate suppresses it on a +// shared store regardless. +func dataBehindAdopter(readOnly bool) *schema.FastForwardAdopter { + return &schema.FastForwardAdopter{ + AheadBehind: func(ctx context.Context, db schema.DBConn, ref string) (int, int, error) { + return versioncontrolops.LocalAheadBehind(ctx, db, ref) + }, + // No FastForward is wired above, so canAutoFastForward is already + // false and this flag changes nothing today. Set anyway so the + // adopter's safety invariant ("ReadOnly means cannot write here") + // does not come to depend on that coincidence. + ReadOnly: readOnly, + } +} + // readThroughRefusedMigration serves a read command from a database the // migration gate refused to migrate: warn, attach at the current schema, and // carry on. Writes remain refused (they open without WithReadOnly), so the @@ -306,6 +370,19 @@ func readThroughRefusedMigration(ctx context.Context, conn *sql.Conn, database s // the refusal, which is the actionable half. return backoff.Permanent(gateErr) } + // #6575: the data-behind stop's remedy is the pull, and the consent verb + // this template names does not unlock a remote-backed shared store at all + // (schema.SharedConsentCommandForced). Print that stop's own block, which + // REPLACES the migrate-or-adopt framing rather than decorating it, and say + // only what read-through adds on top. + if gateErr.IsDataBehind() { + fmt.Fprintf(os.Stderr, + "Warning: %[1]s"+ + " Read-only command: continuing on schema v%[2]d without migrating.\n"+ + " Writes stay blocked until the steps above are done.\n", + gateErr.UserMessage(), gateErr.CurrentVersion) + return nil + } fmt.Fprintf(os.Stderr, "Warning: %[1]v\n"+ " Read-only command: continuing on schema v%[2]d without migrating.\n"+ @@ -686,6 +763,7 @@ func openAndInitSchema(ctx context.Context, ep proxy.Endpoint, database, rootUse expectedProjectID: expectedProjectID, preview: opts.preview, readOnly: opts.readOnly, + proxiedServerMode: opts.proxiedServerMode, } if err := initProvider.initSchema(ctx, database); err != nil { @@ -719,5 +797,6 @@ func openAndInitSchema(ctx context.Context, ep proxy.Endpoint, database, rootUse expectedProjectID: expectedProjectID, preview: opts.preview, readOnly: opts.readOnly, + proxiedServerMode: opts.proxiedServerMode, }, nil } From aa3c43b6bd2ff91c5afe6268283d4e8d88e50e8e Mon Sep 17 00:00:00 2001 From: Jim Wordelman Date: Sat, 26 Sep 2026 14:27:45 -0700 Subject: [PATCH 10/45] feat(storage): add events/bd_events_journal/bd_events_seq to cursorRealityFloor sentinel set (#6547) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * test(feat): red — Add sentinelFlooredTables to cursorRealityFloor so events/bd_events_journal/bd_events_seq self-heal (release/1.3.0) (refs be-u14.1) * feat: green — Add sentinelFlooredTables to cursorRealityFloor so events/bd_events_journal/bd_events_seq self-heal (release/1.3.0) (refs be-u14.1) * chore: release gate PASS for cursor-reality-floor-sentinel-self-heal (be-g69lt) * chore: remove leaked internal release-gate doc from PR branch release-gates/be-g69lt-gate.md Tracked our internal gc-management/beads-rig release-gate checklist and should never have been committed to this upstream branch -- it's process residue from our deploy tooling, not part of the change itself. Per maintainer bee-ghosttrack's request during review of PR #6571. The root cause (our deploy tooling auto-committing this file) is already fixed as of gm-qatxl1; this branch predates that fix, so this commit is cleanup only -- no other content on this branch is touched. Ref: be-lk7o3 * fix(schema): pin the floored-sentinel minimum and generalize creator detection Maintainer review fixups for PR #6547 (adoption review, iteration 1). No behavior change beyond the probe error-message text. - Pin the floored loop's intra-loop minimum. Every existing case removed exactly one floored sentinel, so `st.replayFloor < floor` was never evaluated against a floor an earlier absent sentinel had already set, and inverting it to `>` left the whole package green. Adds an "events and bd_events_journal both absent" case asserting floor 18 and widens allSentinelTablesExcept to variadic; the inversion now fails (floor = 21, want 18). The floors ascend in slice order today (18, 21, 21), so this is what keeps a future reordering or lower-floored addition from silently regressing to "last absent sentinel wins". - Distinguish the floored-table probe's error wrap from the floor-0 loop's. They were byte-identical, so a store-open failure did not say which sentinel class failed, and the consequences differ sharply (floor 0 and a full series replay vs. floor 18/21). - Detect both creator shapes in TestSentinelFlooredTablesAreCreatedByTheSeries rather than keying on the literal table name "events". The PREPAREd conditional CREATE is required of any creator whose table carries a named, database-unique constraint (ignored/0019's own comment), so a future floored sentinel in that shape would have fallen through to the __temp__ regex and failed with a misleading "never created by the series". Both arms are mutation-proved to be load-bearing. - Record in the sentinelFlooredTables comment that events' dolt_ignore pattern is version-gated ({"events", 62}) rather than unconditional, and why the clamp is safe anyway: the ignored plane is only reached after the main pass succeeds, and a main pass from below 62 applies 0062_events_dolt_ignore, whose REPLACE INTO dolt_ignore lands before the ignored plane is touched. Validation: go test -count=1 ./internal/storage/schema/ (ok, 36.5s), go vet and gofmt clean, git diff --check clean. --------- Co-authored-by: Julian Knutsen --- .../storage/schema/cursor_reality_test.go | 192 +++++++++++++++++- internal/storage/schema/lock_test.go | 6 +- internal/storage/schema/schema.go | 68 ++++++- 3 files changed, 260 insertions(+), 6 deletions(-) diff --git a/internal/storage/schema/cursor_reality_test.go b/internal/storage/schema/cursor_reality_test.go index c41144fab..299a2e7f9 100644 --- a/internal/storage/schema/cursor_reality_test.go +++ b/internal/storage/schema/cursor_reality_test.go @@ -46,12 +46,30 @@ func stubSentinelColumns(t *testing.T, present map[string]bool, probeErr error) } // allSentinelTables is the "everything the table sentinels ask for is present" -// shape, so column-sentinel cases can say so without repeating the list. +// shape, so column-sentinel cases can say so without repeating the list. Both +// the unconditional-floor-0 tables and the floored tables share the same +// sentinelTableExists probe, so both must be marked present here or a case +// meant to isolate one sentinel would spuriously trip another. func allSentinelTables() map[string]bool { - present := make(map[string]bool, len(ignoredSource.sentinelTables)) + present := make(map[string]bool, len(ignoredSource.sentinelTables)+len(ignoredSource.sentinelFlooredTables)) for _, table := range ignoredSource.sentinelTables { present[table] = true } + for _, st := range ignoredSource.sentinelFlooredTables { + present[st.table] = true + } + return present +} + +// allSentinelTablesExcept is allSentinelTables with the named tables absent, +// for cases that need every sentinel healthy except the one(s) under test. +// Variadic so a case can contradict two sentinels of the same kind at once, +// which is what pins the minimum a single loop accumulates. +func allSentinelTablesExcept(tables ...string) map[string]bool { + present := allSentinelTables() + for _, table := range tables { + delete(present, table) + } return present } @@ -139,6 +157,55 @@ func TestCursorRealityFloor(t *testing.T) { wantFloor: 0, wantLimited: false, }, + { + name: "events absent", + present: allSentinelTablesExcept("events"), + columns: map[string]bool{"leases.granted_node": true}, + wantFloor: 18, + wantLimited: true, + }, + { + name: "bd_events_journal absent", + present: allSentinelTablesExcept("bd_events_journal"), + columns: map[string]bool{"leases.granted_node": true}, + wantFloor: 21, + wantLimited: true, + }, + { + name: "bd_events_seq absent", + present: allSentinelTablesExcept("bd_events_seq"), + columns: map[string]bool{"leases.granted_node": true}, + wantFloor: 21, + wantLimited: true, + }, + { + // Two floored sentinels contradicted at once: the minimum must win + // WITHIN the floored loop, not only across loops. Without this + // case every case removes exactly one floored table, so the loop + // only ever reaches its !limited arm and `st.replayFloor < floor` + // is never evaluated against a floor an earlier absent sentinel + // already set — inverting it to `>` leaves the package green. The + // floors happen to ascend in slice order (18, 21, 21) today, so + // this is what keeps a future reordering or lower-floored addition + // from silently regressing to "last absent sentinel wins". + name: "events and bd_events_journal both absent", + present: allSentinelTablesExcept("events", "bd_events_journal"), + columns: map[string]bool{"leases.granted_node": true}, + wantFloor: 18, + wantLimited: true, + }, + { + // Combined contradiction across two different sentinel kinds: the + // true minimum (11, from the column) must win over the higher + // floored-table floor (18) the table loop finds first, proving the + // accumulator takes the minimum across both loops rather than + // whichever ran first. + name: "events absent and leases granted_node absent", + present: allSentinelTablesExcept("events"), + columns: map[string]bool{}, + wantFloor: 11, + wantLimited: true, + }, } { t.Run(tc.name, func(t *testing.T) { stubSentinelTables(t, tc.present, nil) @@ -255,6 +322,37 @@ func TestCurrentVersionClampsToSentinelFloor(t *testing.T) { columns: map[string]bool{"leases.granted_node": true}, want: latest, }, + { + name: "events absent, at-latest cursor", + raw: latest, + present: allSentinelTablesExcept("events"), + columns: map[string]bool{"leases.granted_node": true}, + want: 18, + }, + { + name: "bd_events_journal absent, at-latest cursor", + raw: latest, + present: allSentinelTablesExcept("bd_events_journal"), + columns: map[string]bool{"leases.granted_node": true}, + want: 21, + }, + { + name: "bd_events_seq absent, at-latest cursor", + raw: latest, + present: allSentinelTablesExcept("bd_events_seq"), + columns: map[string]bool{"leases.granted_node": true}, + want: 21, + }, + { + // Combined contradiction across two different sentinel kinds: the + // true minimum (11, from the column) must win over the higher + // floored-table floor (18). + name: "events absent and leases column absent, at-latest cursor", + raw: latest, + present: allSentinelTablesExcept("events"), + columns: map[string]bool{}, + want: 11, + }, } { t.Run(tc.name, func(t *testing.T) { stubSentinelTables(t, tc.present, nil) @@ -437,6 +535,84 @@ func TestSentinelColumnsAreCreatedByTheSeries(t *testing.T) { } } +// TestSentinelFlooredTablesAreCreatedByTheSeries is the floored-table +// analogue of TestSentinelColumnsAreCreatedByTheSeries: every sentinel in +// sentinelFlooredTables must be created by the series, strictly above its own +// replayFloor, and that floor must exclude ignored/0007's unguarded restamp +// the same way the column floor does. +// +// events does not use the __temp__ + RENAME pattern the other +// creators do: it carries the named, database-unique constraint +// fk_events_issue, so a temp twin errors "duplicate foreign key constraint +// name" on Dolt 2.2.2 (see ignored/0019's own comment). Its creator guards a +// plain CREATE TABLE behind a PREPAREd conditional on an INFORMATION_SCHEMA +// existence probe instead, so both creation shapes are detected here. +func TestSentinelFlooredTablesAreCreatedByTheSeries(t *testing.T) { + if len(ignoredSource.sentinelFlooredTables) == 0 { + t.Fatal("ignoredSource has no sentinel floored tables; the events/bd_events_journal/bd_events_seq guard is inert") + } + + bodies := make(map[int]string) + for _, mf := range ignoredSource.list() { + blob, err := ignoredSource.files.ReadFile(ignoredSource.dir + "/" + mf.name) + if err != nil { + t.Fatalf("read %s: %v", mf.name, err) + } + bodies[mf.version] = string(blob) + } + + // findVersion returns the lowest migration version whose body matches, or + // 0 when nothing does. + findVersion := func(re *regexp.Regexp) int { + found := 0 + for version, body := range bodies { + if !re.MatchString(body) { + continue + } + if found == 0 || version < found { + found = version + } + } + return found + } + + unguarded := findVersion(regexp.MustCompile(`(?i)UPDATE\s+wisps\s+SET\s+is_blocked\s*=\s*0\s*;`)) + if unguarded == 0 { + t.Fatal("the unguarded `UPDATE wisps SET is_blocked = 0` the replay floor exists to exclude is gone; re-derive the floor") + } + + for _, st := range ignoredSource.sentinelFlooredTables { + // Detect both creation shapes and accept whichever the series actually + // uses, rather than keying on the table's name: the PREPAREd + // conditional CREATE is required of any creator whose table carries a + // named, database-unique constraint (ignored/0019's own comment), so a + // future floored sentinel can legitimately arrive in either shape and + // must not fall through to the misleading "never created" failure. + // findVersion already returns the lowest match, so take the earlier of + // the two shapes if a table somehow uses both. + prepared := findVersion(regexp.MustCompile( + `(?i)IF\(@\w+\s*=\s*0,\s*'CREATE TABLE ` + regexp.QuoteMeta(st.table) + `\s*\(`)) + temp := findVersion(regexp.MustCompile(regexp.QuoteMeta("__temp__" + st.table))) + creator := prepared + if creator == 0 || (temp != 0 && temp < creator) { + creator = temp + } + if creator == 0 { + t.Errorf("sentinel floored table %q is never created by the %s series; re-running it could not repair that table", + st.table, ignoredSource.dir) + continue + } + if creator <= st.replayFloor { + t.Errorf("sentinel floored table %q is created by ignored/%04d, at or below its replayFloor %d; the clamped replay would skip the migration that heals it", + st.table, creator, st.replayFloor) + } + if st.replayFloor < unguarded { + t.Errorf("sentinel floored table %q replayFloor %d does not exclude ignored/%04d, whose unguarded UPDATE restamps wisps.updated_at", + st.table, st.replayFloor, unguarded) + } + } +} + // TestMigrationWorkNeededWhenWispTablesAbsent is gh 5033 end to end, through // the real short-circuit that caused it. Both cursors read at-latest, so // before this change migrationWorkNeeded returned false, MigrateUp did @@ -491,6 +667,12 @@ func TestMigrationWorkNeededWhenLeaseGrantedNodeAbsent(t *testing.T) { mock.ExpectQuery(regexp.QuoteMeta("FROM INFORMATION_SCHEMA.TABLES")). WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) } + // The floored-table sentinels probe the same way and must all corroborate + // too, ahead of the column probe below. + for range ignoredSource.sentinelFlooredTables { + mock.ExpectQuery(regexp.QuoteMeta("FROM INFORMATION_SCHEMA.TABLES")). + WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) + } mock.ExpectQuery(regexp.QuoteMeta("FROM INFORMATION_SCHEMA.COLUMNS")). WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(0)) @@ -590,6 +772,12 @@ func TestMigrateStartsAboveTheFloorUnderColumnContradiction(t *testing.T) { mock.ExpectQuery(regexp.QuoteMeta("FROM INFORMATION_SCHEMA.TABLES")). WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) } + // The floored-table sentinels probe the same way and must all corroborate + // too, ahead of the column probe below. + for range ignoredSource.sentinelFlooredTables { + mock.ExpectQuery(regexp.QuoteMeta("FROM INFORMATION_SCHEMA.TABLES")). + WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) + } mock.ExpectQuery(regexp.QuoteMeta("FROM INFORMATION_SCHEMA.COLUMNS")). WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(0)) diff --git a/internal/storage/schema/lock_test.go b/internal/storage/schema/lock_test.go index 481cbe463..c8cb9dc30 100644 --- a/internal/storage/schema/lock_test.go +++ b/internal/storage/schema/lock_test.go @@ -709,7 +709,7 @@ func TestMigrateUpWithLockFreshBootstrapHealResetsAndRetries(t *testing.T) { // expectIgnoredSentinelProbes mocks the INFORMATION_SCHEMA lookups // currentVersion issues to confirm a non-zero ignored cursor against the // schema it claims (gh 5033). They fire only for a non-zero cursor, in -// ignoredSource's table then column sentinel order. +// ignoredSource's table then floored-table then column sentinel order. func expectIgnoredSentinelProbes(mock sqlmock.Sqlmock, present bool) { count := 0 if present { @@ -720,6 +720,10 @@ func expectIgnoredSentinelProbes(mock sqlmock.Sqlmock, present bool) { WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(count)) } if present { + for range ignoredSource.sentinelFlooredTables { + mock.ExpectQuery(regexp.QuoteMeta("FROM INFORMATION_SCHEMA.TABLES")). + WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) + } for range ignoredSource.sentinelColumns { mock.ExpectQuery(regexp.QuoteMeta("FROM INFORMATION_SCHEMA.COLUMNS")). WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) diff --git a/internal/storage/schema/schema.go b/internal/storage/schema/schema.go index 5ee2df65a..659b82b03 100644 --- a/internal/storage/schema/schema.go +++ b/internal/storage/schema/schema.go @@ -237,6 +237,22 @@ type migrationSource struct { // TestSentinelTablesAreCreatedByTheSeries enforces the creating side only; // the dropping side is this comment. sentinelTables []string + // sentinelFlooredTables are clone-local tables whose absence contradicts an + // otherwise at-latest cursor, but — unlike sentinelTables — only back to + // each table's own replayFloor rather than unconditionally to 0. events, + // bd_events_journal, and bd_events_seq are dolt_ignored (never part of + // committed history, so a fresh clone can arrive at-latest without them) + // but floor-0 would replay ignored/0007's unguarded + // `UPDATE wisps SET is_blocked = 0` (see schemaSentinelColumn.replayFloor) + // for a table absence that says nothing about migrations that predate it. + // + // INVARIANT: no future migration in this series may DROP or RENAME a + // floored sentinel table. Older binaries in the field check their own + // sentinel list against the live schema, so removing one would make every + // healthy newer database read as "contradicted" to them and replay the + // series from their floor. TestSentinelFlooredTablesAreCreatedByTheSeries + // enforces the creating side only; the dropping side is this comment. + sentinelFlooredTables []schemaSentinelTable // sentinelColumns are clone-local columns whose absence contradicts an // otherwise at-latest cursor. Unlike a missing sentinel table, a missing // column disbelieves the cursor only back to the column's replayFloor: it @@ -252,6 +268,15 @@ type migrationSource struct { sentinelColumns []schemaSentinelColumn } +type schemaSentinelTable struct { + table string + // replayFloor is the highest cursor value this sentinel's absence still + // leaves believable — same semantics and the same two constraints as + // schemaSentinelColumn.replayFloor, applied to a whole table's presence + // instead of one column's. + replayFloor int +} + type schemaSentinelColumn struct { table string column string @@ -296,6 +321,28 @@ var ( // missing; wisps is checked too so a partially materialized database // is caught by whichever is absent. sentinelTables: []string{"wisps", "wisp_dependencies"}, + // events is created by ignored/0019 (floor 18: everything at or below + // must stay out of the replay, and 0019 itself must stay in it) and + // bd_events_journal/bd_events_seq are both created by ignored/0022 + // (floor 21, same reasoning). All three are dolt_ignored and never + // part of committed history, so a clone can arrive with the cursor + // at-latest but without them — exactly the leases shape below, except + // a whole-table absence rather than a column-on-an-existing-table one. + // + // The journal tables are ignored unconditionally; events is the one + // version-gated pattern ({"events", 62} below), so strictly its + // ignore is only asserted once the main cursor reaches 0062. That is + // safe here rather than a hole: this replay runs on the ignored plane, + // which migrateUpAfterReconcile only reaches after the main pass + // succeeded, and a main pass from below 62 applies + // 0062_events_dolt_ignore, whose REPLACE INTO dolt_ignore lands + // before the ignored plane is touched. So the clamp can never have + // 0019 re-create events as tracked-at-HEAD. + sentinelFlooredTables: []schemaSentinelTable{ + {table: "events", replayFloor: 18}, + {table: "bd_events_journal", replayFloor: 21}, + {table: "bd_events_seq", replayFloor: 21}, + }, // A historical ignored-v16 ordinal collision can leave the local // leases table present but without the column frozen 0016 adds; a // database materialized out of band can be missing the leases table @@ -1392,7 +1439,7 @@ func (m migrationSource) currentVersion(ctx context.Context, db DBConn) (int, er // corroborates. limited is false when nothing is missing (believe the cursor as // read); otherwise floor is the highest version still believable — 0 for a // missing sentinel table, and the sentinel's replayFloor for a missing sentinel -// column. +// floored table or sentinel column. // // Clamping rather than erroring is deliberate: the series is written to be // re-runnable against a database that already has some of it. @@ -1404,8 +1451,10 @@ func (m migrationSource) currentVersion(ctx context.Context, db DBConn) (int, er // rather than merely diagnose. // // Sentinel tables are probed first and short-circuit the whole check: they -// floor at 0, so no column probe could lower the answer, and skipping it keeps -// this cheap on the path every store open takes. +// floor at 0, so no floored-table or column probe could lower the answer, and +// skipping it keeps this cheap on the path every store open takes. Floored +// tables are then probed together with columns, each clamping floor to the +// lowest replayFloor of anything found absent. func (m migrationSource) cursorRealityFloor(ctx context.Context, db DBConn) (int, bool, error) { for _, table := range m.sentinelTables { present, err := sentinelTableExists(ctx, db, table) @@ -1417,6 +1466,19 @@ func (m migrationSource) cursorRealityFloor(ctx context.Context, db DBConn) (int } } floor, limited := 0, false + for _, st := range m.sentinelFlooredTables { + present, err := sentinelTableExists(ctx, db, st.table) + if err != nil { + return 0, false, fmt.Errorf("checking %s sentinel floored table %s: %w", m.cursorTable, st.table, err) + } + if present { + continue + } + if !limited || st.replayFloor < floor { + floor = st.replayFloor + } + limited = true + } for _, column := range m.sentinelColumns { present, err := sentinelColumnExists(ctx, db, column.table, column.column) if err != nil { From eac25975579953677845ec98bd5dd73f313e058e Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 22 Sep 2026 15:12:46 -0700 Subject: [PATCH 11/45] refactor(proxy): one capability registry with a reason on every refusal (proxied-parity S2) (#6582) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * refactor(proxy): one capability registry, with a reason on every refusal Proxied-server policy lived in three path tables (a maintenance refusal map, a history classification map, a transform row list), each with its own lookup and its own front-door validator, plus a flag matrix. Nothing forced a new command to appear in any of them, so an unclassified command sailed past the gate and died in the store factory with `proxy server store should be uow provider`. That — not the table count — was the root problem. One table now (cmd/bd/capability_registry.go), and a test that fails when a runnable command is missing from it. Adding a command is no longer a place where policy can be forgotten: it is a build failure until someone decides whether the proxied front door lets it through. Every refusal carries a Reason: `design` where refusing is correct and expected to stay (shared history, multi-repo routing, destructive admin, storage-format surgery, strict --readonly), `unimplemented` where it is a gap with a named owner. The reason renders in the refusal JSON, additive to the frozen {code, error, mutates} contract, and is omitted on refusals that report a runtime state rather than a policy so a consumer keying on its presence is never told "" by mistake. The registry also inventories the 57 untyped refusals commands raise from their own RunE. Those rows are not enforced — converting them changes observable output and belongs to the slice that owns each family — but a source scan now keeps the inventory honest in both directions. The store-factory backstop is typed as `proxy.store.unrouted`, the unreachable duplicate readonly refusal in main.go is gone (the proxied branch above it already returns for the only config that reaches it), and with it `backendSupportsStrictReadonly`, which had no other caller. No outcome changes. Every code, message and exit status is identical, which #6576's five-topology matrix proves: 90/90 cells unchanged (HONORED 47, REFUSED_TYPED 24 = 4 design + 20 unimplemented, ERROR 12, WRONG_ANSWER 4, DEGRADED 3), all five topologies exercised, no skips. Co-Authored-By: Claude Fable 5 * refactor(proxy): `migrate hooks` is an unimplemented refusal, not a design one The registry filed `migrate hooks` under "storage-format and workspace surgery against a store other clients may be holding open". That rationale does not describe it: migrate_hooks.go plans and applies GIT HOOK FILE migration through doctor.PlanHookMigration, on the local filesystem, and opens no store at all. Left as it was, proxied-local — the default topology since 1.3.0 — loses the command permanently, because `design` tells every later slice the refusal is semantic and settled. It is not settled, but it is also not free to lift, and the difference is structural rather than semantic. `migrate hooks` appears on none of main.go's store-init skip lists, so permitting it makes the root pre-run open the proxied provider before RunE runs. Measured with the row flipped to permitted, not reasoned about: on a managed-local workspace `bd migrate hooks --dry-run` started the proxy AND its dolt child — both pidfiles present — to print a hook plan for five missing hooks, and on a workspace with an unreadable sidecar it exited 1 with "failed to open uow provider: corrupt proxied-server sidecar ..." where it had cleanly refused before. Trading a typed refusal for a provider-open error is not a lift. So: Reason=unimplemented with a Tracking item that names the actual work — put the command on the store-init skip list first, then permit it — which is S7's, not this slice's. No outcome changes; the row still refuses with the same code, message and exit status. The registry header now says that permitting a path costs a provider open, since that is the fact this row turns on and the one the completeness test cannot see. Also leaves S7 an explicit instruction on the rows next door. `migrate issues`, `migrate-personal` and `admin cleanup`/`admin reset` claim `design` on a topology-blind rationale — "a store other clients may be holding open", "destructive against a possibly-shared store" — that describes external and team-server workspaces and does not describe managed-local, which is 1:1 and bd-owned. They are not reclassified here: the table cannot yet express "refused, except on managed-local", and narrowing them is semantic work nobody has done. The comments now say that is why, so the next reader does not mistake conservatism for a considered verdict. Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Fable 5 (cherry picked from commit 7ccb961f778367fc19ca686c2e9a28d5e161bce1) --- CHANGELOG.md | 15 + cmd/bd/capability_registry.go | 569 ++++++++++++++++++ ...go => capability_registry_history_test.go} | 12 +- cmd/bd/capability_registry_test.go | 399 ++++++++++++ ... => capability_registry_transform_test.go} | 27 +- cmd/bd/errors.go | 8 + cmd/bd/history_capability.go | 28 - cmd/bd/main.go | 21 +- cmd/bd/migrate_dolt_mode.go | 5 + cmd/bd/proxy_capability.go | 183 +++--- cmd/bd/proxy_capability_test.go | 14 +- cmd/bd/readonly_policy_test.go | 20 - cmd/bd/serve.go | 2 +- cmd/bd/store_factory.go | 21 +- cmd/bd/store_factory_nocgo.go | 21 +- cmd/bd/transform_capability.go | 90 --- 16 files changed, 1106 insertions(+), 329 deletions(-) create mode 100644 cmd/bd/capability_registry.go rename cmd/bd/{history_capability_test.go => capability_registry_history_test.go} (92%) create mode 100644 cmd/bd/capability_registry_test.go rename cmd/bd/{transform_capability_test.go => capability_registry_transform_test.go} (88%) delete mode 100644 cmd/bd/history_capability.go delete mode 100644 cmd/bd/transform_capability.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 651478e16..f8ba307ec 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- **Proxied-server refusals now say *why* they refuse.** The JSON a refused + command prints gains a `reason` field next to the existing `code`, `error` + and `mutates`: `design` for a refusal that is expected to stay (shared + history, multi-repo routing, destructive admin, strict `--readonly`) and + `unimplemented` for a capability gap with a named owner. Every existing code, + message and exit status is unchanged, and `reason` is absent on refusals that + report a runtime state rather than a policy, so existing consumers are + unaffected. The policy behind it moved into one registry + (`cmd/bd/capability_registry.go`) that every command must appear in. +- **A command with no proxied-server route now fails with a typed error** + (`proxy.store.unrouted`) instead of the bare string `proxy server store + should be uow provider`. + ## [1.3.1] - 2026-09-16 ### Changed diff --git a/cmd/bd/capability_registry.go b/cmd/bd/capability_registry.go new file mode 100644 index 000000000..276650bbe --- /dev/null +++ b/cmd/bd/capability_registry.go @@ -0,0 +1,569 @@ +package main + +import ( + "sort" + "strings" + + "github.com/spf13/cobra" + "github.com/spf13/pflag" +) + +// The proxied-server capability registry: one table for every command path. +// +// Before this file the policy lived in three separate tables — a maintenance +// refusal map, a history classification map, and a transform row list — each +// with its own lookup and its own front-door validator. Nothing forced a new +// command to appear in any of them, so an unclassified command sailed past the +// gate and died deep in the store factory with an untyped error. The fix is not +// a fourth table: it is one table plus a test that fails when a command is +// missing from it (TestProxyCapabilityRegistryCoversCommandTree). +// +// WHAT A ROW MEANS. Outcome describes what the PRE-PROVIDER GATE does with the +// path on a proxied-server workspace, nothing more: +// +// ProxyOutcomeRefused the gate refuses with a typed ProxyCapabilityError +// before any workspace side effect +// ProxyOutcomeRefusedInRunE the gate lets the path through and the command +// refuses inside its own RunE with an untyped +// string; recorded here so the untyped surface is +// inventoried rather than invisible. Not enforced. +// ProxyOutcomeHonored the gate lets the path through. It does NOT +// promise a proxied route exists: a permitted +// command with no route still fails at the store +// factory with proxy.store.unrouted. +// +// Permitting a path also COSTS something, and the cost is invisible from this +// table: unless the command is on one of main.go's store-init skip lists, the +// root pre-run opens the proxied provider before RunE runs — which starts the +// proxy and its dolt child, creates the database and migrates the schema. For a +// command that genuinely uses the store that is the point. For a store-free one +// it is pure side effect, and on a workspace whose topology cannot be resolved +// it turns a clean typed refusal into a provider-open error. So "this command +// touches no store" is an argument for permitting it only once the command is +// also off the store-init path; see the `migrate hooks` row. Nothing here +// asserts that, and TestProxyCapabilityRegistryCoversCommandTree cannot see it. +// +// REASON is the load-bearing field. "design" means refusing is correct and is +// expected to stay that way — the semantics are undefined or unsafe against a +// backend that may be shared. "unimplemented" means the refusal is a capability +// gap with a named slice that closes it, and Tracking says which. Every later +// slice measures itself by turning "unimplemented" rows into honored ones; the +// count of remaining "unimplemented" rows is the completion signal for the whole +// proxied-parity effort. Assign the field row by row, from the command's actual +// semantics — a bulk assignment would destroy the only information it carries. +// +// The reason also renders in the refusal JSON, additive to the frozen +// {code, error, mutates} contract (see buildJSONCapabilityError). Codes are +// frozen; messages are informative-only. + +// ProxyRefusalReason distinguishes a deliberate policy refusal from a +// not-yet-built one. +type ProxyRefusalReason string + +const ( + // ProxyReasonDesign: the command is refused on purpose and is expected to + // stay refused. Lifting it would need a semantic decision, not plumbing. + ProxyReasonDesign ProxyRefusalReason = "design" + // ProxyReasonUnimplemented: the command could work over the proxied + // provider; nobody has routed it yet. Tracking names the work item. + ProxyReasonUnimplemented ProxyRefusalReason = "unimplemented" +) + +// Tracking values name the plan-of-record work item for an unimplemented +// refusal. These are the slices of the proxied-parity plan; a row gets a GitHub +// issue or bead ID instead once one is filed for it. +const ( + trackBackup = "proxied-parity S3 (backup/restore family)" + trackVersionCtl = "proxied-parity S4 (dolt commit/push/pull/remote)" + trackSync = "proxied-parity S5 (bd sync)" + trackDegradation = "proxied-parity S6 (doctor, config show, degradations)" + trackLongTail = "proxied-parity S7 (long-tail policy pass)" +) + +// capabilityRow is one command-path policy row. ArgSet is the sorted set of +// execution-altering flags that were set (for example "--auto-merge --dry-run"), +// empty for the bare path; a row keyed on an ArgSet wins over the bare row, which +// is how `duplicates --auto-merge --dry-run` stays honored while +// `duplicates --auto-merge` is refused. +type capabilityRow struct { + Path string + ArgSet string + Rule proxyCapabilityRule + History HistoryCapabilityClass + Note string +} + +// display renders the row's key the way its refusal message names it. +func (r capabilityRow) display() string { + if r.ArgSet == "" { + return r.Path + } + return r.Path + " " + r.ArgSet +} + +func (r capabilityRow) withHistory(class HistoryCapabilityClass) capabilityRow { + r.History = class + return r +} + +// asParentGroup marks a row whose path is a command GROUP. Cobra answers a +// non-runnable parent with its help text before PersistentPreRunE, so the gate +// never sees it; the row is kept because it is the parent every child row +// inherits its code from, and because a parent that later gains a Run must not +// silently become permitted. +func (r capabilityRow) asParentGroup() capabilityRow { + r.Note = "parent group: cobra shows help instead of reaching the gate" + return r +} + +func (r capabilityRow) withMessage(message string) capabilityRow { + r.Rule.Message = message + return r +} + +// refusedPath builds a typed front-door refusal for a bare command path. +func refusedPath(path, code string, reason ProxyRefusalReason, tracking string) capabilityRow { + return refusedArgs(path, "", code, reason, tracking) +} + +// refusedArgs builds a typed front-door refusal keyed on a flag set. +func refusedArgs(path, argSet, code string, reason ProxyRefusalReason, tracking string) capabilityRow { + row := capabilityRow{Path: path, ArgSet: argSet} + row.Rule = proxyCapabilityRule{ + Outcome: ProxyOutcomeRefused, + Code: code, + Message: row.display() + " is not supported in proxied-server mode", + ExitCode: 1, + Reason: reason, + Tracking: tracking, + } + return row +} + +// refusedInRunE records a command that the gate permits and that then refuses +// itself with an untyped string from RunE. The row carries no message: the +// string lives in the command, and duplicating it here would rot. The scan in +// TestProxyCapabilityRegistryCoversInlineRefusals keeps the two in step. +// +// The reason is always "unimplemented" and the owner is always the long-tail +// slice, because an untyped string inside RunE is not a considered policy +// statement: a refusal decided on purpose belongs in the gate, typed, where a +// consumer can read it. Anything here is a family nobody has routed yet. +func refusedInRunE(path, site string) capabilityRow { + return capabilityRow{ + Path: path, + Rule: proxyCapabilityRule{ + Outcome: ProxyOutcomeRefusedInRunE, ExitCode: 1, + Reason: ProxyReasonUnimplemented, Tracking: trackLongTail, + }, + Note: "untyped refusal in " + site, + } +} + +// permitted records that the gate lets a path through. See the header: this is +// a statement about the gate, not a promise that a proxied route exists. +func permitted(path string) capabilityRow { + return capabilityRow{Path: path, Rule: proxyCapabilityRule{Outcome: ProxyOutcomeHonored}} +} + +// proxyCapabilityRegistry is the whole path-keyed policy. Rows for permitted +// paths are appended from proxyPermittedPaths by init. +var proxyCapabilityRegistry = []capabilityRow{ + // --- doctor ------------------------------------------------------------- + // Diagnosis needs no version-control surface; the checks simply have no + // proxied dual yet. S6 splits them into client-local and store-backed. + refusedPath("doctor", "proxy.doctor.unsupported", ProxyReasonUnimplemented, trackDegradation), + + // --- backup / restore --------------------------------------------------- + // CALL DOLT_BACKUP(...) is pure SQL over a connection proxied mode already + // owns, so this family is plumbing, not policy. Locality policy (file:// + // URLs naming the server's filesystem on external topologies) is decided + // per topology when S3 lands. + refusedPath("backup", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup). + asParentGroup(), + refusedPath("backup init", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), + refusedPath("backup sync", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), + refusedPath("backup remove", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), + refusedPath("backup status", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), + refusedPath("backup restore", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), + + // --- version control over a shared history ------------------------------ + // Refused by design: these mutate or traverse a history that several + // clients share through one connection, and the per-client semantics + // (whose working set? whose branch head?) are undefined. `diff` is the one + // plausible 1.4 candidate, via the dolt_diff system tables. + refusedPath("branch", "proxy.branch.unsupported", ProxyReasonDesign, ""). + withHistory(HistoryDirectOnly), + refusedPath("diff", "proxy.diff.unsupported", ProxyReasonDesign, "1.4 candidate: dolt_diff system tables"), + refusedPath("flatten", "proxy.flatten.unsupported", ProxyReasonDesign, ""). + withHistory(HistoryDirectOnly), + refusedPath("conflicts", "proxy.conflicts.unsupported", ProxyReasonDesign, ""). + withHistory(HistoryDirectOnly). + asParentGroup(), + refusedPath("conflicts list", "proxy.conflicts.unsupported", ProxyReasonDesign, ""), + refusedPath("conflicts show", "proxy.conflicts.unsupported", ProxyReasonDesign, ""), + refusedPath("conflicts resolve", "proxy.conflicts.unsupported", ProxyReasonDesign, ""), + refusedPath("vc", "proxy.vc.unsupported", ProxyReasonDesign, ""). + withHistory(HistoryDirectOnly). + asParentGroup(), + refusedPath("vc merge", "proxy.vc.unsupported", ProxyReasonDesign, ""), + refusedPath("vc commit", "proxy.vc.unsupported", ProxyReasonDesign, ""), + refusedPath("vc status", "proxy.vc.unsupported", ProxyReasonDesign, ""), + refusedPath("federation", "proxy.federation.unsupported", ProxyReasonDesign, ""). + withHistory(HistoryDirectOnly). + asParentGroup(), + refusedPath("federation sync", "proxy.federation.unsupported", ProxyReasonDesign, ""), + refusedPath("federation status", "proxy.federation.unsupported", ProxyReasonDesign, ""), + refusedPath("federation add-peer", "proxy.federation.unsupported", ProxyReasonDesign, ""), + refusedPath("federation remove-peer", "proxy.federation.unsupported", ProxyReasonDesign, ""), + refusedPath("federation list-peers", "proxy.federation.unsupported", ProxyReasonDesign, ""), + + // --- multi-repo routing ------------------------------------------------- + // Routing to another workspace bypasses the proxied root entirely, which is + // the same reason --repo is N/A mode-wide. + refusedPath("repo", "proxy.repo.unsupported", ProxyReasonDesign, ""). + withHistory(HistoryDirectOnly). + asParentGroup(), + refusedPath("repo add", "proxy.repo.unsupported", ProxyReasonDesign, ""), + refusedPath("repo remove", "proxy.repo.unsupported", ProxyReasonDesign, ""), + refusedPath("repo list", "proxy.repo.unsupported", ProxyReasonDesign, ""), + refusedPath("repo sync", "proxy.repo.unsupported", ProxyReasonDesign, ""), + + // --- dolt version-control verbs ----------------------------------------- + // CALL DOLT_PUSH/DOLT_PULL/DOLT_COMMIT are SQL procedures and the generic + // repository that wraps them is already written; these need a use case and + // a route, not proxy work. `dolt remote remove` is already routed, which is + // the precedent. + refusedPath("dolt commit", "proxy.dolt_commit.unsupported", ProxyReasonUnimplemented, trackVersionCtl). + withHistory(HistoryDirectOnly), + refusedPath("dolt push", "proxy.dolt_push.unsupported", ProxyReasonUnimplemented, trackVersionCtl). + withHistory(HistoryDirectOnly), + refusedPath("dolt pull", "proxy.dolt_pull.unsupported", ProxyReasonUnimplemented, trackVersionCtl). + withHistory(HistoryDirectOnly), + refusedPath("dolt remote", "proxy.dolt_remote.unsupported", ProxyReasonUnimplemented, trackVersionCtl). + withHistory(HistoryDirectOnly). + asParentGroup(), + refusedPath("dolt remote add", "proxy.dolt_remote.unsupported", ProxyReasonUnimplemented, trackVersionCtl). + withHistory(HistoryDirectOnly), + refusedPath("dolt remote list", "proxy.dolt_remote.unsupported", ProxyReasonUnimplemented, trackVersionCtl). + withHistory(HistoryDirectOnly), + refusedPath("dolt remote reset-data", "proxy.dolt_remote.unsupported", ProxyReasonUnimplemented, trackVersionCtl). + withHistory(HistoryDirectOnly), + refusedPath("sync", "proxy.sync.unsupported", ProxyReasonUnimplemented, trackSync). + withHistory(HistoryDirectOnly), + + // --- migration and workspace surgery ------------------------------------ + // Storage-format and workspace surgery against a store other clients may be + // holding open. `migrate schema` is the exception and is already routed: + // the provider open runs it under the migration gate. + // + // S7 MUST RE-EXAMINE THE ROWS BELOW PER TOPOLOGY. "A store other clients may + // be holding open" describes external and team-server workspaces; it does + // not describe managed-local, which is 1:1 and bd-owned — bd spawned the + // dolt child itself. A row cannot yet say "refused, except on managed-local" + // (S3's backup locality policy is the slice that adds per-topology rules), + // so the blanket refusal is the only thing this table can express today. + // These rows keep it because nobody has done the semantic work to narrow + // them, not because the work has been done and come out this way. + refusedPath("migrate", "proxy.migrate.unsupported", ProxyReasonDesign, ""), + refusedPath("migrate sync", "proxy.migrate.unsupported", ProxyReasonDesign, ""), + // `migrate hooks` is the one row in this block the surgery rationale never + // described: it plans and applies GIT HOOK FILE migration (migrate_hooks.go + // -> doctor.PlanHookMigration) on the local filesystem and opens no store at + // all. It is refused for a structural reason instead — the command is wired + // into the store-opening path it does not need. It is on none of main.go's + // store-init skip lists, so permitting it makes the root pre-run open the + // proxied provider before RunE ever runs. Measured, not assumed: with this + // row flipped to permitted, `bd migrate hooks --dry-run` on a managed-local + // workspace started the proxy AND its dolt child (both pidfiles present) to + // print a hook plan, and on a workspace with an unreadable sidecar it exited + // 1 with "failed to open uow provider: corrupt proxied-server sidecar ..." + // where it had cleanly refused before. Trading a typed refusal for a + // connection error is not a lift. The honest fix is to take the command off + // the store path first, which is S7's job, not this slice's. + refusedPath("migrate hooks", "proxy.migrate.unsupported", ProxyReasonUnimplemented, + trackLongTail+" (specifically: put `migrate hooks` on main.go's store-init skip list, then permit it)"), + refusedPath("migrate issues", "proxy.migrate.unsupported", ProxyReasonDesign, ""), + refusedPath("migrate-issues", "proxy.migrate.unsupported", ProxyReasonDesign, ""), + refusedPath("migrate-personal", "proxy.migrate.unsupported", ProxyReasonDesign, ""), + + // --- destructive admin -------------------------------------------------- + // Same topology-blind rationale as the migrate block above, and the same + // instruction to S7: "destructive against a possibly-shared store" is a + // claim about external and team-server shapes. On managed-local the store is + // this workspace's alone, and `bd admin reset` there is no more shared than + // it is in embedded mode, where it is honored. Re-examine per topology. + refusedPath("admin cleanup", "proxy.admin.unsupported", ProxyReasonDesign, ""), + refusedPath("admin reset", "proxy.admin.unsupported", ProxyReasonDesign, ""), + // --- issue compaction --------------------------------------------------- + // `bd admin compact` without --dolt is the description-rewriting pass; only + // the --dolt arm has a proxied route (runCompactDoltProxiedServer). The + // --dolt arm is decided by flag VALUE, so the gate keeps a value-aware + // branch for it and lands here when the flag is off. + refusedPath("admin compact", "proxy.compact.unsupported", ProxyReasonUnimplemented, trackLongTail). + withMessage("only 'compact --dolt' is supported in proxied-server mode"), + // `bd restore` recovers a COMPACTED ISSUE's original text, not a backup — + // the backup verb is `bd backup restore`. Its snapshot read is ordinary + // CRUD; only its fallback path (reconstructing from Dolt history when no + // snapshot was archived) touches version control. + refusedPath("restore", "proxy.restore.unsupported", ProxyReasonUnimplemented, trackLongTail), + + // --- gates, formulas, swarm, merge slots -------------------------------- + // Ordinary CRUD over a store with no version-control surface. They stay + // refused because nothing downstream has asked for them, which is this + // codebase's standing rule for lifting a refusal — not because the proxied + // topology makes them wrong. + refusedPath("gate discover", "proxy.gate.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("cook", "proxy.formula.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("ship", "proxy.formula.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("swarm create", "proxy.swarm.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("swarm list", "proxy.swarm.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("merge-slot create", "proxy.merge_slot.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("merge-slot check", "proxy.merge_slot.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("merge-slot acquire", "proxy.merge_slot.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("merge-slot release", "proxy.merge_slot.unsupported", ProxyReasonUnimplemented, trackLongTail), + + // --- transforms --------------------------------------------------------- + // ID rewrites walk every reference in the store. They need a routed + // multi-statement transaction, which the UOW can express; nobody has + // written it. The arg-set rows are what let the read-only preview of a + // duplicate merge stay honored while the merge itself is refused. + refusedPath("rename", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("rename-prefix", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedArgs("rename-prefix", "--dry-run", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedArgs("rename-prefix", "--repair", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedArgs("rename-prefix", "--dry-run --repair", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("duplicate", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedArgs("duplicate", "--of", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedPath("supersede", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedArgs("supersede", "--with", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + refusedArgs("duplicates", "--auto-merge", "proxy.transform.unsupported", ProxyReasonUnimplemented, trackLongTail), + // The dry run reports what the merge WOULD do without writing anything, so + // it rides the read path plain `duplicates` already uses. + {Path: "duplicates", ArgSet: "--auto-merge --dry-run", Rule: proxyCapabilityRule{Outcome: ProxyOutcomeHonored}}, + + // --- routed history ----------------------------------------------------- + // The two paths that already have a proxied route, recorded so the class is + // asserted rather than remembered. + permitted("history").withHistory(HistoryProxySupported), + permitted("dolt remote remove").withHistory(HistoryProxySupported), + + // --- untyped refusals inside RunE --------------------------------------- + // The gate permits these paths and the command refuses itself with a bare + // string, so a JSON consumer gets no code and no mutates flag. They are + // listed to inventory that surface, not to enforce it: converting them to + // typed refusals changes observable output and belongs to the slice that + // owns each family. Tracker verbs are ordinary store reads plus a network + // call; nothing about the proxied topology makes them wrong. + refusedInRunE("ado status", "ado.go"), + refusedInRunE("ado projects", "ado.go"), + refusedInRunE("ado sync", "ado.go"), + refusedInRunE("github status", "github.go"), + refusedInRunE("github repos", "github.go"), + refusedInRunE("github sync", "github.go"), + refusedInRunE("gitlab status", "gitlab.go"), + refusedInRunE("gitlab projects", "gitlab.go"), + refusedInRunE("gitlab sync", "gitlab.go"), + refusedInRunE("notion status", "notion.go"), + refusedInRunE("notion init", "notion.go"), + refusedInRunE("notion connect", "notion.go"), + refusedInRunE("notion sync", "notion.go"), + refusedInRunE("create-form", "create_form.go"), +} + +// proxyPermittedPaths lists every command path the proxied front door lets +// through. It exists so that a command added without a policy decision fails +// TestProxyCapabilityRegistryCoversCommandTree instead of shipping — the whole +// point of the registry. Being listed here says the GATE permits the path; a +// permitted command with no proxied route still fails at the store factory with +// proxy.store.unrouted. +var proxyPermittedPaths = []string{ + // issue CRUD and queries + "assign", "batch", "blocked", "children", "close", "comment", "count", "create", + "defer", "delete", "duplicates", "edit", "export", "find-duplicates", "forget", + "gc", "graph", "graph check", "heartbeat", "import", "info", "lint", "link", + "list", "note", "orphans", "priority", "promote", "prune", "purge", "q", "query", + "ready", "recompute-blocked", "reclaim", "reopen", "search", "set-state", "show", + "stale", "status", "statuses", "tag", "types", "unclaim", "undefer", "update", + + // comments, dependencies, labels, key-value + "comments", "comments add", "comments list", + "dep", "dep add", "dep cycles", "dep list", "dep relate", "dep remove", "dep tree", "dep unrelate", + "kv clear", "kv get", "kv list", "kv set", + "label add", "label list", "label list-all", "label propagate", "label remove", + + // molecules, epics, todos, state + "epic close-eligible", "epic status", + "mol bond", "mol burn", "mol current", "mol distill", "mol last-activity", "mol pour", + "mol progress", "mol ready", "mol seed", "mol show", "mol squash", "mol stale", + "mol wisp", "mol wisp create", "mol wisp gc", "mol wisp list", + "state", "state list", "todo", "todo add", "todo done", "todo list", + + // gates, rules, provenance, audit, upgrade review + "audit label", "audit record", + "gate add-waiter", "gate check", "gate create", "gate list", "gate resolve", "gate show", + "provenance by-ref", "provenance log", "provenance record", + "rules audit", "rules compact", + "upgrade ack", "upgrade review", "upgrade status", + + // workspace lifecycle and configuration + "bootstrap", "config apply", "config drift", "config get", "config list", "config set", + "config set-many", "config show", "config unset", "config validate", "context", + "init", "init-safety", "onboard", "preflight", "quickstart", "schema", "setup", "where", + "version", "worktree create", "worktree info", "worktree list", "worktree remove", + + // storage, history and migration paths that are already routed or store-free + "compact", "events export", "events prune", "events tail", + "dolt clean-databases", "dolt killall", "dolt set", "dolt show", "dolt status", + "dolt start", "dolt stop", "dolt test", + "migrate from-proxied-server-to-server", "migrate from-proxied-server-to-shared-server", + "migrate from-server-to-proxied-server", "migrate from-shared-server-to-proxied-server", + "migrate legacy-sqlite", "migrate schema", + + // formulas (parser-only), agent memory, swarm reads + "formula convert", "formula list", "formula schema", "formula show", + "mail", "memories", "ping", "prime", "recall", "remember", + "swarm status", "swarm validate", + + // human-in-the-loop queue + "human", "human dismiss", "human list", "human respond", "human stats", + + // trackers: the verbs with no proxied refusal of any kind + "ado pull", "ado push", "github pull", "github push", "gitlab pull", "gitlab push", + "jira pull", "jira push", "jira status", "jira sync", + "linear pull", "linear push", "linear status", "linear sync", "linear teams", + "notion pull", "notion push", + + // hooks, metrics, serving and internal entry points + "hooks install", "hooks list", "hooks run", "hooks uninstall", + "metrics", "metrics example", "metrics off", "metrics on", + "serve", "sql", + "codex-hook", "cursor-hook", "db-proxy-child", "send-metrics", +} + +func init() { + for _, path := range proxyPermittedPaths { + proxyCapabilityRegistry = append(proxyCapabilityRegistry, permitted(path)) + } + proxyCapabilityIndex = make(map[string]capabilityRow, len(proxyCapabilityRegistry)) + for _, row := range proxyCapabilityRegistry { + proxyCapabilityIndex[capabilityKey(row.Path, row.ArgSet)] = row + } +} + +// proxyCapabilityIndex is proxyCapabilityRegistry keyed for lookup. Duplicate +// keys are rejected by TestProxyCapabilityRegistryHasNoDuplicateRows rather +// than by a panic here: a typo must fail the build, not every bd invocation. +var proxyCapabilityIndex map[string]capabilityRow + +func capabilityKey(path, argSet string) string { return path + "\x00" + argSet } + +// LookupCapabilityRow returns the row for an exact path/arg-set key. +func LookupCapabilityRow(path, argSet string) (capabilityRow, bool) { + row, ok := proxyCapabilityIndex[capabilityKey(path, argSet)] + return row, ok +} + +// capabilityRowFor resolves a command to its row, preferring the row keyed on +// the flags the invocation actually set over the bare path. +// +// The fallback to the bare row is what lets one table serve both halves of the +// old policy: a path row refuses whatever flags were passed unless a more +// specific row says otherwise, so `backup sync --dry-run` is still refused by +// the `backup sync` row. The predecessor of the arg-set half matched exactly or +// not at all, which meant an unlisted flag combination on a refused path +// slipped through; no such combination exists among the commands that carry +// these flags, so nothing changes today, and the fallback is why none can +// appear later. +func capabilityRowFor(cmd *cobra.Command) (capabilityRow, bool) { + path := commandRegistryPath(cmd) + if argSet := commandPolicyArgSet(cmd); argSet != "" { + if row, ok := LookupCapabilityRow(path, argSet); ok { + return row, true + } + } + return LookupCapabilityRow(path, "") +} + +// commandRegistryPath is the registry key for a command: its full path with the +// root name removed, for example "dolt remote add". +func commandRegistryPath(cmd *cobra.Command) string { + return strings.TrimSpace(strings.TrimPrefix(cmd.CommandPath(), cmd.Root().Name())) +} + +// proxyPolicyArgFlags are the flags that take part in a capability key. Only +// flags that alter a command's execution belong here: output, tracing and other +// inherited root flags must never bypass a refusal by changing the key. +var proxyPolicyArgFlags = map[string]bool{ + "auto-merge": true, "dry-run": true, "repair": true, "of": true, "with": true, +} + +// commandPolicyArgSet renders the set flags that take part in the key, sorted so +// the key does not depend on the order the user typed them. +func commandPolicyArgSet(cmd *cobra.Command) string { + var args []string + cmd.Flags().Visit(func(f *pflag.Flag) { + if proxyPolicyArgFlags[f.Name] { + args = append(args, "--"+f.Name) + } + }) + sort.Strings(args) + return strings.Join(args, " ") +} + +// validateProxyRegistryBeforeProvider is the path-keyed half of the proxied +// front door: it rejects direct-only commands before migrations, auto-start or +// provider construction, so a refusal leaves the workspace untouched. The +// flag-keyed half is validateProxyCapabilitiesBeforeProvider. +func validateProxyRegistryBeforeProvider(cmd *cobra.Command) error { + if cmd == nil { + return nil + } + // `compact` is the one rule that a flag VALUE decides rather than a flag's + // presence, so it cannot be expressed as an arg-set row. `bd compact` and + // `bd rules compact` have no --dolt flag at all and are never refused here; + // `bd admin compact --dolt` has a proxied route and falls through to it. + if cmd.Name() == "compact" { + if cmd.Flags().Lookup("dolt") == nil { + return nil + } + if dolt, _ := cmd.Flags().GetBool("dolt"); dolt { + return nil + } + } + row, ok := capabilityRowFor(cmd) + if !ok || row.Rule.Outcome != ProxyOutcomeRefused { + return nil + } + return HandleProxyCapabilityError(proxyCapabilityErrorFor(row.Rule)) +} + +func proxyCapabilityErrorFor(rule proxyCapabilityRule) *ProxyCapabilityError { + return &ProxyCapabilityError{ + Code: rule.Code, + Message: rule.Message, + ExitCode: rule.ExitCode, + Mutates: rule.Mutates, + Reason: rule.Reason, + } +} + +// HistoryCapabilityClass classifies history/version-control commands on the +// proxied topology without coupling them to a particular backend engine. +type HistoryCapabilityClass string + +const ( + HistoryProxySupported HistoryCapabilityClass = "proxy-supported" + HistoryDirectOnly HistoryCapabilityClass = "direct-only" +) + +// LookupHistoryCapability classifies an exact command path. Paths with no +// history classification are intentionally absent so callers cannot +// accidentally advertise support. +func LookupHistoryCapability(commandPath string) (HistoryCapabilityClass, bool) { + row, ok := LookupCapabilityRow(commandPath, "") + if !ok || row.History == "" { + return "", false + } + return row.History, true +} diff --git a/cmd/bd/history_capability_test.go b/cmd/bd/capability_registry_history_test.go similarity index 92% rename from cmd/bd/history_capability_test.go rename to cmd/bd/capability_registry_history_test.go index cc277190d..c0319a73b 100644 --- a/cmd/bd/history_capability_test.go +++ b/cmd/bd/capability_registry_history_test.go @@ -31,7 +31,7 @@ func TestSyncRefusesBeforeProvider(t *testing.T) { root := &cobra.Command{Use: "bd"} cmd := &cobra.Command{Use: "sync"} root.AddCommand(cmd) - err := validateProxyMaintenanceBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd) if err == nil { t.Fatalf("sync refusal = %#v", err) } @@ -71,7 +71,7 @@ func TestHistoryDirectOnlyRefusalContract(t *testing.T) { } uowProvider = nil jsonOutput = true - out := captureStdout(t, func() error { _ = validateProxyMaintenanceBeforeProvider(cmd); return nil }) + out := captureStdout(t, func() error { _ = validateProxyRegistryBeforeProvider(cmd); return nil }) var got map[string]any want := expected[path] if err := json.Unmarshal([]byte(out), &got); err != nil || got["code"] != want.code || got["error"] != want.message || got["mutates"] != false { @@ -104,7 +104,7 @@ func TestHistoryNestedFrontDoorsRefuseAndSupportedPathsPass(t *testing.T) { cmd.AddCommand(child) cmd = child } - err := validateProxyMaintenanceBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd) if path == "dolt remote remove" { if err != nil { t.Fatalf("supported %s refused: %v", path, err) @@ -118,9 +118,9 @@ func TestHistoryNestedFrontDoorsRefuseAndSupportedPathsPass(t *testing.T) { t.Fatalf("%s exit=%v, want 1", path, err) } var got map[string]any - // validateProxyMaintenanceBeforeProvider renders the typed refusal to + // validateProxyRegistryBeforeProvider renders the typed refusal to // stdout in JSON mode; the command must retain its nested path. - out := captureStdout(t, func() error { _ = validateProxyMaintenanceBeforeProvider(cmd); return nil }) + out := captureStdout(t, func() error { _ = validateProxyRegistryBeforeProvider(cmd); return nil }) if err := json.Unmarshal([]byte(out), &got); err != nil { t.Fatalf("%s refusal JSON: %v (%q)", path, err, out) } @@ -132,7 +132,7 @@ func TestHistoryNestedFrontDoorsRefuseAndSupportedPathsPass(t *testing.T) { root := &cobra.Command{Use: "bd"} history := &cobra.Command{Use: "history"} root.AddCommand(history) - if err := validateProxyMaintenanceBeforeProvider(history); err != nil { + if err := validateProxyRegistryBeforeProvider(history); err != nil { t.Fatalf("history --events supported path refused: %v", err) } } diff --git a/cmd/bd/capability_registry_test.go b/cmd/bd/capability_registry_test.go new file mode 100644 index 000000000..f6f7f47c1 --- /dev/null +++ b/cmd/bd/capability_registry_test.go @@ -0,0 +1,399 @@ +package main + +import ( + "errors" + "os" + "path/filepath" + "regexp" + "sort" + "strings" + "testing" + + "github.com/spf13/cobra" +) + +// cobraBuiltinCommands are added to the tree by cobra itself, not by bd, and +// only once something calls Execute — which another test in this package does +// (completions_test.go runs __complete). Without this set the completeness test +// below would pass or fail depending on test ORDER, which is the worst possible +// property for a policy gate. +var cobraBuiltinCommands = map[string]bool{ + "help": true, "completion": true, "__complete": true, "__completeNoDesc": true, +} + +// walkCommandPaths returns every command path in the tree, with the runnable +// ones flagged. Runnable is the property that matters: cobra returns ErrHelp +// for a non-runnable parent before PersistentPreRunE, so a group command never +// reaches the capability gate at all. +func walkCommandPaths(root *cobra.Command) map[string]bool { + paths := map[string]bool{} + var walk func(c *cobra.Command) + walk = func(c *cobra.Command) { + if cobraBuiltinCommands[c.Name()] { + return + } + if path := commandRegistryPath(c); path != "" { + paths[path] = c.Runnable() + } + for _, sub := range c.Commands() { + walk(sub) + } + } + walk(root) + return paths +} + +// TestProxyCapabilityRegistryCoversCommandTree is the reason this registry +// exists. Every command that can actually run must state what the proxied front +// door does with it. A new command with no row fails here, at build time, rather +// than shipping and dying in the store factory with an untyped error — which is +// how every scattered refusal in this codebase got there. +// +// Adding a row is not a formality: choose the outcome deliberately. +// - the command works over the proxied provider, or never opens a store at +// all => add its path to proxyPermittedPaths +// - the command must not run on a shared/proxied store => add a refusedPath +// row with Reason=design and say why in a comment +// - the command could work but nobody has routed it => add a refusedPath row +// with Reason=unimplemented and a Tracking item +func TestProxyCapabilityRegistryCoversCommandTree(t *testing.T) { + var missing []string + for path, runnable := range walkCommandPaths(rootCmd) { + if !runnable { + continue + } + if _, ok := LookupCapabilityRow(path, ""); !ok { + missing = append(missing, path) + } + } + sort.Strings(missing) + if len(missing) > 0 { + t.Fatalf("%d command(s) have no proxied-server capability row:\n %s\n\n"+ + "Add each to cmd/bd/capability_registry.go: proxyPermittedPaths if the proxied "+ + "front door should let it through, or a refusedPath row with a Reason if it should not.", + len(missing), strings.Join(missing, "\n ")) + } +} + +// TestProxyCapabilityRegistryHasNoStaleRows is the other direction: a row whose +// command no longer exists is policy for nothing, and would hide the renamed +// command's own missing row behind a green test. +func TestProxyCapabilityRegistryHasNoStaleRows(t *testing.T) { + tree := walkCommandPaths(rootCmd) + var stale []string + for _, row := range proxyCapabilityRegistry { + if _, ok := tree[row.Path]; !ok { + stale = append(stale, row.display()) + } + } + sort.Strings(stale) + if len(stale) > 0 { + t.Fatalf("registry rows for commands that do not exist:\n %s", strings.Join(stale, "\n ")) + } +} + +func TestProxyCapabilityRegistryHasNoDuplicateRows(t *testing.T) { + seen := map[string]bool{} + for _, row := range proxyCapabilityRegistry { + key := capabilityKey(row.Path, row.ArgSet) + if seen[key] { + t.Errorf("duplicate registry row for %q: one of them is dead and the lookup silently picks the last", row.display()) + } + seen[key] = true + } +} + +// TestProxyCapabilityRegistryRowInvariants pins the contract every row must +// satisfy. The Reason rules are the load-bearing ones: a refusal with no reason +// is exactly the tribal knowledge this registry replaces, and an unimplemented +// refusal with no tracking item is a gap nobody owns. +func TestProxyCapabilityRegistryRowInvariants(t *testing.T) { + for _, row := range proxyCapabilityRegistry { + switch row.Rule.Outcome { + case ProxyOutcomeRefused: + if row.Rule.Code == "" || row.Rule.Message == "" || row.Rule.ExitCode != 1 { + t.Errorf("%s: refusal must carry a code, a message and exit 1, got %#v", row.display(), row.Rule) + } + if row.Rule.Mutates { + t.Errorf("%s: a refusal never mutates", row.display()) + } + if !strings.HasPrefix(row.Rule.Code, "proxy.") { + t.Errorf("%s: code %q must be in the stable proxy.* namespace", row.display(), row.Rule.Code) + } + case ProxyOutcomeRefusedInRunE: + if row.Rule.Code != "" { + t.Errorf("%s: an untyped RunE refusal has no code; if it has one it belongs in the gate", row.display()) + } + case ProxyOutcomeHonored: + if row.Rule.Reason != "" { + t.Errorf("%s: Reason explains a refusal; an honored row must not carry one", row.display()) + } + default: + t.Errorf("%s: unexpected outcome %q", row.display(), row.Rule.Outcome) + } + + refusal := row.Rule.Outcome == ProxyOutcomeRefused || row.Rule.Outcome == ProxyOutcomeRefusedInRunE + if refusal { + switch row.Rule.Reason { + case ProxyReasonDesign, ProxyReasonUnimplemented: + default: + t.Errorf("%s: refusal must declare Reason design or unimplemented, got %q", row.display(), row.Rule.Reason) + } + if row.Rule.Reason == ProxyReasonUnimplemented && row.Rule.Tracking == "" { + t.Errorf("%s: an unimplemented refusal needs a Tracking item naming who closes it", row.display()) + } + } + if row.History == HistoryDirectOnly && row.Rule.Outcome != ProxyOutcomeRefused { + t.Errorf("%s: classified direct-only but the gate does not refuse it", row.display()) + } + if row.History == HistoryProxySupported && row.Rule.Outcome != ProxyOutcomeHonored { + t.Errorf("%s: classified proxy-supported but the gate refuses it", row.display()) + } + } +} + +// inlineProxiedRefusal matches the untyped refusals commands raise from their +// own RunE, e.g. HandleErrorRespectJSON("gitlab sync is not supported in +// proxied-server mode"). +var inlineProxiedRefusal = regexp.MustCompile(`HandleErrorRespectJSON\("([^"]*proxied-server mode)"\)`) + +// inlineProxiedMessages classifies the messages that do not name their command +// path in the standard " is not supported ..." shape. A path maps the +// message to its registry row; an empty string records a message that mentions +// proxied mode without being a capability refusal at all. Keeping both explicit +// is what lets the scan below be exhaustive rather than best-effort. +var inlineProxiedMessages = map[string]string{ + "only 'compact --dolt' is supported in proxied-server mode": "admin compact", + // Not a refusal: --database is the flag that only MAKES SENSE on a proxied + // workspace, so this fires on the direct topologies. + "--database (or a --db value naming a database) is only supported in proxied-server mode": "", +} + +// TestProxyCapabilityRegistryCoversInlineRefusals scans the package for +// refusals raised inside RunE and requires each one to resolve to a registry +// row. Those strings are the untyped half of the refusal surface — no code, no +// mutates flag, nothing a JSON consumer can classify — and before the registry +// they were discoverable only by grep. +// +// The scan runs in both directions on purpose: a new inline refusal must be +// registered, and a ProxyOutcomeRefusedInRunE row whose string has been deleted +// must be flipped to its new outcome by the slice that deleted it. +func TestProxyCapabilityRegistryCoversInlineRefusals(t *testing.T) { + entries, err := os.ReadDir(".") + if err != nil { + t.Fatalf("read package dir: %v", err) + } + + found := map[string]string{} // registry path -> file it was found in + for _, entry := range entries { + name := entry.Name() + if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") { + continue + } + src, err := os.ReadFile(filepath.Clean(name)) + if err != nil { + t.Fatalf("read %s: %v", name, err) + } + for _, match := range inlineProxiedRefusal.FindAllStringSubmatch(string(src), -1) { + message := match[1] + path, ok := inlineRefusalPath(message) + if !ok { + t.Errorf("%s: %q mentions proxied-server mode but does not name a command path; "+ + "add it to inlineProxiedMessages so it can be checked against the registry "+ + "(map it to \"\" if it is not a capability refusal)", name, message) + continue + } + if path == "" { + continue // documented non-refusal + } + found[path] = name + } + } + + for path, file := range found { + argSet := "" + if base, args, split := strings.Cut(path, " --"); split { + argSet = "--" + args + path = base + } + row, ok := LookupCapabilityRow(path, argSet) + if !ok { + t.Errorf("%s refuses %q inside RunE with no registry row; the front door and the command disagree about what bd supports", + file, strings.TrimSpace(path+" "+argSet)) + continue + } + if row.Rule.Outcome == ProxyOutcomeHonored { + t.Errorf("%s refuses %q inside RunE, but the registry says the front door honors it", file, row.display()) + } + } + + for _, row := range proxyCapabilityRegistry { + if row.Rule.Outcome != ProxyOutcomeRefusedInRunE { + continue + } + if _, ok := found[row.Path]; !ok { + t.Errorf("%s is recorded as an untyped RunE refusal but no such refusal exists in the package any more (%s); "+ + "flip the row to its new outcome", row.display(), row.Note) + } + } +} + +func inlineRefusalPath(message string) (string, bool) { + if path, ok := inlineProxiedMessages[message]; ok { + return path, true + } + path, ok := strings.CutSuffix(message, " is not supported in proxied-server mode") + if !ok { + return "", false + } + // conflicts.go says "bd conflicts ..."; the registry keys on the path. + return strings.TrimPrefix(path, "bd "), true +} + +// TestProxyCapabilityRegistryReasonsAreAssignedIndividually guards the one way +// this registry could become useless while staying green: reasons assigned in +// bulk. Every later slice reads these to decide what is left to do, so both +// values have to be genuinely represented, and the design refusals — the claim +// a reviewer will challenge — are listed here by name. +func TestProxyCapabilityRegistryReasonsAreAssignedIndividually(t *testing.T) { + wantDesign := map[string]bool{ + "branch": true, "diff": true, "flatten": true, + "conflicts": true, "conflicts list": true, "conflicts show": true, "conflicts resolve": true, + "vc": true, "vc merge": true, "vc commit": true, "vc status": true, + "federation": true, "federation sync": true, "federation status": true, + "federation add-peer": true, "federation remove-peer": true, "federation list-peers": true, + "repo": true, "repo add": true, "repo remove": true, "repo list": true, "repo sync": true, + // `migrate hooks` is deliberately NOT here: it migrates git hook files + // and opens no store, so the surgery rationale the rest of this block + // rests on does not describe it. Its refusal is structural — the + // command is on none of main.go's store-init skip lists — which makes + // it unimplemented with a tracking item, not design. + "migrate": true, "migrate sync": true, "migrate issues": true, + "migrate-issues": true, "migrate-personal": true, + "admin cleanup": true, "admin reset": true, + } + + gotDesign := map[string]bool{} + unimplemented := 0 + for _, row := range proxyCapabilityRegistry { + switch row.Rule.Reason { + case ProxyReasonDesign: + gotDesign[row.display()] = true + case ProxyReasonUnimplemented: + unimplemented++ + } + } + + for path := range wantDesign { + if !gotDesign[path] { + t.Errorf("%q is no longer refused by design; if that is intended, update this test and say why in the PR", path) + } + } + for path := range gotDesign { + if !wantDesign[path] { + t.Errorf("%q claims Reason=design and is not in the reviewed list; a design refusal is a semantic claim that needs review", path) + } + } + if unimplemented == 0 { + t.Error("no unimplemented refusals left — either the work is done or the reasons were bulk-assigned") + } +} + +// TestProxyCapabilityArgSetFallsBackToBarePath pins the lookup rule that lets +// one table carry both the path policy and the flag policy: a flag combination +// with no row of its own inherits the bare path's verdict rather than escaping +// it. Written against the real command tree so it exercises the flags these +// commands actually declare. +func TestProxyCapabilityArgSetFallsBackToBarePath(t *testing.T) { + for _, tc := range []struct { + path string + flags []string + want ProxyCapabilityOutcome + wantMsg string + }{ + // Covered by an explicit row: the message names the flags. + {"rename-prefix", []string{"dry-run"}, ProxyOutcomeRefused, "rename-prefix --dry-run is not supported in proxied-server mode"}, + // No row for this combination: it inherits the refused bare row rather + // than slipping past the gate. + {"rename-prefix", []string{"dry-run", "auto-merge"}, ProxyOutcomeRefused, "rename-prefix is not supported in proxied-server mode"}, + // Inheriting an honored bare row is equally load-bearing: this is the + // read-only preview and it must keep working. + {"duplicates", []string{"dry-run"}, ProxyOutcomeHonored, ""}, + } { + t.Run(tc.path+" "+strings.Join(tc.flags, " "), func(t *testing.T) { + root := &cobra.Command{Use: "bd"} + cmd := &cobra.Command{Use: tc.path} + root.AddCommand(cmd) + for _, flag := range tc.flags { + cmd.Flags().Bool(flag, false, "") + if err := cmd.Flags().Set(flag, "true"); err != nil { + t.Fatal(err) + } + } + row, ok := capabilityRowFor(cmd) + if !ok { + t.Fatalf("no row resolved for %q with %v", tc.path, tc.flags) + } + if row.Rule.Outcome != tc.want || row.Rule.Message != tc.wantMsg { + t.Fatalf("row = outcome %q message %q, want %q / %q", + row.Rule.Outcome, row.Rule.Message, tc.want, tc.wantMsg) + } + }) + } +} + +// TestProxyRefusalJSONCarriesReason pins the additive half of the JSON +// contract. {code, error, mutates} is frozen and parsed downstream; reason is +// new, and it must appear without disturbing the three fields that were already +// there. +func TestProxyRefusalJSONCarriesReason(t *testing.T) { + t.Setenv("BD_JSON_ENVELOPE", "") + oldJSON := jsonOutput + jsonOutput = true + t.Cleanup(func() { jsonOutput = oldJSON }) + + for _, tc := range []struct { + path, code string + reason ProxyRefusalReason + }{ + {"branch", "proxy.branch.unsupported", ProxyReasonDesign}, + {"sync", "proxy.sync.unsupported", ProxyReasonUnimplemented}, + } { + t.Run(tc.path, func(t *testing.T) { + root := &cobra.Command{Use: "bd"} + cmd := &cobra.Command{Use: tc.path} + root.AddCommand(cmd) + out := captureStdout(t, func() error { + _ = validateProxyRegistryBeforeProvider(cmd) + return nil + }) + for _, want := range []string{ + `"code": "` + tc.code + `"`, + `"error": "` + tc.path + ` is not supported in proxied-server mode"`, + `"mutates": false`, + `"reason": "` + string(tc.reason) + `"`, + } { + if !strings.Contains(out, want) { + t.Fatalf("refusal JSON missing %s:\n%s", want, out) + } + } + }) + } +} + +// TestProxiedStoreUnroutedIsTyped covers the backstop: a permitted command with +// no proxied route used to die with a bare string that no consumer could tell +// from a real store failure. +func TestProxiedStoreUnroutedIsTyped(t *testing.T) { + var typed *ProxyCapabilityError + err := errProxiedStoreUnrouted() + if !errors.As(err, &typed) { + t.Fatalf("store backstop error = %#v, want *ProxyCapabilityError", err) + } + if typed.Code != "proxy.store.unrouted" || typed.ExitCode != 1 || typed.Mutates { + t.Fatalf("store backstop = %#v", typed) + } + if typed.Reason != ProxyReasonUnimplemented { + t.Fatalf("store backstop reason = %q, want unimplemented", typed.Reason) + } +} diff --git a/cmd/bd/transform_capability_test.go b/cmd/bd/capability_registry_transform_test.go similarity index 88% rename from cmd/bd/transform_capability_test.go rename to cmd/bd/capability_registry_transform_test.go index 913b4d39f..86ed302b4 100644 --- a/cmd/bd/transform_capability_test.go +++ b/cmd/bd/capability_registry_transform_test.go @@ -58,24 +58,25 @@ func TestProxyTransformCapabilityRows(t *testing.T) { } } } - row, ok := lookupTransformCapability(cmd) + row, ok := capabilityRowFor(cmd) if !ok { t.Fatalf("lookup(%q, %q) missing row", tc.path, tc.argument) } - if row.Path != tc.path || row.Argument != tc.argument || row.Outcome != tc.outcome { + if row.Path != tc.path || row.ArgSet != tc.argument || row.Rule.Outcome != tc.outcome { t.Fatalf("row = %#v, want path=%q argument=%q outcome=%q", row, tc.path, tc.argument, tc.outcome) } if tc.outcome == ProxyOutcomeRefused { - if row.Code != tc.code || row.Message != tc.message || row.ExitCode != tc.exit || row.Mutates != tc.mutates { - t.Fatalf("refusal fields = code=%q message=%q exit=%d mutates=%t", row.Code, row.Message, row.ExitCode, row.Mutates) + if row.Rule.Code != tc.code || row.Rule.Message != tc.message || row.Rule.ExitCode != tc.exit || row.Rule.Mutates != tc.mutates { + t.Fatalf("refusal fields = code=%q message=%q exit=%d mutates=%t", + row.Rule.Code, row.Rule.Message, row.Rule.ExitCode, row.Rule.Mutates) } - typed := transformCapabilityError(row) + var typed error = proxyCapabilityErrorFor(row.Rule) var got *ProxyCapabilityError if !errors.As(typed, &got) || got.Code != tc.code || got.ExitCode != 1 || got.Mutates { t.Fatalf("typed refusal = %#v", typed) } } - if err := validateProxyTransformBeforeProvider(cmd); tc.outcome == ProxyOutcomeHonored && err != nil { + if err := validateProxyRegistryBeforeProvider(cmd); tc.outcome == ProxyOutcomeHonored && err != nil { t.Fatalf("honored transform refused: %v", err) } }) @@ -85,11 +86,11 @@ func TestProxyTransformCapabilityRows(t *testing.T) { func TestProxyDuplicatesAutoMergeDryRunMatrix(t *testing.T) { cmd := transformTestCommand("duplicates", "auto-merge", "dry-run") _ = cmd.Flags().Set("auto-merge", "true") - if err := validateProxyTransformBeforeProvider(cmd); err == nil { + if err := validateProxyRegistryBeforeProvider(cmd); err == nil { t.Fatalf("non-dry auto-merge error = %v", err) } _ = cmd.Flags().Set("dry-run", "true") - if err := validateProxyTransformBeforeProvider(cmd); err != nil { + if err := validateProxyRegistryBeforeProvider(cmd); err != nil { t.Fatalf("dry-run auto-merge refused: %v", err) } } @@ -103,7 +104,7 @@ func TestProxyTransformRefusalRendering(t *testing.T) { jsonOutput = false var textErr string stderr := captureStderr(t, func() { - err := validateProxyTransformBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd) if code, ok := exitCodeFromError(err); !ok || code != 1 { t.Fatalf("exit = %v, want 1", err) } @@ -115,7 +116,7 @@ func TestProxyTransformRefusalRendering(t *testing.T) { jsonOutput = true stdout := captureStdout(t, func() error { - _ = validateProxyTransformBeforeProvider(cmd) + _ = validateProxyRegistryBeforeProvider(cmd) return nil }) var envelope map[string]any @@ -140,8 +141,8 @@ func TestProxyTransformMatcherIgnoresInheritedOutputFlags(t *testing.T) { if err := root.PersistentFlags().Set("json", "true"); err != nil { t.Fatal(err) } - row, ok := lookupTransformCapability(cmd) - if !ok || row.Argument != "--auto-merge" || row.Outcome != ProxyOutcomeRefused { + row, ok := capabilityRowFor(cmd) + if !ok || row.ArgSet != "--auto-merge" || row.Rule.Outcome != ProxyOutcomeRefused { t.Fatalf("row with inherited --json = %#v, ok=%v", row, ok) } } @@ -163,7 +164,7 @@ func TestProxyTransformRefusalHasNoProviderOrFileMutation(t *testing.T) { t.Cleanup(func() { uowProvider = oldProvider }) cmd := transformTestCommand("supersede") - err := validateProxyTransformBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd) if err == nil { t.Fatal("expected transform refusal") } diff --git a/cmd/bd/errors.go b/cmd/bd/errors.go index 80ff1b5b3..bf753777b 100644 --- a/cmd/bd/errors.go +++ b/cmd/bd/errors.go @@ -76,8 +76,16 @@ func buildJSONError(message, hint string) interface{} { return inner } +// buildJSONCapabilityError renders a typed refusal. {code, error, mutates} is a +// frozen contract that downstream consumers parse; `reason` is additive and +// appears only on refusals that came from the capability registry, which is why +// it is omitted rather than emitted empty — a consumer that keys on its presence +// must not be told "" by a runtime state error that has no policy reason. func buildJSONCapabilityError(e *ProxyCapabilityError) interface{} { inner := map[string]interface{}{"error": e.Message, "code": e.Code, "mutates": e.Mutates} + if e.Reason != "" { + inner["reason"] = string(e.Reason) + } if jsonEnvelopeEnabled() { return map[string]interface{}{"schema_version": JSONSchemaVersion, "data": inner} } diff --git a/cmd/bd/history_capability.go b/cmd/bd/history_capability.go deleted file mode 100644 index c3e5fd3f7..000000000 --- a/cmd/bd/history_capability.go +++ /dev/null @@ -1,28 +0,0 @@ -package main - -// HistoryCapabilityClass classifies history/version-control commands on the -// proxied topology without coupling them to a particular backend engine. -type HistoryCapabilityClass string - -const ( - HistoryProxySupported HistoryCapabilityClass = "proxy-supported" - HistoryProxyAPI HistoryCapabilityClass = "proxy-api" - HistoryDirectOnly HistoryCapabilityClass = "direct-only" -) - -var historyCapabilityMatrix = map[string]HistoryCapabilityClass{ - "history": HistoryProxySupported, - "branch": HistoryDirectOnly, "conflicts": HistoryDirectOnly, "repo": HistoryDirectOnly, - "federation": HistoryDirectOnly, "vc": HistoryDirectOnly, "flatten": HistoryDirectOnly, - "dolt push": HistoryDirectOnly, "dolt pull": HistoryDirectOnly, "dolt commit": HistoryDirectOnly, - "dolt remote": HistoryDirectOnly, "dolt remote add": HistoryDirectOnly, "dolt remote list": HistoryDirectOnly, - "dolt remote reset-data": HistoryDirectOnly, "dolt remote remove": HistoryProxySupported, - "sync": HistoryDirectOnly, -} - -// LookupHistoryCapability classifies an exact command path. Unknown paths are -// intentionally absent so callers cannot accidentally advertise support. -func LookupHistoryCapability(commandPath string) (HistoryCapabilityClass, bool) { - c, ok := historyCapabilityMatrix[commandPath] - return c, ok -} diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 5742c07ef..946cefbb1 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -229,13 +229,6 @@ func effectiveRootStorePolicy(cmdName string, strictReadonly bool) rootStorePoli } } -// backendSupportsStrictReadonly reports whether the live backend path can open -// without provisioning or lifecycle changes. Unsupported SQL backends are -// rejected earlier by validateConfiguredBackend; proxied Dolt remains writable-only. -func backendSupportsStrictReadonly(cfg *configfile.Config) bool { - return cfg == nil || !cfg.IsDoltProxiedServerMode() -} - // runsPostCommandMaintenance reports whether PersistentPostRunE should run the // post-command maintenance net — Dolt auto-commit, the tip-metadata commit, // auto-backup, auto-export and auto-push. @@ -1293,11 +1286,11 @@ var rootCmd = &cobra.Command{ } } if cmdName == "doctor" && usesProxiedServer() { - // Refuse only on a real refusal. validateProxyMaintenance... + // Refuse only on a real refusal. The registry validator // returns nil for doctor subcommands, and returning early on // that would skip the legacy-store guard and autocommit-mode // resolution every other skipsStoreInit command still runs. - if err := validateProxyMaintenanceBeforeProvider(cmd); err != nil { + if err := validateProxyRegistryBeforeProvider(cmd); err != nil { return err } } @@ -1504,14 +1497,13 @@ var rootCmd = &cobra.Command{ } // Reject proxy capability combinations before any workspace side effect // (version tracking, migration, auto-start, or provider construction). + // Two validators, one for each half of the policy: flag-keyed rules and + // the path-keyed capability registry. if cfg != nil && cfg.IsDoltProxiedServerMode() { if err := validateProxyCapabilitiesBeforeProvider(cmd); err != nil { return err } - if err := validateProxyMaintenanceBeforeProvider(cmd); err != nil { - return err - } - if err := validateProxyTransformBeforeProvider(cmd); err != nil { + if err := validateProxyRegistryBeforeProvider(cmd); err != nil { return err } } @@ -1522,9 +1514,6 @@ var rootCmd = &cobra.Command{ if readonlyMode && cfg != nil && cfg.IsDoltProxiedServerMode() { return HandleProxyCapabilityError(AssertProxyCapability(ProxyModeProxied, ProxyCapReadonly)) } - if readonlyMode && !backendSupportsStrictReadonly(cfg) { - return HandleError("strict readonly is unavailable for dolt proxied-server backend; refusing to open a store that cannot guarantee mutation-free access") - } // Set actor for audit trail actor = getActorWithGit() diff --git a/cmd/bd/migrate_dolt_mode.go b/cmd/bd/migrate_dolt_mode.go index 387624e93..684212738 100644 --- a/cmd/bd/migrate_dolt_mode.go +++ b/cmd/bd/migrate_dolt_mode.go @@ -199,12 +199,17 @@ func migrateFault(phase migratePhase) error { return nil } +// externalMigrationRefusal is a policy refusal rather than a registry row: it +// depends on the workspace's endpoint, not on which command was typed. The +// reason is design because someone else owns that endpoint and its schema — +// no amount of plumbing on this side makes migrating it bd's business. func externalMigrationRefusal(message string) error { return HandleProxyCapabilityError(&ProxyCapabilityError{ Code: "proxy.migrate.external_endpoint", Message: message, ExitCode: 1, Mutates: false, + Reason: ProxyReasonDesign, }) } diff --git a/cmd/bd/proxy_capability.go b/cmd/bd/proxy_capability.go index 8c188183e..774323944 100644 --- a/cmd/bd/proxy_capability.go +++ b/cmd/bd/proxy_capability.go @@ -2,7 +2,6 @@ package main import ( "fmt" - "strings" "github.com/spf13/cobra" ) @@ -47,17 +46,24 @@ const ( ProxyOutcomeRefused ProxyCapabilityOutcome = "refused" ProxyOutcomeDelegated ProxyCapabilityOutcome = "delegated" ProxyOutcomeNA ProxyCapabilityOutcome = "N/A" + // ProxyOutcomeRefusedInRunE records a path the gate permits and that the + // command then refuses itself, untyped, from RunE. It is inventory, not + // policy: nothing enforces it. See capability_registry.go. + ProxyOutcomeRefusedInRunE ProxyCapabilityOutcome = "refused-in-run" ) // proxyCapabilityRule is the stable contract for one command/argument/topology // capability. Mutates is false for all refusals; ExitCode is used by the CLI -// when rendering a typed refusal. +// when rendering a typed refusal. Reason and Tracking say why a refusal exists +// and who closes it — see the header of capability_registry.go. type proxyCapabilityRule struct { Outcome ProxyCapabilityOutcome Code string Message string ExitCode int Mutates bool + Reason ProxyRefusalReason + Tracking string } // ProxyCapabilityError is a machine-identifiable front-door refusal. @@ -66,12 +72,39 @@ type ProxyCapabilityError struct { Message string ExitCode int Mutates bool + Reason ProxyRefusalReason } func (e *ProxyCapabilityError) Error() string { return e.Message } -func refused(code, message string) proxyCapabilityRule { - return proxyCapabilityRule{Outcome: ProxyOutcomeRefused, Code: code, Message: message, ExitCode: 1} +// errProxiedStoreUnrouted is the backstop for a command that the front door +// permitted and that then asked for a classic store on a proxied-server +// workspace. The proxied topology has no such store — the provider owns the +// connection — so the factory refuses rather than opening the proxied root a +// second time behind the proxy's back, which is the one failure mode every +// slice of this work must not regress. +// +// Reaching this is a routing gap in the command, not a policy decision: the +// registry says the gate permits the path, and permitting a path is not a +// promise that a proxied route exists for it. The owner is therefore whichever +// slice routes the command that landed here, which is why the reason is +// "unimplemented" and no single tracking item fits. It is typed so that a +// caller can classify it; before the registry there was no way to tell this +// apart from a genuine store failure. +func errProxiedStoreUnrouted() error { + return &ProxyCapabilityError{ + Code: "proxy.store.unrouted", + Message: "this command has no proxied-server route; bd will not open a proxied-server workspace as a direct store", + ExitCode: 1, + Reason: ProxyReasonUnimplemented, + } +} + +func refused(code, message string, reason ProxyRefusalReason, tracking string) proxyCapabilityRule { + return proxyCapabilityRule{ + Outcome: ProxyOutcomeRefused, Code: code, Message: message, ExitCode: 1, + Reason: reason, Tracking: tracking, + } } func honored() proxyCapabilityRule { @@ -97,104 +130,6 @@ type ProxyCapabilityRow struct { Rule proxyCapabilityRule } -var proxyMaintenanceRefusals = map[string]proxyCapabilityRule{ - "doctor": refused("proxy.doctor.unsupported", "doctor is not supported in proxied-server mode"), - "backup": refused("proxy.backup.unsupported", "backup is not supported in proxied-server mode"), - "restore": refused("proxy.restore.unsupported", "restore is not supported in proxied-server mode"), - "diff": refused("proxy.diff.unsupported", "diff is not supported in proxied-server mode"), - "flatten": refused("proxy.flatten.unsupported", "flatten is not supported in proxied-server mode"), - "migrate": refused("proxy.migrate.unsupported", "migrate is not supported in proxied-server mode"), - "migrate-personal": refused("proxy.migrate.unsupported", "migrate-personal is not supported in proxied-server mode"), - "branch": refused("proxy.branch.unsupported", "branch is not supported in proxied-server mode"), - "conflicts": refused("proxy.conflicts.unsupported", "conflicts is not supported in proxied-server mode"), - "vc": refused("proxy.vc.unsupported", "vc is not supported in proxied-server mode"), - "federation": refused("proxy.federation.unsupported", "federation is not supported in proxied-server mode"), - "repo": refused("proxy.repo.unsupported", "repo is not supported in proxied-server mode"), - // Wording matches the long-standing compact.go refusal this pre-provider - // gate now short-circuits, so the user-facing message does not change. - "compact": refused("proxy.compact.unsupported", "only 'compact --dolt' is supported in proxied-server mode"), - "backup init": refused("proxy.backup.unsupported", "backup init is not supported in proxied-server mode"), - "backup sync": refused("proxy.backup.unsupported", "backup sync is not supported in proxied-server mode"), - "backup remove": refused("proxy.backup.unsupported", "backup remove is not supported in proxied-server mode"), - "backup status": refused("proxy.backup.unsupported", "backup status is not supported in proxied-server mode"), - "backup restore": refused("proxy.backup.unsupported", "backup restore is not supported in proxied-server mode"), - "migrate sync": refused("proxy.migrate.unsupported", "migrate sync is not supported in proxied-server mode"), - "migrate-issues": refused("proxy.migrate.unsupported", "migrate-issues is not supported in proxied-server mode"), - "gate discover": refused("proxy.gate.unsupported", "gate discover is not supported in proxied-server mode"), - "admin cleanup": refused("proxy.admin.unsupported", "admin cleanup is not supported in proxied-server mode"), - "admin reset": refused("proxy.admin.unsupported", "admin reset is not supported in proxied-server mode"), - "dolt push": refused("proxy.dolt_push.unsupported", "dolt push is not supported in proxied-server mode"), - "dolt pull": refused("proxy.dolt_pull.unsupported", "dolt pull is not supported in proxied-server mode"), - "dolt commit": refused("proxy.dolt_commit.unsupported", "dolt commit is not supported in proxied-server mode"), - "dolt remote": refused("proxy.dolt_remote.unsupported", "dolt remote is not supported in proxied-server mode"), - "dolt remote add": refused("proxy.dolt_remote.unsupported", "dolt remote add is not supported in proxied-server mode"), - "dolt remote list": refused("proxy.dolt_remote.unsupported", "dolt remote list is not supported in proxied-server mode"), - "cook": refused("proxy.formula.unsupported", "cook is not supported in proxied-server mode"), - "ship": refused("proxy.formula.unsupported", "ship is not supported in proxied-server mode"), - "swarm create": refused("proxy.swarm.unsupported", "swarm create is not supported in proxied-server mode"), - "swarm list": refused("proxy.swarm.unsupported", "swarm list is not supported in proxied-server mode"), - "merge-slot create": refused("proxy.merge_slot.unsupported", "merge-slot create is not supported in proxied-server mode"), - "merge-slot check": refused("proxy.merge_slot.unsupported", "merge-slot check is not supported in proxied-server mode"), - "merge-slot acquire": refused("proxy.merge_slot.unsupported", "merge-slot acquire is not supported in proxied-server mode"), - "merge-slot release": refused("proxy.merge_slot.unsupported", "merge-slot release is not supported in proxied-server mode"), - "dolt remote reset-data": refused("proxy.dolt_remote.unsupported", "dolt remote reset-data is not supported in proxied-server mode"), - "sync": refused("proxy.sync.unsupported", "sync is not supported in proxied-server mode"), -} - -func init() { - for parent, children := range map[string][]string{ - "vc": {"merge", "commit", "status"}, - "federation": {"sync", "status", "add-peer", "remove-peer", "list-peers"}, - "repo": {"add", "remove", "list", "sync"}, - "conflicts": {"list", "show", "resolve"}, - "migrate": {"hooks", "issues"}, - } { - rule := proxyMaintenanceRefusals[parent] - for _, child := range children { - path := parent + " " + child - proxyMaintenanceRefusals[path] = refused(rule.Code, path+" is not supported in proxied-server mode") - } - } -} - -// validateProxyMaintenanceBeforeProvider rejects known direct-only commands -// before migrations, auto-start, or provider construction. -func validateProxyMaintenanceBeforeProvider(cmd *cobra.Command) error { - if cmd == nil { - return nil - } - name := cmd.Name() - path := strings.TrimSpace(strings.TrimPrefix(cmd.CommandPath(), cmd.Root().Name())) - if name == "compact" { - if cmd.Flags().Lookup("dolt") == nil { - return nil // root `bd compact` is the Dolt history command - } - dolt, _ := cmd.Flags().GetBool("dolt") - if dolt { - return nil - } - rule := proxyMaintenanceRefusals["compact"] - return HandleProxyCapabilityError(&ProxyCapabilityError{Code: rule.Code, Message: rule.Message, ExitCode: rule.ExitCode}) - } - if rule, ok := proxyMaintenanceRefusals[path]; ok { - return HandleProxyCapabilityError(&ProxyCapabilityError{Code: rule.Code, Message: rule.Message, ExitCode: rule.ExitCode}) - } - if class, ok := LookupHistoryCapability(path); ok && class == HistoryDirectOnly { - rule := refused("proxy.history.unsupported", path+" is not supported in proxied-server mode") - if specific, found := proxyMaintenanceRefusals[path]; found { - rule = specific - } - return HandleProxyCapabilityError(&ProxyCapabilityError{Code: rule.Code, Message: rule.Message, ExitCode: rule.ExitCode, Mutates: rule.Mutates}) - } - if strings.Contains(path, " ") { - return nil - } - if rule, ok := proxyMaintenanceRefusals[name]; ok { - return HandleProxyCapabilityError(&ProxyCapabilityError{Code: rule.Code, Message: rule.Message, ExitCode: rule.ExitCode}) - } - return nil -} - // LookupProxyCapabilityFor returns the command/argument-specific rule. An // absent command row falls back to the topology-wide default. func LookupProxyCapabilityFor(command, argument string, mode ProxyMode) (proxyCapabilityRule, bool) { @@ -212,26 +147,50 @@ func LookupProxyCapabilityFor(command, argument string, mode ProxyMode) (proxyCa return LookupProxyCapability(mode, capability) } +// Shared by the mode-wide rule and its per-command overrides, so a command that +// refuses one of these refuses it with the same words and the same reason. +var ( + // The cap is refused rather than silently dropped, which is right; the fix + // is threading it through the UOW reader the way `list` already does, so + // the refusal itself is the gap. + maxRowsRefusal = refused("proxy.max_rows.unsupported", "--max-rows / BEADS_MAX_ROWS is not supported in proxied-server mode", ProxyReasonUnimplemented, trackLongTail) + // `list --watch` is routed, so watching is possible over the provider; + // nothing else has been wired to it. + watchRefusal = refused("proxy.watch.unsupported", "watch mode not supported in proxied-server mode", ProxyReasonUnimplemented, trackLongTail) +) + +// proxyCapabilityMatrix is the flag-keyed half of the policy; the path-keyed +// half is the registry in capability_registry.go. Reasons follow the same rule +// there and here. +// +// Strict --readonly is the one capability that needs real backend work rather +// than routing: it is a GUARANTEE, the proxied provider carries no read-only +// posture, and the only honest implementation is a read-only SQL principal on +// the backend — which on external topologies is the operator's configuration, +// not bd's. A best-effort version would betray exactly the workflows that ask +// for it, so it is refused by design until that principal exists. --repo is +// design for a simpler reason: it routes to another workspace, bypassing the +// proxied root entirely. var proxyCapabilityMatrix = map[ProxyMode]map[ProxyCapability]proxyCapabilityRule{ ProxyModeDirect: { ProxyCapReadonly: honored(), ProxyCapMaxRows: honored(), ProxyCapWatch: honored(), ProxyCapRepo: honored(), }, ProxyModeProxied: { - ProxyCapReadonly: refused("proxy.readonly.unsupported", "strict readonly is unavailable for dolt proxied-server backend; refusing to open a store that cannot guarantee mutation-free access"), - ProxyCapMaxRows: refused("proxy.max_rows.unsupported", "--max-rows / BEADS_MAX_ROWS is not supported in proxied-server mode"), - ProxyCapWatch: refused("proxy.watch.unsupported", "watch mode not supported in proxied-server mode"), - ProxyCapRepo: refused("proxy.repo.unsupported", "--repo is not supported with --proxied-server"), + ProxyCapReadonly: refused("proxy.readonly.unsupported", "strict readonly is unavailable for dolt proxied-server backend; refusing to open a store that cannot guarantee mutation-free access", ProxyReasonDesign, "1.4 candidate: read-only SQL principal on the backend"), + ProxyCapMaxRows: maxRowsRefusal, + ProxyCapWatch: watchRefusal, + ProxyCapRepo: refused("proxy.repo.unsupported", "--repo is not supported with --proxied-server", ProxyReasonDesign, ""), }, } var proxyCommandCapabilities = map[string]map[ProxyMode]map[ProxyCapability]proxyCapabilityRule{ - "show": {ProxyModeProxied: {ProxyCapWatch: refused("proxy.watch.unsupported", "watch mode not supported in proxied-server mode")}}, + "show": {ProxyModeProxied: {ProxyCapWatch: watchRefusal}}, "list": {ProxyModeProxied: {ProxyCapWatch: honored(), ProxyCapMaxRows: honored(), ProxyCapRepo: notApplicable()}}, "dep tree": {ProxyModeProxied: {ProxyCapMaxRows: honored()}}, - "ready": {ProxyModeProxied: {ProxyCapMaxRows: refused("proxy.max_rows.unsupported", "--max-rows / BEADS_MAX_ROWS is not supported in proxied-server mode")}}, - "graph": {ProxyModeProxied: {ProxyCapMaxRows: refused("proxy.max_rows.unsupported", "--max-rows / BEADS_MAX_ROWS is not supported in proxied-server mode")}}, - "find-duplicates": {ProxyModeProxied: {ProxyCapMaxRows: refused("proxy.max_rows.unsupported", "--max-rows / BEADS_MAX_ROWS is not supported in proxied-server mode")}}, + "ready": {ProxyModeProxied: {ProxyCapMaxRows: maxRowsRefusal}}, + "graph": {ProxyModeProxied: {ProxyCapMaxRows: maxRowsRefusal}}, + "find-duplicates": {ProxyModeProxied: {ProxyCapMaxRows: maxRowsRefusal}}, } // proxyCapabilityRows materializes the policy for every supported proxied @@ -305,7 +264,7 @@ func AssertProxyCommandCapability(command string, mode ProxyMode, capability Pro return nil } if rule.Code != "" { - return &ProxyCapabilityError{Code: rule.Code, Message: rule.Message, ExitCode: rule.ExitCode, Mutates: rule.Mutates} + return proxyCapabilityErrorFor(rule) } return fmt.Errorf("%s", rule.Message) } @@ -314,7 +273,7 @@ func AssertProxyCommandCapability(command string, mode ProxyMode, capability Pro rule, ok := LookupProxyCapability(mode, capability) if !ok || (rule.Outcome != ProxyOutcomeHonored && rule.Outcome != ProxyOutcomeDelegated) { if rule.Code != "" { - return &ProxyCapabilityError{Code: rule.Code, Message: rule.Message, ExitCode: rule.ExitCode, Mutates: rule.Mutates} + return proxyCapabilityErrorFor(rule) } if rule.Message != "" { return fmt.Errorf("%s", rule.Message) diff --git a/cmd/bd/proxy_capability_test.go b/cmd/bd/proxy_capability_test.go index 54656af26..3b7718cf0 100644 --- a/cmd/bd/proxy_capability_test.go +++ b/cmd/bd/proxy_capability_test.go @@ -151,7 +151,7 @@ func TestProxyMaintenanceNestedPathsRefuseBeforeProvider(t *testing.T) { root.AddCommand(parent) parent.AddCommand(child) out := captureStdout(t, func() error { - _ = validateProxyMaintenanceBeforeProvider(child) + _ = validateProxyRegistryBeforeProvider(child) return nil }) if !strings.Contains(out, `"code":`) { @@ -171,7 +171,7 @@ func TestProxyFormulaSwarmMergeSlotRefusals(t *testing.T) { cmd.AddCommand(child) cmd = child } - err := validateProxyMaintenanceBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd) if err == nil { t.Fatalf("%s unexpectedly allowed", path) } @@ -209,7 +209,7 @@ func TestProxyWorkflowRefusalContractAndNoMutation(t *testing.T) { if !ok || row.Code != tc.code || row.Message != tc.message || row.ExitCode != 1 || row.Mutates { t.Fatalf("row = %#v, ok=%v", row, ok) } - var typed *ProxyCapabilityError = &ProxyCapabilityError{Code: row.Code, Message: row.Message, ExitCode: row.ExitCode, Mutates: row.Mutates} + typed := proxyCapabilityErrorFor(row) if typed.Code != tc.code || typed.Message != tc.message || typed.ExitCode != 1 || typed.Mutates { t.Fatalf("typed refusal = %#v", typed) } @@ -232,7 +232,7 @@ func TestProxyWorkflowRefusalContractAndNoMutation(t *testing.T) { oldDidWrite := commandDidWrite.Load() commandDidWrite.Store(false) t.Cleanup(func() { commandDidWrite.Store(oldDidWrite) }) - out := captureStdout(t, func() error { _ = validateProxyMaintenanceBeforeProvider(cmd); return nil }) + out := captureStdout(t, func() error { _ = validateProxyRegistryBeforeProvider(cmd); return nil }) var got map[string]any if err := json.Unmarshal([]byte(out), &got); err != nil || got["code"] != tc.code || got["error"] != tc.message { t.Fatalf("JSON refusal = %q (%v)", out, err) @@ -251,8 +251,8 @@ func TestProxyWorkflowRefusalContractAndNoMutation(t *testing.T) { } func lookupProxyMaintenanceRuleForTest(path string) (proxyCapabilityRule, bool) { - rule, ok := proxyMaintenanceRefusals[path] - return rule, ok + row, ok := LookupCapabilityRow(path, "") + return row.Rule, ok } func TestProxyMaintenanceRefusalLeavesFilesUntouched(t *testing.T) { @@ -269,7 +269,7 @@ func TestProxyMaintenanceRefusalLeavesFilesUntouched(t *testing.T) { oldProvider := uowProvider uowProvider = nil t.Cleanup(func() { uowProvider = oldProvider }) - err := validateProxyMaintenanceBeforeProvider(hooks) + err := validateProxyRegistryBeforeProvider(hooks) if err == nil { t.Fatal("expected typed maintenance refusal") } diff --git a/cmd/bd/readonly_policy_test.go b/cmd/bd/readonly_policy_test.go index cd5d78f91..afa6329c2 100644 --- a/cmd/bd/readonly_policy_test.go +++ b/cmd/bd/readonly_policy_test.go @@ -68,26 +68,6 @@ func TestEffectiveRootStorePolicy(t *testing.T) { } } -func TestStrictReadonlyBackendSupport(t *testing.T) { - tests := []struct { - name string - cfg *configfile.Config - want bool - }{ - {name: "fresh embedded default", cfg: nil, want: true}, - {name: "dolt server", cfg: &configfile.Config{Backend: configfile.BackendDolt, DoltMode: configfile.DoltModeServer}, want: true}, - {name: "proxied server", cfg: &configfile.Config{Backend: configfile.BackendDolt, DoltMode: configfile.DoltModeProxiedServer}, want: false}, - } - - for _, tc := range tests { - t.Run(tc.name, func(t *testing.T) { - if got := backendSupportsStrictReadonly(tc.cfg); got != tc.want { - t.Fatalf("backendSupportsStrictReadonly() = %v, want %v", got, tc.want) - } - }) - } -} - type strictReadonlyPostRunStore struct { storage.DoltStorage metadataWrites int diff --git a/cmd/bd/serve.go b/cmd/bd/serve.go index c3df2abcc..f26e51ca7 100644 --- a/cmd/bd/serve.go +++ b/cmd/bd/serve.go @@ -603,7 +603,7 @@ func serveDatabaseSource(beadsDir string) (serveDatabase, error) { // the server, which no client can discover before connecting. bd already // answers this question the same way one layer down, where a backend that // cannot guarantee mutation-free access is turned away rather than opened -// anyway (backendSupportsStrictReadonly, cmd/bd/main.go). +// anyway (the proxy.readonly.unsupported refusal in cmd/bd/main.go). // // The value is read from the global rather than a flag lookup because // `readonly` is also a config key, and PersistentPreRunE has already folded diff --git a/cmd/bd/store_factory.go b/cmd/bd/store_factory.go index 00f1afb32..0bf5fe521 100644 --- a/cmd/bd/store_factory.go +++ b/cmd/bd/store_factory.go @@ -70,9 +70,7 @@ func newRegisteredBackendStore(ctx context.Context, name, beadsDir string, readO func newDoltStore(ctx context.Context, cfg *dolt.Config) (s storage.DoltStorage, err error) { defer func() { s, err = activateEventsJournalStore(cfg.BeadsDir, s, err) }() if cfg.ProxiedServer { - // TODO: this should not be a store - // it should be a uow provider - return nil, fmt.Errorf("proxy server store should be uow provider") + return nil, errProxiedStoreUnrouted() } if cfg.ServerMode { return dolt.New(ctx, cfg) @@ -172,13 +170,7 @@ func newDoltStoreFromConfig(ctx context.Context, beadsDir string) (s storage.Dol return backend.Open(ctx, beadsDir) } if cfg != nil && cfg.IsDoltProxiedServerMode() { - // TODO: this needs to be uow provider - return nil, fmt.Errorf("proxy server store should be uow provider") - // return newProxiedServerStore(ctx, &dolt.Config{ - // BeadsDir: beadsDir, - // Database: cfg.GetDoltDatabase(), - // ProxiedServer: true, - // }) + return nil, errProxiedStoreUnrouted() } if cfg != nil && cfg.IsDoltServerMode() { return dolt.NewFromConfig(ctx, beadsDir) @@ -280,14 +272,7 @@ func openNonMutatingStoreFromConfig(ctx context.Context, beadsDir string, previe return backend.OpenReadOnly(ctx, beadsDir) } if cfg != nil && cfg.IsDoltProxiedServerMode() { - // TODO: this needs to be uow provider - return nil, fmt.Errorf("proxy server store needs to be uow provider") - // return newProxiedServerStore(ctx, &dolt.Config{ - // BeadsDir: beadsDir, - // Database: cfg.GetDoltDatabase(), - // ProxiedServer: true, - // ReadOnly: true, - // }) + return nil, errProxiedStoreUnrouted() } if cfg != nil && cfg.IsDoltServerMode() { return dolt.NewFromConfigWithOptions(ctx, beadsDir, &dolt.Config{ReadOnly: true}) diff --git a/cmd/bd/store_factory_nocgo.go b/cmd/bd/store_factory_nocgo.go index 3266e1aa9..152ceb294 100644 --- a/cmd/bd/store_factory_nocgo.go +++ b/cmd/bd/store_factory_nocgo.go @@ -50,9 +50,7 @@ func newRegisteredBackendStore(ctx context.Context, name, beadsDir string, readO func newDoltStore(ctx context.Context, cfg *dolt.Config) (s storage.DoltStorage, err error) { defer func() { s, err = activateEventsJournalStore(cfg.BeadsDir, s, err) }() if cfg.ProxiedServer { - // TODO: this should not be a store - // it should be a uow provider - return nil, fmt.Errorf("proxy server store should be uow provider") + return nil, errProxiedStoreUnrouted() } if !cfg.ServerMode { return nil, fmt.Errorf("%s", nocgoEmbeddedErrMsg) @@ -83,13 +81,7 @@ func newDoltStoreFromConfig(ctx context.Context, beadsDir string) (s storage.Dol return backend.Open(ctx, beadsDir) } if cfg != nil && cfg.IsDoltProxiedServerMode() { - // TODO: this needs to be uow provider - return nil, fmt.Errorf("proxy server store should be uow provider") - // return newProxiedServerStore(ctx, &dolt.Config{ - // BeadsDir: beadsDir, - // Database: cfg.GetDoltDatabase(), - // ProxiedServer: true, - // }) + return nil, errProxiedStoreUnrouted() } if cfg != nil && cfg.IsDoltServerMode() { return dolt.NewFromConfig(ctx, beadsDir) @@ -114,14 +106,7 @@ func newReadOnlyStoreFromConfig(ctx context.Context, beadsDir string) (storage.D return backend.OpenReadOnly(ctx, beadsDir) } if cfg != nil && cfg.IsDoltProxiedServerMode() { - // TODO: this needs to be uow provider - return nil, fmt.Errorf("proxy server store needs to be uow provider") - // return newProxiedServerStore(ctx, &dolt.Config{ - // BeadsDir: beadsDir, - // Database: cfg.GetDoltDatabase(), - // ProxiedServer: true, - // ReadOnly: true, - // }) + return nil, errProxiedStoreUnrouted() } if cfg != nil && cfg.IsDoltServerMode() { return dolt.NewFromConfigWithOptions(ctx, beadsDir, &dolt.Config{ReadOnly: true}) diff --git a/cmd/bd/transform_capability.go b/cmd/bd/transform_capability.go deleted file mode 100644 index 2265a1b78..000000000 --- a/cmd/bd/transform_capability.go +++ /dev/null @@ -1,90 +0,0 @@ -package main - -import ( - "sort" - "strings" - - "github.com/spf13/cobra" - "github.com/spf13/pflag" -) - -type TransformCapability struct { - Path string - Argument string - Outcome ProxyCapabilityOutcome - Code string - Message string - ExitCode int - Mutates bool -} - -var transformCapabilityRows = []TransformCapability{ - transformRefused("rename"), - transformRefused("rename-prefix"), - transformRefusedWithArgs("rename-prefix", "--dry-run"), - transformRefusedWithArgs("rename-prefix", "--repair"), - transformRefusedWithArgs("rename-prefix", "--dry-run --repair"), - transformRefused("duplicate"), - transformRefusedWithArgs("duplicate", "--of"), - transformRefused("supersede"), - transformRefusedWithArgs("supersede", "--with"), - {Path: "duplicates", Outcome: ProxyOutcomeHonored}, - transformRefusedWithArgs("duplicates", "--auto-merge"), - {Path: "duplicates", Argument: "--auto-merge --dry-run", Outcome: ProxyOutcomeHonored}, -} - -func transformRefused(path string) TransformCapability { - return transformRefusedWithArgs(path, "") -} - -func transformRefusedWithArgs(path, argument string) TransformCapability { - displayPath := path - if argument != "" { - displayPath += " " + argument - } - return TransformCapability{ - Path: path, Argument: argument, Outcome: ProxyOutcomeRefused, - Code: "proxy.transform.unsupported", Message: displayPath + " is not supported in proxied-server mode", ExitCode: 1, - } -} - -func transformCapabilityError(row TransformCapability) *ProxyCapabilityError { - return &ProxyCapabilityError{Code: row.Code, Message: row.Message, ExitCode: row.ExitCode, Mutates: row.Mutates} -} - -func lookupTransformCapability(cmd *cobra.Command) (TransformCapability, bool) { - path := cmd.CommandPath() - path = strings.TrimSpace(strings.TrimPrefix(path, cmd.Root().Name())) - var args []string - cmd.Flags().Visit(func(f *pflag.Flag) { - if transformPolicyFlags[f.Name] { - args = append(args, "--"+f.Name) - } - }) - sort.Strings(args) - arg := strings.Join(args, " ") - for _, row := range transformCapabilityRows { - if row.Path == path && row.Argument == arg { - return row, true - } - } - return TransformCapability{}, false -} - -// Only flags that alter a transform's execution belong in the capability key. -// Output, tracing, and other inherited root flags must not bypass a refusal. -var transformPolicyFlags = map[string]bool{ - "auto-merge": true, "dry-run": true, "repair": true, "of": true, "with": true, -} - -func validateProxyTransformBeforeProvider(cmd *cobra.Command) error { - if cmd == nil { - return nil - } - if row, ok := lookupTransformCapability(cmd); ok { - if row.Outcome == ProxyOutcomeRefused { - return HandleProxyCapabilityError(transformCapabilityError(row)) - } - } - return nil -} From ca37d8f9fcdb5f680fb6607f8486b58163115d58 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 23 Sep 2026 02:16:45 -0700 Subject: [PATCH 12/45] feat(backup): route the backup family over proxied-server, refused by design off-host (proxied-parity S3) (#6584) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * refactor(proxy): one capability registry, with a reason on every refusal Proxied-server policy lived in three path tables (a maintenance refusal map, a history classification map, a transform row list), each with its own lookup and its own front-door validator, plus a flag matrix. Nothing forced a new command to appear in any of them, so an unclassified command sailed past the gate and died in the store factory with `proxy server store should be uow provider`. That — not the table count — was the root problem. One table now (cmd/bd/capability_registry.go), and a test that fails when a runnable command is missing from it. Adding a command is no longer a place where policy can be forgotten: it is a build failure until someone decides whether the proxied front door lets it through. Every refusal carries a Reason: `design` where refusing is correct and expected to stay (shared history, multi-repo routing, destructive admin, storage-format surgery, strict --readonly), `unimplemented` where it is a gap with a named owner. The reason renders in the refusal JSON, additive to the frozen {code, error, mutates} contract, and is omitted on refusals that report a runtime state rather than a policy so a consumer keying on its presence is never told "" by mistake. The registry also inventories the 57 untyped refusals commands raise from their own RunE. Those rows are not enforced — converting them changes observable output and belongs to the slice that owns each family — but a source scan now keeps the inventory honest in both directions. The store-factory backstop is typed as `proxy.store.unrouted`, the unreachable duplicate readonly refusal in main.go is gone (the proxied branch above it already returns for the only config that reaches it), and with it `backendSupportsStrictReadonly`, which had no other caller. No outcome changes. Every code, message and exit status is identical, which #6576's five-topology matrix proves: 90/90 cells unchanged (HONORED 47, REFUSED_TYPED 24 = 4 design + 20 unimplemented, ERROR 12, WRONG_ANSWER 4, DEGRADED 3), all five topologies exercised, no skips. Co-Authored-By: Claude Fable 5 * refactor(proxy): `migrate hooks` is an unimplemented refusal, not a design one The registry filed `migrate hooks` under "storage-format and workspace surgery against a store other clients may be holding open". That rationale does not describe it: migrate_hooks.go plans and applies GIT HOOK FILE migration through doctor.PlanHookMigration, on the local filesystem, and opens no store at all. Left as it was, proxied-local — the default topology since 1.3.0 — loses the command permanently, because `design` tells every later slice the refusal is semantic and settled. It is not settled, but it is also not free to lift, and the difference is structural rather than semantic. `migrate hooks` appears on none of main.go's store-init skip lists, so permitting it makes the root pre-run open the proxied provider before RunE runs. Measured with the row flipped to permitted, not reasoned about: on a managed-local workspace `bd migrate hooks --dry-run` started the proxy AND its dolt child — both pidfiles present — to print a hook plan for five missing hooks, and on a workspace with an unreadable sidecar it exited 1 with "failed to open uow provider: corrupt proxied-server sidecar ..." where it had cleanly refused before. Trading a typed refusal for a provider-open error is not a lift. So: Reason=unimplemented with a Tracking item that names the actual work — put the command on the store-init skip list first, then permit it — which is S7's, not this slice's. No outcome changes; the row still refuses with the same code, message and exit status. The registry header now says that permitting a path costs a provider open, since that is the fact this row turns on and the one the completeness test cannot see. Also leaves S7 an explicit instruction on the rows next door. `migrate issues`, `migrate-personal` and `admin cleanup`/`admin reset` claim `design` on a topology-blind rationale — "a store other clients may be holding open", "destructive against a possibly-shared store" — that describes external and team-server workspaces and does not describe managed-local, which is 1:1 and bd-owned. They are not reclassified here: the table cannot yet express "refused, except on managed-local", and narrowing them is semantic work nobody has done. The comments now say that is why, so the next reader does not mistake conservatism for a considered verdict. Co-Authored-By: Claude Fable 5 * feat(backup): route the backup family over proxied-server, refuse it by design off-host A proxied-server workspace had no backup path at all: all five verbs — init, sync, remove, status, restore — refused. Proxied has been the default topology since 1.3.0, which made this the highest user-weight gap in the proxied-parity plan. The mechanism needed no proxy work. CALL DOLT_BACKUP(...) is ordinary SQL, RunNonTx already hands out a raw pinned connection (the seam compact --dolt and gc run on), the UOW DSN sets a connect timeout and no read deadline, and an active connection keeps the idle reaper from arming — so a backup that takes minutes is safe from both the driver and the 30s reaper. The work was use cases, route files and policy rows, and the premise held. The policy is the interesting half, and it is the first genuinely topology-differentiated rule in the capability registry. A backup destination is resolved by the SERVER: `file:///…` names a path on the machine running dolt. On managed-local that machine and that user are bd's own, because bd spawned the child. Anywhere else — an external host or socket, or a beads-team-server database — the URL describes a filesystem bd cannot see, so honoring the command would either fail at the point of use or, far worse, report success having written the backup where the operator will never look. Those shapes stay refused with Reason=design: no plumbing on this side gives bd a path on somebody else's host. Team-server is classified by OWNERSHIP, not transport. bts owns the store's schema and identity whether bd reaches it over a port, a socket, or a child bd started, and one client registering a server-side backup remote acts on every client's data — the shape of the storm this command's own help text warns about. Restore quiesces before it replaces. DOLT_BACKUP('restore') swaps the database under the server, which is undefined with clients connected, so: the exclusive workspace gate `bd backup restore` already takes excludes every other gated bd command (including `bd serve`); this closes the provider the pre-run opened and shuts the proxy and its dolt child down; it reopens a provider of its own to run the restore as the only client; then it puts the topology down and reopens once more to reconcile the project identity and the backup remote — which also proves the restored database is serviceable. The final shutdown leaves the workspace quiescent for the next command to relaunch lazily. Registry rows gain a Topology map and honoredOn(), stated as "refuse, except here" so a shape nobody thought about — a new transport, or the unknown a corrupt sidecar produces — inherits the refusal rather than the exemption. resolveProxiedTopology reads the shape from the workspace and reports unknown rather than guessing. `bd backup restore --json` now prints a result object on every topology; it printed nothing before, which left a caller unable to tell a completed restore from a skipped one. The four backup subcommands that lacked SilenceUsage/ SilenceErrors no longer follow their error message with "Error: exit code 1" and a usage block. Verified by round trip, not by exit codes: TestManagedLocalProxiedBackupRoundTrip takes a backup on a live managed-local workspace, creates an issue after it, restores, and asserts the pre-backup issue is back and the post-backup one is gone, with both pidfiles absent afterwards. TestProxiedServerBackupRefusedOnExternalTopology pins the typed design refusal against the shared container and proves a refused command starts nothing and writes nothing. Flips four cells of #6576's matrix: `backup status` and `backup sync` go REFUSED_TYPED/unimplemented -> HONORED and -> ERROR ("no backup destination configured", the command's own error, same as every direct topology) on proxied-local, and unimplemented -> design on proxied-external-tcp. Co-Authored-By: Claude Fable 5 * fix(backup): say why proxied backup is refused, and stop calling a cleanup failure a restore failure Review follow-ups on the proxied backup slice. None of them changes which commands are honored on which topology; what changes is what the refusal CLAIMS, and what a restore says when only its teardown failed. The refusal rationale was true of one destination scheme out of five. It said "a backup destination is resolved on the server's filesystem, not this client's" — which describes `file://` and nothing else. resolveDoltBackupURL passes `https://`, `aws://` and `gs://` straight through, and `bd backup`'s own help recommends DoltHub by name; for those the server pushes over the network from its own environment and no filesystem of anybody's is involved. An operator running `bd backup init https://doltremoteapi.dolthub.com/...` on an external topology got a permanent verdict explained by a filesystem the command never touches. The refusal itself stands, on the reason that is true of every scheme: `CALL DOLT_BACKUP('add', …)` registers the remote ON THE SERVER, where it is global to every client connected to it, so one workspace's backup decision becomes everyone's — the same shape as the auto-backup storm this command's help text already warns about. The file:// locality problem is now stated as the second, scheme-specific problem it is. The message keeps its frozen " is not supported in proxied-server mode" prefix and its proxy.backup.unsupported code. The remote-scheme case gets a Tracking item, the way `diff` got "1.4 candidate". The outcome is design; that particular CASE is an open question, and a design label with no tracking reads as "never" to every later slice. It is the same server-env credential question S4 has to answer for `dolt push` on these same topologies. Recorded there too: restore-from-a-remote is already half-broken in DIRECT mode, because validateBackupRestoreDir stats its argument, so `bd backup restore https://...` fails everywhere today. A restore that succeeded and then failed to shut the proxy down reported "restore failed". The operator's data was back; they were told it was not, and the obvious next move — run the restore again — was the wrong one. withQuiescedProxiedProvider now returns teardown failures wrapped in *proxiedTeardownError instead of flattening them into the operation's error, and the message the caller renders is split into a function a test can hold still, because the judgement it encodes (did the data come back) is the point. Both call sites distinguish now; the second one's "did not reopen" wording was also wrong for a teardown failure, since it had reopened. Two smaller things while in here. The guard-rail comment on newProxiedServerUOWProviderAdopting still said "only two callers legitimately have no workspace identity to assert" — this slice added a third. That comment is the register of identity-assertion bypasses, so it now names the third and why, and says what to do about a fourth. And DirToFileURL was "file://" + filepath.Abs with no scheme check: it would have turned a DoltHub URL into file:///cwd/https:/doltremoteapi.dolthub.com/... , a valid-looking URL naming a directory nobody asked for. Unreachable today, and every caller is a restore path, which is exactly the capability the tracking item above describes. CHANGELOG additionally discloses two direct-mode changes this slice makes that were previously only visible in the diff: `bd backup restore --json` prints a result object on EVERY topology where it used to print nothing, and the four backup subcommands that gained SilenceUsage/SilenceErrors no longer follow a failure with "Error: exit code 1" and a usage block, on every topology. Co-Authored-By: Claude Fable 5 * fix(backup): quiesce failed restore reopens and pin JSON output (bd-obos) Preserve the primary constructor or operation error while reporting cleanup failures without a successful-restore marker. Exercise a real malformed-backup reopen failure and assert both pidfiles are removed; retain the successful round trip and assert its JSON result. Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Julian Knutsen --------- Co-authored-by: Claude Fable 5 (cherry picked from commit c69dd3061a25f176f0159ebde13426d2143b426c) --- CHANGELOG.md | 36 ++ cmd/bd/backup.go | 4 +- cmd/bd/backup_dolt.go | 38 +- ...ackup_proxied_external_integration_test.go | 107 ++++ .../backup_proxied_local_integration_test.go | 208 ++++++++ cmd/bd/backup_proxied_policy_test.go | 285 +++++++++++ cmd/bd/backup_proxied_server.go | 477 ++++++++++++++++++ cmd/bd/backup_restore.go | 25 +- cmd/bd/backup_status_test.go | 69 ++- cmd/bd/capability_registry.go | 140 ++++- cmd/bd/capability_registry_history_test.go | 10 +- cmd/bd/capability_registry_test.go | 90 ++-- cmd/bd/capability_registry_transform_test.go | 12 +- cmd/bd/main.go | 10 +- cmd/bd/proxy_capability.go | 20 +- cmd/bd/proxy_capability_test.go | 13 +- cmd/bd/proxy_topology.go | 54 ++ cmd/bd/uow_factory.go | 23 +- internal/storage/versioncontrolops/backup.go | 13 + .../storage/versioncontrolops/backup_test.go | 29 ++ 20 files changed, 1548 insertions(+), 115 deletions(-) create mode 100644 cmd/bd/backup_proxied_external_integration_test.go create mode 100644 cmd/bd/backup_proxied_local_integration_test.go create mode 100644 cmd/bd/backup_proxied_policy_test.go create mode 100644 cmd/bd/backup_proxied_server.go create mode 100644 cmd/bd/proxy_topology.go diff --git a/CHANGELOG.md b/CHANGELOG.md index f8ba307ec..46b484813 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,31 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- **`bd backup` works on a proxied-server workspace bd runs the Dolt server + for.** `bd backup init`, `sync`, `remove`, `status` and `restore` are routed + over the proxied provider; before this, a proxied workspace — the default + topology since 1.3.0 — had no backup path at all. `bd backup restore` stops + the proxy and its Dolt child before replacing the database and leaves the + workspace quiescent, so the next command relaunches against the restored + data. + + The family stays **refused by design** on a proxied workspace pointed at a + Dolt server bd does not own (an external host, socket, or a beads-team-server + database): `CALL DOLT_BACKUP('add', …)` registers the backup remote on the + *server*, where it is global to every client connected to it, so one + workspace's backup decision would silently become everyone's — the shape + `bd backup`'s own help text already warns about for auto-backup. A `file://` + destination has a second problem on top, since the server resolves that path + on its own filesystem. The refusal keeps the `proxy.backup.unsupported` code + and now carries `"reason": "design"`; a backup story for those deployments + belongs to whoever runs the server. Whether a *remote-scheme* destination + (DoltHub, `aws://`, `gs://`) should eventually be allowed there is recorded as + an open question rather than settled — the server pushes those over the + network from its own environment, which is the same credential question the + next slice has to answer for `dolt push`. + ### Changed - **Proxied-server refusals now say *why* they refuse.** The JSON a refused @@ -22,6 +47,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 (`proxy.store.unrouted`) instead of the bare string `proxy server store should be uow provider`. +- **`bd backup restore --json` prints a result object on every topology** + (`{"restored": true, "source": ""}`). It previously printed nothing at + all, which left a caller unable to tell a completed restore from a silently + skipped one. This applies to direct/embedded workspaces too, not only proxied + ones. +- **`bd backup init`, `sync`, `remove` and `restore` no longer print a usage + block after a failure.** They now silence cobra's own error and usage + rendering like the rest of the CLI, so a failure is one error line on stderr + instead of the message followed by `Error: exit code 1` and the full usage + text. Exit statuses are unchanged; this applies on every topology. + ## [1.3.1] - 2026-09-16 ### Changed diff --git a/cmd/bd/backup.go b/cmd/bd/backup.go index 4f5249ef1..6e2e40325 100644 --- a/cmd/bd/backup.go +++ b/cmd/bd/backup.go @@ -56,7 +56,9 @@ func newBackupStatusCommand(sizeDatabase backupSizeFunc) *cobra.Command { SilenceErrors: true, RunE: func(cmd *cobra.Command, args []string) error { if usesProxiedServer() { - return HandleErrorRespectJSON("backup status is not supported in proxied-server mode") + if err := requireLocalProxiedBackup("backup status"); err != nil { + return err + } } evt := metrics.NewCommandEvent("backup-status") defer func() { diff --git a/cmd/bd/backup_dolt.go b/cmd/bd/backup_dolt.go index 2e42ad2c0..5ee084260 100644 --- a/cmd/bd/backup_dolt.go +++ b/cmd/bd/backup_dolt.go @@ -46,10 +46,13 @@ DoltHub (recommended for cloud backup): After adding, run 'bd backup sync' to push your data.`, Args: cobra.ExactArgs(1), + // As on `backup status`: the failures here are configuration and I/O + // errors, not usage errors, and cobra's default prints the message, then + // "Error: exit code 1", then the whole usage block — three things where one + // was wanted, and on --json it lands after the JSON object. + SilenceUsage: true, + SilenceErrors: true, RunE: func(cmd *cobra.Command, args []string) error { - if usesProxiedServer() { - return HandleErrorRespectJSON("backup init is not supported in proxied-server mode") - } evt := metrics.NewCommandEvent("backup-init") defer func() { if c := metrics.Global(); c != nil { @@ -60,6 +63,10 @@ After adding, run 'bd backup sync' to push your data.`, ctx := rootCtx rawPath := args[0] + if usesProxiedServer() { + return runBackupInitProxied(ctx, rawPath) + } + if store == nil { return fmt.Errorf("no store available") } @@ -124,10 +131,9 @@ backup location configured with 'bd backup init'. The backup is atomic — if the sync fails, the previous backup state is preserved. Run 'bd backup init ' first to configure a destination.`, + SilenceUsage: true, + SilenceErrors: true, RunE: func(cmd *cobra.Command, args []string) error { - if usesProxiedServer() { - return HandleErrorRespectJSON("backup sync is not supported in proxied-server mode") - } evt := metrics.NewCommandEvent("backup-sync") defer func() { if c := metrics.Global(); c != nil { @@ -136,6 +142,9 @@ Run 'bd backup init ' first to configure a destination.`, }() ctx := rootCtx + if usesProxiedServer() { + return runBackupSyncProxied(ctx) + } if store == nil { return fmt.Errorf("no store available") } @@ -358,7 +367,14 @@ func showDoltBackupStatusJSON() map[string]interface{} { // doltBackupSize returns the active database size when the current store // instance can measure its storage locally. Unsupported backends preserve the // optional status-field contract instead of failing the command. +// +// Proxied mode has no store to ask — the provider owns the connection — so the +// fork lives here, where "which backend measures this database" is already the +// question being answered. func doltBackupSize(ctx context.Context) (int64, bool, error) { + if usesProxiedServer() { + return doltBackupSizeProxied(ctx) + } if store == nil { return 0, false, fmt.Errorf("no storage backend is open") } @@ -405,11 +421,10 @@ var backupRemoveCmd = &cobra.Command{ This unregisters the backup remote from Dolt and removes the local backup configuration. The backup data at the destination is not deleted.`, - Aliases: []string{"rm"}, + Aliases: []string{"rm"}, + SilenceUsage: true, + SilenceErrors: true, RunE: func(cmd *cobra.Command, args []string) error { - if usesProxiedServer() { - return HandleErrorRespectJSON("backup remove is not supported in proxied-server mode") - } evt := metrics.NewCommandEvent("backup-remove") defer func() { if c := metrics.Global(); c != nil { @@ -418,6 +433,9 @@ backup configuration. The backup data at the destination is not deleted.`, }() ctx := rootCtx + if usesProxiedServer() { + return runBackupRemoveProxied(ctx) + } if store == nil { return fmt.Errorf("no store available") } diff --git a/cmd/bd/backup_proxied_external_integration_test.go b/cmd/bd/backup_proxied_external_integration_test.go new file mode 100644 index 000000000..5f3ce3f56 --- /dev/null +++ b/cmd/bd/backup_proxied_external_integration_test.go @@ -0,0 +1,107 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" + "github.com/steveyegge/beads/internal/storage/dbproxy/server" +) + +// TestProxiedServerBackupRefusedOnExternalTopology is the other half of slice +// S3: the backup family is honored on managed-local and refused BY DESIGN +// against a Dolt server bd does not own. +// +// The refusal matters more than the capability. `CALL DOLT_BACKUP('add', …)` +// registers the remote on the server, where it is global to every client of +// that server, so one workspace's backup decision becomes everyone's. With a +// file:/// destination there is a second problem on top: the server resolves +// that path on its own filesystem, so honoring it here would either fail at the +// point of use or write the backup to a directory on the server's host that the +// operator asking for it will never look in. The second outcome is worse than +// any refusal. +// +// Named TestProxiedServer* so .github/scripts/proxied-test-shard.sh runs it in +// the external lane, which is the only one with an external Dolt server. +func TestProxiedServerBackupRefusedOnExternalTopology(t *testing.T) { + requireProxiedServerEnv(t) + + bd := buildEmbeddedBD(t) + p := newSharedProxiedProject(t, bd, "bkext") + dest := filepath.Join(t.TempDir(), "dolt-backup") + + // Refusals are decided before the provider is constructed, so take the + // topology down first: a pidfile appearing during one of these commands + // would mean the refusal came too late to have cost nothing. + if err := proxy.Shutdown(p.proxyRoot); err != nil { + t.Logf("proxy.Shutdown(%s) before the refusal probes: %v", p.proxyRoot, err) + } + + for _, tc := range []struct { + name string + args []string + }{ + {"status", []string{"backup", "status"}}, + {"init", []string{"backup", "init", dest}}, + {"sync", []string{"backup", "sync"}}, + {"remove", []string{"backup", "remove"}}, + {"restore", []string{"backup", "restore", dest}}, + } { + t.Run(tc.name, func(t *testing.T) { + args := append([]string{"--json"}, tc.args...) + out, err := bdProxiedRun(t, bd, p.dir, args...) + if err == nil { + t.Fatalf("bd %s succeeded on an external proxied topology:\n%s", + strings.Join(tc.args, " "), out) + } + + var refusal struct { + Code string `json:"code"` + Error string `json:"error"` + Mutates bool `json:"mutates"` + Reason string `json:"reason"` + } + start := strings.Index(string(out), "{") + if start < 0 { + t.Fatalf("no typed refusal JSON:\n%s", out) + } + if err := json.Unmarshal(out[start:], &refusal); err != nil { + t.Fatalf("parse refusal JSON: %v\n%s", err, out) + } + if refusal.Code != "proxy.backup.unsupported" { + t.Errorf("code = %q, want proxy.backup.unsupported", refusal.Code) + } + if refusal.Mutates { + t.Errorf("refusal reported mutates=true") + } + // design, not unimplemented: no later slice is going to lift this. + if refusal.Reason != string(ProxyReasonDesign) { + t.Errorf("reason = %q, want design", refusal.Reason) + } + // The reason has to be the one that is true of every destination + // scheme. "resolved on the server's filesystem" is a file:// + // property, and `bd backup init` also takes https/aws/gs. + if !strings.Contains(refusal.Error, "the backup remote is registered on the server, where it is global to every client") { + t.Errorf("refusal message does not say why: %q", refusal.Error) + } + + // Nothing was started and nothing was written: the refusal is free. + for _, name := range []string{proxy.PIDFileName, server.PIDFileName} { + if _, err := os.Stat(filepath.Join(p.proxyRoot, name)); err == nil { + t.Errorf("a refused backup command started the topology (%s)", name) + } + } + if _, err := os.Stat(filepath.Join(p.beadsDir, "dolt-backup.json")); err == nil { + t.Errorf("a refused backup command wrote the local backup config") + } + if _, err := os.Stat(dest); err == nil { + t.Errorf("a refused backup command created %s", dest) + } + }) + } +} diff --git a/cmd/bd/backup_proxied_local_integration_test.go b/cmd/bd/backup_proxied_local_integration_test.go new file mode 100644 index 000000000..166d9ab94 --- /dev/null +++ b/cmd/bd/backup_proxied_local_integration_test.go @@ -0,0 +1,208 @@ +//go:build cgo && unix + +package main + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "slices" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" + "github.com/steveyegge/beads/internal/storage/dbproxy/server" + "github.com/steveyegge/beads/internal/storage/versioncontrolops" +) + +// TestManagedLocalProxiedBackupRoundTrip is the claim slice S3 makes: a +// managed-local proxied workspace can take a Dolt-native backup and get its +// data back from it. +// +// It is deliberately a ROUND TRIP rather than a pair of exit-code assertions. +// A backup command that reports success without producing a restorable +// artifact is the failure that matters here, and the only way to rule it out is +// to restore the artifact and look at the issues that come back: the issue +// created BEFORE the backup must survive, and the one created AFTER it must be +// gone. Both directions are load-bearing — a restore that changed nothing would +// pass the first check alone. +// +// Named TestManagedLocalProxied* so the proxied-local smoke lane +// (.github/workflows/proxied-local-smoke.yml) runs it; that lane is the only +// one that sets BEADS_TEST_PROXIED_LOCAL. +func TestManagedLocalProxiedBackupRoundTrip(t *testing.T) { + requireManagedLocalProxiedEnv(t) + + bd := buildEmbeddedBD(t) + p := bdManagedLocalInit(t, bd, "bkrt", 5*time.Minute) + + before := bdProxiedCreate(t, bd, p.dir, "survives the restore", "-p", "1") + + // Status first: it is the read-only probe that proves the route exists at + // all, and it must answer with a real measured size rather than the + // "optional field omitted" the external topologies produce. + status := readProxiedBackupStatus(t, bd, p) + if status.Dolt.Configured { + t.Fatalf("fresh workspace reports a configured backup: %+v", status.Dolt) + } + if status.DatabaseSize.Bytes <= 0 { + t.Fatalf("backup status measured no database: %+v", status.DatabaseSize) + } + + dest := filepath.Join(t.TempDir(), "dolt-backup") + if out, err := bdProxiedRun(t, bd, p.dir, "backup", "init", dest); err != nil { + t.Fatalf("bd backup init %s: %v\n%s", dest, err, out) + } + if out, err := bdProxiedRun(t, bd, p.dir, "backup", "sync"); err != nil { + t.Fatalf("bd backup sync: %v\n%s", err, out) + } + // A Dolt backup is a chunk store with a manifest; an empty directory here + // would mean the command reported a success it did not perform. + if _, err := os.Stat(filepath.Join(dest, "manifest")); err != nil { + t.Fatalf("backup sync produced no manifest at %s: %v", dest, err) + } + + after := bdProxiedCreate(t, bd, p.dir, "must not survive the restore", "-p", "1") + + out, err := bdProxiedRun(t, bd, p.dir, "backup", "restore", dest, "--force", "--json") + if err != nil { + t.Fatalf("bd backup restore %s --force: %v\n%s", dest, err, out) + } + var result struct { + Restored bool `json:"restored"` + Source string `json:"source"` + } + if err := json.Unmarshal(out, &result); err != nil { + t.Fatalf("parse restore JSON: %v\n%s", err, out) + } + if !result.Restored || result.Source != dest { + t.Fatalf("unexpected restore result: %+v; want restored=true, source=%s", result, dest) + } + + // The restore quiesces the topology and leaves it down, so the next command + // relaunches against the restored data. Assert that rather than assume it: + // a restore that ran with the old server still up would be the undefined + // case this design exists to avoid. + for _, name := range []string{proxy.PIDFileName, server.PIDFileName} { + if _, err := os.Stat(filepath.Join(p.proxyRoot, name)); err == nil { + t.Fatalf("restore left %s in place; the topology should be quiescent afterwards", name) + } + } + + issues := bdProxiedListJSON(t, bd, p) + var ids []string + for _, issue := range issues { + ids = append(ids, issue.ID) + } + if !slices.Contains(ids, before.ID) { + t.Fatalf("restore lost the issue the backup contained (%s); got %v", before.ID, ids) + } + if slices.Contains(ids, after.ID) { + t.Fatalf("restore did not replace the database: %s was created after the backup and is still here; got %v", + after.ID, ids) + } + + // The restore registers its source as the backup destination, so a plain + // `bd backup sync` works straight afterwards with no second init. + restored := readProxiedBackupStatus(t, bd, p) + if !restored.Dolt.Configured || !strings.HasSuffix(restored.Dolt.BackupURL, dest) { + t.Fatalf("restore did not register %s as the backup destination: %+v", dest, restored.Dolt) + } + if out, err := bdProxiedRun(t, bd, p.dir, "backup", "sync"); err != nil { + t.Fatalf("bd backup sync after restore: %v\n%s", err, out) + } + + if out, err := bdProxiedRun(t, bd, p.dir, "backup", "remove"); err != nil { + t.Fatalf("bd backup remove: %v\n%s", err, out) + } + removed := readProxiedBackupStatus(t, bd, p) + if removed.Dolt.Configured { + t.Fatalf("backup remove left the destination configured: %+v", removed.Dolt) + } + // Removing the destination unregisters the remote; it does not delete the + // backup, which is what makes `bd backup remove` safe to run. + if _, err := os.Stat(filepath.Join(dest, "manifest")); err != nil { + t.Fatalf("backup remove deleted the backup data at %s: %v", dest, err) + } + if out, err := bdProxiedRun(t, bd, p.dir, "backup", "sync"); err == nil { + t.Fatalf("bd backup sync succeeded with no destination configured:\n%s", out) + } +} + +// A backup with a malformed migration table can replace the database but fail +// the following provider construction at schema initialization. That failure must +// still leave both processes down, even though no provider was returned. +func TestManagedLocalProxiedBackupRestoreReopenFailure(t *testing.T) { + requireManagedLocalProxiedEnv(t) + bd := buildEmbeddedBD(t) + p := bdManagedLocalInit(t, bd, "bkrf", 5*time.Minute) + dest := filepath.Join(t.TempDir(), "malformed-backup") + if out, err := bdProxiedRun(t, bd, p.dir, "backup", "init", dest); err != nil { + t.Fatalf("backup init: %v\n%s", err, out) + } + + // Damage the migration table in the backup, then repair the live table so + // the command's initial provider can open normally. Only its post-restore + // constructor will see the malformed table and fail after starting Dolt. + held := openHeldManagedProxiedConn(t, bd, p) + ctx := context.Background() + if _, err := held.Conn.ExecContext(ctx, "ALTER TABLE schema_migrations RENAME COLUMN version TO broken_version"); err != nil { + t.Fatalf("damage migration table: %v", err) + } + if _, err := held.Conn.ExecContext(ctx, "CALL DOLT_COMMIT('-Am', 'Simulate a malformed migration table in the backup')"); err != nil { + t.Fatalf("commit malformed migration table: %v", err) + } + if err := versioncontrolops.BackupSync(ctx, held.Conn, proxiedBackupTargetName); err != nil { + t.Fatalf("back up malformed schema: %v", err) + } + if _, err := held.Conn.ExecContext(ctx, "ALTER TABLE schema_migrations RENAME COLUMN broken_version TO version"); err != nil { + t.Fatalf("repair live migration table: %v", err) + } + held.Release() + + out, err := bdProxiedRun(t, bd, p.dir, "backup", "restore", dest, "--force") + if err == nil { + t.Fatalf("restore unexpectedly reopened a malformed schema:\n%s", out) + } + if !strings.Contains(string(out), "the restored database did not reopen") || !strings.Contains(string(out), "schema") { + t.Fatalf("expected post-restore schema initialization failure, got:\n%s", out) + } + if strings.Contains(string(out), "restored and reconciled") { + t.Fatalf("failed reopen reported successful reconciliation:\n%s", out) + } + for _, name := range []string{proxy.PIDFileName, server.PIDFileName} { + if _, err := os.Stat(filepath.Join(p.proxyRoot, name)); !os.IsNotExist(err) { + t.Errorf("failed reopen should remove %s; stat error: %v", name, err) + } + } +} + +type proxiedBackupStatus struct { + Dolt struct { + Configured bool `json:"configured"` + BackupURL string `json:"backup_url"` + BackupName string `json:"backup_name"` + } `json:"dolt"` + DatabaseSize struct { + Bytes int64 `json:"bytes"` + } `json:"database_size"` +} + +func readProxiedBackupStatus(t *testing.T, bd string, p proxiedProject) proxiedBackupStatus { + t.Helper() + out, err := bdProxiedRun(t, bd, p.dir, "backup", "status", "--json") + if err != nil { + t.Fatalf("bd backup status --json: %v\n%s", err, out) + } + start := strings.Index(string(out), "{") + if start < 0 { + t.Fatalf("bd backup status --json produced no JSON:\n%s", out) + } + var status proxiedBackupStatus + if err := json.Unmarshal(out[start:], &status); err != nil { + t.Fatalf("parse backup status JSON: %v\n%s", err, out) + } + return status +} diff --git a/cmd/bd/backup_proxied_policy_test.go b/cmd/bd/backup_proxied_policy_test.go new file mode 100644 index 000000000..db2db3a70 --- /dev/null +++ b/cmd/bd/backup_proxied_policy_test.go @@ -0,0 +1,285 @@ +package main + +import ( + "encoding/json" + "errors" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/spf13/cobra" + "github.com/steveyegge/beads/internal/configfile" +) + +// TestProxiedRestoreFailureMessage pins the one distinction the operator acts +// on: whether the data came back. A teardown failure happens after the step +// succeeded, so reporting it as "restore failed" is not a wording nit — it +// tells someone whose database IS restored to restore it again. +func TestProxiedRestoreFailureMessage(t *testing.T) { + boom := errors.New("proxy shutdown: connection refused") + + for _, tc := range []struct { + name string + afterReconcile bool + err error + wantContains []string + wantMissing []string + }{ + { + name: "restore itself failed", + err: errors.New("DOLT_BACKUP: no such backup"), + // No claim that anything was restored. + wantContains: []string{"restore failed", "no such backup"}, + wantMissing: []string{"the data is restored"}, + }, + { + name: "restore succeeded, teardown failed", + err: &proxiedTeardownError{err: boom}, + wantContains: []string{"the data is restored", "bd dolt stop --force", "bd backup init"}, + wantMissing: []string{"restore failed"}, + }, + { + name: "reconcile failed for real", + afterReconcile: true, + err: errors.New("open provider: schema is behind"), + wantContains: []string{"the restored database did not reopen", "schema is behind"}, + wantMissing: []string{"restore failed"}, + }, + { + name: "reconcile succeeded, teardown failed", + afterReconcile: true, + err: &proxiedTeardownError{err: boom}, + wantContains: []string{"the data is restored and reconciled", "bd dolt stop --force"}, + // Nothing left to re-register: the reconcile already ran. + wantMissing: []string{"restore failed", "bd backup init"}, + }, + } { + t.Run(tc.name, func(t *testing.T) { + got := proxiedRestoreFailureMessage("/tmp/backup", tc.afterReconcile, tc.err) + for _, want := range tc.wantContains { + if !strings.Contains(got, want) { + t.Errorf("message %q does not contain %q", got, want) + } + } + for _, unwanted := range tc.wantMissing { + if strings.Contains(got, unwanted) { + t.Errorf("message %q must not contain %q", got, unwanted) + } + } + }) + } +} + +// backupFamilyPaths is every registry path the backup family owns. Keeping the +// list in one place is what makes "the whole family moves together" an +// assertion rather than a convention. +var backupFamilyPaths = []string{ + "backup", "backup init", "backup sync", "backup remove", "backup status", "backup restore", +} + +// TestProxiedBackupHonoredOnlyOnManagedLocal is the locality policy. +// +// A Dolt backup destination is resolved by the SERVER: CALL DOLT_BACKUP('add', +// name, 'file:///…') names a path on the machine running dolt, not on the +// machine running bd. On managed-local that is the same machine and the same +// user, because bd spawned the child itself. On every other proxied shape it is +// somebody else's filesystem, so honoring the command would either fail at the +// point of use or — much worse — succeed and write the backup somewhere the +// operator cannot see. That is why these stay refused with Reason=design. +func TestProxiedBackupHonoredOnlyOnManagedLocal(t *testing.T) { + for _, path := range backupFamilyPaths { + row, ok := LookupCapabilityRow(path, "") + if !ok { + t.Fatalf("%q has no registry row", path) + } + + if got := row.ruleFor(ProxyTopologyManagedLocal); got.Outcome != ProxyOutcomeHonored { + t.Errorf("%q on managed-local: outcome %q, want honored", path, got.Outcome) + } + + for _, topology := range []ProxyTopology{ + ProxyTopologyExternalTCP, ProxyTopologyExternalUnix, + ProxyTopologyTeamServer, ProxyTopologyUnknown, + } { + got := row.ruleFor(topology) + if got.Outcome != ProxyOutcomeRefused { + t.Errorf("%q on %s: outcome %q, want refused", path, topology, got.Outcome) + } + if got.Reason != ProxyReasonDesign { + t.Errorf("%q on %s: reason %q, want design", path, topology, got.Reason) + } + if got.Code != "proxy.backup.unsupported" { + t.Errorf("%q on %s: code %q, want the frozen proxy.backup.unsupported", path, topology, got.Code) + } + if !strings.Contains(got.Message, "is not supported in proxied-server mode") { + t.Errorf("%q on %s: message %q dropped the frozen prefix", path, topology, got.Message) + } + } + } +} + +// TestProxiedBackupGateRefusesByTopology exercises the gate itself rather than +// the table, so a row that is right while the validator ignores the topology +// still fails. +func TestProxiedBackupGateRefusesByTopology(t *testing.T) { + newBackupCmd := func(sub string) *cobra.Command { + root := &cobra.Command{Use: "bd"} + parent := &cobra.Command{Use: "backup"} + child := &cobra.Command{Use: sub} + parent.AddCommand(child) + root.AddCommand(parent) + return child + } + + for _, sub := range []string{"init", "sync", "remove", "status", "restore"} { + t.Run(sub, func(t *testing.T) { + if err := validateProxyRegistryBeforeProvider(newBackupCmd(sub), ProxyTopologyManagedLocal); err != nil { + t.Fatalf("managed-local: gate refused %q: %v", sub, err) + } + err := validateProxyRegistryBeforeProvider(newBackupCmd(sub), ProxyTopologyExternalTCP) + if err == nil { + t.Fatalf("external-tcp: gate permitted %q", sub) + } + }) + } +} + +// TestProxiedBackupRefusalJSONCarriesDesignReason pins what a consumer sees. +// The three frozen fields are unchanged and reason now says "design", which is +// the difference that matters: this refusal is not waiting on a later slice. +func TestProxiedBackupRefusalJSONCarriesDesignReason(t *testing.T) { + t.Setenv("BD_JSON_ENVELOPE", "") + oldJSON := jsonOutput + jsonOutput = true + t.Cleanup(func() { jsonOutput = oldJSON }) + + root := &cobra.Command{Use: "bd"} + parent := &cobra.Command{Use: "backup"} + child := &cobra.Command{Use: "sync"} + parent.AddCommand(child) + root.AddCommand(parent) + + out := captureStdout(t, func() error { + _ = validateProxyRegistryBeforeProvider(child, ProxyTopologyExternalTCP) + return nil + }) + for _, want := range []string{ + `"code": "proxy.backup.unsupported"`, + `"mutates": false`, + `"reason": "design"`, + } { + if !strings.Contains(out, want) { + t.Fatalf("refusal JSON missing %s:\n%s", want, out) + } + } +} + +// TestResolveProxiedTopology covers the resolution the gate depends on. The +// team-server case is the one worth reading twice: ownership beats transport, +// because a bts-managed database is somebody else's store whether bd reaches it +// over a socket, a port, or a child it spawned itself. +func TestResolveProxiedTopology(t *testing.T) { + for _, tc := range []struct { + name string + cfg configfile.Config + sidecar *configfile.ProxiedServerClientInfo + rawCfg string // when set, written verbatim instead of cfg + rawCar string // when set, written verbatim instead of sidecar + want ProxyTopology + }{ + { + name: "managed local has no external block", + cfg: configfile.Config{DoltMode: configfile.DoltModeProxiedServer}, + sidecar: &configfile.ProxiedServerClientInfo{}, + want: ProxyTopologyManagedLocal, + }, + { + name: "no sidecar at all is still managed local", + cfg: configfile.Config{DoltMode: configfile.DoltModeProxiedServer}, + sidecar: nil, + want: ProxyTopologyManagedLocal, + }, + { + name: "external host and port", + cfg: configfile.Config{DoltMode: configfile.DoltModeProxiedServer}, + sidecar: &configfile.ProxiedServerClientInfo{ + External: &configfile.ExternalDoltConfig{Host: "db.example.com", Port: 3306}, + }, + want: ProxyTopologyExternalTCP, + }, + { + name: "external unix socket", + cfg: configfile.Config{DoltMode: configfile.DoltModeProxiedServer}, + sidecar: &configfile.ProxiedServerClientInfo{ + External: &configfile.ExternalDoltConfig{Socket: "/var/run/dolt.sock"}, + }, + want: ProxyTopologyExternalUnix, + }, + { + name: "team server beats the transport", + cfg: configfile.Config{DoltMode: configfile.DoltModeProxiedServer, DoltTeamServer: true}, + sidecar: &configfile.ProxiedServerClientInfo{ + External: &configfile.ExternalDoltConfig{Host: "bts.example.com", Port: 3306}, + }, + want: ProxyTopologyTeamServer, + }, + { + name: "team server on a local child is still team server", + cfg: configfile.Config{DoltMode: configfile.DoltModeProxiedServer, DoltTeamServer: true}, + sidecar: &configfile.ProxiedServerClientInfo{}, + want: ProxyTopologyTeamServer, + }, + { + name: "unreadable sidecar fails closed", + cfg: configfile.Config{DoltMode: configfile.DoltModeProxiedServer}, + rawCar: "{ not json", + want: ProxyTopologyUnknown, + }, + { + name: "unreadable workspace config fails closed", + rawCfg: "{ not json", + want: ProxyTopologyUnknown, + }, + } { + t.Run(tc.name, func(t *testing.T) { + beadsDir := filepath.Join(t.TempDir(), ".beads") + if err := os.MkdirAll(beadsDir, 0o750); err != nil { + t.Fatal(err) + } + switch { + case tc.rawCfg != "": + writeTestFile(t, configfile.ConfigPath(beadsDir), tc.rawCfg) + default: + cfg := tc.cfg + cfg.Backend = configfile.BackendDolt + data, err := json.Marshal(cfg) + if err != nil { + t.Fatal(err) + } + writeTestFile(t, configfile.ConfigPath(beadsDir), string(data)) + } + switch { + case tc.rawCar != "": + writeTestFile(t, configfile.ProxiedServerClientInfoPath(beadsDir), tc.rawCar) + case tc.sidecar != nil: + data, err := json.Marshal(tc.sidecar) + if err != nil { + t.Fatal(err) + } + writeTestFile(t, configfile.ProxiedServerClientInfoPath(beadsDir), string(data)) + } + + if got := resolveProxiedTopology(beadsDir); got != tc.want { + t.Fatalf("resolveProxiedTopology = %q, want %q", got, tc.want) + } + }) + } +} + +func writeTestFile(t *testing.T, path, body string) { + t.Helper() + if err := os.WriteFile(path, []byte(body), 0o600); err != nil { + t.Fatal(err) + } +} diff --git a/cmd/bd/backup_proxied_server.go b/cmd/bd/backup_proxied_server.go new file mode 100644 index 000000000..13bfaacac --- /dev/null +++ b/cmd/bd/backup_proxied_server.go @@ -0,0 +1,477 @@ +package main + +import ( + "context" + "database/sql" + "errors" + "fmt" + "os" + "path/filepath" + "strings" + "time" + + "github.com/steveyegge/beads/internal/beads" + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/storage/uow" + "github.com/steveyegge/beads/internal/storage/versioncontrolops" +) + +// Dolt-native backup over the proxied-server provider. +// +// The mechanism needed no proxy work at all: CALL DOLT_BACKUP(...) is ordinary +// SQL, RunNonTx already hands out a raw pinned connection (the seam `compact +// --dolt` and `gc` run on), the UOW DSN sets a connect timeout but no read +// deadline, and an active connection keeps the idle reaper from arming — so a +// backup that takes minutes is safe from both the driver and the 30s reaper. +// +// What the family DID need is the locality policy, which lives in the registry +// (see capability_registry.go's backup rows) because it is what the front door +// enforces. The guard below repeats it at the route because these functions are +// reachable from RunE, and a route that trusts the gate is one refactor away +// from writing a backup onto a host it does not own. +// +// None of these routes touch commandDidWrite or commandDidExplicitDoltCommit. +// Those two drive the post-run Dolt auto-commit, which is an embedded/direct +// concern — PersistentPostRunE's proxied branch never reads them — and the +// other proxied routes leave them alone for the same reason. + +// proxiedBackupTargetName is the Dolt backup remote bd registers. Same name the +// direct path uses, so a workspace that changes topology still finds its +// destination. +const proxiedBackupTargetName = defaultDoltBackupName + +// requireLocalProxiedBackup re-checks the locality policy at the route. +// Defense in depth, not a second policy: the rule and its wording come from the +// same registry row the pre-provider gate consults, so the two cannot drift. +func requireLocalProxiedBackup(path string) error { + row, ok := LookupCapabilityRow(path, "") + if !ok { + // Unreachable: TestProxyCapabilityRegistryCoversCommandTree fails the + // build for a backup path with no row. Refuse rather than guess. + return HandleErrorRespectJSON("%s has no proxied-server capability row", path) + } + rule := row.ruleFor(resolveProxiedTopology(beads.FindBeadsDir())) + if rule.Outcome == ProxyOutcomeHonored { + return nil + } + return HandleProxyCapabilityError(proxyCapabilityErrorFor(rule)) +} + +// proxiedActiveDatabase asks the server which database the provider opened, +// rather than re-deriving it from the workspace config. The provider already +// resolved that question (metadata.json, then --database/--db), and a second +// ladder here would be a second answer waiting to disagree with it. +func proxiedActiveDatabase(ctx context.Context) (string, error) { + var database string + err := runProxiedNonTx(ctx, func(ctx context.Context, conn *sql.Conn) error { + var name sql.NullString + if err := conn.QueryRowContext(ctx, "SELECT DATABASE()").Scan(&name); err != nil { + return fmt.Errorf("resolve active database: %w", err) + } + if !name.Valid || name.String == "" { + return fmt.Errorf("the proxied connection has no database selected") + } + database = name.String + return nil + }) + return database, err +} + +// runBackupInitProxied registers the backup destination, mirroring the direct +// path's conflict handling: Dolt refuses a second remote at an address another +// one already holds, and a re-init to a new path has to replace the old entry +// rather than fail. +func runBackupInitProxied(ctx context.Context, rawPath string) error { + if err := requireLocalProxiedBackup("backup init"); err != nil { + return err + } + backupURL := resolveDoltBackupURL(rawPath) + + err := runProxiedNonTx(ctx, func(ctx context.Context, conn *sql.Conn) error { + addErr := versioncontrolops.BackupAdd(ctx, conn, proxiedBackupTargetName, backupURL) + if addErr == nil { + return nil + } + switch { + case strings.Contains(addErr.Error(), "already exists"): + // Same name, different URL: replace it. + _ = versioncontrolops.BackupRemove(ctx, conn, proxiedBackupTargetName) + case versioncontrolops.ExtractAddressConflictName(addErr) != "": + // A different name (auto-export registers "backup_export") already + // points at this URL; take the address over. + _ = versioncontrolops.BackupRemove(ctx, conn, versioncontrolops.ExtractAddressConflictName(addErr)) + default: + return addErr + } + return versioncontrolops.BackupAdd(ctx, conn, proxiedBackupTargetName, backupURL) + }) + if err != nil { + return HandleErrorRespectJSON("failed to add backup destination: %v", err) + } + + if err := saveDoltBackupConfig(backupURL); err != nil { + fmt.Fprintf(os.Stderr, "Warning: backup registered but failed to save config: %v\n", err) + } + + if jsonOutput { + return outputJSON(map[string]interface{}{ + "backup_url": backupURL, + "backup_name": proxiedBackupTargetName, + "initialized": true, + }) + } + fmt.Printf("Backup destination configured: %s\n", backupURL) + fmt.Println("Run 'bd backup sync' to push your data.") + return nil +} + +// runBackupSyncProxied pushes the database to the configured destination. +func runBackupSyncProxied(ctx context.Context) error { + if err := requireLocalProxiedBackup("backup sync"); err != nil { + return err + } + + start := time.Now() + err := runProxiedNonTx(ctx, func(ctx context.Context, conn *sql.Conn) error { + if err := commitWorkingSetBeforeBackup(ctx, conn); err != nil { + // Same posture as the direct path: a failed pre-backup commit + // means the backup may miss the newest writes, which is worth + // saying out loud and is not worth refusing to back up over. + fmt.Fprintf(os.Stderr, "Warning: failed to commit pending changes: %v\n", err) + } + return versioncontrolops.BackupSync(ctx, conn, proxiedBackupTargetName) + }) + if err != nil { + if strings.Contains(err.Error(), "no backup") || strings.Contains(err.Error(), "not found") { + return HandleErrorRespectJSON("no backup destination configured. Run 'bd backup init ' first") + } + return HandleErrorRespectJSON("backup sync failed: %v", err) + } + + elapsed := time.Since(start) + if err := updateDoltBackupState(elapsed); err != nil { + fmt.Fprintf(os.Stderr, "Warning: backup synced but failed to update state: %v\n", err) + } + + if jsonOutput { + return outputJSON(map[string]interface{}{ + "synced": true, + "duration": elapsed.String(), + }) + } + fmt.Printf("Backup synced in %s\n", elapsed.Round(time.Millisecond)) + return nil +} + +// commitWorkingSetBeforeBackup is the proxied twin of the direct path's +// "bd: pre-backup commit". DOLT_BACKUP syncs COMMITTED history, so anything +// still in the working set would silently be missing from the backup. +// +// It is normally a no-op: every proxied write transaction ends in +// DOLT_COMMIT('-Am') (uow doltServerTx.Commit), so the working set is already +// clean. The pending check is what keeps this from minting an empty commit on +// every sync — Dolt rejects those server-side and logs each one. +func commitWorkingSetBeforeBackup(ctx context.Context, conn *sql.Conn) error { + pending, err := issueops.HasPendingChanges(ctx, conn) + if err != nil { + return fmt.Errorf("check pending changes: %w", err) + } + if !pending { + return nil + } + if _, err := conn.ExecContext(ctx, "CALL DOLT_COMMIT('-Am', ?)", "bd: pre-backup commit"); err != nil { + if issueops.IsNothingToCommitError(err) { + return nil + } + return err + } + return nil +} + +// runBackupRemoveProxied unregisters the destination and drops the local +// sidecars. The backup data itself is left alone, as in direct mode. +func runBackupRemoveProxied(ctx context.Context) error { + if err := requireLocalProxiedBackup("backup remove"); err != nil { + return err + } + + err := runProxiedNonTx(ctx, func(ctx context.Context, conn *sql.Conn) error { + if err := versioncontrolops.BackupRemove(ctx, conn, proxiedBackupTargetName); err != nil { + return err + } + // auto-export may have registered a second remote at the same URL. + _ = versioncontrolops.BackupRemove(ctx, conn, "backup_export") + return nil + }) + if err != nil { + if strings.Contains(err.Error(), "not found") || strings.Contains(err.Error(), "no backup") { + return HandleErrorRespectJSON("no backup destination configured") + } + return HandleErrorRespectJSON("failed to remove backup: %v", err) + } + + if path, err := doltBackupConfigPath(); err == nil { + _ = os.Remove(path) + } + if path, err := doltBackupStatePath(); err == nil { + _ = os.Remove(path) + } + + if jsonOutput { + return outputJSON(map[string]interface{}{"removed": true}) + } + fmt.Println("Backup destination removed.") + return nil +} + +// doltBackupSizeProxied measures the managed-local database on disk, which is +// what `bd backup status` reports on every other topology that can answer. +// +// It reads the directory rather than asking Dolt because that is what the +// direct stores do (DoltStore.ActiveDatabaseSize measures +// localActiveDatabaseDir), so the number means the same thing in both modes. +// The database NAME still comes from the server, so the directory measured is +// the one this connection is actually using. +func doltBackupSizeProxied(ctx context.Context) (int64, bool, error) { + beadsDir := beads.FindBeadsDir() + if beadsDir == "" { + return 0, false, nil + } + root, err := resolveProxiedServerRootPath(beadsDir) + if err != nil { + return 0, false, nil + } + + database, err := proxiedActiveDatabase(ctx) + if err != nil { + return 0, false, err + } + + dir := filepath.Join(root, database) + if _, err := os.Stat(dir); err != nil { + // No local directory means nothing bd can measure; the status contract + // treats size as optional rather than failing the command. + return 0, false, nil + } + size, err := storage.MeasureDirectorySize(ctx, dir) + if err != nil { + return 0, false, fmt.Errorf("measure active database directory %q: %w", dir, err) + } + return size, true, nil +} + +// runBackupRestoreProxied restores the database from a Dolt backup, quiescing +// the topology first. +// +// DOLT_BACKUP('restore') REPLACES the database under the server. Doing that +// while clients hold connections to it is undefined, so the restore runs with +// bd as the only client and nothing else pointed at the store: +// +// 1. `bd backup restore` already holds the workspace and physical-root gates +// EXCLUSIVELY (commandNeedsExclusiveGate), so no other gated bd command — +// including `bd serve` — is using this workspace. +// 2. This function closes the provider the root pre-run opened and shuts the +// proxy and its dolt child down, so no connection survives into the +// replace. +// 3. It reopens a provider of its own, which relaunches the topology with +// exactly one client, and runs the restore on that. +// 4. It shuts the topology down again and reopens once more to finish: the +// reopen proves the restored database is serviceable, and it is where the +// project identity and the backup remote are reconciled. A connection +// pinned across the replace would be reading a database that no longer +// exists underneath it. +// +// The final shutdown leaves the workspace quiescent, so the next bd command +// relaunches lazily against the restored data. +func runBackupRestoreProxied(ctx context.Context, dir string, force bool) error { + if err := requireLocalProxiedBackup("backup restore"); err != nil { + return err + } + beadsDir := beads.FindBeadsDir() + if beadsDir == "" { + return HandleErrorRespectJSON("%s", activeWorkspaceNotFoundError()) + } + root, err := resolveProxiedServerRootPath(beadsDir) + if err != nil { + return HandleErrorRespectJSON("%v", err) + } + backupURL, err := versioncontrolops.DirToFileURL(dir) + if err != nil { + return HandleErrorRespectJSON("%v", err) + } + + // Ask the live provider which database to replace before taking it away. + database, err := proxiedActiveDatabase(ctx) + if err != nil { + return HandleErrorRespectJSON("%v", err) + } + + if err := quiesceProxiedTopology(ctx, root); err != nil { + return HandleErrorRespectJSON("%v", err) + } + + if err := withQuiescedProxiedProvider(ctx, beadsDir, database, root, func(ctx context.Context, conn *sql.Conn) error { + return versioncontrolops.BackupRestore(ctx, conn, backupURL, database, force) + }); err != nil { + return HandleErrorRespectJSON("%s", proxiedRestoreFailureMessage(dir, false, err)) + } + + if err := withQuiescedProxiedProvider(ctx, beadsDir, database, root, func(ctx context.Context, conn *sql.Conn) error { + return reconcileRestoredProxiedWorkspace(ctx, conn, beadsDir, dir, force) + }); err != nil { + return HandleErrorRespectJSON("%s", proxiedRestoreFailureMessage(dir, true, err)) + } + + return nil +} + +// proxiedRestoreFailureMessage renders what the operator is told when one of +// the two quiesced steps fails. +// +// The distinction it makes is the only one that matters here: DID THE DATA COME +// BACK. A teardown failure happens after the step succeeded, so saying "restore +// failed" would be false, and the obvious response to it — run the restore +// again — is the wrong one. Split out from the call sites so that judgement is +// something a test can hold still (TestProxiedRestoreFailureMessage). +// +// afterReconcile distinguishes the two steps, because a teardown failure costs +// the operator different follow-up work either side of the reconcile: before +// it, the backup destination has not been re-registered yet. +func proxiedRestoreFailureMessage(dir string, afterReconcile bool, err error) string { + var teardown *proxiedTeardownError + if errors.As(err, &teardown) { + if afterReconcile { + return fmt.Sprintf("restored from %s, but shutting the proxied server down afterwards failed: %v; "+ + "the data is restored and reconciled — run 'bd dolt stop --force' before the next command", dir, teardown) + } + return fmt.Sprintf("restored from %s, but shutting the proxied server down afterwards failed: %v; "+ + "the data is restored — run 'bd dolt stop --force', then 'bd backup init %s' to re-register the "+ + "backup destination, which this run did not reach", dir, teardown, dir) + } + if afterReconcile { + return fmt.Sprintf("restored from %s, but the restored database did not reopen: %v", dir, err) + } + return fmt.Sprintf("restore failed: %v", err) +} + +// quiesceProxiedTopology drops this process's provider and stops the proxy and +// its dolt child, so nothing holds the database that is about to be replaced. +func quiesceProxiedTopology(ctx context.Context, root string) error { + if uowProvider != nil { + if err := uowProvider.Close(ctx); err != nil { + return fmt.Errorf("close the proxied provider before restoring: %w", err) + } + uowProvider = nil + } + if err := proxy.Shutdown(root); err != nil { + return fmt.Errorf("stop the proxied server before restoring: %w; "+ + "run 'bd dolt stop --force' and retry", err) + } + return nil +} + +// proxiedTeardownError marks a failure that happened AFTER the operation +// succeeded, while putting the topology back down. It exists so the caller can +// tell the two apart in what it prints. Flattening them is what made a restore +// that completed and then failed to shut the proxy down report "restore +// failed": the operator is told their data did not come back when it did, and +// the obvious response — run the restore again — is the wrong one. +type proxiedTeardownError struct{ err error } + +func (e *proxiedTeardownError) Error() string { return e.err.Error() } +func (e *proxiedTeardownError) Unwrap() error { return e.err } + +// withQuiescedProxiedProvider relaunches the topology for one operation and +// puts it back down afterwards, so each step either side of the replace runs on +// a connection that was opened after it. +// +// The provider ADOPTS whatever identity the database carries: a restore is +// precisely the operation after which the workspace's recorded project id and +// the database's may legitimately differ, and asserting the old one would +// refuse the connection that is supposed to reconcile them. That makes this the +// third caller of newProxiedServerUOWProviderAdopting; its doc comment is the +// register of identity-assertion bypasses and names this one. +// +// A teardown failure is returned wrapped in *proxiedTeardownError, and only +// when the operation itself succeeded. If opening or running the operation +// failed, preserve that error and append any cleanup failure without the +// success marker. Shutdown is attempted even if construction fails after +// starting the topology but before returning a provider. +func withQuiescedProxiedProvider(ctx context.Context, beadsDir, database, root string, fn func(context.Context, *sql.Conn) error) (err error) { + var provider uow.UnitOfWorkProvider + defer func() { + var closeErr error + if provider != nil { + closeErr = provider.Close(ctx) + } + stopErr := proxy.Shutdown(root) + if teardownErr := errors.Join(closeErr, stopErr); teardownErr != nil { + if err != nil { + err = errors.Join(err, fmt.Errorf("shut down the proxied server after failure: %w", teardownErr)) + } else { + err = &proxiedTeardownError{err: teardownErr} + } + } + }() + provider, err = newProxiedServerUOWProviderAdopting(ctx, beadsDir, database) + if err != nil { + return err + } + + maintenance, ok := provider.(uow.MaintenanceProvider) + if !ok { + return fmt.Errorf("the proxied provider does not support maintenance operations") + } + return maintenance.RunNonTx(ctx, fn) +} + +// reconcileRestoredProxiedWorkspace is the post-restore bookkeeping, mirroring +// the direct path: adopt the restored database's project identity when the +// restore overwrote one, and register the restore source as the backup +// destination so `bd backup sync` works without a separate init. +// +// Both are best-effort with a warning, as in direct mode. The restore itself +// has already succeeded by this point, and failing the command over bookkeeping +// would tell the operator their data did not come back when it did. +func reconcileRestoredProxiedWorkspace(ctx context.Context, conn *sql.Conn, beadsDir, dir string, force bool) error { + if force { + if err := syncProjectIDFromProxiedDB(ctx, conn, beadsDir); err != nil { + fmt.Fprintf(os.Stderr, "Warning: failed to sync project ID after restore: %v\n", err) + } + } + + backupURL := resolveDoltBackupURL(dir) + _ = versioncontrolops.BackupRemove(ctx, conn, proxiedBackupTargetName) + if err := versioncontrolops.BackupAdd(ctx, conn, proxiedBackupTargetName, backupURL); err != nil { + fmt.Fprintf(os.Stderr, "Warning: failed to register backup remote: %v\n", err) + return nil + } + if err := saveDoltBackupConfig(backupURL); err != nil { + fmt.Fprintf(os.Stderr, "Warning: backup registered but failed to save config: %v\n", err) + } + return nil +} + +// syncProjectIDFromProxiedDB copies the restored database's _project_id into +// metadata.json, so the identity check does not reject later connections. The +// direct path reads it through the store; there is no store in proxied mode, so +// it goes through the same issueops accessor the store itself uses rather than +// a second copy of the query. +func syncProjectIDFromProxiedDB(ctx context.Context, conn *sql.Conn, beadsDir string) error { + dbID, err := issueops.GetMetadataInTx(ctx, conn, "_project_id") + if err != nil || dbID == "" { + return err + } + cfg, err := configfile.Load(beadsDir) + if err != nil { + return err + } + if cfg == nil || cfg.ProjectID == dbID { + return nil + } + cfg.ProjectID = dbID + return cfg.Save(beadsDir) +} diff --git a/cmd/bd/backup_restore.go b/cmd/bd/backup_restore.go index 7d1496181..aa508838a 100644 --- a/cmd/bd/backup_restore.go +++ b/cmd/bd/backup_restore.go @@ -30,11 +30,10 @@ Use --force to overwrite an existing database with the backup contents. The database must already be initialized (run 'bd init' first if needed). To initialize and restore in one step, use: bd init && bd backup restore`, - Args: cobra.MaximumNArgs(1), + Args: cobra.MaximumNArgs(1), + SilenceUsage: true, + SilenceErrors: true, RunE: func(cmd *cobra.Command, args []string) error { - if usesProxiedServer() { - return HandleErrorRespectJSON("backup restore is not supported in proxied-server mode") - } evt := metrics.NewCommandEvent("backup-restore") defer func() { if c := metrics.Global(); c != nil { @@ -61,14 +60,24 @@ To initialize and restore in one step, use: bd init && bd backup restore`, force, _ := cmd.Flags().GetBool("force") - if err := runBackupRestore(ctx, store, dir, force); err != nil { + if usesProxiedServer() { + if err := runBackupRestoreProxied(ctx, dir, force); err != nil { + return err + } + } else if err := runBackupRestore(ctx, store, dir, force); err != nil { return err } - if !jsonOutput { - fmt.Printf("%s Restore complete\n", ui.RenderPass("✓")) + // One success report for both topologies. Under --json this used to + // print nothing at all, which left a caller unable to tell a completed + // restore from a silently skipped one. + if jsonOutput { + return outputJSON(map[string]interface{}{ + "restored": true, + "source": dir, + }) } - + fmt.Printf("%s Restore complete\n", ui.RenderPass("✓")) return nil }, } diff --git a/cmd/bd/backup_status_test.go b/cmd/bd/backup_status_test.go index 0d6159b61..d06d0db78 100644 --- a/cmd/bd/backup_status_test.go +++ b/cmd/bd/backup_status_test.go @@ -10,6 +10,7 @@ import ( "testing" "github.com/spf13/cobra" + "github.com/steveyegge/beads/internal/configfile" ) func TestBackupStatusSizeErrorRespectsOutputMode(t *testing.T) { @@ -159,25 +160,62 @@ func TestBackupStatusIncludesAvailableDatabaseSize(t *testing.T) { }) } -func TestBackupStatusProxiedGuardDoesNotMeasureSize(t *testing.T) { - prepareBackupStatusTest(t) - proxiedServerMode = true - called := false - sizeDatabase := func(context.Context) (int64, bool, error) { - called = true - return 1, true, nil - } +// TestBackupStatusProxiedGuardFollowsLocality replaces a test that asserted a +// blanket proxied refusal. Since slice S3 the guard is scoped to locality: bd +// owns the dolt process on a managed-local workspace and can honor the command +// there, while an external one is still refused — and a refused command must +// not go on to measure anything, because the directory it would measure belongs +// to a server on another host. +func TestBackupStatusProxiedGuardFollowsLocality(t *testing.T) { + for _, tc := range []struct { + name string + sidecar string + wantMeasure bool + }{ + { + name: "external topology is refused without measuring", + sidecar: `{"external":{"host":"db.example.com","port":3306}}`, + wantMeasure: false, + }, + { + name: "managed local is honored and measures", + sidecar: `{}`, + wantMeasure: true, + }, + } { + t.Run(tc.name, func(t *testing.T) { + beadsDir := prepareBackupStatusTest(t) + proxiedServerMode = true + if err := os.WriteFile(configfile.ProxiedServerClientInfoPath(beadsDir), []byte(tc.sidecar), 0o600); err != nil { + t.Fatalf("write proxied sidecar: %v", err) + } - cmd := newBackupStatusTestRoot(sizeDatabase) - cmd.SetArgs([]string{"backup", "status"}) - _, _, err := executeBackupStatusCommand(t, cmd) - assertExitCode(t, err, 1) - if called { - t.Fatal("proxied-server guard called database size provider") + measured := false + sizeDatabase := func(context.Context) (int64, bool, error) { + measured = true + return 1, true, nil + } + + cmd := newBackupStatusTestRoot(sizeDatabase) + cmd.SetArgs([]string{"backup", "status"}) + _, _, err := executeBackupStatusCommand(t, cmd) + if tc.wantMeasure { + if err != nil { + t.Fatalf("backup status on managed-local: %v", err) + } + } else { + assertExitCode(t, err, 1) + } + if measured != tc.wantMeasure { + t.Fatalf("size provider called = %v, want %v", measured, tc.wantMeasure) + } + }) } } -func prepareBackupStatusTest(t *testing.T) { +// prepareBackupStatusTest builds a throwaway workspace and returns its .beads +// directory, so a caller can drop workspace files (the proxied sidecar) into it. +func prepareBackupStatusTest(t *testing.T) string { t.Helper() beadsDir := filepath.Join(t.TempDir(), ".beads") @@ -198,6 +236,7 @@ func prepareBackupStatusTest(t *testing.T) { proxiedServerMode = oldProxiedServerMode resetCommandContext() }) + return beadsDir } func newBackupStatusTestRoot(sizeDatabase backupSizeFunc) *cobra.Command { diff --git a/cmd/bd/capability_registry.go b/cmd/bd/capability_registry.go index 276650bbe..5ee3b2aa5 100644 --- a/cmd/bd/capability_registry.go +++ b/cmd/bd/capability_registry.go @@ -72,8 +72,9 @@ const ( // Tracking values name the plan-of-record work item for an unimplemented // refusal. These are the slices of the proxied-parity plan; a row gets a GitHub // issue or bead ID instead once one is filed for it. +// S3 has no entry: it landed, and its rows carry Reason=design rather than a +// tracking item. const ( - trackBackup = "proxied-parity S3 (backup/restore family)" trackVersionCtl = "proxied-parity S4 (dolt commit/push/pull/remote)" trackSync = "proxied-parity S5 (bd sync)" trackDegradation = "proxied-parity S6 (doctor, config show, degradations)" @@ -85,12 +86,40 @@ const ( // empty for the bare path; a row keyed on an ArgSet wins over the bare row, which // is how `duplicates --auto-merge --dry-run` stays honored while // `duplicates --auto-merge` is refused. +// +// Topology overrides Rule on the proxied shapes named in it and is empty for +// every row whose answer is the same everywhere — which is all of them but the +// backup family. See honoredOn. type capabilityRow struct { - Path string - ArgSet string - Rule proxyCapabilityRule - History HistoryCapabilityClass - Note string + Path string + ArgSet string + Rule proxyCapabilityRule + Topology map[ProxyTopology]proxyCapabilityRule + History HistoryCapabilityClass + Note string +} + +// ruleFor resolves the row's policy on one proxied topology. +func (r capabilityRow) ruleFor(topology ProxyTopology) proxyCapabilityRule { + if rule, ok := r.Topology[topology]; ok { + return rule + } + return r.Rule +} + +// honoredOn narrows a refusal to the shapes where it is actually true: the row +// goes on refusing everywhere except the topologies named here. +// +// The direction is deliberate. A row states its refusal first and then lists +// the exceptions, so a shape nobody thought about — a new transport, or the +// unknown a corrupt sidecar produces — inherits the refusal rather than the +// exemption. +func (r capabilityRow) honoredOn(topologies ...ProxyTopology) capabilityRow { + r.Topology = make(map[ProxyTopology]proxyCapabilityRule, len(topologies)) + for _, topology := range topologies { + r.Topology[topology] = proxyCapabilityRule{Outcome: ProxyOutcomeHonored} + } + return r } // display renders the row's key the way its refusal message names it. @@ -166,6 +195,50 @@ func permitted(path string) capabilityRow { return capabilityRow{Path: path, Rule: proxyCapabilityRule{Outcome: ProxyOutcomeHonored}} } +// backupRefusal builds one backup-family row: honored where bd owns the Dolt +// server, refused by design everywhere else. The code stays +// proxy.backup.unsupported — consumers have been keying on it since 1.3.0 and +// the refusal it names has not changed, only its reason and its scope — and the +// message keeps the frozen " is not supported in proxied-server mode" +// prefix, with the reason appended. +// +// The message leads with SERVER-GLOBAL STATE rather than with the filesystem, +// because only the filesystem half is scheme-specific. `bd backup init` accepts +// https/aws/gs as well as file:// (resolveDoltBackupURL) and the help text +// recommends DoltHub; for those a server-side push over the network with the +// server's own credentials is a coherent thing to ask for, and "the destination +// is on the server's disk" says nothing about it. What is true of every scheme +// is that `CALL DOLT_BACKUP('add', …)` registers the remote ON THE SERVER, +// where it is global to every client of that server — one workspace's backup +// decision silently becomes everyone's, which is the shape of the auto-backup +// storm this command's own help text warns about. +// +// backupRemoteSchemeTracking is why these rows carry a Tracking item despite +// being Reason=design: the outcome is settled, the remote-scheme CASE is not. +func backupRefusal(path string) capabilityRow { + return refusedPath(path, "proxy.backup.unsupported", ProxyReasonDesign, backupRemoteSchemeTracking). + withMessage(path + " is not supported in proxied-server mode against a Dolt server bd does not own: " + + "the backup remote is registered on the server, where it is global to every client, " + + "and a file:// destination names a filesystem this client cannot see"). + honoredOn(ProxyTopologyManagedLocal) +} + +// backupRemoteSchemeTracking records the one part of the backup refusal that is +// a question rather than a verdict, so refusing the family off-host does not +// silently foreclose it. A remote-scheme destination (DoltHub, aws://, gs://) +// needs no filesystem bd can see — the server pushes it over the network from +// its own environment, which is exactly the credential story S4 has to settle +// for `dolt push` on these same topologies. Whether a per-client backup remote +// on a shared server is ever acceptable is the open part; the plumbing is not. +// +// Note also that the DoltHub story is already half-built in DIRECT mode: +// `bd backup init https://…` and `bd backup sync` work, but +// `bd backup restore https://…` does not, because validateBackupRestoreDir +// os.Stats its argument and rejects anything that is not an existing directory +// on every topology. Pre-existing, not something this slice changed. +const backupRemoteSchemeTracking = "open question, not a verdict: remote-scheme (DoltHub/aws/gs) backup on a server " + + "bd does not own — revisit with S4, which settles the same server-env credential question for dolt push" + // proxyCapabilityRegistry is the whole path-keyed policy. Rows for permitted // paths are appended from proxyPermittedPaths by init. var proxyCapabilityRegistry = []capabilityRow{ @@ -176,16 +249,37 @@ var proxyCapabilityRegistry = []capabilityRow{ // --- backup / restore --------------------------------------------------- // CALL DOLT_BACKUP(...) is pure SQL over a connection proxied mode already - // owns, so this family is plumbing, not policy. Locality policy (file:// - // URLs naming the server's filesystem on external topologies) is decided - // per topology when S3 lands. - refusedPath("backup", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup). - asParentGroup(), - refusedPath("backup init", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), - refusedPath("backup sync", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), - refusedPath("backup remove", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), - refusedPath("backup status", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), - refusedPath("backup restore", "proxy.backup.unsupported", ProxyReasonUnimplemented, trackBackup), + // owns, so routing this family needed no proxy work at all — see + // backup_proxied_server.go. + // + // What it did need is the ownership policy, which is the first genuinely + // topology-differentiated rule in this table. A backup remote belongs to the + // SERVER, not to the client that asked for it: `CALL DOLT_BACKUP('add', …)` + // registers it in the server's own configuration, where it is global to + // every client connected to that server. On managed-local "every client" is + // this workspace, because bd spawned the child itself. Anywhere else one + // operator's backup decision silently becomes everyone's — the same shape as + // the auto-backup storm this command's help text already warns about, and + // the reason team-server is the sharpest case: bts owns the store. + // + // A file:// destination adds a second, scheme-specific problem on top: it + // names a path on the machine running dolt, so off-host the command either + // fails at the point of use or, far worse, reports success having written + // the backup somewhere the operator will never look for it. + // + // Hence: honored on managed-local, refused by design everywhere else. The + // OUTCOME is design — server-global state is not something plumbing on this + // side can make per-client, and a backup story for those shapes belongs to + // whoever runs the server. The remote-scheme case (DoltHub, aws://, gs://, + // which resolveDoltBackupURL accepts and backup.go recommends) is the part + // that is a question rather than a verdict, and it is not silently + // foreclosed: see backupRemoteSchemeTracking, which every row below carries. + backupRefusal("backup").asParentGroup(), + backupRefusal("backup init"), + backupRefusal("backup sync"), + backupRefusal("backup remove"), + backupRefusal("backup status"), + backupRefusal("backup restore"), // --- version control over a shared history ------------------------------ // Refused by design: these mutate or traverse a history that several @@ -515,7 +609,11 @@ func commandPolicyArgSet(cmd *cobra.Command) string { // front door: it rejects direct-only commands before migrations, auto-start or // provider construction, so a refusal leaves the workspace untouched. The // flag-keyed half is validateProxyCapabilitiesBeforeProvider. -func validateProxyRegistryBeforeProvider(cmd *cobra.Command) error { +// +// topology is the workspace's proxied shape (resolveProxiedTopology). It is a +// parameter rather than a lookup inside so the gate can be exercised for every +// shape without a workspace on disk. +func validateProxyRegistryBeforeProvider(cmd *cobra.Command, topology ProxyTopology) error { if cmd == nil { return nil } @@ -532,10 +630,14 @@ func validateProxyRegistryBeforeProvider(cmd *cobra.Command) error { } } row, ok := capabilityRowFor(cmd) - if !ok || row.Rule.Outcome != ProxyOutcomeRefused { + if !ok { return nil } - return HandleProxyCapabilityError(proxyCapabilityErrorFor(row.Rule)) + rule := row.ruleFor(topology) + if rule.Outcome != ProxyOutcomeRefused { + return nil + } + return HandleProxyCapabilityError(proxyCapabilityErrorFor(rule)) } func proxyCapabilityErrorFor(rule proxyCapabilityRule) *ProxyCapabilityError { diff --git a/cmd/bd/capability_registry_history_test.go b/cmd/bd/capability_registry_history_test.go index c0319a73b..87d1fd44d 100644 --- a/cmd/bd/capability_registry_history_test.go +++ b/cmd/bd/capability_registry_history_test.go @@ -31,7 +31,7 @@ func TestSyncRefusesBeforeProvider(t *testing.T) { root := &cobra.Command{Use: "bd"} cmd := &cobra.Command{Use: "sync"} root.AddCommand(cmd) - err := validateProxyRegistryBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal) if err == nil { t.Fatalf("sync refusal = %#v", err) } @@ -71,7 +71,7 @@ func TestHistoryDirectOnlyRefusalContract(t *testing.T) { } uowProvider = nil jsonOutput = true - out := captureStdout(t, func() error { _ = validateProxyRegistryBeforeProvider(cmd); return nil }) + out := captureStdout(t, func() error { _ = validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal); return nil }) var got map[string]any want := expected[path] if err := json.Unmarshal([]byte(out), &got); err != nil || got["code"] != want.code || got["error"] != want.message || got["mutates"] != false { @@ -104,7 +104,7 @@ func TestHistoryNestedFrontDoorsRefuseAndSupportedPathsPass(t *testing.T) { cmd.AddCommand(child) cmd = child } - err := validateProxyRegistryBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal) if path == "dolt remote remove" { if err != nil { t.Fatalf("supported %s refused: %v", path, err) @@ -120,7 +120,7 @@ func TestHistoryNestedFrontDoorsRefuseAndSupportedPathsPass(t *testing.T) { var got map[string]any // validateProxyRegistryBeforeProvider renders the typed refusal to // stdout in JSON mode; the command must retain its nested path. - out := captureStdout(t, func() error { _ = validateProxyRegistryBeforeProvider(cmd); return nil }) + out := captureStdout(t, func() error { _ = validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal); return nil }) if err := json.Unmarshal([]byte(out), &got); err != nil { t.Fatalf("%s refusal JSON: %v (%q)", path, err, out) } @@ -132,7 +132,7 @@ func TestHistoryNestedFrontDoorsRefuseAndSupportedPathsPass(t *testing.T) { root := &cobra.Command{Use: "bd"} history := &cobra.Command{Use: "history"} root.AddCommand(history) - if err := validateProxyRegistryBeforeProvider(history); err != nil { + if err := validateProxyRegistryBeforeProvider(history, ProxyTopologyManagedLocal); err != nil { t.Fatalf("history --events supported path refused: %v", err) } } diff --git a/cmd/bd/capability_registry_test.go b/cmd/bd/capability_registry_test.go index f6f7f47c1..6725729fb 100644 --- a/cmd/bd/capability_registry_test.go +++ b/cmd/bd/capability_registry_test.go @@ -109,38 +109,12 @@ func TestProxyCapabilityRegistryHasNoDuplicateRows(t *testing.T) { // refusal with no tracking item is a gap nobody owns. func TestProxyCapabilityRegistryRowInvariants(t *testing.T) { for _, row := range proxyCapabilityRegistry { - switch row.Rule.Outcome { - case ProxyOutcomeRefused: - if row.Rule.Code == "" || row.Rule.Message == "" || row.Rule.ExitCode != 1 { - t.Errorf("%s: refusal must carry a code, a message and exit 1, got %#v", row.display(), row.Rule) - } - if row.Rule.Mutates { - t.Errorf("%s: a refusal never mutates", row.display()) - } - if !strings.HasPrefix(row.Rule.Code, "proxy.") { - t.Errorf("%s: code %q must be in the stable proxy.* namespace", row.display(), row.Rule.Code) - } - case ProxyOutcomeRefusedInRunE: - if row.Rule.Code != "" { - t.Errorf("%s: an untyped RunE refusal has no code; if it has one it belongs in the gate", row.display()) - } - case ProxyOutcomeHonored: - if row.Rule.Reason != "" { - t.Errorf("%s: Reason explains a refusal; an honored row must not carry one", row.display()) - } - default: - t.Errorf("%s: unexpected outcome %q", row.display(), row.Rule.Outcome) - } - - refusal := row.Rule.Outcome == ProxyOutcomeRefused || row.Rule.Outcome == ProxyOutcomeRefusedInRunE - if refusal { - switch row.Rule.Reason { - case ProxyReasonDesign, ProxyReasonUnimplemented: - default: - t.Errorf("%s: refusal must declare Reason design or unimplemented, got %q", row.display(), row.Rule.Reason) - } - if row.Rule.Reason == ProxyReasonUnimplemented && row.Rule.Tracking == "" { - t.Errorf("%s: an unimplemented refusal needs a Tracking item naming who closes it", row.display()) + assertCapabilityRuleInvariants(t, row.display(), row.Rule) + for topology, rule := range row.Topology { + assertCapabilityRuleInvariants(t, row.display()+" on "+string(topology), rule) + if topology == ProxyTopologyUnknown { + t.Errorf("%s: a row must not name the unknown topology; unknown exists so an unclassifiable "+ + "workspace INHERITS the refusal, and exempting it would invert that", row.display()) } } if row.History == HistoryDirectOnly && row.Rule.Outcome != ProxyOutcomeRefused { @@ -152,6 +126,46 @@ func TestProxyCapabilityRegistryRowInvariants(t *testing.T) { } } +// assertCapabilityRuleInvariants checks one rule — a row's default or one of +// its per-topology overrides. Both have to satisfy the same contract: a +// topology override is policy a consumer sees, not a footnote. +func assertCapabilityRuleInvariants(t *testing.T, where string, rule proxyCapabilityRule) { + t.Helper() + switch rule.Outcome { + case ProxyOutcomeRefused: + if rule.Code == "" || rule.Message == "" || rule.ExitCode != 1 { + t.Errorf("%s: refusal must carry a code, a message and exit 1, got %#v", where, rule) + } + if rule.Mutates { + t.Errorf("%s: a refusal never mutates", where) + } + if !strings.HasPrefix(rule.Code, "proxy.") { + t.Errorf("%s: code %q must be in the stable proxy.* namespace", where, rule.Code) + } + case ProxyOutcomeRefusedInRunE: + if rule.Code != "" { + t.Errorf("%s: an untyped RunE refusal has no code; if it has one it belongs in the gate", where) + } + case ProxyOutcomeHonored: + if rule.Reason != "" { + t.Errorf("%s: Reason explains a refusal; an honored row must not carry one", where) + } + default: + t.Errorf("%s: unexpected outcome %q", where, rule.Outcome) + } + + if rule.Outcome == ProxyOutcomeRefused || rule.Outcome == ProxyOutcomeRefusedInRunE { + switch rule.Reason { + case ProxyReasonDesign, ProxyReasonUnimplemented: + default: + t.Errorf("%s: refusal must declare Reason design or unimplemented, got %q", where, rule.Reason) + } + if rule.Reason == ProxyReasonUnimplemented && rule.Tracking == "" { + t.Errorf("%s: an unimplemented refusal needs a Tracking item naming who closes it", where) + } + } +} + // inlineProxiedRefusal matches the untyped refusals commands raise from their // own RunE, e.g. HandleErrorRespectJSON("gitlab sync is not supported in // proxied-server mode"). @@ -271,6 +285,16 @@ func TestProxyCapabilityRegistryReasonsAreAssignedIndividually(t *testing.T) { "migrate": true, "migrate sync": true, "migrate issues": true, "migrate-issues": true, "migrate-personal": true, "admin cleanup": true, "admin reset": true, + // The backup family's DEFAULT rule is the design refusal; managed-local + // overrides it to honored (TestProxiedBackupHonoredOnlyOnManagedLocal). + // It is design because a backup remote is registered on the SERVER, + // where it is global to every client of that server — true of every + // destination scheme, unlike the filesystem half, which is a file:// + // property only. These rows carry a Tracking item anyway + // (backupRemoteSchemeTracking): the outcome is settled, the + // remote-scheme case is an open question and must not read as closed. + "backup": true, "backup init": true, "backup sync": true, + "backup remove": true, "backup status": true, "backup restore": true, } gotDesign := map[string]bool{} @@ -364,7 +388,7 @@ func TestProxyRefusalJSONCarriesReason(t *testing.T) { cmd := &cobra.Command{Use: tc.path} root.AddCommand(cmd) out := captureStdout(t, func() error { - _ = validateProxyRegistryBeforeProvider(cmd) + _ = validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal) return nil }) for _, want := range []string{ diff --git a/cmd/bd/capability_registry_transform_test.go b/cmd/bd/capability_registry_transform_test.go index 86ed302b4..7862323a2 100644 --- a/cmd/bd/capability_registry_transform_test.go +++ b/cmd/bd/capability_registry_transform_test.go @@ -76,7 +76,7 @@ func TestProxyTransformCapabilityRows(t *testing.T) { t.Fatalf("typed refusal = %#v", typed) } } - if err := validateProxyRegistryBeforeProvider(cmd); tc.outcome == ProxyOutcomeHonored && err != nil { + if err := validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal); tc.outcome == ProxyOutcomeHonored && err != nil { t.Fatalf("honored transform refused: %v", err) } }) @@ -86,11 +86,11 @@ func TestProxyTransformCapabilityRows(t *testing.T) { func TestProxyDuplicatesAutoMergeDryRunMatrix(t *testing.T) { cmd := transformTestCommand("duplicates", "auto-merge", "dry-run") _ = cmd.Flags().Set("auto-merge", "true") - if err := validateProxyRegistryBeforeProvider(cmd); err == nil { + if err := validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal); err == nil { t.Fatalf("non-dry auto-merge error = %v", err) } _ = cmd.Flags().Set("dry-run", "true") - if err := validateProxyRegistryBeforeProvider(cmd); err != nil { + if err := validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal); err != nil { t.Fatalf("dry-run auto-merge refused: %v", err) } } @@ -104,7 +104,7 @@ func TestProxyTransformRefusalRendering(t *testing.T) { jsonOutput = false var textErr string stderr := captureStderr(t, func() { - err := validateProxyRegistryBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal) if code, ok := exitCodeFromError(err); !ok || code != 1 { t.Fatalf("exit = %v, want 1", err) } @@ -116,7 +116,7 @@ func TestProxyTransformRefusalRendering(t *testing.T) { jsonOutput = true stdout := captureStdout(t, func() error { - _ = validateProxyRegistryBeforeProvider(cmd) + _ = validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal) return nil }) var envelope map[string]any @@ -164,7 +164,7 @@ func TestProxyTransformRefusalHasNoProviderOrFileMutation(t *testing.T) { t.Cleanup(func() { uowProvider = oldProvider }) cmd := transformTestCommand("supersede") - err := validateProxyRegistryBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal) if err == nil { t.Fatal("expected transform refusal") } diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 946cefbb1..e93b6dd54 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -1285,18 +1285,18 @@ var rootCmd = &cobra.Command{ fmt.Fprintf(os.Stderr, "warning: %v\n", err) } } + if beadsDir == "" { + beadsDir = beads.FindBeadsDir() + } if cmdName == "doctor" && usesProxiedServer() { // Refuse only on a real refusal. The registry validator // returns nil for doctor subcommands, and returning early on // that would skip the legacy-store guard and autocommit-mode // resolution every other skipsStoreInit command still runs. - if err := validateProxyRegistryBeforeProvider(cmd); err != nil { + if err := validateProxyRegistryBeforeProvider(cmd, resolveProxiedTopology(beadsDir)); err != nil { return err } } - if beadsDir == "" { - beadsDir = beads.FindBeadsDir() - } if err := guardLegacyNoStoreCommand(cmd, beadsDir); err != nil { isMigrationCommand := false for current := cmd; current != nil; current = current.Parent() { @@ -1503,7 +1503,7 @@ var rootCmd = &cobra.Command{ if err := validateProxyCapabilitiesBeforeProvider(cmd); err != nil { return err } - if err := validateProxyRegistryBeforeProvider(cmd); err != nil { + if err := validateProxyRegistryBeforeProvider(cmd, resolveProxiedTopology(beadsDir)); err != nil { return err } } diff --git a/cmd/bd/proxy_capability.go b/cmd/bd/proxy_capability.go index 774323944..73869cdc7 100644 --- a/cmd/bd/proxy_capability.go +++ b/cmd/bd/proxy_capability.go @@ -25,10 +25,11 @@ const ( ProxyModeProxied ProxyMode = "proxied-server" ) -// ProxyTopology distinguishes the provider deployment shape. Capability -// policy is currently identical across proxied shapes, but retaining this -// dimension prevents an external TCP server from being conflated with a local -// managed one as more surfaces are added. +// ProxyTopology distinguishes the provider deployment shape. The backup family +// is the first policy that genuinely differs across these: a Dolt backup +// destination is resolved on the machine running dolt, so "the server bd +// spawned itself" and "a server somebody else runs" are different answers to +// the same command (see capability_registry.go's backup rows). type ProxyTopology string const ( @@ -36,6 +37,17 @@ const ( ProxyTopologyManagedLocal ProxyTopology = "managed-local" ProxyTopologyExternalTCP ProxyTopology = "external-tcp" ProxyTopologyExternalUnix ProxyTopology = "external-unix" + // ProxyTopologyTeamServer is a workspace whose database is owned by + // beads-team-server. It is a shape rather than a transport: bts owns the + // store's schema and identity whether bd reaches it over a port, a socket, + // or a child bd started, so the ownership fact decides capability policy and + // the transport does not. + ProxyTopologyTeamServer ProxyTopology = "team-server" + // ProxyTopologyUnknown is what a workspace bd cannot classify reports. It is + // never honored by a topology-keyed row, which is the fail-closed half of + // the policy: a workspace that cannot prove it owns its Dolt server does not + // get the privileges of one that can. + ProxyTopologyUnknown ProxyTopology = "unknown" ) // ProxyCapabilityOutcome describes what the front door does with a feature. diff --git a/cmd/bd/proxy_capability_test.go b/cmd/bd/proxy_capability_test.go index 3b7718cf0..6c70f17ff 100644 --- a/cmd/bd/proxy_capability_test.go +++ b/cmd/bd/proxy_capability_test.go @@ -139,6 +139,11 @@ func TestProxyCapabilityCommandRows(t *testing.T) { } } +// TestProxyMaintenanceNestedPathsRefuseBeforeProvider proves the gate resolves +// a PARENT+CHILD path, which is where a nested command used to slip through. +// It runs on external-tcp because that is the topology on which every path +// listed here still refuses: the backup family is honored on managed-local +// since slice S3, and asserting a refusal there would be asserting the bug. func TestProxyMaintenanceNestedPathsRefuseBeforeProvider(t *testing.T) { oldJSON := jsonOutput jsonOutput = true @@ -151,7 +156,7 @@ func TestProxyMaintenanceNestedPathsRefuseBeforeProvider(t *testing.T) { root.AddCommand(parent) parent.AddCommand(child) out := captureStdout(t, func() error { - _ = validateProxyRegistryBeforeProvider(child) + _ = validateProxyRegistryBeforeProvider(child, ProxyTopologyExternalTCP) return nil }) if !strings.Contains(out, `"code":`) { @@ -171,7 +176,7 @@ func TestProxyFormulaSwarmMergeSlotRefusals(t *testing.T) { cmd.AddCommand(child) cmd = child } - err := validateProxyRegistryBeforeProvider(cmd) + err := validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal) if err == nil { t.Fatalf("%s unexpectedly allowed", path) } @@ -232,7 +237,7 @@ func TestProxyWorkflowRefusalContractAndNoMutation(t *testing.T) { oldDidWrite := commandDidWrite.Load() commandDidWrite.Store(false) t.Cleanup(func() { commandDidWrite.Store(oldDidWrite) }) - out := captureStdout(t, func() error { _ = validateProxyRegistryBeforeProvider(cmd); return nil }) + out := captureStdout(t, func() error { _ = validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal); return nil }) var got map[string]any if err := json.Unmarshal([]byte(out), &got); err != nil || got["code"] != tc.code || got["error"] != tc.message { t.Fatalf("JSON refusal = %q (%v)", out, err) @@ -269,7 +274,7 @@ func TestProxyMaintenanceRefusalLeavesFilesUntouched(t *testing.T) { oldProvider := uowProvider uowProvider = nil t.Cleanup(func() { uowProvider = oldProvider }) - err := validateProxyRegistryBeforeProvider(hooks) + err := validateProxyRegistryBeforeProvider(hooks, ProxyTopologyManagedLocal) if err == nil { t.Fatal("expected typed maintenance refusal") } diff --git a/cmd/bd/proxy_topology.go b/cmd/bd/proxy_topology.go new file mode 100644 index 000000000..c8f8568ff --- /dev/null +++ b/cmd/bd/proxy_topology.go @@ -0,0 +1,54 @@ +package main + +import ( + "github.com/steveyegge/beads/internal/beads" + "github.com/steveyegge/beads/internal/configfile" +) + +// resolveProxiedTopology reports which proxied-server SHAPE a workspace is, for +// the capability rows whose answer depends on it. +// +// The shapes are ordered by how much of the store bd owns, and the first one +// that matches wins: +// +// team-server the database belongs to beads-team-server, which owns its +// schema and identity. Ownership beats transport — a +// bts-managed database is somebody else's store even when the +// dolt process happens to be local. +// external-* an operator's Dolt server that bd only connects to. The +// sidecar's External block is what records it, and the socket +// field is what separates the two transports. +// managed-local no External block: bd spawned this dolt sql-server itself, +// under this user, on this filesystem. +// +// A workspace bd cannot read is reported as unknown rather than guessed at. +// Every topology-keyed row treats unknown as refused, so a corrupt sidecar +// costs a capability instead of buying one; the provider open that follows +// fails on the same file with a much better message than this function could +// produce, so this is a policy decision, not the error path. +func resolveProxiedTopology(beadsDir string) ProxyTopology { + if beadsDir == "" { + beadsDir = beads.FindBeadsDir() + } + if beadsDir == "" { + return ProxyTopologyUnknown + } + cfg, err := configfile.Load(beadsDir) + if err != nil { + return ProxyTopologyUnknown + } + if cfg != nil && cfg.IsTeamServerManaged() { + return ProxyTopologyTeamServer + } + info, err := configfile.LoadProxiedServerClientInfo(beadsDir) + if err != nil { + return ProxyTopologyUnknown + } + if info == nil || info.External == nil { + return ProxyTopologyManagedLocal + } + if info.External.Socket != "" { + return ProxyTopologyExternalUnix + } + return ProxyTopologyExternalTCP +} diff --git a/cmd/bd/uow_factory.go b/cmd/bd/uow_factory.go index adbfb8aa7..aeeb24a89 100644 --- a/cmd/bd/uow_factory.go +++ b/cmd/bd/uow_factory.go @@ -171,11 +171,24 @@ func newProxiedServerUOWProvider(ctx context.Context, beadsDir, databaseOverride return openProxiedServerUOWProvider(ctx, beadsDir, databaseOverride, assertWorkspaceIdentity, opts...) } -// newProxiedServerUOWProviderAdopting skips that assertion. Only two callers -// legitimately have no workspace identity to assert: `bd init --team-server`, -// which ADOPTS the identity the shared database already carries (asserting the -// locally-minted placeholder would reject every correct init), and server-wide -// database maintenance, which is not scoped to one project's database. +// newProxiedServerUOWProviderAdopting skips that assertion. Three callers +// legitimately have no workspace identity to assert: +// +// - `bd init --team-server`, which ADOPTS the identity the shared database +// already carries (asserting the locally-minted placeholder would reject +// every correct init); +// - server-wide database maintenance, which is not scoped to one project's +// database; +// - withQuiescedProxiedProvider (backup_proxied_server.go), the post-restore +// reopen. A restore is precisely the operation after which the workspace's +// recorded project id and the database's may legitimately differ, and it is +// the connection whose job is to reconcile them — asserting the pre-restore +// identity would refuse the one open that can fix the mismatch. The full +// reasoning is at that function's doc comment. +// +// Anything else reaching this constructor is bypassing an identity assertion +// that exists to stop a workspace writing into another project's database. Add +// a fourth entry here, with its reason, or use newProxiedServerUOWProvider. func newProxiedServerUOWProviderAdopting(ctx context.Context, beadsDir, databaseOverride string, opts ...uow.ProviderOption) (uow.UnitOfWorkProvider, error) { return openProxiedServerUOWProvider(ctx, beadsDir, databaseOverride, adoptWorkspaceIdentity, opts...) } diff --git a/internal/storage/versioncontrolops/backup.go b/internal/storage/versioncontrolops/backup.go index 2206d729e..56acacf27 100644 --- a/internal/storage/versioncontrolops/backup.go +++ b/internal/storage/versioncontrolops/backup.go @@ -48,7 +48,20 @@ func BackupRestore(ctx context.Context, db DBConn, url, dbName string, force boo } // DirToFileURL resolves dir to an absolute path and returns a file:// URL. +// +// An input that already carries a scheme is rejected rather than prefixed. +// filepath.Abs would happily turn "https://doltremoteapi.dolthub.com/u/r" into +// "file:///cwd/https:/doltremoteapi.dolthub.com/u/r" — a syntactically valid +// URL naming a local directory that does not exist, which DOLT_BACKUP would +// then fail on, or worse create. No caller can reach that today (`bd backup +// restore` os.Stats its argument first, so a URL never gets here), but every +// caller is a restore path and restore-from-a-remote is an open capability +// question — see backupRemoteSchemeTracking in cmd/bd/capability_registry.go. +// Whoever closes it should get an error here, not a mangled path. func DirToFileURL(dir string) (string, error) { + if scheme, _, found := strings.Cut(dir, "://"); found { + return "", fmt.Errorf("%q is a %s URL, not a directory: this path takes a local directory to turn into a file:// URL", dir, scheme) + } abs, err := filepath.Abs(dir) if err != nil { return "", fmt.Errorf("resolve absolute path: %w", err) diff --git a/internal/storage/versioncontrolops/backup_test.go b/internal/storage/versioncontrolops/backup_test.go index a3278b982..d58df7660 100644 --- a/internal/storage/versioncontrolops/backup_test.go +++ b/internal/storage/versioncontrolops/backup_test.go @@ -2,9 +2,38 @@ package versioncontrolops import ( "fmt" + "strings" "testing" ) +// TestDirToFileURLRejectsSchemes pins the guard that stops a URL being turned +// into a bogus local path. filepath.Abs treats "https://host/repo" as a +// relative path, so without the check this helper would hand DOLT_BACKUP +// "file:///cwd/https:/host/repo" and the failure would name a directory nobody +// asked for. No caller can reach it today — `bd backup restore` stats its +// argument first — but every caller is a restore path, and restore-from-a-remote +// is the open capability that would reach it. +func TestDirToFileURLRejectsSchemes(t *testing.T) { + for _, dir := range []string{ + "https://doltremoteapi.dolthub.com/user/repo", + "file:///already/a/url", + "aws://bucket/key", + "gs://bucket/key", + } { + if got, err := DirToFileURL(dir); err == nil { + t.Errorf("DirToFileURL(%q) = %q, want an error naming the scheme", dir, got) + } + } + + got, err := DirToFileURL("backups/nightly") + if err != nil { + t.Fatalf("DirToFileURL on a plain relative dir: %v", err) + } + if !strings.HasPrefix(got, "file://") || strings.Contains(got, "://backups") { + t.Fatalf("DirToFileURL(%q) = %q", "backups/nightly", got) + } +} + func TestExtractAddressConflictName(t *testing.T) { tests := []struct { name string From 74dc7154c266c40990a803c45fdb9b46e4d70263 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Wed, 23 Sep 2026 18:53:16 -0700 Subject: [PATCH 13/45] fix(backup): stop attributing proxied auto-backup OFF to a missing git remote (#6694) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-merge review of #6584 (landed 7ccb961f7..c69dd3061). Honoring the backup family on a managed-local proxied server made `bd backup status` reachable on that topology for the first time — and it rendered the embedded-mode footnote `(auto: no git remote)`. That reason was never true there: `isBackupAutoEnabled` returns false because `usesSQLServer()` short-circuits it, whether or not a remote exists. The same command's help then told the operator to set `backup.enabled=true` to opt in, which on proxied changes nothing — the proxied arm of `PersistentPostRunE` never calls `runPostRunAutoBackup`, so an operator following bd's own instructions gets silent no-backup. The status note now comes from `backupAutoStatusNote`, which narrates the reason the decision actually used: proxied-server first (and regardless of the config source, since an explicit `backup.enabled=true` is inert there and a bare `enabled=true` is the misleading half), then sql-server mode — the reason `bd config get backup.enabled` already reports for the same condition — then the unchanged embedded git-remote arms. The `backup` help stops advertising the opt-in on proxied and points at explicit `bd backup sync`, which is honored on the one proxied shape that reaches these surfaces. Wiring an auto-backup hook through the new proxied routes is the other half of the reviewer's fix and is deliberately not taken here: all three reviewers classified the missing hook as a pre-existing gap #6584 did not claim. Only the newly produced misinformation is corrected. `TestBackupAutoStatusNote` pins all seven shapes against the value they are printed beside; reverting the helper to the pre-fix ordering fails exactly the three server-mode cases and leaves the four embedded ones green. (cherry picked from commit e677d90d88b771e985f5eb838f9346777828194e) --- CHANGELOG.md | 7 ++ cmd/bd/backup.go | 16 ++--- cmd/bd/backup_auto.go | 37 ++++++++++ cmd/bd/backup_auto_test.go | 139 +++++++++++++++++++++++++++++++++++++ 4 files changed, 191 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 46b484813..8e4d27cf0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 network from its own environment, which is the same credential question the next slice has to answer for `dolt push`. + Backups on this topology are **explicit only**. Auto-backup has no + post-command hook on the proxied path, so `backup.enabled=true` does nothing + there; `bd backup status` and `bd backup` help now say so instead of + attributing the OFF to a missing git remote, which was never the reason on a + server topology. Wiring an auto-backup hook through the new routes is left to + a later slice. + ### Changed - **Proxied-server refusals now say *why* they refuse.** The JSON a refused diff --git a/cmd/bd/backup.go b/cmd/bd/backup.go index 6e2e40325..270536bcf 100644 --- a/cmd/bd/backup.go +++ b/cmd/bd/backup.go @@ -40,7 +40,12 @@ Auto-backup default: dir and full-sync the whole database — a self-amplifying storm. To back up a shared server, run 'bd backup' explicitly (or set backup.enabled=true and coordinate destinations). 'bd config get backup.enabled' shows the effective - value and its source.`, + value and its source. + + On a proxied-server workspace that opt-in does not apply: auto-backup has + no post-command hook on that path, so backup.enabled=true is inert there. + The five backup verbs above are honored on a managed-local proxied server — + run 'bd backup sync' explicitly to back one up.`, GroupID: "sync", } @@ -126,14 +131,9 @@ func newBackupStatusCommand(sizeDatabase backupSizeFunc) *cobra.Command { // Show config (effective values with source) enabled := isBackupAutoEnabled() interval := config.GetDuration("backup.interval") - enabledSource := config.GetValueSource("backup.enabled") enabledNote := "" - if enabledSource == config.SourceDefault { - if enabled { - enabledNote = " (auto: git remote detected)" - } else { - enabledNote = " (auto: no git remote)" - } + if note := backupAutoStatusNote(enabled); note != "" { + enabledNote = " (" + note + ")" } fmt.Printf("\nConfig: enabled=%v%s interval=%s\n", enabled, enabledNote, interval) diff --git a/cmd/bd/backup_auto.go b/cmd/bd/backup_auto.go index 210bf5299..8974439e2 100644 --- a/cmd/bd/backup_auto.go +++ b/cmd/bd/backup_auto.go @@ -40,6 +40,43 @@ func isBackupAutoEnabled() bool { return primeHasGitRemote() } +// backupAutoStatusNote returns the parenthetical `bd backup status` +// prints after the effective backup.enabled value, or "" when the value +// speaks for itself. Callers pass the value isBackupAutoEnabled() +// already computed so the note cannot disagree with the number beside +// it — and so status does not re-run the git-remote probe. +// +// The note narrates the reason the decision actually used. Proxied-server +// is checked first and ignores the config source, because on that topology +// auto-backup cannot run at all whatever backup.enabled says: the proxied +// arm of PersistentPostRunE never calls runPostRunAutoBackup (main.go), so +// an explicit backup.enabled=true is inert and reporting a bare +// "enabled=true" would send the operator away believing backups happen. +// The five backup verbs are honored on managed-local (the only proxied +// shape that reaches this line — every other one is refused by +// requireLocalProxiedBackup), so `bd backup sync` is the real answer there. +// +// Every other sql-server shape keeps its post-run hook, so there the +// default is merely off — the reason `bd config get backup.enabled` +// already reports for the same condition. Before this arm existed, status +// attributed that OFF to a missing git remote, which is untrue: server +// mode short-circuits isBackupAutoEnabled whether or not a remote exists. +func backupAutoStatusNote(enabled bool) string { + if usesProxiedServer() { + return "auto-backup does not run on proxied-server; use 'bd backup sync'" + } + if config.GetValueSource("backup.enabled") != config.SourceDefault { + return "" + } + if usesSQLServer() { + return "auto: off in sql-server mode" + } + if enabled { + return "auto: git remote detected" + } + return "auto: no git remote" +} + // clientServerShareFilesystem reports whether the configured Dolt // server runs on a filesystem the bd client can also see — i.e. // whether a file:// URL constructed on the client is meaningful to diff --git a/cmd/bd/backup_auto_test.go b/cmd/bd/backup_auto_test.go index 4dbd14d9b..5e64195db 100644 --- a/cmd/bd/backup_auto_test.go +++ b/cmd/bd/backup_auto_test.go @@ -123,6 +123,145 @@ func TestIsBackupAutoEnabled(t *testing.T) { } } +// TestBackupAutoStatusNote pins the reason `bd backup status` prints beside +// the effective backup.enabled value against the reason isBackupAutoEnabled +// actually used — the two are computed in different files and drifted apart +// once already. +// +// The regression this locks down arrived with the proxied-parity S3 slice +// (PR #6584): honoring the backup family on managed-local made `bd backup +// status` reachable on a proxied server for the first time, where it rendered +// the embedded-mode footnote "(auto: no git remote)". On any sql-server shape +// the value is false because usesSQLServer() short-circuits, whether or not a +// remote exists, so that reason was simply untrue; and on proxied specifically +// even an explicit backup.enabled=true changes nothing, because the proxied +// arm of PersistentPostRunE never calls runPostRunAutoBackup. +// +// Cannot be parallel: mutates the proxiedServerMode global, primeHasGitRemote +// and env vars. +func TestBackupAutoStatusNote(t *testing.T) { + tests := []struct { + name string + envVal string // "\x00" = not set, otherwise explicit backup.enabled + hasRemote bool + sharedServer bool // BEADS_DOLT_SHARED_SERVER=1 → usesSQLServer() true + proxied bool // proxiedServerMode → usesProxiedServer() true + wantEnabled bool + wantNote string + }{ + { + name: "embedded default + git remote → unchanged", + envVal: "\x00", + hasRemote: true, + wantEnabled: true, + wantNote: "auto: git remote detected", + }, + { + name: "embedded default + no git remote → unchanged", + envVal: "\x00", + hasRemote: false, + wantEnabled: false, + wantNote: "auto: no git remote", + }, + { + name: "embedded explicit → no note, the source explains it", + envVal: "true", + hasRemote: false, + wantEnabled: true, + wantNote: "", + }, + { + // Pre-fix this case printed "auto: no git remote" even with a + // remote configured: server mode, not the remote, is the reason. + name: "sql-server default + git remote → server mode, not the remote", + envVal: "\x00", + hasRemote: true, + sharedServer: true, + wantEnabled: false, + wantNote: "auto: off in sql-server mode", + }, + { + // Non-proxied server mode keeps its post-run hook, so an explicit + // opt-in really does work there and needs no caveat. + name: "sql-server explicit opt-in → no note", + envVal: "true", + hasRemote: false, + sharedServer: true, + wantEnabled: true, + wantNote: "", + }, + { + name: "proxied default → names the missing hook, not the remote", + envVal: "\x00", + hasRemote: true, + proxied: true, + wantEnabled: false, + wantNote: "auto-backup does not run on proxied-server; use 'bd backup sync'", + }, + { + // The dead opt-in: enabled=true with nothing to run it. The note + // must survive an explicit source here, or status reports a bare + // "enabled=true" and the operator believes backups are happening. + name: "proxied explicit opt-in → still inert, still annotated", + envVal: "true", + hasRemote: false, + proxied: true, + wantEnabled: true, + wantNote: "auto-backup does not run on proxied-server; use 'bd backup sync'", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + // Same isolation as TestIsBackupAutoEnabled: a leaked BEADS_DIR or + // a cwd inside the repo tree lets Initialize() load an ambient + // config.yaml whose explicit backup.enabled wins over the computed + // default these cases assert on (be-yjp4z). + t.Chdir(t.TempDir()) + t.Setenv("BEADS_DIR", "") + t.Setenv("BEADS_TEST_IGNORE_REPO_CONFIG", "1") + + orig := primeHasGitRemote + primeHasGitRemote = func() bool { return tt.hasRemote } + t.Cleanup(func() { primeHasGitRemote = orig }) + + origProxied := proxiedServerMode + proxiedServerMode = tt.proxied + t.Cleanup(func() { proxiedServerMode = origProxied }) + + if tt.sharedServer { + t.Setenv("BEADS_DOLT_SHARED_SERVER", "1") + } else { + os.Unsetenv("BEADS_DOLT_SHARED_SERVER") + t.Cleanup(func() { os.Unsetenv("BEADS_DOLT_SHARED_SERVER") }) + } + + if tt.envVal == "\x00" { + os.Unsetenv("BD_BACKUP_ENABLED") + t.Cleanup(func() { os.Unsetenv("BD_BACKUP_ENABLED") }) + } else { + t.Setenv("BD_BACKUP_ENABLED", tt.envVal) + } + + config.ResetForTesting() + t.Cleanup(func() { config.ResetForTesting() }) + if err := config.Initialize(); err != nil { + t.Fatalf("config.Initialize: %v", err) + } + + // Take the value the way backup status does, so the note is pinned + // against the number it is printed beside rather than a literal. + enabled := isBackupAutoEnabled() + if enabled != tt.wantEnabled { + t.Fatalf("isBackupAutoEnabled() = %v, want %v", enabled, tt.wantEnabled) + } + if got := backupAutoStatusNote(enabled); got != tt.wantNote { + t.Errorf("backupAutoStatusNote(%v) = %q, want %q", enabled, got, tt.wantNote) + } + }) + } +} + // TestClientServerShareFilesystem_GH3523 pins the gating logic that // suppresses auto-backup's file:// register when the Dolt server // runs on a different filesystem from the client. Pre-fix every From 7c2568c3234443b4d56f7617e2d4f278fdb9cf7a Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 28 Sep 2026 00:57:24 -0700 Subject: [PATCH 14/45] feat(purge): live-dependent protection, --wisps-plane selector, hour-precision --older-than, --limit (#6883) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(purge): live-dependent protection, --wisps-plane selector, hour-precision --older-than An orchestrator's wisp retention step deletes every closed row in the wisps table older than N hours, except a closed wisp that an open/hooked/in_progress wisp depends on through parent-child, tracks or blocks. `bd purge` could not express that: it selected the ephemeral TIER (so closed --no-history beads, which live in the wisps table, were left behind), it deleted a closed molecule root whose step was still open, and it floored --older-than hours to whole days (36h swept rows 24h old). All three land together so `bd purge --wisps-plane --older-than 168h --force` covers that step. Live-dependent protection (issueops.SweepRequest.ProtectLiveDependents, SweepSkips.LiveDependent): a candidate is held back when a live row depends on it through a SweepLiveDependentEdgeTypes edge (parent-child, tracks, blocks), read from both dependency tables. "Live" is the same not-done set the reference scan uses, custom statuses included, so it is a superset of open/hooked/in_progress. `bd purge` always asks for it (no opt-out: removing live work's root is `bd delete`'s job); `bd prune` does not. Reported as live_dependent_skipped in --json. Wisps-plane selector (issueops.SweepWispsPlane, `--wisps-plane`): the candidates are every closed row in the wisps TABLE, whatever its ephemeral/no_history/wisp_type values, via a new single-plane search (issueops.SearchWispsPlaneInTx) that the store backends call directly and the unit-of-work provider reaches through IssueUseCase.SearchWispsPlane, so embedded, server and proxied share one body. It reaches durable-tier (no-history) rows, so it carries the durable tier's require-a-filter gate. --older-than (purge and prune): day values keep their meaning (7, 7d, 2w); anything else is a Go duration taken exactly (36h, 168h, 90m). Tests: three new Sweeper contract cases (wisps plane clears the whole table, wisps plane requires a filter, protects live dependents) run on dolt, embeddeddolt and the uow provider; workapi unit tests; a CLI retention scenario run in embedded, shared proxied and managed-local proxied modes (fails against origin/main); TestParseOlderThan. Co-Authored-By: Claude Opus 5.5 * test(uow): classify IssueUseCase.SearchWispsPlane as a read in the notifying parity partition TestRecordingIssueUseCaseCoversItsSurface requires every IssueUseCase method to be either wrapped by the notifying decorator or exempted with the reason it fires no hook. SearchWispsPlane reads state and changes none, like SearchIssues beside it. Co-Authored-By: Claude Opus 5.5 * fix(purge): refuse out-of-range --older-than; add --limit for bounded drains; cross-plane protection test Review follow-ups on #6883. --older-than: a day count past what a time.Duration holds (213504d, 30501w) used to wrap silently — 213504d became 25m26s, which on a destructive command selected almost every closed row. It is now refused as out of range; ParseDuration already refused oversized h/m/s values. --limit N (issueops.SweepRequest.Limit, SweepResult.Remaining): one purge deletes at most N rows, oldest closed_at first (then id), after every protection, and reports the rest. --json adds remaining and has_more when --limit is given. At ~15ms/row on the proxied path a single unbounded sweep of a large backlog can outlive a caller's deadline, roll back, and never clear; a bounded loop cannot. Negative limits are ErrValidation at the role and refused at the CLI. Tests: TestParseOlderThan out-of-range rows; contract cases LimitTakesTheOldestClosedFirst and ProtectsLiveDependentsAcrossPlanes (a live issues-table bead depending on a closed wisp via parent-child, tracks and blocks keeps it) on dolt, embeddeddolt and uow; workapi units for the ordering and the negative-limit refusal; the CLI retention scenario now drains with --limit 1 and checks the overflow refusal. CHANGELOG notes that sub-day Nh values used to round UP to a day (12h is now taken exactly) and lists the protection under Changed. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 (cherry picked from commit 2ec0b2f07205bb519e67f81207a5f2a4c99b4e6b) --- .github/workflows/proxied-local-smoke.yml | 5 +- CHANGELOG.md | 38 +++ backend/conformance/role_bundle_cases.go | 5 + backend/conformance/sweeper_contract.go | 239 +++++++++++++++++- cmd/bd/prune.go | 2 +- cmd/bd/purge.go | 221 ++++++++++++---- cmd/bd/purge_test.go | 62 +++-- cmd/bd/purge_wisps_plane_integration_test.go | 207 +++++++++++++++ .../purge_wisps_plane_managed_local_test.go | 19 ++ .../storage/dolt/sweeper_contract_test.go | 25 ++ internal/storage/domain/db/issue.go | 7 + internal/storage/domain/issue.go | 16 ++ .../embeddeddolt/sweeper_contract_test.go | 25 ++ internal/storage/issueops/search.go | 29 +++ internal/storage/issueops/sweep.go | 56 +++- internal/storage/uow/notifying_parity_test.go | 1 + internal/storage/uow/sweeper.go | 82 +++++- internal/storage/uow/sweeper_contract_test.go | 29 +++ internal/workapi/sweep.go | 135 +++++++++- internal/workapi/sweep_test.go | 116 +++++++++ issueops/dependencyeditor.go | 1 + issueops/sweeper.go | 74 +++++- 22 files changed, 1299 insertions(+), 95 deletions(-) create mode 100644 cmd/bd/purge_wisps_plane_integration_test.go create mode 100644 cmd/bd/purge_wisps_plane_managed_local_test.go diff --git a/.github/workflows/proxied-local-smoke.yml b/.github/workflows/proxied-local-smoke.yml index e19f4f8dd..263e182e4 100644 --- a/.github/workflows/proxied-local-smoke.yml +++ b/.github/workflows/proxied-local-smoke.yml @@ -23,6 +23,7 @@ on: - 'cmd/bd/proxied_server*.go' - 'cmd/bd/dolt*.go' - 'cmd/bd/*_proxied_server.go' + - 'cmd/bd/purge*.go' - 'cmd/bd/db_proxy_child.go' - 'cmd/bd/init*.go' - 'cmd/bd/proxied_integration_helpers_test.go' @@ -42,6 +43,7 @@ on: - 'cmd/bd/proxied_server*.go' - 'cmd/bd/dolt*.go' - 'cmd/bd/*_proxied_server.go' + - 'cmd/bd/purge*.go' - 'cmd/bd/db_proxy_child.go' - 'cmd/bd/init*.go' - 'cmd/bd/proxied_integration_helpers_test.go' @@ -110,7 +112,8 @@ jobs: TestManagedLocalProxiedDoltStatusReportsLiveTopology \ TestManagedLocalProxiedDoltStartRefusesOverLiveProxy \ TestManagedLocalProxiedDoltStartRefusesWithProxyDown \ - TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone + TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone \ + TestManagedLocalProxiedPurgeWispsPlaneRetention do if ! grep -qx "$test_name" <<<"$listed"; then echo "managed-local lifecycle test $test_name is missing; got: '$listed'" >&2 diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e4d27cf0..e188daf97 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 server topology. Wiring an auto-backup hook through the new routes is left to a later slice. +- **`bd purge` covers an orchestrator's wisp retention sweep.** Three changes + that together let `bd purge --wisps-plane --older-than 168h --force` replace + a raw `DELETE FROM wisps` retention step: + - **Live-dependent protection, always on.** `bd purge` no longer deletes a + closed bead that a live bead depends on through a `parent-child`, + `tracks` or `blocks` edge — a closed molecule root whose step is still + open, a closed wisp a live convoy tracks, a closed blocker of live work. + "Live" is any status that is not done, including custom statuses. The + held-back count is reported as `live_dependent_skipped` in `--json` (and + on its own line in text output). The protection is on the role + (`issueops.SweepRequest.ProtectLiveDependents`); `bd prune` does not ask + for it. + - **`--wisps-plane`** selects every closed row stored in the wisps table, + including `--no-history` beads, which the default ephemeral selection + leaves to `bd prune`. It works on embedded, server and proxied + workspaces, and because it reaches durable-tier rows it requires + `--older-than` or `--pattern`, like `bd prune` + (`issueops.SweepWispsPlane`). + - **`--older-than` accepts hour and finer durations** (`36h`, `168h`, + `90m`, down to `1s`) on both `bd purge` and `bd prune`. Day values (`7`, + `7d`, `2w`) are unchanged. An `Nh` value used to be converted to whole + days: floored above a day (`36h` swept rows only 24 hours old) and + rounded UP to one day below it (`12h` kept everything younger than 24 + hours). Both are now taken exactly, so `bd prune --older-than 12h` now + deletes rows closed 12–24 hours ago that it used to keep. A value too + large to represent (e.g. `213504d`) is now refused instead of silently + wrapping to a tiny age. + - **`--limit N`** caps one `bd purge` run at N beads, oldest-closed first + (`issueops.SweepRequest.Limit`), so a large backlog drains in bounded + transactions; `--json` then adds `remaining` and `has_more`. Loop while + `has_more` is true. + ### Changed - **Proxied-server refusals now say *why* they refuse.** The JSON a refused @@ -65,6 +97,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 instead of the message followed by `Error: exit code 1` and the full usage text. Exit statuses are unchanged; this applies on every topology. +- **`bd purge` keeps closed beads a live bead depends on.** A closed bead + that a not-done bead depends on through `parent-child`, `tracks` or + `blocks` is no longer purged (reported as `live_dependent_skipped`), so a + purge that used to delete a closed molecule root under a live step now + leaves it. See the `--wisps-plane` entry under Added. + ## [1.3.1] - 2026-09-16 ### Changed diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index 2169bbe6a..e2dee40d8 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -637,6 +637,11 @@ var roleContractCases = []roleContract{ RunSweeperEmptyMatchIsZeroAndNil, RunSweeperRecordsExactlyOneHistoryEntry, RunSweeperDoesNotMutateTheCallerRequest, + RunSweeperWispsPlaneClearsTheWholeWispsTable, + RunSweeperWispsPlaneRequiresAFilter, + RunSweeperProtectsLiveDependents, + RunSweeperProtectsLiveDependentsAcrossPlanes, + RunSweeperLimitTakesTheOldestClosedFirst, ), roleCases("TreeWalker", "TreeWalker()", oncePerRole, diff --git a/backend/conformance/sweeper_contract.go b/backend/conformance/sweeper_contract.go index db70d027b..bae89c58d 100644 --- a/backend/conformance/sweeper_contract.go +++ b/backend/conformance/sweeper_contract.go @@ -89,6 +89,11 @@ type SweeperFixture struct { // by hand. A nil Exec means "this backend cannot seed raw rows", and the // case that needs it SKIPS with that reason rather than passing quietly. Exec func(ctx context.Context, statements []SQLStatement) error + // AddDependencies wires edges between seeded rows, filled from the + // backend's DependencyEditor role so each adapter routes an edge to the + // dependency table of its source's plane itself. The live-dependent case + // needs it; a nil hook SKIPS that case with the reason. + AddDependencies func(ctx context.Context, req publicops.AddDependenciesRequest) error } // sweeperClosedAt is the stamp every seeded candidate carries, and @@ -345,7 +350,7 @@ func RunSweeperDryRunChangesNothing(t *testing.T, ctx context.Context, fixture S IDPattern: sweeperPattern(fixture, "dry"), } - preview := sweeperSweep(t, ctx, fixture, sweeperWithDryRun(request, true)) + preview := sweeperSweep(t, ctx, fixture, sweeperAsDryRun(request)) if !preview.DryRun { t.Error("preview.DryRun = false; the result must echo the request it answered") } @@ -356,7 +361,7 @@ func RunSweeperDryRunChangesNothing(t *testing.T, ctx context.Context, fixture S // A second preview: a dry run that had deleted anything reports a smaller // number here. - if again := sweeperSweep(t, ctx, fixture, sweeperWithDryRun(request, true)); again.Swept != preview.Swept { + if again := sweeperSweep(t, ctx, fixture, sweeperAsDryRun(request)); again.Swept != preview.Swept { t.Fatalf("second preview Swept = %d, first = %d: the first preview changed the store", again.Swept, preview.Swept) } @@ -556,7 +561,7 @@ func RunSweeperRecordsExactlyOneHistoryEntry(t *testing.T, ctx context.Context, } before := sweeperHistory(t, ctx, fixture) - sweeperSweep(t, ctx, fixture, sweeperWithDryRun(request, true)) + sweeperSweep(t, ctx, fixture, sweeperAsDryRun(request)) if after := sweeperHistory(t, ctx, fixture); after != before { t.Errorf("history went %d -> %d across a DRY RUN, want no entry at all", before, after) } @@ -609,6 +614,230 @@ func RunSweeperDoesNotMutateTheCallerRequest(t *testing.T, ctx context.Context, } } +// RunSweeperWispsPlaneClearsTheWholeWispsTable pins issueops.SweepWispsPlane: +// the selection is the wisps TABLE, so a closed no-history bead — durable-tier, +// and left alone by SweepEphemeral (RunSweeperLeavesNoHistoryBeadsToTheDurableTier) +// — goes with the closed ephemeral wisp, while a closed issues-plane row the +// same pattern admits stays. +func RunSweeperWispsPlaneClearsTheWholeWispsTable(t *testing.T, ctx context.Context, fixture SweeperFixture) { + t.Helper() + wisp := sweeperSeedClosedIssue(t, ctx, fixture, "plane", true) + durable := sweeperSeedClosedIssue(t, ctx, fixture, "plane", false) + noHistory := sweeperIssue(fixture, "plane", "nohist", false) + noHistory.NoHistory = true + if err := fixture.CreateWisp(ctx, noHistory, "sweeper-seed"); err != nil { + t.Fatalf("seeding %s: %v", noHistory.ID, err) + } + pinned := sweeperSeed(t, ctx, fixture, sweeperIssue(fixture, "plane", "pinned", true), func(issue *types.Issue) { + issue.Pinned = true + }) + + request := publicops.SweepRequest{ + Tier: publicops.SweepWispsPlane, + IDPattern: sweeperPattern(fixture, "plane"), + ClosedBefore: &sweeperCutoff, + } + preview := sweeperSweep(t, ctx, fixture, sweeperAsDryRun(request)) + if preview.Swept != 2 { + t.Fatalf("dry-run Swept = %d, want 2 (the ephemeral wisp and the no-history bead)", preview.Swept) + } + sweeperAssertWispRows(t, ctx, fixture, 3, wisp, noHistory.ID, pinned) + + result := sweeperSweep(t, ctx, fixture, request) + if result.Swept != 2 { + t.Errorf("Swept = %d, want 2 (the ephemeral wisp and the no-history bead)", result.Swept) + } + if result.Skipped.Pinned != 1 { + t.Errorf("Skipped.Pinned = %d, want 1 — the plane selector does not override pinning", result.Skipped.Pinned) + } + sweeperAssertWispRows(t, ctx, fixture, 0, wisp, noHistory.ID) + sweeperAssertWispRows(t, ctx, fixture, 1, pinned) + sweeperAssertIssueRows(t, ctx, fixture, 1, durable) +} + +// RunSweeperWispsPlaneRequiresAFilter pins that the wisps-plane tier carries +// the durable tier's require-a-filter gate: it reaches no-history beads, which +// are durable-tier. The refusal is asserted with its effect, as the durable +// case does. +func RunSweeperWispsPlaneRequiresAFilter(t *testing.T, ctx context.Context, fixture SweeperFixture) { + t.Helper() + id := sweeperSeedClosedIssue(t, ctx, fixture, "planegate", true) + + result, err := fixture.Sweeper.Sweep(ctx, publicops.SweepRequest{Tier: publicops.SweepWispsPlane}) + if !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("Sweep(wisps-plane, no cutoff, no pattern) error = %v, want ErrValidation", err) + } + if result.Swept != 0 { + t.Errorf("refused sweep reported Swept = %d, want 0", result.Swept) + } + sweeperAssertWispRows(t, ctx, fixture, 1, id) +} + +// RunSweeperProtectsLiveDependents pins issueops.SweepRequest.ProtectLiveDependents: +// a closed row a LIVE row depends on through parent-child, tracks or blocks is +// held back and counted, while a closed row reached only by a non-structural +// edge, or only by a dependent that is itself closed, is swept. +// +// The live sources carry three different not-done statuses (open, +// in_progress, hooked), because "live" is the not-done set and not "open". +// It runs on the wisps-plane tier with a cutoff — the request an +// orchestrator's retention sweep sends — after proving on a dry run that the +// same rows ARE candidates without the protection. +func RunSweeperProtectsLiveDependents(t *testing.T, ctx context.Context, fixture SweeperFixture) { + t.Helper() + if fixture.AddDependencies == nil { + t.Skip("fixture has no AddDependencies hook; cannot wire the edges this case needs") + } + live := func(name string, status types.Status) string { + issue := sweeperIssue(fixture, "live", name, true) + issue.Status = status + issue.ClosedAt = nil + return sweeperSeed(t, ctx, fixture, issue, nil) + } + closed := func(name string) string { + return sweeperSeed(t, ctx, fixture, sweeperIssue(fixture, "live", name, true), nil) + } + + parent := closed("parent") + openChild := live("child-open", types.StatusOpen) + tracked := closed("tracked") + convoy := live("convoy", types.StatusInProgress) + blocker := closed("blocker") + hooked := live("hooked", types.StatusHooked) + related := closed("related") + relater := live("relater", types.StatusOpen) + doneParent := closed("done-parent") + doneChild := closed("done-child") + + if err := fixture.AddDependencies(ctx, publicops.AddDependenciesRequest{ + Actor: "sweeper-seed", + Edges: []publicops.DependencyEdge{ + {IssueID: openChild, DependsOnID: parent, Type: publicops.DepParentChild}, + {IssueID: convoy, DependsOnID: tracked, Type: publicops.DepTracks}, + {IssueID: hooked, DependsOnID: blocker, Type: publicops.DepBlocks}, + {IssueID: relater, DependsOnID: related, Type: publicops.DepRelated}, + {IssueID: doneChild, DependsOnID: doneParent, Type: publicops.DepParentChild}, + }, + }); err != nil { + t.Fatalf("wiring dependencies: %v", err) + } + + request := publicops.SweepRequest{ + Tier: publicops.SweepWispsPlane, + IDPattern: sweeperPattern(fixture, "live"), + ClosedBefore: &sweeperCutoff, + } + unprotected := sweeperSweep(t, ctx, fixture, sweeperAsDryRun(request)) + if unprotected.Swept != 6 || unprotected.Skipped.LiveDependent != 0 { + t.Fatalf("unprotected dry run: Swept = %d, LiveDependent = %d; want 6 and 0 — "+ + "the protection must be asked for", unprotected.Swept, unprotected.Skipped.LiveDependent) + } + + request.ProtectLiveDependents = true + result := sweeperSweep(t, ctx, fixture, request) + if result.Skipped.LiveDependent != 3 { + t.Errorf("Skipped.LiveDependent = %d, want 3 (the parent, the tracked bead and the blocker)", + result.Skipped.LiveDependent) + } + if result.Swept != 3 { + t.Errorf("Swept = %d, want 3 (the related-only bead and the closed parent/child pair)", result.Swept) + } + sweeperAssertWispRows(t, ctx, fixture, 3, parent, tracked, blocker) + sweeperAssertWispRows(t, ctx, fixture, 0, related, doneParent, doneChild) + sweeperAssertWispRows(t, ctx, fixture, 4, openChild, convoy, hooked, relater) +} + +// RunSweeperProtectsLiveDependentsAcrossPlanes pins that the live-dependent +// protection reads BOTH dependency tables: a closed wisp that a live DURABLE +// (issues-table) bead depends on through parent-child, tracks or blocks is +// kept, whatever not-done status the durable source carries. +func RunSweeperProtectsLiveDependentsAcrossPlanes(t *testing.T, ctx context.Context, fixture SweeperFixture) { + t.Helper() + if fixture.AddDependencies == nil { + t.Skip("fixture has no AddDependencies hook; cannot wire the edges this case needs") + } + durableLive := func(name string, status types.Status) string { + issue := sweeperIssue(fixture, "xplane", name, false) + issue.Status = status + issue.ClosedAt = nil + return sweeperSeed(t, ctx, fixture, issue, nil) + } + closedWisp := func(name string) string { + return sweeperSeed(t, ctx, fixture, sweeperIssue(fixture, "xplane", name, true), nil) + } + + parent := closedWisp("parent") + tracked := closedWisp("tracked") + blocker := closedWisp("blocker") + loose := closedWisp("loose") + child := durableLive("child", types.StatusOpen) + convoy := durableLive("convoy", types.StatusInProgress) + waiter := durableLive("waiter", types.StatusDeferred) + + if err := fixture.AddDependencies(ctx, publicops.AddDependenciesRequest{ + Actor: "sweeper-seed", + Edges: []publicops.DependencyEdge{ + {IssueID: child, DependsOnID: parent, Type: publicops.DepParentChild}, + {IssueID: convoy, DependsOnID: tracked, Type: publicops.DepTracks}, + {IssueID: waiter, DependsOnID: blocker, Type: publicops.DepBlocks}, + }, + }); err != nil { + t.Fatalf("wiring cross-plane dependencies: %v", err) + } + + result := sweeperSweep(t, ctx, fixture, publicops.SweepRequest{ + Tier: publicops.SweepWispsPlane, + IDPattern: sweeperPattern(fixture, "xplane"), + ClosedBefore: &sweeperCutoff, + ProtectLiveDependents: true, + }) + if result.Skipped.LiveDependent != 3 || result.Swept != 1 { + t.Errorf("Swept = %d, LiveDependent = %d; want 1 (the unlinked wisp) and 3", result.Swept, result.Skipped.LiveDependent) + } + sweeperAssertWispRows(t, ctx, fixture, 3, parent, tracked, blocker) + sweeperAssertWispRows(t, ctx, fixture, 0, loose) + sweeperAssertIssueRows(t, ctx, fixture, 3, child, convoy, waiter) +} + +// RunSweeperLimitTakesTheOldestClosedFirst pins issueops.SweepRequest.Limit: +// one call deletes at most Limit rows, oldest closed_at first, and reports the +// rest as Remaining; the next call picks up where it stopped. +func RunSweeperLimitTakesTheOldestClosedFirst(t *testing.T, ctx context.Context, fixture SweeperFixture) { + t.Helper() + seed := func(name string, closed time.Time) string { + return sweeperSeed(t, ctx, fixture, sweeperIssue(fixture, "limit", name, true), func(issue *types.Issue) { + issue.ClosedAt = &closed + }) + } + newest := seed("newest", sweeperClosedAt) + middle := seed("middle", sweeperClosedAt.Add(-24*time.Hour)) + oldest := seed("oldest", sweeperClosedAt.Add(-48*time.Hour)) + + request := publicops.SweepRequest{ + Tier: publicops.SweepEphemeral, + IDPattern: sweeperPattern(fixture, "limit"), + Limit: 2, + } + first := sweeperSweep(t, ctx, fixture, request) + if first.Swept != 2 || first.Remaining != 1 { + t.Fatalf("first limited sweep: Swept = %d, Remaining = %d; want 2 and 1", first.Swept, first.Remaining) + } + sweeperAssertWispRows(t, ctx, fixture, 0, oldest, middle) + sweeperAssertWispRows(t, ctx, fixture, 1, newest) + + second := sweeperSweep(t, ctx, fixture, request) + if second.Swept != 1 || second.Remaining != 0 { + t.Fatalf("second limited sweep: Swept = %d, Remaining = %d; want 1 and 0", second.Swept, second.Remaining) + } + sweeperAssertWispRows(t, ctx, fixture, 0, newest) + + if _, err := fixture.Sweeper.Sweep(ctx, publicops.SweepRequest{ + Tier: publicops.SweepEphemeral, IDPattern: sweeperPattern(fixture, "limit"), Limit: -1, + }); !errors.Is(err, publicops.ErrValidation) { + t.Fatalf("negative Limit error = %v, want ErrValidation", err) + } +} + // --- fixture helpers ------------------------------------------------------- // sweeperPattern is the glob that scopes one case's ids: a sweep is asked of a @@ -664,8 +893,8 @@ func sweeperSeedOpen(t *testing.T, ctx context.Context, fixture SweeperFixture, return sweeperSeed(t, ctx, fixture, issue, nil) } -func sweeperWithDryRun(request publicops.SweepRequest, dryRun bool) publicops.SweepRequest { - request.DryRun = dryRun +func sweeperAsDryRun(request publicops.SweepRequest) publicops.SweepRequest { + request.DryRun = true return request } diff --git a/cmd/bd/prune.go b/cmd/bd/prune.go index 510a62101..e230e199c 100644 --- a/cmd/bd/prune.go +++ b/cmd/bd/prune.go @@ -73,7 +73,7 @@ func init() { pruneCmd.Flags().BoolP("force", "f", false, "Actually prune (without this, shows preview)") pruneCmd.Flags().Bool("ignore-references", false, "Delete closed beads even when referenced by open beads (use with care; see --help for details)") pruneCmd.Flags().Bool("dry-run", false, "Preview what would be pruned with stats") - pruneCmd.Flags().String("older-than", "", "Only prune beads closed more than N ago (e.g., 30d, 2w, 60)") + pruneCmd.Flags().String("older-than", "", "Only prune beads closed more than N ago (e.g., 30d, 2w, 60, 36h)") pruneCmd.Flags().String("pattern", "", "Only prune beads matching ID glob pattern (e.g., 'gm-old-*')") rootCmd.AddCommand(pruneCmd) } diff --git a/cmd/bd/purge.go b/cmd/bd/purge.go index 0445e5ee7..0337db793 100644 --- a/cmd/bd/purge.go +++ b/cmd/bd/purge.go @@ -2,6 +2,7 @@ package main import ( "fmt" + "math" "strconv" "strings" "time" @@ -36,6 +37,12 @@ type purgeScope struct { // is not done. `bd prune` asks unless --ignore-references; `bd purge` // never does, because a wisp's citations are as transient as the wisp. protectReferenced bool + // protectLiveDependents asks the role to skip candidates a live bead + // depends on through a parent-child, tracks or blocks edge. `bd purge` + // always asks: deleting a closed wisp whose child, tracker or blocked bead + // is still live cuts that live work loose from its root, and there is no + // retention reason to want that — a deliberate removal is `bd delete`. + protectLiveDependents bool // reportsReferences publishes the reference-skip keys under --json. It is // separate from protectReferenced because `bd prune --ignore-references` // still publishes them, as zeroes, which is the shipped shape. @@ -52,7 +59,21 @@ Closed ephemeral beads (wisps, transient molecules) accumulate rapidly and have no value once closed. This command removes them to reclaim storage. Deletes: issues, dependencies, labels, events, and comments for matching beads. -Skips: pinned beads (protected). +Skips: pinned beads, and closed beads a live bead still depends on through a +parent-child, tracks or blocks edge (a closed molecule root whose step is open, +a closed bead a live convoy tracks). Live means any status that is not done. + +--wisps-plane selects by storage plane instead: every closed row stored in the +wisps table, including --no-history beads, which the default ephemeral +selection leaves to ` + "`bd prune`" + `. It reaches durable-tier rows, so it requires +--older-than or --pattern, like ` + "`bd prune`" + `. + +--limit N caps one run at N beads, oldest-closed first, so a large backlog can +be drained in bounded transactions; --json then reports "remaining" and +"has_more". Loop while has_more is true. + +--older-than takes days (7, 7d), weeks (2w) or a duration with hour precision +(36h, 168h, 90m). To delete closed non-ephemeral beads (regular tasks, features, bugs, etc.) use ` + "`bd prune`" + ` instead. @@ -65,6 +86,9 @@ EXAMPLES: bd purge --force # Delete all closed ephemeral beads bd purge --older-than 7d --force # Only purge items closed 7+ days ago bd purge --pattern "*-wisp-*" # Only purge matching ID pattern + bd purge --older-than 36h --force # Hour precision: closed 36+ hours ago + bd purge --wisps-plane --older-than 168h --force + # Every closed wisps-table row, incl. no-history bd purge --dry-run # Detailed preview with stats`, SilenceUsage: true, SilenceErrors: true, @@ -76,14 +100,20 @@ EXAMPLES: } }() - return runPurgeOrPrune(cmd, purgeScope{ - cmdName: "purge", - pastTense: "purged", - countKey: "purged_count", - dryRunCountKey: "purge_count", - subjectNoun: "closed ephemeral bead", - tier: issueops.SweepEphemeral, - }) + scope := purgeScope{ + cmdName: "purge", + pastTense: "purged", + countKey: "purged_count", + dryRunCountKey: "purge_count", + subjectNoun: "closed ephemeral bead", + tier: issueops.SweepEphemeral, + protectLiveDependents: true, + } + if wispsPlane, _ := cmd.Flags().GetBool("wisps-plane"); wispsPlane { + scope.subjectNoun = "closed wisps-plane bead" + scope.tier = issueops.SweepWispsPlane + } + return runPurgeOrPrune(cmd, scope) }, } @@ -110,6 +140,11 @@ func runPurgeOrPrune(cmd *cobra.Command, scope purgeScope) error { dryRun, _ := cmd.Flags().GetBool("dry-run") olderThan, _ := cmd.Flags().GetString("older-than") pattern, _ := cmd.Flags().GetString("pattern") + // Only `bd purge` defines --limit; GetInt on an undefined flag is 0, "no cap". + limit, _ := cmd.Flags().GetInt("limit") + if limit < 0 { + return HandleErrorRespectJSON("invalid --limit value %d: must be 0 (no limit) or positive", limit) + } // The ROLE refuses an unfiltered durable sweep — that guard is // workapi.ValidateSweepRequest, below every front door. This branch is here @@ -117,30 +152,36 @@ func runPurgeOrPrune(cmd *cobra.Command, scope purgeScope) error { // flags a person who typed `bd prune --force` has to reach for. The contract // case RunSweeperRefusesAnUnfilteredDurableSweep proves the guard survives // this branch being deleted. - if scope.tier == issueops.SweepDurable && olderThan == "" && pattern == "" { + if scope.tier != issueops.SweepEphemeral && olderThan == "" && pattern == "" { + command := scope.cmdName + if scope.tier == issueops.SweepWispsPlane { + command += " --wisps-plane" + } return HandleErrorWithHint( - fmt.Sprintf("bd %s requires --older-than or --pattern", scope.cmdName), + fmt.Sprintf("bd %s requires --older-than or --pattern", command), "Protects against accidental bulk deletion. Use `--pattern '*'` to\n"+ " include all closed beads in this scope, or `--older-than 1d`\n"+ " / `--pattern ''` to narrow the deletion.") } request := issueops.SweepRequest{ - Actor: actor, - Tier: scope.tier, - IDPattern: pattern, - ProtectReferenced: scope.protectReferenced, + Actor: actor, + Tier: scope.tier, + IDPattern: pattern, + ProtectReferenced: scope.protectReferenced, + ProtectLiveDependents: scope.protectLiveDependents, + Limit: limit, // A --dry-run and an UNCONFIRMED run ask the role the same question — // "what would this do" — so both send DryRun. --force is this // command's confirmation, not a request field. DryRun: dryRun || !force, } if olderThan != "" { - days, err := parseHumanDuration(olderThan) + age, err := parseOlderThan(olderThan) if err != nil { return HandleErrorRespectJSON("invalid --older-than value %q: %v", olderThan, err) } - cutoff := time.Now().UTC().AddDate(0, 0, -days) + cutoff := time.Now().UTC().Add(-age) request.ClosedBefore = &cutoff } @@ -165,16 +206,16 @@ func runPurgeOrPrune(cmd *cobra.Command, scope purgeScope) error { switch { case result.Swept == 0: - return emitSweepEmpty(scope, olderThan, pattern, result) + return emitSweepEmpty(scope, olderThan, pattern, limit, result) case dryRun: - return emitSweepDryRun(scope, result) + return emitSweepDryRun(scope, limit, result) case !force: return emitSweepConfirm(scope, olderThan, pattern, result) } commandDidWrite.Store(true) commandMayEmptyJSONLExport.Store(true) - return emitSweepResult(scope, result) + return emitSweepResult(scope, limit, result) } // warnSweepDefenseSkips reports the candidates the role's own recheck threw @@ -208,12 +249,40 @@ func addReferenceStats(scope purgeScope, stats map[string]interface{}, result is } } -func emitSweepEmpty(scope purgeScope, olderThan, pattern string, result issueops.SweepResult) error { +// addLimitStats reports what --limit left for the next run. It is attached +// only when the caller asked for a limit, so the unlimited shape is unchanged. +func addLimitStats(stats map[string]interface{}, limit int, result issueops.SweepResult) { + if limit <= 0 { + return + } + stats["remaining"] = result.Remaining + stats["has_more"] = result.Remaining > 0 +} + +// printLimitNote tells a text reader that --limit left rows behind. +func printLimitNote(result issueops.SweepResult) { + if result.Remaining > 0 { + fmt.Printf(" Remaining (over --limit): %d — run again to continue\n", result.Remaining) + } +} + +// addLiveDependentStats attaches the live-dependent skip count to a --json +// payload when the protection held anything back, the way pinned_skipped is +// reported. +func addLiveDependentStats(stats map[string]interface{}, result issueops.SweepResult) { + if result.Skipped.LiveDependent > 0 { + stats["live_dependent_skipped"] = result.Skipped.LiveDependent + } +} + +func emitSweepEmpty(scope purgeScope, olderThan, pattern string, limit int, result issueops.SweepResult) error { if jsonOutput { stats := map[string]interface{}{ scope.countKey: 0, "message": fmt.Sprintf("No %ss to %s", scope.subjectNoun, scope.cmdName), } + addLiveDependentStats(stats, result) + addLimitStats(stats, limit, result) addReferenceStats(scope, stats, result) return outputJSON(stats) } @@ -225,6 +294,10 @@ func emitSweepEmpty(scope purgeScope, olderThan, pattern string, result issueops msg += fmt.Sprintf(" (matching %q)", pattern) } fmt.Println(msg) + if result.Skipped.LiveDependent > 0 { + fmt.Println(ui.MutedStyle.Render(fmt.Sprintf( + " (%d closed bead(s) protected by live dependents)", result.Skipped.LiveDependent))) + } if result.Skipped.Referenced > 0 { fmt.Println(ui.MutedStyle.Render(fmt.Sprintf( " (%d closed bead(s) protected by open-bead references — use --ignore-references to override)", @@ -233,7 +306,7 @@ func emitSweepEmpty(scope purgeScope, olderThan, pattern string, result issueops return nil } -func emitSweepDryRun(scope purgeScope, result issueops.SweepResult) error { +func emitSweepDryRun(scope purgeScope, limit int, result issueops.SweepResult) error { if jsonOutput { stats := map[string]interface{}{ "dry_run": true, @@ -245,6 +318,8 @@ func emitSweepDryRun(scope purgeScope, result issueops.SweepResult) error { if result.Skipped.Pinned > 0 { stats["pinned_skipped"] = result.Skipped.Pinned } + addLiveDependentStats(stats, result) + addLimitStats(stats, limit, result) addReferenceStats(scope, stats, result) return outputJSON(stats) } @@ -255,6 +330,9 @@ func emitSweepDryRun(scope purgeScope, result issueops.SweepResult) error { if result.Skipped.Pinned > 0 { fmt.Printf(" Pinned (skipped): %d\n", result.Skipped.Pinned) } + if result.Skipped.LiveDependent > 0 { + fmt.Printf(" Live dependent (skipped): %d\n", result.Skipped.LiveDependent) + } if result.Skipped.Referenced > 0 { fmt.Printf(" %s %d\n", ui.MutedStyle.Render("Referenced (skipped):"), result.Skipped.Referenced) sample := result.ReferencedIDs @@ -271,6 +349,7 @@ func emitSweepDryRun(scope purgeScope, result issueops.SweepResult) error { } fmt.Printf(" %s %s%s\n", ui.MutedStyle.Render("Referenced IDs (sample):"), strings.Join(idStrs, ", "), suffix) } + printLimitNote(result) fmt.Printf("\n(Dry-run mode — no changes made)\n") return nil } @@ -280,10 +359,16 @@ func emitSweepConfirm(scope purgeScope, olderThan, pattern string, result issueo if result.Skipped.Pinned > 0 { fmt.Printf("Skipping %d pinned bead(s)\n", result.Skipped.Pinned) } + if result.Skipped.LiveDependent > 0 { + fmt.Printf("Skipping %d bead(s) a live bead depends on\n", result.Skipped.LiveDependent) + } if result.Skipped.Referenced > 0 { fmt.Println(ui.MutedStyle.Render(fmt.Sprintf("Skipping %d referenced bead(s)", result.Skipped.Referenced))) } hint := fmt.Sprintf("bd %s --force", scope.cmdName) + if scope.tier == issueops.SweepWispsPlane { + hint += " --wisps-plane" + } if olderThan != "" { hint += " --older-than " + olderThan } @@ -295,7 +380,7 @@ func emitSweepConfirm(scope purgeScope, olderThan, pattern string, result issueo fmt.Sprintf("Use --force to confirm or --dry-run to preview.\n %s", hint)) } -func emitSweepResult(scope purgeScope, result issueops.SweepResult) error { +func emitSweepResult(scope purgeScope, limit int, result issueops.SweepResult) error { if jsonOutput { stats := map[string]interface{}{ scope.countKey: result.Swept, @@ -306,6 +391,8 @@ func emitSweepResult(scope purgeScope, result issueops.SweepResult) error { if result.Skipped.Pinned > 0 { stats["pinned_skipped"] = result.Skipped.Pinned } + addLiveDependentStats(stats, result) + addLimitStats(stats, limit, result) addReferenceStats(scope, stats, result) return outputJSON(stats) } @@ -316,9 +403,13 @@ func emitSweepResult(scope purgeScope, result issueops.SweepResult) error { if result.Skipped.Pinned > 0 { fmt.Printf(" Pinned (skipped): %d\n", result.Skipped.Pinned) } + if result.Skipped.LiveDependent > 0 { + fmt.Printf(" Live dependent (skipped): %d\n", result.Skipped.LiveDependent) + } if result.Skipped.Referenced > 0 { fmt.Printf(" %s %d\n", ui.MutedStyle.Render("Referenced (skipped):"), result.Skipped.Referenced) } + printLimitNote(result) return nil } @@ -329,53 +420,77 @@ func capitalize(s string) string { return strings.ToUpper(s[:1]) + s[1:] } -// parseHumanDuration parses a human-friendly duration string into days. -// Accepts: "7d", "30d", "24h", "2w", or just a number (treated as days). -func parseHumanDuration(s string) (int, error) { +// parseOlderThan parses an --older-than value into an age. +// +// Day-denominated values keep the meaning they always had: a bare number is +// days ("7"), and "7d" / "2w" are days and weeks. Anything else is a Go +// duration with hour precision or finer — "36h", "168h", "90m", "1h30m" — and +// is taken exactly; "Nh" used to be floored to whole days, which made "36h" +// sweep rows only 24 hours old. The suffix letters are case-insensitive, as +// they always were. The age must be positive. +func parseOlderThan(s string) (time.Duration, error) { s = strings.TrimSpace(s) if s == "" { return 0, fmt.Errorf("empty duration") } - // Plain number = days if days, err := strconv.Atoi(s); err == nil { - if days <= 0 { - return 0, fmt.Errorf("duration must be positive") - } - return days, nil + return daysAge(int64(days)) } - // Parse suffix unit := s[len(s)-1] - numStr := s[:len(s)-1] - num, err := strconv.Atoi(numStr) - if err != nil { - return 0, fmt.Errorf("invalid number %q", numStr) + switch unit { + case 'd', 'D', 'w', 'W': + num, err := strconv.Atoi(s[:len(s)-1]) + if err != nil { + return 0, fmt.Errorf("invalid number %q", s[:len(s)-1]) + } + days := int64(num) + if unit == 'w' || unit == 'W' { + if days > maxOlderThanDays/7 || days < -maxOlderThanDays/7 { + return 0, errOlderThanRange + } + days *= 7 + } + return daysAge(days) } - if num <= 0 { + + age, err := time.ParseDuration(strings.ToLower(s)) + if err != nil { + // time.ParseDuration refuses an out-of-range value ("9999999h") the + // same way it refuses a malformed one, so neither can wrap around. + return 0, fmt.Errorf("invalid or out-of-range duration %q (use a number of days, Nd, Nw, or a duration such as 36h or 90m)", s) + } + return positiveAge(age) +} + +// maxOlderThanDays is the largest day count a time.Duration can hold. A larger +// value used to overflow silently — 213504d wrapped to 25 minutes — which on a +// destructive command selected nearly everything instead of nothing. +const maxOlderThanDays = math.MaxInt64 / int64(24*time.Hour) + +var errOlderThanRange = fmt.Errorf("duration out of range (at most %d days)", maxOlderThanDays) + +func daysAge(days int64) (time.Duration, error) { + if days > maxOlderThanDays { + return 0, errOlderThanRange + } + return positiveAge(time.Duration(days) * 24 * time.Hour) +} + +func positiveAge(age time.Duration) (time.Duration, error) { + if age <= 0 { return 0, fmt.Errorf("duration must be positive") } - - switch unit { - case 'h', 'H': - days := num / 24 - if days == 0 { - days = 1 // minimum 1 day - } - return days, nil - case 'd', 'D': - return num, nil - case 'w', 'W': - return num * 7, nil - default: - return 0, fmt.Errorf("unknown unit %q (use h, d, or w)", string(unit)) - } + return age, nil } func init() { purgeCmd.Flags().BoolP("force", "f", false, "Actually purge (without this, shows preview)") purgeCmd.Flags().Bool("dry-run", false, "Preview what would be purged with stats") - purgeCmd.Flags().String("older-than", "", "Only purge beads closed more than N ago (e.g., 7d, 2w, 30)") + purgeCmd.Flags().String("older-than", "", "Only purge beads closed more than N ago (e.g., 7d, 2w, 30, 36h)") + purgeCmd.Flags().Int("limit", 0, "Purge at most N beads this run, oldest-closed first (0 = no limit); --json reports remaining/has_more") + purgeCmd.Flags().Bool("wisps-plane", false, "Select every closed row in the wisps table, including --no-history beads (requires --older-than or --pattern)") purgeCmd.Flags().String("pattern", "", "Only purge beads matching ID glob pattern (e.g., *-wisp-*)") rootCmd.AddCommand(purgeCmd) } diff --git a/cmd/bd/purge_test.go b/cmd/bd/purge_test.go index 89d84bb1b..d44cf8221 100644 --- a/cmd/bd/purge_test.go +++ b/cmd/bd/purge_test.go @@ -1,40 +1,68 @@ package main -import "testing" +import ( + "testing" + "time" +) -func TestParseHumanDuration(t *testing.T) { +// TestParseOlderThan pins --older-than on both `bd purge` and `bd prune`. +// Day-denominated values keep their meaning; hour (and finer) values are taken +// exactly rather than floored to whole days — "36h" used to become 24h and +// sweep rows younger than the caller asked for. +func TestParseOlderThan(t *testing.T) { + const day = 24 * time.Hour tests := []struct { input string - want int + want time.Duration wantErr bool }{ - {"7", 7, false}, - {"30", 30, false}, - {"7d", 7, false}, - {"30d", 30, false}, - {"2w", 14, false}, - {"1w", 7, false}, - {"48h", 2, false}, - {"12h", 1, false}, // rounds up to 1 day minimum - {"7D", 7, false}, - {"2W", 14, false}, + // Unchanged day semantics. + {"7", 7 * day, false}, + {"30", 30 * day, false}, + {"7d", 7 * day, false}, + {"30d", 30 * day, false}, + {"2w", 14 * day, false}, + {"1w", 7 * day, false}, + {"7D", 7 * day, false}, + {"2W", 14 * day, false}, + // Hour precision, no longer floored to days. + {"48h", 48 * time.Hour, false}, + {"168h", 7 * day, false}, + {"36h", 36 * time.Hour, false}, + {"12h", 12 * time.Hour, false}, + {"36H", 36 * time.Hour, false}, + {"90m", 90 * time.Minute, false}, + {"1h30m", 90 * time.Minute, false}, + // Refusals. {"", 0, true}, {"0", 0, true}, {"-1", 0, true}, {"0d", 0, true}, + {"0h", 0, true}, + {"-36h", 0, true}, {"abc", 0, true}, {"7x", 0, true}, + {"d", 0, true}, + // Out of range must be refused, never wrapped: 213504d used to + // overflow to 25m26s and select almost every closed row. + {"213504d", 0, true}, + {"213504", 0, true}, + {"30501w", 0, true}, + {"106752d", 0, true}, + {"9223372036854775807d", 0, true}, + {"99999999999h", 0, true}, + // The largest representable day count still parses. + {"106751d", 106751 * day, false}, } for _, tt := range tests { t.Run(tt.input, func(t *testing.T) { - got, err := parseHumanDuration(tt.input) + got, err := parseOlderThan(tt.input) if (err != nil) != tt.wantErr { - t.Errorf("parseHumanDuration(%q) error = %v, wantErr %v", tt.input, err, tt.wantErr) - return + t.Fatalf("parseOlderThan(%q) error = %v, wantErr %v", tt.input, err, tt.wantErr) } if got != tt.want { - t.Errorf("parseHumanDuration(%q) = %d, want %d", tt.input, got, tt.want) + t.Errorf("parseOlderThan(%q) = %s, want %s", tt.input, got, tt.want) } }) } diff --git a/cmd/bd/purge_wisps_plane_integration_test.go b/cmd/bd/purge_wisps_plane_integration_test.go new file mode 100644 index 000000000..c7147f3ee --- /dev/null +++ b/cmd/bd/purge_wisps_plane_integration_test.go @@ -0,0 +1,207 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "os" + "os/exec" + "strings" + "testing" + "time" +) + +// purgeScenarioRunner drives one storage mode for the purge retention +// scenario. run never fatals: several steps assert a refusal. +type purgeScenarioRunner struct { + mode string + run func(t *testing.T, args ...string) (stdout, stderr string, err error) +} + +func (r purgeScenarioRunner) must(t *testing.T, args ...string) string { + t.Helper() + stdout, stderr, err := r.run(t, args...) + if err != nil { + t.Fatalf("%s: bd %s failed: %v\nstdout:\n%s\nstderr:\n%s", + r.mode, strings.Join(args, " "), err, stdout, stderr) + } + return stdout +} + +func (r purgeScenarioRunner) create(t *testing.T, title string, args ...string) string { + t.Helper() + id := strings.TrimSpace(r.must(t, append([]string{"create", title, "--silent"}, args...)...)) + if id == "" { + t.Fatalf("%s: bd create %q printed no id", r.mode, title) + } + return id +} + +func (r purgeScenarioRunner) exists(t *testing.T, id string) bool { + t.Helper() + _, _, err := r.run(t, "show", id) + return err == nil +} + +// purgeJSON runs a purge that must succeed and decodes its --json object. +func (r purgeScenarioRunner) purgeJSON(t *testing.T, args ...string) map[string]any { + t.Helper() + out := r.must(t, append(append([]string{"purge"}, args...), "--json")...) + start := strings.Index(out, "{") + if start < 0 { + t.Fatalf("%s: bd purge %s --json printed no object:\n%s", r.mode, strings.Join(args, " "), out) + } + var result map[string]any + if err := json.Unmarshal([]byte(out[start:]), &result); err != nil { + t.Fatalf("%s: parse purge JSON: %v\n%s", r.mode, err, out) + } + return result +} + +func jsonCount(result map[string]any, key string) int { + n, _ := result[key].(float64) + return int(n) +} + +// runPurgeRetentionScenario is the retention sweep an orchestrator runs over +// the wisps plane — `bd purge --wisps-plane --older-than --force` — and +// the three things it has to get right: +// +// - the wisps PLANE is the selection, so a closed --no-history bead goes +// with the closed ephemeral wisp while a closed durable bead stays; +// - a closed wisp a live wisp depends on (parent-child, tracks) is kept and +// counted, on the default purge as much as the plane one; +// - --older-than has sub-day precision: rows closed seconds ago are swept +// by "1s" and kept by "36h" (which used to be floored to 24h), and an +// out-of-range value is refused rather than wrapped; +// - --limit drains a backlog in bounded runs, reporting remaining/has_more. +func runPurgeRetentionScenario(t *testing.T, r purgeScenarioRunner) { + t.Helper() + + wisp := r.create(t, "closed wisp", "--ephemeral") + noHistory := r.create(t, "closed no-history bead", "--no-history") + durable := r.create(t, "closed durable bead") + + parent := r.create(t, "closed molecule root", "--ephemeral") + child := r.create(t, "open step", "--ephemeral") + r.must(t, "dep", "add", child, parent, "--type", "parent-child") + + tracked := r.create(t, "closed tracked wisp", "--ephemeral") + convoy := r.create(t, "live convoy", "--ephemeral") + r.must(t, "dep", "add", convoy, tracked, "--type", "tracks") + r.must(t, "update", convoy, "--status", "in_progress") + + r.must(t, "close", wisp, noHistory, durable, tracked) + r.must(t, "close", parent, "--force") + + // Everything above closed within the last few seconds. closed_at is + // stored at second precision, so wait comfortably past one second before + // asking "--older-than 1s" for it. + time.Sleep(2500 * time.Millisecond) + + // The plane selector reaches durable-tier rows, so it takes the durable + // tier's require-a-filter gate. + if stdout, stderr, err := r.run(t, "purge", "--wisps-plane", "--force"); err == nil || + !strings.Contains(stdout+stderr, "requires --older-than or --pattern") { + t.Fatalf("%s: unfiltered purge --wisps-plane: err=%v\n%s%s", r.mode, err, stdout, stderr) + } + + // Hour precision: nothing here is 36 hours old. + if got := r.purgeJSON(t, "--wisps-plane", "--older-than", "36h", "--dry-run"); jsonCount(got, "purge_count") != 0 { + t.Fatalf("%s: --older-than 36h selected rows closed seconds ago: %v", r.mode, got) + } + + preview := r.purgeJSON(t, "--wisps-plane", "--older-than", "1s", "--dry-run") + if jsonCount(preview, "purge_count") != 2 || jsonCount(preview, "live_dependent_skipped") != 2 { + t.Fatalf("%s: dry run = %v; want purge_count 2 (wisp + no-history) and live_dependent_skipped 2 (root + tracked)", + r.mode, preview) + } + + // --limit drains in bounded runs and says whether more remain. + first := r.purgeJSON(t, "--wisps-plane", "--older-than", "1s", "--limit", "1", "--force") + if jsonCount(first, "purged_count") != 1 || jsonCount(first, "remaining") != 1 || first["has_more"] != true { + t.Fatalf("%s: purge --limit 1 = %v; want purged_count 1, remaining 1, has_more true", r.mode, first) + } + result := r.purgeJSON(t, "--wisps-plane", "--older-than", "1s", "--limit", "1", "--force") + if jsonCount(result, "purged_count") != 1 || jsonCount(result, "live_dependent_skipped") != 2 || + jsonCount(result, "remaining") != 0 || result["has_more"] != false { + t.Fatalf("%s: second purge --limit 1 = %v; want purged_count 1, live_dependent_skipped 2, remaining 0, has_more false", + r.mode, result) + } + if stdout, stderr, err := r.run(t, "purge", "--limit", "-1", "--force"); err == nil { + t.Fatalf("%s: purge --limit -1 succeeded:\n%s%s", r.mode, stdout, stderr) + } + if stdout, stderr, err := r.run(t, "purge", "--older-than", "213504d", "--force"); err == nil || + !strings.Contains(stdout+stderr, "out of range") { + t.Fatalf("%s: purge --older-than 213504d must refuse as out of range: err=%v\n%s%s", r.mode, err, stdout, stderr) + } + for id, want := range map[string]bool{ + wisp: false, noHistory: false, + parent: true, child: true, tracked: true, convoy: true, durable: true, + } { + if got := r.exists(t, id); got != want { + t.Errorf("%s: after purge --wisps-plane, %s exists = %v, want %v", r.mode, id, got, want) + } + } + + // The default (ephemeral-tier) purge protects live dependents too, and + // still leaves no-history beads alone. + root2 := r.create(t, "second closed root", "--ephemeral") + step2 := r.create(t, "second open step", "--ephemeral") + r.must(t, "dep", "add", step2, root2, "--type", "parent-child") + r.must(t, "close", root2, "--force") + noHistory2 := r.create(t, "second no-history bead", "--no-history") + r.must(t, "close", noHistory2) + plain := r.purgeJSON(t, "--force") + if jsonCount(plain, "live_dependent_skipped") < 1 { + t.Errorf("%s: default purge reported no live-dependent skip: %v", r.mode, plain) + } + if !r.exists(t, root2) { + t.Errorf("%s: default purge deleted %s, whose step %s is still open", r.mode, root2, step2) + } + if !r.exists(t, noHistory2) { + t.Errorf("%s: default purge deleted no-history bead %s; only --wisps-plane selects it", r.mode, noHistory2) + } +} + +func embeddedPurgeScenarioRunner(bd, dir string) purgeScenarioRunner { + return purgeScenarioRunner{ + mode: "embedded", + run: func(t *testing.T, args ...string) (string, string, error) { + t.Helper() + cmd := exec.Command(bd, args...) + cmd.Dir = dir + cmd.Env = bdEnv(dir) + stdout, stderr, err := runCommandBuffers(t, cmd) + return stdout.String(), stderr.String(), err + }, + } +} + +func proxiedPurgeScenarioRunner(bd string, p proxiedProject) purgeScenarioRunner { + return purgeScenarioRunner{ + mode: "proxied", + run: func(t *testing.T, args ...string) (string, string, error) { + t.Helper() + return bdProxiedRunBuffers(t, bd, p.dir, args...) + }, + } +} + +func TestEmbeddedPurgeWispsPlaneRetention(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + bd := buildEmbeddedBD(t) + dir, _, _ := bdInit(t, bd, "--prefix", "pwp") + runPurgeRetentionScenario(t, embeddedPurgeScenarioRunner(bd, dir)) +} + +func TestProxiedServerPurgeWispsPlaneRetention(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + p := newSharedProxiedProject(t, bd, "pwx") + runPurgeRetentionScenario(t, proxiedPurgeScenarioRunner(bd, p)) +} diff --git a/cmd/bd/purge_wisps_plane_managed_local_test.go b/cmd/bd/purge_wisps_plane_managed_local_test.go new file mode 100644 index 000000000..0a4d6a741 --- /dev/null +++ b/cmd/bd/purge_wisps_plane_managed_local_test.go @@ -0,0 +1,19 @@ +//go:build cgo && unix + +package main + +import ( + "testing" + "time" +) + +// TestManagedLocalProxiedPurgeWispsPlaneRetention runs the purge retention +// scenario on the managed-local proxied topology, the default for a proxied +// workspace bd starts the Dolt server for. Named TestManagedLocalProxied* so +// the proxied-local smoke lane runs it. +func TestManagedLocalProxiedPurgeWispsPlaneRetention(t *testing.T) { + requireManagedLocalProxiedEnv(t) + bd := buildEmbeddedBD(t) + p := bdManagedLocalInit(t, bd, "pwl", 5*time.Minute) + runPurgeRetentionScenario(t, proxiedPurgeScenarioRunner(bd, p)) +} diff --git a/internal/storage/dolt/sweeper_contract_test.go b/internal/storage/dolt/sweeper_contract_test.go index 9c025b9b0..d01d2e96b 100644 --- a/internal/storage/dolt/sweeper_contract_test.go +++ b/internal/storage/dolt/sweeper_contract_test.go @@ -35,6 +35,21 @@ func TestSweeperContract(t *testing.T) { t.Run("TreatsALegacyTypedWispAsEphemeralTier", func(t *testing.T) { conformance.RunSweeperTreatsALegacyTypedWispAsEphemeralTier(t, ctx, fixture) }) + t.Run("WispsPlaneClearsTheWholeWispsTable", func(t *testing.T) { + conformance.RunSweeperWispsPlaneClearsTheWholeWispsTable(t, ctx, fixture) + }) + t.Run("WispsPlaneRequiresAFilter", func(t *testing.T) { + conformance.RunSweeperWispsPlaneRequiresAFilter(t, ctx, fixture) + }) + t.Run("ProtectsLiveDependents", func(t *testing.T) { + conformance.RunSweeperProtectsLiveDependents(t, ctx, fixture) + }) + t.Run("ProtectsLiveDependentsAcrossPlanes", func(t *testing.T) { + conformance.RunSweeperProtectsLiveDependentsAcrossPlanes(t, ctx, fixture) + }) + t.Run("LimitTakesTheOldestClosedFirst", func(t *testing.T) { + conformance.RunSweeperLimitTakesTheOldestClosedFirst(t, ctx, fixture) + }) t.Run("LeavesNoHistoryBeadsToTheDurableTier", func(t *testing.T) { conformance.RunSweeperLeavesNoHistoryBeadsToTheDurableTier(t, ctx, fixture) }) @@ -98,6 +113,16 @@ func newDoltSweeperFixture(t *testing.T, prefix string) (conformance.SweeperFixt } return nil }, + AddDependencies: func(ctx context.Context, req issueops.AddDependenciesRequest) error { + // Through the DependencyEditor ROLE, which routes each edge to its + // source plane's dependency table itself. + editor, err := store.DependencyEditor() + if err != nil { + return err + } + _, err = editor.AddDependencies(ctx, req) + return err + }, AddComment: func(ctx context.Context, issueID, author, text string) error { // Through the Commenter ROLE, which resolves the plane itself, so // the case can cite from a wisp's comment without knowing how this diff --git a/internal/storage/domain/db/issue.go b/internal/storage/domain/db/issue.go index 6905afb62..b44cdc2a1 100644 --- a/internal/storage/domain/db/issue.go +++ b/internal/storage/domain/db/issue.go @@ -933,6 +933,13 @@ func (r *issueSQLRepositoryImpl) SearchAcrossIssuesAndWispsWithCounts(ctx contex return r.searchAcrossIssuesAndWispsWithCounts(ctx, query, filter) } +// SearchWispsPlane runs the shared wisps-plane search body on this +// repository's transaction, so the unit-of-work provider and the two store +// backends read the plane through one function. +func (r *issueSQLRepositoryImpl) SearchWispsPlane(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) { + return issueops.SearchWispsPlaneInTx(ctx, r.runner, query, filter) +} + func (r *issueSQLRepositoryImpl) SearchIssueIDs(ctx context.Context, query string, filter types.IssueFilter) ([]string, error) { return issueops.SearchIssueIDsInTx(ctx, r.runner, query, filter) } diff --git a/internal/storage/domain/issue.go b/internal/storage/domain/issue.go index cadb0fd22..a8137338a 100644 --- a/internal/storage/domain/issue.go +++ b/internal/storage/domain/issue.go @@ -72,6 +72,10 @@ type IssueSQLRepository interface { SearchAcrossIssuesAndWisps(ctx context.Context, query string, filter types.IssueFilter) (SearchPage, error) SearchAcrossIssuesAndWispsWithCounts(ctx context.Context, query string, filter types.IssueFilter) (SearchCountsPage, error) SearchIssueIDs(ctx context.Context, query string, filter types.IssueFilter) ([]string, error) + // SearchWispsPlane searches the wisps table ALONE, whatever each row's + // ephemeral, no_history or wisp_type values, and never the issues table + // (issueops.SearchWispsPlaneInTx). + SearchWispsPlane(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) GetReadyWork(ctx context.Context, filter types.WorkFilter) (SearchPage, error) GetReadyWorkWithCounts(ctx context.Context, filter types.WorkFilter) (SearchCountsPage, error) GetDescendants(ctx context.Context, rootID string, filter types.IssueFilter) ([]*types.Issue, error) @@ -308,6 +312,10 @@ type IssueUseCase interface { SearchIssues(ctx context.Context, query string, filter types.IssueFilter) (SearchPage, error) SearchIssuesWithCounts(ctx context.Context, query string, filter types.IssueFilter) (SearchCountsPage, error) SearchIssueIDs(ctx context.Context, query string, filter types.IssueFilter) ([]string, error) + // SearchWispsPlane searches the wisps table ALONE, whatever each row's + // ephemeral, no_history or wisp_type values, and never the issues table + // (issueops.SearchWispsPlaneInTx). + SearchWispsPlane(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) GetReadyWork(ctx context.Context, filter types.WorkFilter) (SearchPage, error) GetReadyWorkWithCounts(ctx context.Context, filter types.WorkFilter) (SearchCountsPage, error) GetDescendants(ctx context.Context, rootID string, filter types.IssueFilter) ([]*types.Issue, error) @@ -843,6 +851,14 @@ func (u *issueUseCaseImpl) SearchIssues(ctx context.Context, query string, filte return out, nil } +func (u *issueUseCaseImpl) SearchWispsPlane(ctx context.Context, query string, filter types.IssueFilter) ([]*types.Issue, error) { + out, err := u.issueRepo.SearchWispsPlane(ctx, query, filter) + if err != nil { + return nil, fmt.Errorf("SearchWispsPlane: %w", err) + } + return out, nil +} + func (u *issueUseCaseImpl) SearchIssueIDs(ctx context.Context, query string, filter types.IssueFilter) ([]string, error) { out, err := u.issueRepo.SearchIssueIDs(ctx, query, filter) if err != nil { diff --git a/internal/storage/embeddeddolt/sweeper_contract_test.go b/internal/storage/embeddeddolt/sweeper_contract_test.go index 2d221219c..7a2e550e0 100644 --- a/internal/storage/embeddeddolt/sweeper_contract_test.go +++ b/internal/storage/embeddeddolt/sweeper_contract_test.go @@ -40,6 +40,21 @@ func TestSweeperContract(t *testing.T) { t.Run("TreatsALegacyTypedWispAsEphemeralTier", func(t *testing.T) { conformance.RunSweeperTreatsALegacyTypedWispAsEphemeralTier(t, ctx, fixture) }) + t.Run("WispsPlaneClearsTheWholeWispsTable", func(t *testing.T) { + conformance.RunSweeperWispsPlaneClearsTheWholeWispsTable(t, ctx, fixture) + }) + t.Run("WispsPlaneRequiresAFilter", func(t *testing.T) { + conformance.RunSweeperWispsPlaneRequiresAFilter(t, ctx, fixture) + }) + t.Run("ProtectsLiveDependents", func(t *testing.T) { + conformance.RunSweeperProtectsLiveDependents(t, ctx, fixture) + }) + t.Run("ProtectsLiveDependentsAcrossPlanes", func(t *testing.T) { + conformance.RunSweeperProtectsLiveDependentsAcrossPlanes(t, ctx, fixture) + }) + t.Run("LimitTakesTheOldestClosedFirst", func(t *testing.T) { + conformance.RunSweeperLimitTakesTheOldestClosedFirst(t, ctx, fixture) + }) t.Run("LeavesNoHistoryBeadsToTheDurableTier", func(t *testing.T) { conformance.RunSweeperLeavesNoHistoryBeadsToTheDurableTier(t, ctx, fixture) }) @@ -99,6 +114,16 @@ func newEmbeddedSweeperFixture(t *testing.T, te *testEnv, prefix string) conform } return nil }, + AddDependencies: func(ctx context.Context, req issueops.AddDependenciesRequest) error { + // Through the DependencyEditor ROLE, which routes each edge to its + // source plane's dependency table itself. + editor, err := te.store.DependencyEditor() + if err != nil { + return err + } + _, err = editor.AddDependencies(ctx, req) + return err + }, AddComment: func(ctx context.Context, issueID, author, text string) error { // Through the Commenter ROLE, which resolves the plane itself, so // the case can cite from a wisp's comment without knowing how this diff --git a/internal/storage/issueops/search.go b/internal/storage/issueops/search.go index bd19def52..df0ff5b95 100644 --- a/internal/storage/issueops/search.go +++ b/internal/storage/issueops/search.go @@ -32,6 +32,35 @@ func SearchIssueIDsInTx(ctx context.Context, tx DBTX, query string, filter types return searchInTx(ctx, tx, query, filter, idProjection) } +// SearchWispsPlaneInTx searches the wisps plane ALONE: every row stored in +// the wisps table, whatever its ephemeral, no_history or wisp_type values, and +// never the issues table. +// +// It is not SearchIssuesInTx with Ephemeral=true. That filter adds an +// "ephemeral = 1" clause, which drops the no-history rows stored beside the +// wisps, and it falls back to the issues table when the wisps plane is empty. +// This answers "what is in the wisps table" — the unit +// issueops.SweepWispsPlane selects by — and a database with no wisps table +// answers with nothing. +func SearchWispsPlaneInTx(ctx context.Context, tx DBTX, query string, filter types.IssueFilter) ([]*types.Issue, error) { + proj := issueProjection + if filter.Lite { + proj = issueLiteProjection + } + results, err := searchTableInTxT(ctx, tx, query, filter, WispsFilterTables, proj) + if err != nil { + if missingOptionalWispTable(err) { + return nil, nil + } + return nil, fmt.Errorf("search wisps plane: %w", err) + } + results = trimToSearchLimit(results, filter.Limit) + if err := EnforceMaxRowsCap(len(results), filter.MaxRows, filter.MaxRowsSource); err != nil { + return nil, err + } + return results, nil +} + // searchProjection describes how to project, scan, and dedup search results. // Adding a narrow-projection variant means adding a new projection literal — // not a parallel top-level function or wisp-merge wrapper, which is how the diff --git a/internal/storage/issueops/sweep.go b/internal/storage/issueops/sweep.go index a609579e6..e27a3b671 100644 --- a/internal/storage/issueops/sweep.go +++ b/internal/storage/issueops/sweep.go @@ -31,7 +31,11 @@ import ( func SweepInTx(ctx context.Context, tx *sql.Tx, req publicops.SweepRequest) (publicops.SweepResult, error) { result := publicops.SweepResult{DryRun: req.DryRun} - candidates, err := SearchIssuesInTx(ctx, tx, "", workapi.BuildSweepCandidateFilter(req)) + search := SearchIssuesInTx + if workapi.SweepSearchesWispsPlaneOnly(req) { + search = SearchWispsPlaneInTx + } + candidates, err := search(ctx, tx, "", workapi.BuildSweepCandidateFilter(req)) if err != nil { return publicops.SweepResult{}, fmt.Errorf("listing sweep candidates: %w", err) } @@ -39,6 +43,14 @@ func SweepInTx(ctx context.Context, tx *sql.Tx, req publicops.SweepRequest) (pub kept, skips := workapi.FilterSweepCandidates(candidates, req.IDPattern, req.ClosedBefore) result.Skipped = skips + if req.ProtectLiveDependents { + protected, err := sweepLiveDependentsInTx(ctx, tx, kept) + if err != nil { + return publicops.SweepResult{}, err + } + kept, result.Skipped.LiveDependent = workapi.PartitionSweepLiveDependents(kept, protected) + } + if req.ProtectReferenced { referenced, err := sweepReferencedInTx(ctx, tx, kept) if err != nil { @@ -49,6 +61,8 @@ func SweepInTx(ctx context.Context, tx *sql.Tx, req publicops.SweepRequest) (pub result.Skipped.Referenced = count } + kept, result.Remaining = workapi.LimitSweepCandidates(kept, req.Limit) + if len(kept) == 0 { return result, nil } @@ -73,6 +87,46 @@ func SweepInTx(ctx context.Context, tx *sql.Tx, req publicops.SweepRequest) (pub return result, nil } +// sweepLiveDependentsInTx returns which of the candidates a live row depends +// on through a protecting edge (issueops.SweepRequest.ProtectLiveDependents), +// on the SAME transaction the candidates came off. Edges are read from both +// dependency tables, and the sources' statuses from both planes. +// +// Reading the custom statuses is required for the reason sweepReferencedInTx +// gives: a missed custom active status would under-protect. +func sweepLiveDependentsInTx(ctx context.Context, tx *sql.Tx, candidates []*types.Issue) (map[string]bool, error) { + if len(candidates) == 0 { + return nil, nil + } + ids := make([]string, len(candidates)) + for i, issue := range candidates { + ids[i] = issue.ID + } + incoming, err := GetDependentRecordsForIssuesInTx(ctx, tx, ids) + if err != nil { + return nil, fmt.Errorf("reading dependents for live-dependent protection: %w", err) + } + sources := workapi.SweepLiveDependentSources(incoming) + if len(sources) == 0 { + return nil, nil + } + custom, err := ResolveCustomStatusesDetailedInTx(ctx, tx) + if err != nil { + return nil, fmt.Errorf("reading custom statuses for live-dependent protection: %w", err) + } + liveRows, err := SearchIssuesInTx(ctx, tx, "", workapi.BuildSweepLiveDependentScanFilter(sources, custom)) + if err != nil { + return nil, fmt.Errorf("reading dependent statuses for live-dependent protection: %w", err) + } + live := make(map[string]bool, len(liveRows)) + for _, issue := range liveRows { + if issue != nil { + live[issue.ID] = true + } + } + return workapi.SweepLiveDependentTargets(incoming, live), nil +} + // sweepReferencedInTx returns which of the candidates are cited by a row that // is not done, reading the not-done set and its comments on the SAME // transaction the candidates came off. diff --git a/internal/storage/uow/notifying_parity_test.go b/internal/storage/uow/notifying_parity_test.go index bf1066d58..c9fd5d7fd 100644 --- a/internal/storage/uow/notifying_parity_test.go +++ b/internal/storage/uow/notifying_parity_test.go @@ -159,6 +159,7 @@ func TestRecordingIssueUseCaseCoversItsSurface(t *testing.T) { "PreviewDeleteWisp": reads, "SearchIssueIDs": reads, "SearchIssues": reads, + "SearchWispsPlane": reads, "SearchIssuesWithCounts": reads, "DeleteIssue": deletes, "DeleteIssues": deletes, diff --git a/internal/storage/uow/sweeper.go b/internal/storage/uow/sweeper.go index 3ce242c43..415833178 100644 --- a/internal/storage/uow/sweeper.go +++ b/internal/storage/uow/sweeper.go @@ -4,6 +4,7 @@ import ( "context" "fmt" + "github.com/steveyegge/beads/internal/storage/dberrors" "github.com/steveyegge/beads/internal/storage/domain" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/workapi" @@ -76,14 +77,22 @@ func (s *sweeper) Sweep(ctx context.Context, req publicops.SweepRequest) (public func sweepInUOW(ctx context.Context, uw UnitOfWork, req publicops.SweepRequest) (publicops.SweepResult, error) { result := publicops.SweepResult{DryRun: req.DryRun} - page, err := uw.IssueUseCase().SearchIssues(ctx, "", workapi.BuildSweepCandidateFilter(req)) + candidates, err := sweepCandidatesInUOW(ctx, uw, req) if err != nil { return publicops.SweepResult{}, fmt.Errorf("listing sweep candidates: %w", err) } - kept, skips := workapi.FilterSweepCandidates(page.Items, req.IDPattern, req.ClosedBefore) + kept, skips := workapi.FilterSweepCandidates(candidates, req.IDPattern, req.ClosedBefore) result.Skipped = skips + if req.ProtectLiveDependents { + protected, err := sweepLiveDependentsInUOW(ctx, uw, kept) + if err != nil { + return publicops.SweepResult{}, err + } + kept, result.Skipped.LiveDependent = workapi.PartitionSweepLiveDependents(kept, protected) + } + if req.ProtectReferenced { referenced, err := sweepReferencedInUOW(ctx, uw, kept) if err != nil { @@ -94,6 +103,8 @@ func sweepInUOW(ctx context.Context, uw UnitOfWork, req publicops.SweepRequest) result.Skipped.Referenced = count } + kept, result.Remaining = workapi.LimitSweepCandidates(kept, req.Limit) + if len(kept) == 0 { return result, nil } @@ -116,6 +127,73 @@ func sweepInUOW(ctx context.Context, uw UnitOfWork, req publicops.SweepRequest) return result, nil } +// sweepCandidatesInUOW reads the request's candidate rows: the wisps table +// alone for the wisps-plane tier, a tier query across both planes otherwise. +func sweepCandidatesInUOW(ctx context.Context, uw UnitOfWork, req publicops.SweepRequest) ([]*types.Issue, error) { + filter := workapi.BuildSweepCandidateFilter(req) + if workapi.SweepSearchesWispsPlaneOnly(req) { + return uw.IssueUseCase().SearchWispsPlane(ctx, "", filter) + } + page, err := uw.IssueUseCase().SearchIssues(ctx, "", filter) + if err != nil { + return nil, err + } + return page.Items, nil +} + +// sweepLiveDependentsInUOW returns which of the candidates a live row depends +// on through a protecting edge, on the unit of work the candidates came off. +// +// Edges are read from BOTH dependency tables: a wisp's edges live in +// wisp_dependencies and an issue's in dependencies, and the source of an edge +// pointing at a candidate can be either. The sources' statuses are then read +// across both planes by SearchIssues. +func sweepLiveDependentsInUOW(ctx context.Context, uw UnitOfWork, candidates []*types.Issue) (map[string]bool, error) { + if len(candidates) == 0 { + return nil, nil + } + ids := make([]string, len(candidates)) + for i, issue := range candidates { + ids[i] = issue.ID + } + edges := domain.DepListFilter{Types: publicops.SweepLiveDependentEdgeTypes, Direction: domain.DepDirectionIn} + durable, err := uw.DependencyUseCase().ListByIssueIDs(ctx, ids, edges) + if err != nil { + return nil, fmt.Errorf("reading dependents for live-dependent protection: %w", err) + } + wisp, err := uw.DependencyUseCase().ListByWispIDs(ctx, ids, edges) + if err != nil && !dberrors.IsTableNotExist(err) { + return nil, fmt.Errorf("reading wisp dependents for live-dependent protection: %w", err) + } + incoming := make(map[string][]*types.Dependency, len(durable.Incoming)+len(wisp.Incoming)) + for target, deps := range durable.Incoming { + incoming[target] = append(incoming[target], deps...) + } + for target, deps := range wisp.Incoming { + incoming[target] = append(incoming[target], deps...) + } + + sources := workapi.SweepLiveDependentSources(incoming) + if len(sources) == 0 { + return nil, nil + } + custom, err := uw.ConfigUseCase().GetCustomStatuses(ctx) + if err != nil { + return nil, fmt.Errorf("reading custom statuses for live-dependent protection: %w", err) + } + page, err := uw.IssueUseCase().SearchIssues(ctx, "", workapi.BuildSweepLiveDependentScanFilter(sources, custom)) + if err != nil { + return nil, fmt.Errorf("reading dependent statuses for live-dependent protection: %w", err) + } + live := make(map[string]bool, len(page.Items)) + for _, issue := range page.Items { + if issue != nil { + live[issue.ID] = true + } + } + return workapi.SweepLiveDependentTargets(incoming, live), nil +} + // sweepReferencedInUOW returns which of the candidates a not-done row cites, // reading the not-done set and its comments on the same unit of work the // candidates came off. Comments are read for the whole not-done set in ONE diff --git a/internal/storage/uow/sweeper_contract_test.go b/internal/storage/uow/sweeper_contract_test.go index 4ccec5a9a..e054b5226 100644 --- a/internal/storage/uow/sweeper_contract_test.go +++ b/internal/storage/uow/sweeper_contract_test.go @@ -35,6 +35,21 @@ func TestSweeperContract(t *testing.T) { t.Run("TreatsALegacyTypedWispAsEphemeralTier", func(t *testing.T) { conformance.RunSweeperTreatsALegacyTypedWispAsEphemeralTier(t, ctx, fixture) }) + t.Run("WispsPlaneClearsTheWholeWispsTable", func(t *testing.T) { + conformance.RunSweeperWispsPlaneClearsTheWholeWispsTable(t, ctx, fixture) + }) + t.Run("WispsPlaneRequiresAFilter", func(t *testing.T) { + conformance.RunSweeperWispsPlaneRequiresAFilter(t, ctx, fixture) + }) + t.Run("ProtectsLiveDependents", func(t *testing.T) { + conformance.RunSweeperProtectsLiveDependents(t, ctx, fixture) + }) + t.Run("ProtectsLiveDependentsAcrossPlanes", func(t *testing.T) { + conformance.RunSweeperProtectsLiveDependentsAcrossPlanes(t, ctx, fixture) + }) + t.Run("LimitTakesTheOldestClosedFirst", func(t *testing.T) { + conformance.RunSweeperLimitTakesTheOldestClosedFirst(t, ctx, fixture) + }) t.Run("LeavesNoHistoryBeadsToTheDurableTier", func(t *testing.T) { conformance.RunSweeperLeavesNoHistoryBeadsToTheDurableTier(t, ctx, fixture) }) @@ -98,6 +113,20 @@ func newUOWSweeperFixture(t *testing.T, ctx context.Context, prefix string) conf return "seed legacy sweep rows", nil }) }, + AddDependencies: func(ctx context.Context, req issueops.AddDependenciesRequest) error { + // Through the DependencyEditor ROLE, which routes each edge to its + // source plane's dependency table itself. + ds, ok := provider.(DependencyEditorSource) + if !ok { + return fmt.Errorf("provider %T does not offer the DependencyEditor accessor", provider) + } + editor, err := ds.DependencyEditor() + if err != nil { + return err + } + _, err = editor.AddDependencies(ctx, req) + return err + }, AddComment: func(ctx context.Context, issueID, author, text string) error { // Through the Commenter ROLE, which resolves the plane itself, so // the case can cite from a wisp's comment without knowing how this diff --git a/internal/workapi/sweep.go b/internal/workapi/sweep.go index 3466558e2..24af4ec50 100644 --- a/internal/workapi/sweep.go +++ b/internal/workapi/sweep.go @@ -29,13 +29,13 @@ import ( // by calling the role. See issueops.SweepRequest. func ValidateSweepRequest(in issueops.SweepRequest) error { switch in.Tier { - case issueops.SweepEphemeral, issueops.SweepDurable: + case issueops.SweepEphemeral, issueops.SweepDurable, issueops.SweepWispsPlane: case "": - return fmt.Errorf("%w: sweep requires a tier (%q or %q), and has no default", - issueops.ErrValidation, issueops.SweepEphemeral, issueops.SweepDurable) + return fmt.Errorf("%w: sweep requires a tier (%q, %q or %q), and has no default", + issueops.ErrValidation, issueops.SweepEphemeral, issueops.SweepDurable, issueops.SweepWispsPlane) default: - return fmt.Errorf("%w: %q is not a sweep tier; use %q or %q", - issueops.ErrValidation, in.Tier, issueops.SweepEphemeral, issueops.SweepDurable) + return fmt.Errorf("%w: %q is not a sweep tier; use %q, %q or %q", + issueops.ErrValidation, in.Tier, issueops.SweepEphemeral, issueops.SweepDurable, issueops.SweepWispsPlane) } if in.IDPattern != "" { // filepath.Match reports a malformed pattern on any subject, so one @@ -47,14 +47,32 @@ func ValidateSweepRequest(in issueops.SweepRequest) error { issueops.ErrValidation, in.IDPattern, err) } } - if in.Tier == issueops.SweepDurable && in.ClosedBefore == nil && in.IDPattern == "" { - return fmt.Errorf("%w: a durable sweep requires a closed-before cutoff or an id pattern; "+ + if in.Limit < 0 { + return fmt.Errorf("%w: sweep limit %d is negative; use 0 for no limit", issueops.ErrValidation, in.Limit) + } + if sweepRequiresFilter(in.Tier) && in.ClosedBefore == nil && in.IDPattern == "" { + return fmt.Errorf("%w: a %s sweep requires a closed-before cutoff or an id pattern; "+ "pass the pattern \"*\" to sweep every closed issue deliberately", - issueops.ErrValidation) + issueops.ErrValidation, in.Tier) } return nil } +// sweepRequiresFilter reports whether a tier reaches durable-tier rows and so +// carries the require-a-filter gate. The wisps plane does: it holds no-history +// beads (issueops.SweepWispsPlane). +func sweepRequiresFilter(tier issueops.SweepTier) bool { + return tier == issueops.SweepDurable || tier == issueops.SweepWispsPlane +} + +// SweepSearchesWispsPlaneOnly reports whether a request's candidates come from +// the wisps table alone (issueops.SweepWispsPlane) rather than from a tier +// query merged across both planes. Every Sweeper implementation branches on +// this one answer, so the plane a tier reads is decided in one place. +func SweepSearchesWispsPlaneOnly(in issueops.SweepRequest) bool { + return in.Tier == issueops.SweepWispsPlane +} + // BuildSweepCandidateFilter turns a sweep request into the storage-level // filter that selects its CANDIDATES: the closed rows of one tier, bounded by // the cutoff. @@ -66,8 +84,21 @@ func ValidateSweepRequest(in issueops.SweepRequest) error { // // Call ValidateSweepRequest first; this builder assumes a validated request // and does not re-refuse one. +// +// For SweepWispsPlane the filter carries NO tier constraint: the plane is the +// selection, and the implementation applies it by searching the wisps table +// alone (SweepSearchesWispsPlaneOnly). Every closed row there is a candidate, +// whatever its ephemeral, no_history or wisp_type values. func BuildSweepCandidateFilter(in issueops.SweepRequest) types.IssueFilter { closed := types.StatusClosed + if SweepSearchesWispsPlaneOnly(in) { + filter := types.IssueFilter{Status: &closed} + if in.ClosedBefore != nil { + cutoff := *in.ClosedBefore + filter.ClosedBefore = &cutoff + } + return filter + } ephemeral := in.Tier == issueops.SweepEphemeral // EphemeralTier, not Ephemeral: the tier owns typed wisps whether or not // their minting set the ephemeral flag (older creators set wisp_type but @@ -271,3 +302,91 @@ func PartitionSweepReferenced(candidates []*types.Issue, referenced map[string]b } return kept, referencedCount, sample } + +// BuildSweepLiveDependentScanFilter selects which of a candidate set's +// dependents are LIVE for issueops.SweepRequest.ProtectLiveDependents: the +// named sources, restricted to the not-done statuses. It is the same status +// vocabulary the reference scan asks, so "live" means one thing in a sweep. +func BuildSweepLiveDependentScanFilter(dependentIDs []string, custom []types.CustomStatus) types.IssueFilter { + return types.IssueFilter{IDs: dependentIDs, Statuses: NotDoneStatusesForSweep(custom)} +} + +// SweepLiveDependentSources returns the ids of the rows that depend on a +// candidate through a protecting edge (issueops.SweepLiveDependentEdgeTypes), +// given the edges pointing at the candidates keyed by target. The result is +// sorted and de-duplicated, so the follow-up status read is deterministic. +func SweepLiveDependentSources(incoming map[string][]*types.Dependency) []string { + seen := make(map[string]bool) + for _, deps := range incoming { + for _, dep := range deps { + if dep != nil && isSweepProtectingEdge(dep.Type) { + seen[dep.IssueID] = true + } + } + } + ids := make([]string, 0, len(seen)) + for id := range seen { + ids = append(ids, id) + } + sort.Strings(ids) + return ids +} + +// SweepLiveDependentTargets returns the candidates a live row depends on: the +// targets of protecting edges whose source is in live. +func SweepLiveDependentTargets(incoming map[string][]*types.Dependency, live map[string]bool) map[string]bool { + protected := make(map[string]bool) + for target, deps := range incoming { + for _, dep := range deps { + if dep != nil && isSweepProtectingEdge(dep.Type) && live[dep.IssueID] { + protected[target] = true + break + } + } + } + return protected +} + +func isSweepProtectingEdge(t types.DependencyType) bool { + for _, protecting := range issueops.SweepLiveDependentEdgeTypes { + if t == protecting { + return true + } + } + return false +} + +// PartitionSweepLiveDependents splits candidates into the ones no live row +// depends on and a count of the ones some live row does, preserving candidate +// order. +func PartitionSweepLiveDependents(candidates []*types.Issue, protected map[string]bool) (kept []*types.Issue, protectedCount int) { + kept = make([]*types.Issue, 0, len(candidates)) + for _, issue := range candidates { + if protected[issue.ID] { + protectedCount++ + continue + } + kept = append(kept, issue) + } + return kept, protectedCount +} + +// LimitSweepCandidates applies issueops.SweepRequest.Limit to the candidates +// that survived every protection: the oldest-closed first (closed_at +// ascending, then id, so two runs over one snapshot pick the same rows), at +// most limit of them. It returns the kept rows and how many were left behind. +// A non-positive limit keeps everything. The input slice is not reordered. +func LimitSweepCandidates(candidates []*types.Issue, limit int) (kept []*types.Issue, remaining int) { + if limit <= 0 || len(candidates) <= limit { + return candidates, 0 + } + ordered := append([]*types.Issue(nil), candidates...) + sort.SliceStable(ordered, func(i, j int) bool { + a, b := ordered[i], ordered[j] + if a.ClosedAt != nil && b.ClosedAt != nil && !a.ClosedAt.Equal(*b.ClosedAt) { + return a.ClosedAt.Before(*b.ClosedAt) + } + return a.ID < b.ID + }) + return ordered[:limit], len(ordered) - limit +} diff --git a/internal/workapi/sweep_test.go b/internal/workapi/sweep_test.go index 9c4093cd0..af26fd9fb 100644 --- a/internal/workapi/sweep_test.go +++ b/internal/workapi/sweep_test.go @@ -353,3 +353,119 @@ func sweepCandidateIDs(issues []*types.Issue) []string { } return ids } + +func TestValidateSweepRequestGatesTheWispsPlane(t *testing.T) { + cutoff := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + if err := ValidateSweepRequest(issueops.SweepRequest{Tier: issueops.SweepWispsPlane}); !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("unfiltered wisps-plane sweep error = %v, want ErrValidation: it reaches no-history (durable-tier) rows", err) + } + for _, req := range []issueops.SweepRequest{ + {Tier: issueops.SweepWispsPlane, ClosedBefore: &cutoff}, + {Tier: issueops.SweepWispsPlane, IDPattern: "*"}, + } { + if err := ValidateSweepRequest(req); err != nil { + t.Errorf("ValidateSweepRequest(%+v) = %v, want nil", req, err) + } + } +} + +func TestBuildSweepCandidateFilterForTheWispsPlaneHasNoTierConstraint(t *testing.T) { + cutoff := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + req := issueops.SweepRequest{Tier: issueops.SweepWispsPlane, ClosedBefore: &cutoff} + if !SweepSearchesWispsPlaneOnly(req) { + t.Fatal("SweepSearchesWispsPlaneOnly(wisps-plane) = false; the plane IS the selection") + } + filter := BuildSweepCandidateFilter(req) + if filter.Status == nil || *filter.Status != types.StatusClosed { + t.Fatalf("Status = %v, want closed", filter.Status) + } + if filter.EphemeralTier != nil || filter.Ephemeral != nil { + t.Fatalf("wisps-plane filter carries a flag constraint (EphemeralTier=%v, Ephemeral=%v); "+ + "that would drop no-history rows", filter.EphemeralTier, filter.Ephemeral) + } + if filter.ClosedBefore == nil || !filter.ClosedBefore.Equal(cutoff) || filter.ClosedBefore == &cutoff { + t.Fatalf("ClosedBefore = %v, want a copy of %v", filter.ClosedBefore, cutoff) + } + for _, tier := range []issueops.SweepTier{issueops.SweepEphemeral, issueops.SweepDurable} { + if SweepSearchesWispsPlaneOnly(issueops.SweepRequest{Tier: tier}) { + t.Errorf("SweepSearchesWispsPlaneOnly(%s) = true; tier queries merge both planes", tier) + } + } +} + +func TestSweepLiveDependentsCountOnlyProtectingEdgesFromLiveSources(t *testing.T) { + dep := func(source, target string, typ types.DependencyType) *types.Dependency { + return &types.Dependency{IssueID: source, DependsOnID: target, Type: typ} + } + incoming := map[string][]*types.Dependency{ + "root": {dep("step-open", "root", types.DepParentChild), dep("step-done", "root", types.DepParentChild)}, + "tracked": {dep("convoy", "tracked", types.DepTracks)}, + "gate": {dep("waiter", "gate", types.DepBlocks)}, + "linked": {dep("reader", "linked", types.DepRelated)}, + "orphan": {dep("step-done", "orphan", types.DepParentChild), nil}, + } + + sources := SweepLiveDependentSources(incoming) + if want := []string{"convoy", "step-done", "step-open", "waiter"}; strings.Join(sources, ",") != strings.Join(want, ",") { + t.Fatalf("SweepLiveDependentSources = %v, want %v (sorted, protecting edges only)", sources, want) + } + + live := map[string]bool{"step-open": true, "convoy": true, "waiter": true, "reader": true} + protected := SweepLiveDependentTargets(incoming, live) + for _, id := range []string{"root", "tracked", "gate"} { + if !protected[id] { + t.Errorf("%s not protected; a live source depends on it through a protecting edge", id) + } + } + for _, id := range []string{"linked", "orphan"} { + if protected[id] { + t.Errorf("%s protected; its only live source is a non-protecting edge or none is live", id) + } + } + + candidates := []*types.Issue{{ID: "root"}, {ID: "linked"}, {ID: "tracked"}, {ID: "orphan"}} + kept, count := PartitionSweepLiveDependents(candidates, protected) + if count != 2 || len(kept) != 2 || kept[0].ID != "linked" || kept[1].ID != "orphan" { + t.Fatalf("PartitionSweepLiveDependents = %d kept %v, count %d; want [linked orphan] and 2", len(kept), kept, count) + } +} + +func TestValidateSweepRequestRefusesANegativeLimit(t *testing.T) { + err := ValidateSweepRequest(issueops.SweepRequest{Tier: issueops.SweepEphemeral, Limit: -1}) + if !errors.Is(err, issueops.ErrValidation) { + t.Fatalf("negative Limit error = %v, want ErrValidation", err) + } + if err := ValidateSweepRequest(issueops.SweepRequest{Tier: issueops.SweepEphemeral, Limit: 0}); err != nil { + t.Fatalf("zero Limit (no cap) error = %v, want nil", err) + } +} + +func TestLimitSweepCandidatesTakesTheOldestClosedFirst(t *testing.T) { + at := func(day int) *time.Time { + ts := time.Date(2026, 1, day, 0, 0, 0, 0, time.UTC) + return &ts + } + candidates := []*types.Issue{ + {ID: "c", ClosedAt: at(3)}, + {ID: "b2", ClosedAt: at(1)}, + {ID: "a", ClosedAt: at(2)}, + {ID: "b1", ClosedAt: at(1)}, + } + + kept, remaining := LimitSweepCandidates(candidates, 3) + var got []string + for _, issue := range kept { + got = append(got, issue.ID) + } + if strings.Join(got, ",") != "b1,b2,a" || remaining != 1 { + t.Fatalf("LimitSweepCandidates(3) = %v, remaining %d; want [b1 b2 a] (closed_at, then id) and 1", got, remaining) + } + if candidates[0].ID != "c" { + t.Fatal("LimitSweepCandidates reordered the caller's slice") + } + for _, limit := range []int{0, 4, 10} { + if kept, remaining := LimitSweepCandidates(candidates, limit); len(kept) != 4 || remaining != 0 { + t.Errorf("LimitSweepCandidates(%d) kept %d, remaining %d; want all 4 and 0", limit, len(kept), remaining) + } + } +} diff --git a/issueops/dependencyeditor.go b/issueops/dependencyeditor.go index 2951dd66e..f9b0683eb 100644 --- a/issueops/dependencyeditor.go +++ b/issueops/dependencyeditor.go @@ -29,6 +29,7 @@ const ( DepDiscoveredFrom = types.DepDiscoveredFrom DepRepliesTo = types.DepRepliesTo DepWaitsFor = types.DepWaitsFor + DepTracks = types.DepTracks ) // DependencyEdge is one directed edge: IssueID depends on DependsOnID, with diff --git a/issueops/sweeper.go b/issueops/sweeper.go index 46f4e528b..4e60eda0a 100644 --- a/issueops/sweeper.go +++ b/issueops/sweeper.go @@ -27,6 +27,21 @@ const ( // it — carries the require-a-filter refusal SweepRequest.ClosedBefore // describes. SweepDurable SweepTier = "durable" + // SweepWispsPlane clears CLOSED rows by STORAGE PLANE rather than by + // tier: every closed row stored in the wisps table, whatever its flags — + // ephemeral wisps and no-history beads alike. `bd purge --wisps-plane`. + // + // It is the one tier that is NOT disjoint from the other two, and that is + // the point of it. A no-history bead lives in the wisps plane but is + // durable-tier (it is not ephemeral and carries no wisp_type), so neither + // SweepEphemeral nor SweepDurable can say "the wisps table" — the unit an + // orchestrator's retention policy is written in. Conversely a legacy + // typed wisp stored in the issues table is ephemeral-tier and NOT in this + // plane. + // + // Because it reaches durable-tier rows, it carries the durable tier's + // require-a-filter gate (SweepRequest.ClosedBefore). + SweepWispsPlane SweepTier = "wisps-plane" ) // SweepRequest describes one bulk clearance of closed rows: which tier, which @@ -68,6 +83,9 @@ type SweepRequest struct { // says so with IDPattern "*", which is a deliberate keystroke rather than // an omitted one. // + // SweepWispsPlane carries the same gate, because the wisps plane holds + // no-history beads, which are durable-tier. + // // The EPHEMERAL tier carries no such gate: an unfiltered sweep of closed // wisps is the ordinary use of `bd purge`, and the tier is transient by // definition. @@ -108,6 +126,33 @@ type SweepRequest struct { // `bd purge` does not, because a wisp's citations are as transient as the // wisp. ProtectReferenced bool + // ProtectLiveDependents skips candidates that a LIVE row depends on + // through a structural edge — a closed parent whose child is still open, + // a closed bead a live convoy tracks, a closed blocker of a live bead — + // so a sweep never deletes the root, container or gate of work that is + // still in progress. + // + // WHICH EDGES are SweepLiveDependentEdgeTypes: parent-child, tracks and + // blocks, from either plane's dependency table. Other edge types (related, + // discovered-from, ...) are links rather than structure and do not protect. + // + // WHICH ROWS ARE LIVE is the same not-done set ProtectReferenced asks: the + // built-in active statuses (open, in_progress, blocked, deferred, pinned, + // hooked) plus every configured custom status whose category is not + // "done", with the same REQUIRED read of the custom statuses. + // + // Unlike ProtectReferenced it is a stored-edge lookup, not a text scan, so + // it costs one read of the edges pointing at the candidates and one of + // their sources. `bd purge` always asks for it; `bd prune` does not. + ProtectLiveDependents bool + // Limit, when positive, caps how many rows ONE sweep deletes, taking the + // OLDEST-CLOSED first (closed_at ascending, then id). The rest are left + // for the next call and counted in SweepResult.Remaining, so a caller can + // drain a large backlog in bounded transactions instead of one that grows + // with the backlog and can outlive the caller's deadline. It applies after + // every protection, so it caps rows that would actually go. Zero means no + // cap; a negative value is ErrValidation. + Limit int // DryRun reports what the sweep WOULD do and deletes nothing. The result // is otherwise the same result — the same counts, the same skips, the same // refusals — computed against the same snapshot the real sweep would have @@ -132,6 +177,9 @@ type SweepSkips struct { // 0 when that field is false; a caller that reads a 0 without having asked // for the protection has learned nothing about whether rows are cited. Referenced int + // LiveDependent counts candidates skipped by ProtectLiveDependents, under + // the same always-0-when-not-asked rule as Referenced. + LiveDependent int // NotClosed, UnknownClosedAt and ClosedAtOrAfterCutoff count candidates // the tier's own query returned and the sweep rechecked and rejected: a // status that is not closed, a closed row with no closed_at stamp at all, @@ -173,6 +221,10 @@ type SweepResult struct { Events int // Skipped reports the candidates the sweep held back and why. Skipped SweepSkips + // Remaining is how many rows the sweep WOULD also have deleted but left + // because of SweepRequest.Limit. Zero when there is no limit or the + // backlog fit; a caller loops while it is non-zero. + Remaining int // ReferencedIDs is a BOUNDED SAMPLE of the ids Skipped.Referenced counts — // at most SweepReferencedSampleLimit of them, in the order the candidate // query returned them. It is a sample and not the set: a caller that needs @@ -184,6 +236,11 @@ type SweepResult struct { ReferencedIDs []string } +// SweepLiveDependentEdgeTypes are the edge types through which a live row +// protects a closed candidate under SweepRequest.ProtectLiveDependents. It is +// published so a caller can state the same policy without copying the list. +var SweepLiveDependentEdgeTypes = []DependencyType{DepParentChild, DepTracks, DepBlocks} + // SweepReferencedSampleLimit bounds SweepResult.ReferencedIDs. It is published // so a caller can tell a truncated sample from a complete one by comparing the // slice's length against it, rather than against a number copied out of an @@ -255,17 +312,20 @@ type Sweeper interface { // // THE ORDER THE NARROWING HAPPENS IN IS PART OF THE ANSWER, because the // skip counters are counted along the way: the tier's closed rows, then - // IDPattern, then the pinned and closed_at rechecks, then the reference - // protection. A pinned row excluded by the pattern is therefore NOT - // counted in Skipped.Pinned — it was never a candidate — and the counters - // describe the set the request actually reached. + // IDPattern, then the pinned and closed_at rechecks, then the + // live-dependent protection, then the reference protection, then Limit + // (oldest-closed first). A pinned row + // excluded by the pattern is therefore NOT counted in Skipped.Pinned — it + // was never a candidate — and the counters describe the set the request + // actually reached. // // REFUSALS, all ErrValidation and all before anything is read: // // - an unset or unrecognized Tier; - // - a SweepDurable request with neither ClosedBefore nor IDPattern (see - // SweepRequest.ClosedBefore for why this lives here); - // - an IDPattern that is not a well-formed glob. + // - a SweepDurable or SweepWispsPlane request with neither ClosedBefore + // nor IDPattern (see SweepRequest.ClosedBefore for why this lives here); + // - an IDPattern that is not a well-formed glob; + // - a negative Limit. // // A REQUEST THAT MATCHES NOTHING IS A ZERO RESULT AND A NIL ERROR, not a // not-found: an empty set of closed rows is the steady state of a swept From 2b27259cb4cb7a3df488dae58d2a3e92c0f32c98 Mon Sep 17 00:00:00 2001 From: Gas City Adopt-PR Date: Mon, 28 Sep 2026 15:56:08 +0000 Subject: [PATCH 15/45] test(release-1.3.x): adapt release-only capability tests to the backported proxied backup [NOT a cherry-pick] Release-only adaptation, not a copy of any main commit. The tests it touches exist only on release/1.3.0, so there is nothing on main to pick; every other commit on this branch is a faithful cherry-pick -x. - topology_capability_matrix_test.go (#6579, release-only S0 matrix): flip backup status/sync as #6584's own description says. On proxied-local, status is HONORED and sync is ERROR "no backup destination configured"; other proxied topologies are REFUSED_TYPED with reason design. The runner already orders refusal rows before honored ones, so the honored cells no longer start the proxy ahead of the no-side-effect refusal checks (the red smoke-lane cells review found). - proxy_capability_test.go: TestProxyMaintenanceAllowsTheSharedConsentVerb (#6695, release-only) called validateProxyMaintenanceBeforeProvider, which #6582 removes; it now calls validateProxyRegistryBeforeProvider for managed-local, with unchanged assertions. Until this commit the cmd/bd test package does not compile on the picks above it. - proxied-local-smoke.yml: run the capability matrix and the backup round trip in the compiled-in check, and trigger on cmd/bd/backup*.go. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/proxied-local-smoke.yml | 6 +++++- cmd/bd/proxy_capability_test.go | 16 ++++++++-------- cmd/bd/topology_capability_matrix_test.go | 18 +++++++++++++++--- 3 files changed, 28 insertions(+), 12 deletions(-) diff --git a/.github/workflows/proxied-local-smoke.yml b/.github/workflows/proxied-local-smoke.yml index 263e182e4..eee9b7b19 100644 --- a/.github/workflows/proxied-local-smoke.yml +++ b/.github/workflows/proxied-local-smoke.yml @@ -23,6 +23,7 @@ on: - 'cmd/bd/proxied_server*.go' - 'cmd/bd/dolt*.go' - 'cmd/bd/*_proxied_server.go' + - 'cmd/bd/backup*.go' - 'cmd/bd/purge*.go' - 'cmd/bd/db_proxy_child.go' - 'cmd/bd/init*.go' @@ -43,6 +44,7 @@ on: - 'cmd/bd/proxied_server*.go' - 'cmd/bd/dolt*.go' - 'cmd/bd/*_proxied_server.go' + - 'cmd/bd/backup*.go' - 'cmd/bd/purge*.go' - 'cmd/bd/db_proxy_child.go' - 'cmd/bd/init*.go' @@ -113,7 +115,9 @@ jobs: TestManagedLocalProxiedDoltStartRefusesOverLiveProxy \ TestManagedLocalProxiedDoltStartRefusesWithProxyDown \ TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone \ - TestManagedLocalProxiedPurgeWispsPlaneRetention + TestManagedLocalProxiedPurgeWispsPlaneRetention \ + TestManagedLocalProxiedCapabilityMatrix \ + TestManagedLocalProxiedBackupRoundTrip do if ! grep -qx "$test_name" <<<"$listed"; then echo "managed-local lifecycle test $test_name is missing; got: '$listed'" >&2 diff --git a/cmd/bd/proxy_capability_test.go b/cmd/bd/proxy_capability_test.go index 6c70f17ff..4e104b1a5 100644 --- a/cmd/bd/proxy_capability_test.go +++ b/cmd/bd/proxy_capability_test.go @@ -17,12 +17,12 @@ import ( // #6575 proxied data-behind guidance is written against: the consent command // that guidance prescribes is NOT refused at the proxied front door. // -// The refusal table keys on the full command path. It has rows for the bare -// `migrate` verb and for `migrate hooks` / `migrate issues` / `migrate sync`, -// but none for `migrate schema` — so that path falls through -// validateProxyMaintenanceBeforeProvider's multi-word arm to `bd migrate -// schema`'s own proxied arm, which reports the migration the provider open -// already applied under this verb's consent. +// The capability registry keys on the full command path. It refuses the bare +// `migrate` verb and `migrate hooks` / `migrate issues` / `migrate sync`, but +// permits `migrate schema` — so that path passes +// validateProxyRegistryBeforeProvider to `bd migrate schema`'s own proxied +// arm, which reports the migration the provider open already applied under +// this verb's consent. // // An earlier revision of that guidance asserted the opposite on three runtime // surfaces and nothing failed, because no test called this function with this @@ -51,7 +51,7 @@ func TestProxyMaintenanceAllowsTheSharedConsentVerb(t *testing.T) { var allowErr error out := captureStdout(t, func() error { - allowErr = validateProxyMaintenanceBeforeProvider(target) + allowErr = validateProxyRegistryBeforeProvider(target, ProxyTopologyManagedLocal) return nil }) if allowErr != nil { @@ -70,7 +70,7 @@ func TestProxyMaintenanceAllowsTheSharedConsentVerb(t *testing.T) { } var bareErr error refusal := captureStdout(t, func() error { - bareErr = validateProxyMaintenanceBeforeProvider(bare) + bareErr = validateProxyRegistryBeforeProvider(bare, ProxyTopologyManagedLocal) return nil }) if bareErr == nil { diff --git a/cmd/bd/topology_capability_matrix_test.go b/cmd/bd/topology_capability_matrix_test.go index 3e00ceb20..5d79a2bd0 100644 --- a/cmd/bd/topology_capability_matrix_test.go +++ b/cmd/bd/topology_capability_matrix_test.go @@ -274,10 +274,16 @@ var capabilityProbes = []probe{ direct: expectation{ outcome: outcomeHonored, substr: `"configured"`, reason: reasonNA, }, + // Refused by design where bd does not own the server (#6584): the + // backup remote is registered on the server, global to every client. proxied: expectation{ outcome: outcomeRefusedTyped, code: "proxy.backup.unsupported", substr: "backup status is not supported in proxied-server mode", - reason: reasonUnimplemented, + reason: reasonDesign, + }, + // Honored on managed-local, where bd spawned the Dolt child (#6584). + override: map[string]expectation{ + topoProxiedLocal: {outcome: outcomeHonored, substr: `"configured"`, reason: reasonNA}, }, }, { @@ -289,7 +295,13 @@ var capabilityProbes = []probe{ proxied: expectation{ outcome: outcomeRefusedTyped, code: "proxy.backup.unsupported", substr: "backup sync is not supported in proxied-server mode", - reason: reasonUnimplemented, + reason: reasonDesign, + }, + // Honored on managed-local (#6584): it reaches the command, which + // fails on its own merits with no destination configured — the same + // answer every direct topology gives. + override: map[string]expectation{ + topoProxiedLocal: {outcome: outcomeError, substr: "no backup destination configured", reason: reasonNA}, }, }, @@ -464,7 +476,7 @@ var deliberatelyNotProbed = map[string]string{ "restore": "`bd restore` restores an ISSUE, not a backup; the backup verb is `bd backup restore`, which is already " + "covered by the backup rows. Probing it adds a row that says nothing about the backup family.", "backup restore": "would need a real backup destination to distinguish a topology refusal from a missing-backup error. " + - "Covered indirectly by `backup status` / `backup sync`; revisit in slice S3 when the family gains a proxied route.", + "Covered indirectly by `backup status` / `backup sync`, and directly by TestManagedLocalProxiedBackupRoundTrip since slice S3 (#6584).", } // --------------------------------------------------------------------------- From e352b96d602532dce10ea414df5cf0ca17016a31 Mon Sep 17 00:00:00 2001 From: Alex Date: Sun, 27 Sep 2026 00:56:42 -0700 Subject: [PATCH 16/45] fix(dolt): recheck dependents' blocked state after a close commits (#6719) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(dolt): recheck dependents' blocked state after a close commits (be-cmn) A close recomputes its dependents' is_blocked inside its own transaction, against that transaction's start snapshot. Two concurrent closes of sibling blockers each see the other blocker still open, so neither writes the shared dependent's row; both commit without a conflicting cell, and the dependent stays is_blocked=1 and hidden from `bd ready` until `bd recompute-blocked`. The in-transaction close and status-change paths now record the dependents they recomputed (never the closed issue itself) in a transaction-scoped note, the same shape as the events-journal scope. The store's write-transaction runners take that note once the transaction has committed and, only when it is non-empty, run one more short retried transaction that recomputes exactly those ids on a fresh snapshot. The recheck runs after the retry loop around the write, so a recheck failure is reported as "write committed; blocked-state recheck failed" and never replays a write that already landed. When the recheck changes an issues row it mints a Dolt commit naming the closed issue, so the corrected flag reaches history like the close's own rows instead of sitting dirty in the working set. A close with no dependents records nothing and issues no extra SQL. This covers every DoltStore write path, including the lifecycle and batch closers the CLI uses, and the embedded store's withConn. The RunInTransaction path (doltTransaction.CloseIssue) keeps its pinned-connection commit and is not scoped; nothing on the CLI's close or update path reaches a close through it. Fixes #6716 Co-Authored-By: Claude Fable 5.1 * fix(dolt): recheck blocked state after dependency removals and deletes too (be-cmn) Only a close and an update to an inactive status recorded a post-commit blocked recheck. A dependency removal and the three delete paths run the same in-transaction recompute, can likewise only unblock a dependent, and so reproduce gastownhall/beads#6716 when they commit last: a `bd close A` overlapping a `bd dep remove C B` (or a delete of B) leaves C blocked on every committed snapshot with no recheck to settle it. Generalize the recording so each unblocking write records the dependents it recomputed: noteBlockedRecheck now takes a human label for the recheck commit message and an explicit exclusion set. A close or status update excludes its own row (a concurrent writer conflicts on it rather than racing past it); a delete excludes the rows that no longer exist; a dependency removal excludes nothing, because its dependent is exactly the row that needs rechecking. Labels stay bounded: a bulk delete of more than three ids is named by count. The close path's commit message, "bd: recheck blocked after close of ", is unchanged, and a close through update mints the same message as a close through CloseIssue. The server tier gains the close-versus-dependency-removal race and a check that a delete records the deleted row's dependents; the embedded tier pins the recording contract for a close, a removal and a delete on a real engine; the issueops unit test pins deduplication, exclusion and the labels. Co-Authored-By: Claude Fable 5.1 * fix(dolt): name a committed write's recheck failure with a sentinel (be-cmn) A recheck failure reaches the caller as an error from the same store call that just committed its write, indistinguishable from a write that never landed. Add issueops.ErrBlockedRecheckFailed and wrap both stores' recheck failures with it, keeping the underlying cause reachable through a second %w, so a caller can errors.Is a committed-but-unrechecked write apart from an uncommitted one. The stale flag it leaves is the one bd doctor and bd recompute-blocked already repair. The CLI's exit-code treatment is a separate follow-up; this is the storage layer only. Correct the embedded store's comment, which claimed two withConn callers in one process overlap as two server sessions do. They do not: commitConn opens a fresh OpenSQL handle per transaction and OpenSQL waits, with no elapsed-time limit, for the engine, so a second handle blocks until the first is cleaned up. Embedded transactions serialize and the skew of gastownhall/beads#6716 cannot occur there; the recheck stays so the embedded store honors the same contract as the server store and runs after the first handle's cleanup, so it cannot deadlock on itself. Document that the recheck scope is keyed by the DBTX interface value: a write records into it only when the very *sql.Tx the store scoped reaches it, and a future DBTX wrapper would silently record nothing, the same shape as the events-journal scope. Co-Authored-By: Claude Fable 5.1 * docs(issueops): state the blocked recheck's scope precisely (be-cmn) The recorded writes are the ones that take a blocker away, not writes that "can only unblock": removing the one closed child of an any-children gate blocks its waiter, and the recheck, being a full recompute, settles that too. Name the writes that add a blocker (reopen, dependency add, parent-child add) as recomputing the same way without a recheck, so the remaining gap is written down where the mechanism is. The exclude list filters one call's ids only; say so, and that a stale pending id for a row a later write deleted is a harmless no-op. Co-Authored-By: Claude Fable 5.1 * fix(dolt): keep a failed blocked recheck out of the caller, and scope the coverage claim Maintainer fixup for the adoption review of #6719 (fixes gastownhall/beads#6716). Three review majors, plus the cheap non-gating cleanups that sit next to them. The contributor's commits are left exactly as authored. 1. A post-commit recheck failure no longer reaches the caller of a committed write. ErrBlockedRecheckFailed had no consumer, so every store-facing caller — CLI, HTTP, the batch closers — still read it as "the write did not land" and would retry an applied mutation: the double-apply inversion internal/storage/dolt/issue_operations_tx.go documents. The stores now wrap the failure with the sentinel and log it. What it leaves behind is the stale is_blocked flag `bd doctor` and `bd recompute-blocked` already repair, i.e. the state every write had before the recheck existed. This also removes #6723 as a prerequisite for shipping. 2. The recheck runs on issueops.BlockedRecheckContext: detached from the caller's cancellation — the write it repairs has already committed, so a CLI deadline, a disconnected HTTP client or shutdown must not skip the repair and leave exactly the stale flag #6716 is about — with its own ceiling, and marked so a recheck cannot start another. Termination previously rested only on RecomputeIsBlockedInTxWithResult never recording, an invariant nothing enforced. 3. The coverage claim is scoped where it was false. Recording only becomes a recheck on a transaction one of the two store write runners scoped, and `bd batch` reaches neither: it dispatches close, `update status=closed` and `dep remove` onto storage.Transaction when local (cmd/bd/batch.go:177) and onto uow.RunTxResult when proxied (cmd/bd/batch_proxied_server.go:41). 5531d0592's message ("This covers every DoltStore write path … nothing on the CLI's close or update path reaches a close through it") and the PR description said otherwise. The CHANGELOG, blocked_recheck.go's doc comment and the PR description now scope the claim and name the uncovered routes: uow/domain-db, RunInTransaction (`bd batch`, `bd cook`, `bd mol squash`, `bd mol burn`), the legacy RemoveDependencyWithOptions (`bd duplicates --merge`), and the wisp writers. Also non-gating: the recheck's Dolt commit message is bounded (it names up to three writes and counts the rest), and a write that contributed no id is no longer named in it. New tests pin the recheck context, the bounded message, and the store seam the wrapper's unit test could not reach: a close whose recheck transaction fails still returns nil and stays durable, and the failure the withConn/commitConn path produces carries ErrBlockedRecheckFailed. Second review iteration: the scorecard required no changes, and all four findings were non-gating (one minor, three nits). Each indicted text this fixup itself introduced rather than the contributor's, so they are fixed here instead of deferred. 4. A dropped recheck is now counted, not only logged. Swallowing the failure in 1 made that log line the only signal a dependent is stale, while the convention it was modelled on — post_tx_commit_dropped, for the same "post-tx step failed, the write is durable" situation — is a counter *and* a line. bd.db.blocked_recheck_dropped is registered beside it and bumped on the Dolt side, and both stores' warnings now name the rows left unrechecked and point at `bd recompute-blocked`; an operator reading one line previously could not tell which issues needed it. The sentence is shared from issueops next to the sentinel, which also retires the near-duplicate doc comment the two stores carried. 5. The bounded commit message keeps the names it retained: past the limit it appends "and N more" instead of collapsing to a bare count, so the commit that is the only record of which write triggered a repair no longer discards the labels kept for exactly that purpose. N counts the unnamed writes, which also removes a skew — Sources deduplicates and SourceCount does not, so two equal-sized bulk deletes rendered one label and took the count-only form at two writes, not at "more than three" as documented. 6. The CHANGELOG's wisp clause now matches the doc comment and the PR description: closes, updates, deletes and demote-to-wisp. An update to an inactive status also takes a blocker away, and the narrower list under-counted the uncovered routes on the surface users actually read. 7. The Dolt tier's swallow is pinned. Every other test in close_recheck_blocked_test.go calls recheckBlockedAfterCommit directly, so nothing crossed either runner's tail on the tier #6716 was reported on — restoring `return s.recheckBlockedAfterCommit(...)` turned nothing red. TestCloseRecheckBlocked_FailureKeepsTheWrite runs a close through withRetryTx and through withWriteTx with the recheck forced to fail (marking the store closed inside fn, the analog of the embedded test's readOnly flip) and asserts nil plus durability. Gates, this iteration: gofmt clean; go build ./internal/storage/... and go vet on issueops, dolt and embeddeddolt clean; go test ./internal/storage/issueops/ ok; go test -tags gms_pure_go ./internal/storage/embeddeddolt/ -run BlockedRecheck ok (9s, real engine); go test ./internal/storage/dolt/ -run CloseRecheckBlocked ok (27s, dolt testcontainer). Iteration 1 additionally ran the full embedded package (98s) and the dolt subset 'RunInTransaction|BatchCloser|IsBlocked_|CloseRecheck|LifecycleCloseReopen' (156s). Every new assertion was proved non-vacuous by mutating the change it pins — reverting CommitMessage to the count-only form, counting all writes rather than the unnamed ones, dropping the ids and the repair pointer from the warning, unbounding that id list, and restoring the propagation on each runner tail — six mutations, each reddening only the assertions that defend it. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Fable 5.1 Co-authored-by: Julian Knutsen (cherry picked from commit 1ed5b3330375de39d8f7a9d55950e5176d35fa1a) Hand-ported to hotfix/1.3.1 (release/1.3.0 base); the pick did not apply cleanly: - internal/storage/dolt/store.go (commitWriteTx) and internal/storage/embeddeddolt/store.go (commitConn): main's context lines install scopeVersionedHistoryTransaction / ScopeVersionedHistoryTransaction, a main-only feature absent on the release line. Resolved by keeping the release code and adding only the two ScopeBlockedRecheckTransaction lines; the versioned-history scope is NOT brought over. Every other hunk in both files (runner split, recheck, logging, metric) applied unchanged. - CHANGELOG.md: main's entry sits in an Unreleased/Fixed list the release branch does not have. The release CHANGELOG is kept and the #6716 entry is added verbatim under a new Unreleased "### Fixed" heading. No schema change; issueops, dolt and embeddeddolt build and vet clean. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 16 + .../dolt/close_recheck_blocked_test.go | 278 ++++++++++++++++ internal/storage/dolt/store.go | 116 +++++-- .../blocked_recheck_internal_test.go | 164 ++++++++++ internal/storage/embeddeddolt/store.go | 67 +++- internal/storage/issueops/blocked_recheck.go | 302 ++++++++++++++++++ .../storage/issueops/blocked_recheck_test.go | 213 ++++++++++++ internal/storage/issueops/bulk_ops.go | 1 + internal/storage/issueops/close.go | 1 + internal/storage/issueops/delete.go | 2 + internal/storage/issueops/dependencies.go | 3 + internal/storage/issueops/update.go | 3 + 12 files changed, 1145 insertions(+), 21 deletions(-) create mode 100644 internal/storage/dolt/close_recheck_blocked_test.go create mode 100644 internal/storage/embeddeddolt/blocked_recheck_internal_test.go create mode 100644 internal/storage/issueops/blocked_recheck.go create mode 100644 internal/storage/issueops/blocked_recheck_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index e188daf97..4a025b100 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -103,6 +103,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 purge that used to delete a closed molecule root under a live step now leaves it. See the `--wisps-plane` entry under Added. +### Fixed + +- Unblocking two blockers of one dependent at the same time — closing both, + or a close racing a `bd dep remove` or a delete of the other — no longer + leaves the dependent stuck as blocked and hidden from `bd ready` until + `bd recompute-blocked`: a write that takes a blocker away (a close, an + update to an inactive status, a dependency removal, a delete) and runs + through a Dolt store write transaction now rechecks the dependents it + recomputed once that transaction has committed + ([#6716](https://github.com/gastownhall/beads/issues/6716)). Writes that + reach the database another way still need the `bd doctor` / + `bd recompute-blocked` repair they needed before: `bd batch` (on both its + plain and its proxied transaction), `bd cook`, `bd mol squash`, + `bd mol burn`, `bd duplicates --merge`, and the wisp writes — closes, + updates, deletes and demote-to-wisp. + ## [1.3.1] - 2026-09-16 ### Changed diff --git a/internal/storage/dolt/close_recheck_blocked_test.go b/internal/storage/dolt/close_recheck_blocked_test.go new file mode 100644 index 000000000..f7fcca35d --- /dev/null +++ b/internal/storage/dolt/close_recheck_blocked_test.go @@ -0,0 +1,278 @@ +package dolt + +import ( + "context" + "database/sql" + "errors" + "slices" + "testing" + + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// seedSiblingBlockers creates rb-a and rb-b, both blocking rb-c. +func seedSiblingBlockers(t *testing.T, ctx context.Context, store *DoltStore) { + t.Helper() + for _, id := range []string{"rb-a", "rb-b", "rb-c"} { + createPerm(t, ctx, store, id) + } + addDependency(t, ctx, store, "rb-c", "rb-a", types.DepBlocks) + addDependency(t, ctx, store, "rb-c", "rb-b", types.DepBlocks) + assertIsBlocked(t, ctx, store, "issues", "rb-c", true) +} + +// beginRacingWrite opens a transaction on its own pooled connection, pins its +// snapshot by reading both blockers, and runs one unblocking write inside it +// with the ordinary in-transaction recompute, exactly as the store's write +// paths do. It returns the dependents the write recorded for the post-commit +// recheck and a commit step that lands the write and frees the connection. +func beginRacingWrite(t *testing.T, ctx context.Context, store *DoltStore, name string, write func(tx *sql.Tx) error) (pending issueops.BlockedRecheck, commit func()) { + t.Helper() + conn, err := store.db.Conn(ctx) + if err != nil { + t.Fatalf("acquire connection for %s: %v", name, err) + } + tx, err := conn.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("begin tx for %s: %v", name, err) + } + clearScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearScope() + var open int + if err := tx.QueryRowContext(ctx, + "SELECT COUNT(*) FROM issues WHERE id IN ('rb-a', 'rb-b') AND status <> 'closed'").Scan(&open); err != nil { + t.Fatalf("pin snapshot for %s: %v", name, err) + } + if open != 2 { + t.Fatalf("snapshot for %s sees %d open blockers, want 2", name, open) + } + if err := write(tx); err != nil { + t.Fatalf("%s in tx: %v", name, err) + } + return issueops.TakeBlockedRecheck(tx), func() { + if err := tx.Commit(); err != nil { + t.Fatalf("commit %s: %v", name, err) + } + _ = conn.Close() + } +} + +// beginRacingClose is beginRacingWrite for a close of id. +func beginRacingClose(t *testing.T, ctx context.Context, store *DoltStore, id string) (pending issueops.BlockedRecheck, commit func()) { + t.Helper() + return beginRacingWrite(t, ctx, store, "close of "+id, func(tx *sql.Tx) error { + _, err := issueops.CloseIssueInTx(ctx, tx, id, "done", "tester", "") + return err + }) +} + +// TestCloseRecheckBlocked_ConcurrentSiblingCloses is gastownhall/beads#6716: +// two closes of sibling blockers whose transactions both began before either +// committed each see the other blocker open, so neither writes the dependent's +// row and both commit without a conflicting cell. The in-transaction recompute +// cannot see the other close; the post-commit recheck over the same ids can. +func TestCloseRecheckBlocked_ConcurrentSiblingCloses(t *testing.T) { + store, cleanup := setupConcurrentTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + seedSiblingBlockers(t, ctx, store) + + pendingA, commitA := beginRacingClose(t, ctx, store, "rb-a") + pendingB, commitB := beginRacingClose(t, ctx, store, "rb-b") + for _, pending := range []issueops.BlockedRecheck{pendingA, pendingB} { + if !slices.Contains(pending.IssueIDs, "rb-c") || slices.Contains(pending.IssueIDs, "rb-a") || slices.Contains(pending.IssueIDs, "rb-b") { + t.Fatalf("recorded recheck ids = %v, want the dependent rb-c and never the closed issue", pending.IssueIDs) + } + } + + commitA() + assertIsBlocked(t, ctx, store, "issues", "rb-c", true) + if err := store.recheckBlockedAfterCommit(ctx, pendingA); err != nil { + t.Fatalf("recheck after closing rb-a: %v", err) + } + // rb-b is still open on every committed snapshot: rb-c stays blocked. + assertIsBlocked(t, ctx, store, "issues", "rb-c", true) + + commitB() + // The stale state the in-transaction recompute leaves behind. + assertIsBlocked(t, ctx, store, "issues", "rb-c", true) + if err := store.recheckBlockedAfterCommit(ctx, pendingB); err != nil { + t.Fatalf("recheck after closing rb-b: %v", err) + } + assertIsBlocked(t, ctx, store, "issues", "rb-c", false) + requireCleanTables(ctx, t, store, "issues") + if !doltHasCommitMessage(ctx, t, store, "bd: recheck blocked after close of rb-b") { + t.Fatal("the recheck corrected rb-c but minted no Dolt commit for it") + } +} + +// TestCloseRecheckBlocked_PublicClosePaths covers the store methods that own +// the recheck: a close with dependents leaves the dependent settled and the +// working set clean, and a close with no dependents records nothing to recheck. +func TestCloseRecheckBlocked_PublicClosePaths(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + seedSiblingBlockers(t, ctx, store) + createPerm(t, ctx, store, "rb-lone") + + if err := store.CloseIssue(ctx, "rb-a", "done", "tester", ""); err != nil { + t.Fatalf("close rb-a: %v", err) + } + assertIsBlocked(t, ctx, store, "issues", "rb-c", true) + if err := store.UpdateIssue(ctx, "rb-b", map[string]interface{}{"status": types.StatusClosed}, "tester"); err != nil { + t.Fatalf("close rb-b through update: %v", err) + } + assertIsBlocked(t, ctx, store, "issues", "rb-c", false) + requireCleanTables(ctx, t, store, "issues") + + var pending issueops.BlockedRecheck + if err := store.withWriteTx(ctx, func(tx *sql.Tx) error { + if _, err := issueops.CloseIssueInTx(ctx, tx, "rb-lone", "done", "tester", ""); err != nil { + return err + } + pending = issueops.TakeBlockedRecheck(tx) + return nil + }); err != nil { + t.Fatalf("close rb-lone: %v", err) + } + if !pending.Empty() { + t.Fatalf("close of an issue with no dependents recorded %+v to recheck, want nothing", pending) + } +} + +// TestCloseRecheckBlocked_ConcurrentCloseAndDependencyRemoval is #6716 with a +// dependency removal as the second writer: a close of rb-a racing a removal of +// the rb-c -> rb-b edge. Each transaction still sees the other blocker in +// place, neither writes rb-c, and the one that commits last leaves rb-c stale +// unless its own recorded recheck settles it. +func TestCloseRecheckBlocked_ConcurrentCloseAndDependencyRemoval(t *testing.T) { + store, cleanup := setupConcurrentTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + seedSiblingBlockers(t, ctx, store) + + pendingClose, commitClose := beginRacingClose(t, ctx, store, "rb-a") + pendingRemove, commitRemove := beginRacingWrite(t, ctx, store, "dependency removal rb-c -> rb-b", func(tx *sql.Tx) error { + _, err := issueops.RemoveDependencyInTx(ctx, tx, "rb-c", "rb-b", "tester", true) + return err + }) + if !slices.Contains(pendingRemove.IssueIDs, "rb-c") { + t.Fatalf("dependency removal recorded %v, want the dependent rb-c", pendingRemove.IssueIDs) + } + if want := []string{"dependency removal rb-c -> rb-b"}; !slices.Equal(pendingRemove.Sources, want) { + t.Fatalf("dependency removal Sources = %v, want %v", pendingRemove.Sources, want) + } + + commitClose() + if err := store.recheckBlockedAfterCommit(ctx, pendingClose); err != nil { + t.Fatalf("recheck after closing rb-a: %v", err) + } + // The rb-c -> rb-b edge is still present on every committed snapshot. + assertIsBlocked(t, ctx, store, "issues", "rb-c", true) + + commitRemove() + // The stale state the in-transaction recompute leaves behind. + assertIsBlocked(t, ctx, store, "issues", "rb-c", true) + if err := store.recheckBlockedAfterCommit(ctx, pendingRemove); err != nil { + t.Fatalf("recheck after removing rb-c -> rb-b: %v", err) + } + assertIsBlocked(t, ctx, store, "issues", "rb-c", false) + requireCleanTables(ctx, t, store, "issues") + if !doltHasCommitMessage(ctx, t, store, "bd: recheck blocked after dependency removal rb-c -> rb-b") { + t.Fatal("the recheck corrected rb-c but minted no Dolt commit for it") + } +} + +// TestCloseRecheckBlocked_DeleteRecordsDependents: a delete of a blocker +// records its dependents for the recheck and never the deleted row itself. +func TestCloseRecheckBlocked_DeleteRecordsDependents(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + seedSiblingBlockers(t, ctx, store) + + var pending issueops.BlockedRecheck + if err := store.withWriteTx(ctx, func(tx *sql.Tx) error { + if err := issueops.DeleteIssueInTx(ctx, tx, "rb-b", "tester"); err != nil { + return err + } + pending = issueops.TakeBlockedRecheck(tx) + return nil + }); err != nil { + t.Fatalf("delete rb-b: %v", err) + } + if !slices.Contains(pending.IssueIDs, "rb-c") || slices.Contains(pending.IssueIDs, "rb-b") { + t.Fatalf("delete recorded %v, want the dependent rb-c and never the deleted rb-b", pending.IssueIDs) + } + if want := []string{"delete of rb-b"}; !slices.Equal(pending.Sources, want) { + t.Fatalf("delete Sources = %v, want %v", pending.Sources, want) + } +} + +// TestCloseRecheckBlocked_FailureKeepsTheWrite pins the swallow on this tier — +// the one #6716 was reported on, and the one every other test here reaches by +// calling recheckBlockedAfterCommit directly, which crosses neither runner's +// tail. Both runners log a recheck failure and return nil, so a close whose +// recheck failed is durable and its caller is never told to retry a write that +// landed; restoring `return s.recheckBlockedAfterCommit(...)` on either would +// otherwise turn nothing red. It mirrors the embedded tier's +// TestEmbeddedBlockedRecheckFailureKeepsTheWrite. +func TestCloseRecheckBlocked_FailureKeepsTheWrite(t *testing.T) { + store, cleanup := setupTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + seedSiblingBlockers(t, ctx, store) + + // Fail the recheck and nothing else, the way the embedded test flips + // readOnly: commitWriteTx has already passed its closed check by the time fn + // runs, so marking the store here leaves this close committed and refuses + // only the transaction the recheck opens next. ErrStoreClosed is not + // retryable, so that refusal is immediate and never reaches the breaker. + closeInTx := func(id string) func(tx *sql.Tx) error { + return func(tx *sql.Tx) error { + if _, err := issueops.CloseIssueInTx(ctx, tx, id, "done", "tester", ""); err != nil { + return err + } + store.closed.Store(true) + return nil + } + } + + if err := store.withRetryTx(ctx, closeInTx("rb-a")); err != nil { + store.closed.Store(false) + t.Fatalf("withRetryTx: a committed close whose recheck failed returned %v, want nil", err) + } + store.closed.Store(false) + + if err := store.withWriteTx(ctx, closeInTx("rb-b")); err != nil { + store.closed.Store(false) + t.Fatalf("withWriteTx: a committed close whose recheck failed returned %v, want nil", err) + } + store.closed.Store(false) + + for _, id := range []string{"rb-a", "rb-b"} { + got, err := store.GetIssue(ctx, id) + if err != nil { + t.Fatalf("get %s: %v", id, err) + } + if got.Status != types.StatusClosed { + t.Fatalf("%s is %s after a close whose recheck failed, want the write to be durable", id, got.Status) + } + } + + // The failure the runners swallow is the one that names itself, so the + // warning they log tells a stale dependent from a write that never landed. + store.closed.Store(true) + recheckErr := store.recheckBlockedAfterCommit(ctx, issueops.BlockedRecheck{IssueIDs: []string{"rb-c"}}) + store.closed.Store(false) + if !errors.Is(recheckErr, issueops.ErrBlockedRecheckFailed) { + t.Fatalf("a failed recheck returned %v, want it to carry ErrBlockedRecheckFailed", recheckErr) + } +} diff --git a/internal/storage/dolt/store.go b/internal/storage/dolt/store.go index bc9c17968..d077cff5e 100644 --- a/internal/storage/dolt/store.go +++ b/internal/storage/dolt/store.go @@ -873,19 +873,20 @@ var doltTracer = otel.Tracer("github.com/steveyegge/beads/storage/dolt") // Instruments are registered against the global delegating provider at init time, // so they automatically forward to the real provider once telemetry.Init() runs. var doltMetrics struct { - retryCount metric.Int64Counter - lockWaitMs metric.Float64Histogram - circuitTrips metric.Int64Counter - circuitRejected metric.Int64Counter - serializationErrors metric.Int64Counter - writeRetries metric.Int64Counter - postTxCommitDropped metric.Int64Counter - connAcquireMs metric.Float64Histogram - poolWaitCount metric.Int64Counter - poolWaitMs metric.Float64Histogram - claimVerifyLost metric.Int64Counter - claimVerifyRecovered metric.Int64Counter - ignoredTxFreshPool metric.Int64Counter + retryCount metric.Int64Counter + lockWaitMs metric.Float64Histogram + circuitTrips metric.Int64Counter + circuitRejected metric.Int64Counter + serializationErrors metric.Int64Counter + writeRetries metric.Int64Counter + postTxCommitDropped metric.Int64Counter + blockedRecheckDropped metric.Int64Counter + connAcquireMs metric.Float64Histogram + poolWaitCount metric.Int64Counter + poolWaitMs metric.Float64Histogram + claimVerifyLost metric.Int64Counter + claimVerifyRecovered metric.Int64Counter + ignoredTxFreshPool metric.Int64Counter } func init() { @@ -918,6 +919,10 @@ func init() { metric.WithDescription("Post-tx dolt commits abandoned after retries; the data landed but no dolt commit was minted (change rides the next commit on the branch)"), metric.WithUnit("{commit}"), ) + doltMetrics.blockedRecheckDropped, _ = m.Int64Counter("bd.db.blocked_recheck_dropped", + metric.WithDescription("Post-commit blocked-state rechecks abandoned; the write landed but dependents may carry a stale is_blocked flag until `bd recompute-blocked`"), + metric.WithUnit("{recheck}"), + ) doltMetrics.connAcquireMs, _ = m.Float64Histogram("bd.db.conn_acquire_ms", metric.WithDescription("Time to acquire a pooled connection for a Dolt transaction"), metric.WithUnit("ms"), @@ -1162,8 +1167,10 @@ func (s *DoltStore) withRetryTx(ctx context.Context, fn func(tx *sql.Tx) error) if s.serverMode { bo.MaxElapsedTime = 15 * time.Second } - return backoff.Retry(func() error { - err := s.withWriteTx(ctx, fn) + var pending issueops.BlockedRecheck + if err := backoff.Retry(func() error { + var err error + pending, err = s.commitWriteTx(ctx, fn) if err == nil { if !circuitWriteManaged(ctx) && s.breaker != nil { s.breaker.RecordSuccess() @@ -1206,28 +1213,97 @@ func (s *DoltStore) withRetryTx(ctx context.Context, fn func(tx *sql.Tx) error) return err // pre-commit transient: retryable } return backoff.Permanent(err) - }, backoff.WithContext(bo, ctx)) + }, backoff.WithContext(bo, ctx)); err != nil { + return err + } + logBlockedRecheckFailure(ctx, pending, s.recheckBlockedAfterCommit(ctx, pending)) + return nil } func (s *DoltStore) withWriteTx(ctx context.Context, fn func(tx *sql.Tx) error) error { + pending, err := s.commitWriteTx(ctx, fn) + if err != nil { + return err + } + logBlockedRecheckFailure(ctx, pending, s.recheckBlockedAfterCommit(ctx, pending)) + return nil +} + +// commitWriteTx runs fn in one write transaction and, once it has committed, +// hands back the dependents its unblocking writes recorded for a post-commit +// recheck. The caller runs that recheck outside any retry loop around fn: a +// recheck failure must never replay a write that has already landed. +func (s *DoltStore) commitWriteTx(ctx context.Context, fn func(tx *sql.Tx) error) (issueops.BlockedRecheck, error) { if s.closed.Load() { - return ErrStoreClosed + return issueops.BlockedRecheck{}, ErrStoreClosed } tx, err := s.db.BeginTx(ctx, nil) if err != nil { - return fmt.Errorf("begin write tx: %w", err) + return issueops.BlockedRecheck{}, fmt.Errorf("begin write tx: %w", err) } clearJournalScope := s.scopeEventsJournalTransaction(tx) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() if err := fn(tx); err != nil { - return errors.Join(err, tx.Rollback()) + return issueops.BlockedRecheck{}, errors.Join(err, tx.Rollback()) } if err := tx.Commit(); err != nil { - return wrapSQLCommitError("commit write tx", err) + return issueops.BlockedRecheck{}, wrapSQLCommitError("commit write tx", err) + } + return issueops.TakeBlockedRecheck(tx), nil +} + +// recheckBlockedAfterCommit recomputes the blocked state of the dependents a +// committed unblocking write recorded, on a snapshot that includes every +// concurrent commit (gastownhall/beads#6716). It runs no SQL when nothing was +// recorded, and mints a Dolt commit only when it changed an issues row, so +// the corrected flag reaches history the way the close's own rows did rather +// than sitting dirty in the working set. +// +// It runs on issueops.BlockedRecheckContext: the write it follows is durable, +// so the repair must outlive that write's cancellation, and a recheck must +// never start another recheck. The returned failure is for the caller to log, +// never to return — see logBlockedRecheckFailure. +func (s *DoltStore) recheckBlockedAfterCommit(ctx context.Context, pending issueops.BlockedRecheck) error { + if pending.Empty() || issueops.InBlockedRecheck(ctx) { + return nil + } + ctx, cancel := issueops.BlockedRecheckContext(ctx) + defer cancel() + err := s.withRetryTx(ctx, func(tx *sql.Tx) error { + result, err := issueops.RecomputeIsBlockedInTxWithResult(ctx, tx, pending.IssueIDs, pending.WispIDs) + if err != nil || !result.IssueRowsChanged { + return err + } + return s.doltAddAndCommitInTx(ctx, tx, []string{"issues"}, pending.CommitMessage()) + }) + if err != nil { + return issueops.BlockedRecheckFailed(err) } return nil } +// logBlockedRecheckFailure reports a post-commit recheck failure instead of +// returning it. The write it followed is committed and durable, and every +// caller of a store write reads an error as "the mutation did not land": +// surfacing this one would make automated callers retry and double-apply, +// the inversion issue_operations_tx.go documents. What is left behind is the +// stale is_blocked flag `bd doctor` and `bd recompute-blocked` repair, which +// is the state every write had before the recheck existed. +// +// Because the failure stops here, this counter and line are its only trace — +// the same pair post_tx_commit_dropped uses for the same situation, a post-tx +// step that failed while its write stayed durable (issue_operations_tx.go). The +// counter is what a fleet alerts on; the line names the rows to repair. +func logBlockedRecheckFailure(ctx context.Context, pending issueops.BlockedRecheck, err error) { + if err == nil { + return + } + doltMetrics.blockedRecheckDropped.Add(ctx, 1) + log.Printf("warning: %s", issueops.BlockedRecheckFailureMessage(pending, err)) +} + // SetEventsJournalEnabled activates the journal for this store instance only. func (s *DoltStore) SetEventsJournalEnabled(enabled bool) { s.eventsJournalEnabled.Store(enabled) diff --git a/internal/storage/embeddeddolt/blocked_recheck_internal_test.go b/internal/storage/embeddeddolt/blocked_recheck_internal_test.go new file mode 100644 index 000000000..d96b235d4 --- /dev/null +++ b/internal/storage/embeddeddolt/blocked_recheck_internal_test.go @@ -0,0 +1,164 @@ +//go:build cgo + +package embeddeddolt + +import ( + "context" + "database/sql" + "errors" + "path/filepath" + "slices" + "testing" + + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// TestEmbeddedBlockedRecheckRecordsUnblockingWrites pins, on a real engine, +// that an unblocking write running on the embedded store's own scoped +// transaction records the dependents it recomputed for the post-commit +// recheck: a close, a dependency removal, a delete. The embedded store +// serializes its transactions so the skew of gastownhall/beads#6716 cannot +// occur here; the recording contract is what this tier can verify, and the +// server tier's close_recheck_blocked_test.go races it. +func TestEmbeddedBlockedRecheckRecordsUnblockingWrites(t *testing.T) { + ctx := context.Background() + store, err := Open(ctx, filepath.Join(t.TempDir(), ".beads"), "recheck", "main") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + if err := store.SetConfig(ctx, "issue_prefix", "rb"); err != nil { + t.Fatal(err) + } + for _, id := range []string{"rb-a", "rb-b", "rb-c"} { + iss := &types.Issue{ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := store.CreateIssue(ctx, iss, "tester"); err != nil { + t.Fatalf("create %s: %v", id, err) + } + } + for _, blocker := range []string{"rb-a", "rb-b"} { + if err := store.AddDependency(ctx, &types.Dependency{IssueID: "rb-c", DependsOnID: blocker, Type: types.DepBlocks}, "tester"); err != nil { + t.Fatalf("add dependency rb-c -> %s: %v", blocker, err) + } + } + + // record runs one unblocking write in a scoped transaction and returns what + // it recorded, taking it before the store's own post-commit recheck would. + record := func(name string, write func(tx *sql.Tx) error) issueops.BlockedRecheck { + t.Helper() + var pending issueops.BlockedRecheck + if err := store.withConn(ctx, true, func(tx *sql.Tx) error { + if err := write(tx); err != nil { + return err + } + pending = issueops.TakeBlockedRecheck(tx) + return nil + }); err != nil { + t.Fatalf("%s: %v", name, err) + } + return pending + } + expect := func(name string, pending issueops.BlockedRecheck, source string, excluded ...string) { + t.Helper() + if !slices.Contains(pending.IssueIDs, "rb-c") { + t.Fatalf("%s recorded %v, want the dependent rb-c", name, pending.IssueIDs) + } + for _, id := range excluded { + if slices.Contains(pending.IssueIDs, id) { + t.Fatalf("%s recorded %v, must never recheck %s", name, pending.IssueIDs, id) + } + } + if want := []string{source}; !slices.Equal(pending.Sources, want) { + t.Fatalf("%s Sources = %v, want %v", name, pending.Sources, want) + } + } + + expect("close", record("close rb-a", func(tx *sql.Tx) error { + _, err := issueops.CloseIssueInTx(ctx, tx, "rb-a", "done", "tester", "") + return err + }), "close of rb-a", "rb-a") + + expect("dependency removal", record("remove rb-c -> rb-b", func(tx *sql.Tx) error { + _, err := issueops.RemoveDependencyInTx(ctx, tx, "rb-c", "rb-b", "tester", true) + return err + }), "dependency removal rb-c -> rb-b") + + // Re-add the edge so the delete has a dependent to record. + if err := store.AddDependency(ctx, &types.Dependency{IssueID: "rb-c", DependsOnID: "rb-b", Type: types.DepBlocks}, "tester"); err != nil { + t.Fatalf("re-add dependency: %v", err) + } + expect("delete", record("delete rb-b", func(tx *sql.Tx) error { + return issueops.DeleteIssueInTx(ctx, tx, "rb-b", "") + }), "delete of rb-b", "rb-b") + + // Through the public surface the recheck runs after commit and the graph + // ends settled: both blockers gone, rb-c unblocked. + got, err := store.GetIssue(ctx, "rb-c") + if err != nil { + t.Fatalf("get rb-c: %v", err) + } + if got.IsBlocked { + t.Fatal("rb-c is still blocked after its last blocker was deleted") + } +} + +// TestEmbeddedBlockedRecheckFailureKeepsTheWrite pins what a *failing* recheck +// does at the store seam, which is the half of the contract the wrapper's unit +// test cannot reach: the write it follows has already committed, so the store +// call still succeeds and the row is durable, and the failure the store's own +// withConn/commitConn path produces carries issueops.ErrBlockedRecheckFailed +// for the warning that reports it. +func TestEmbeddedBlockedRecheckFailureKeepsTheWrite(t *testing.T) { + ctx := context.Background() + store, err := Open(ctx, filepath.Join(t.TempDir(), ".beads"), "recheckfail", "main") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + if err := store.SetConfig(ctx, "issue_prefix", "rb"); err != nil { + t.Fatal(err) + } + for _, id := range []string{"rb-a", "rb-b", "rb-c"} { + iss := &types.Issue{ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := store.CreateIssue(ctx, iss, "tester"); err != nil { + t.Fatalf("create %s: %v", id, err) + } + } + for _, blocker := range []string{"rb-a", "rb-b"} { + if err := store.AddDependency(ctx, &types.Dependency{IssueID: "rb-c", DependsOnID: blocker, Type: types.DepBlocks}, "tester"); err != nil { + t.Fatalf("add dependency rb-c -> %s: %v", blocker, err) + } + } + + // Fail the recheck and nothing else: the write's own commitConn has + // already passed the read-only check when fn runs, so flipping the flag + // here leaves the close committed and refuses only the transaction the + // recheck opens afterwards. + if err := store.withConn(ctx, true, func(tx *sql.Tx) error { + if _, err := issueops.CloseIssueInTx(ctx, tx, "rb-a", "done", "tester", ""); err != nil { + return err + } + store.readOnly = true + return nil + }); err != nil { + t.Fatalf("a committed write whose recheck failed returned %v, want nil", err) + } + store.readOnly = false + + closed, err := store.GetIssue(ctx, "rb-a") + if err != nil { + t.Fatalf("get rb-a: %v", err) + } + if closed.Status != types.StatusClosed { + t.Fatalf("rb-a is %s after a close whose recheck failed, want the write to be durable", closed.Status) + } + + // The failure itself, on the path that produces it. + store.readOnly = true + recheckErr := store.recheckBlockedAfterCommit(ctx, issueops.BlockedRecheck{IssueIDs: []string{"rb-c"}}) + store.readOnly = false + if !errors.Is(recheckErr, issueops.ErrBlockedRecheckFailed) { + t.Fatalf("a failed recheck returned %v, want it to carry ErrBlockedRecheckFailed", recheckErr) + } +} diff --git a/internal/storage/embeddeddolt/store.go b/internal/storage/embeddeddolt/store.go index 58c96b785..986ad4af7 100644 --- a/internal/storage/embeddeddolt/store.go +++ b/internal/storage/embeddeddolt/store.go @@ -261,7 +261,69 @@ func openReadOnly(ctx context.Context, beadsDir, database, branch string, checkB // returns regardless of outcome. // // The database must already exist (created during initSchema). -func (s *EmbeddedDoltStore) withConn(ctx context.Context, commit bool, fn func(tx *sql.Tx) error) (err error) { +func (s *EmbeddedDoltStore) withConn(ctx context.Context, commit bool, fn func(tx *sql.Tx) error) error { + pending, err := s.commitConn(ctx, commit, fn) + if err != nil { + return err + } + logBlockedRecheckFailure(pending, s.recheckBlockedAfterCommit(ctx, pending)) + return nil +} + +// logBlockedRecheckFailure reports a post-commit recheck failure instead of +// returning it. The write it followed is committed and durable, and every +// caller of a store write reads an error as "the mutation did not land": +// surfacing this one would make automated callers retry and double-apply. +// What is left behind is the stale is_blocked flag `bd doctor` and +// `bd recompute-blocked` repair, which is the state every write had before +// the recheck existed. +// +// The sentence is issueops.BlockedRecheckFailureMessage, shared with the Dolt +// store; only the sink differs. This store has no metrics registry, so unlike +// the Dolt store's counter this line is the whole signal. +func logBlockedRecheckFailure(pending issueops.BlockedRecheck, err error) { + if err == nil { + return + } + fmt.Fprintf(os.Stderr, "warning: %s\n", issueops.BlockedRecheckFailureMessage(pending, err)) +} + +// recheckBlockedAfterCommit recomputes the blocked state of the dependents a +// committed unblocking write recorded, on a fresh snapshot +// (gastownhall/beads#6716). +// +// Embedded transactions serialize: commitConn opens a fresh OpenSQL handle +// per transaction, and OpenSQL (open.go) waits in a backoff with no elapsed +// time limit for the engine, so a second handle blocks until the first has +// been cleaned up. Two transactions therefore never run against overlapping +// snapshots in one process, and the skew of #6716 cannot occur here; no +// embedded reproduction exists. The recheck is kept so the embedded store +// honors the same contract as the server store — a stale row recorded by +// one transaction is settled after its commit — and it runs only after the +// first handle's cleanup, so it cannot deadlock on itself. It runs no SQL +// when nothing was recorded. +// +// It runs on issueops.BlockedRecheckContext: the write it follows is durable, +// so the repair must outlive that write's cancellation, and a recheck must +// never start another recheck. The returned failure is for the caller to log, +// never to return — see logBlockedRecheckFailure. +func (s *EmbeddedDoltStore) recheckBlockedAfterCommit(ctx context.Context, pending issueops.BlockedRecheck) error { + if pending.Empty() || issueops.InBlockedRecheck(ctx) { + return nil + } + ctx, cancel := issueops.BlockedRecheckContext(ctx) + defer cancel() + if _, err := s.commitConn(ctx, true, func(tx *sql.Tx) error { + return issueops.RecomputeIsBlockedInTx(ctx, tx, pending.IssueIDs, pending.WispIDs) + }); err != nil { + return issueops.BlockedRecheckFailed(err) + } + return nil +} + +// commitConn is withConn's transaction: it hands back the dependents the +// transaction's unblocking writes recorded once it has committed. +func (s *EmbeddedDoltStore) commitConn(ctx context.Context, commit bool, fn func(tx *sql.Tx) error) (pending issueops.BlockedRecheck, err error) { if s.closed.Load() { err = errClosed return @@ -291,6 +353,8 @@ func (s *EmbeddedDoltStore) withConn(ctx context.Context, commit bool, fn func(t } clearJournalScope := issueops.ScopeEventsJournalTransaction(tx, s.eventsJournalEnabled.Load()) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() if fnErr := fn(tx); fnErr != nil { err = errors.Join(fnErr, tx.Rollback()) @@ -307,6 +371,7 @@ func (s *EmbeddedDoltStore) withConn(ctx context.Context, commit bool, fn func(t return } committed = true + pending = issueops.TakeBlockedRecheck(tx) return } diff --git a/internal/storage/issueops/blocked_recheck.go b/internal/storage/issueops/blocked_recheck.go new file mode 100644 index 000000000..7204704bd --- /dev/null +++ b/internal/storage/issueops/blocked_recheck.go @@ -0,0 +1,302 @@ +package issueops + +import ( + "context" + "errors" + "fmt" + "strings" + "sync" + "time" + + "github.com/steveyegge/beads/internal/types" +) + +// BlockedRecheck names the dependents whose is_blocked a write recomputed +// inside a transaction, so the store can recompute them again on a fresh +// snapshot once that transaction has committed. +// +// The in-transaction recompute reads the transaction's start snapshot. The +// writes that record here are the ones that take a blocker away from a +// dependent: a close, an update to an inactive status, a dependency removal, +// a delete. Two such writes racing on the blockers of one dependent each see +// the other blocker still in place, each leaves the dependent blocked without +// writing its row, and both commit with no conflicting cell — so the +// dependent stays hidden from `bd ready` until a repair +// (gastownhall/beads#6716, first seen as two closes of sibling blockers). A +// recheck over the same ids after commit sees both writes and settles the +// flag. The recheck is a full recompute, so it is correct whichever way the +// flag has to move — removing the one closed child of an any-children gate +// can block its waiter, and the recheck settles that too. Writes that add a +// blocker (a reopen, a dependency add, a parent-child add) recompute the same +// way and are not recorded here; a racing pair where the last committer adds +// a blocker can leave a dependent flagged ready while blocked. +// +// # What is scoped, and what is not +// +// Recording only reaches a recheck through a transaction a store scoped (see +// ScopeBlockedRecheckTransaction), and exactly two write runners scope one: +// DoltStore's commitWriteTx and EmbeddedDoltStore's commitConn. A write that +// reaches this package on any other transaction records nothing and behaves +// exactly as it did before the recheck existed — the skew above survives on +// it. Those routes are known and deliberately out of scope here: +// +// - the uow/domain-db routing system (internal/storage/uow, with the +// in-transaction recomputes at internal/storage/domain/db/issue.go and +// dependency.go), which serves `bd batch` when proxied and every write a +// uow provider serves. Scoping it needs more than these two lines: +// doltServerTx.Runner() hands out the pinned *sql.Conn, not the *sql.Tx +// this package keys its scope on, so the key has to be threaded through +// the wrapper — the caveat below, already real rather than hypothetical. +// - storage.Transaction (internal/storage/dolt/transaction.go, reached +// through DoltStore.RunInTransaction on a pinned connection), which is +// how `bd batch` closes, status updates and dep removals land when not +// proxied, and how `bd cook`'s proto-subgraph delete, `bd mol squash` and +// `bd mol burn` delete and close. +// - the legacy DoltStore.RemoveDependencyWithOptions (raw BeginTx in +// internal/storage/dolt/dependencies.go), which `bd duplicates --merge` +// uses to remove parent-child edges. +// - the wisp writers (internal/storage/dolt/wisps.go closeWisp/updateWisp/ +// deleteWisp and ephemeral_routing.go's demote-to-wisp), which run raw +// unscoped transactions. The recompute joins depends_on_wisp_id, so a +// permanent issue blocked by two wisps whose removals race keeps the same +// stale flag. +// +// Every one of these leaves the pre-existing, operator-repairable state +// (`bd doctor`, `bd recompute-blocked`); none is made worse by the recheck. +type BlockedRecheck struct { + IssueIDs []string + WispIDs []string + // Sources label the writes that recorded the dependents, one short human + // phrase each ("close of rb-a", "dependency removal rb-c -> rb-a", + // "delete of rb-b"); they name the Dolt commit a recheck mints. At most + // recheckSourceLimit are kept — SourceCount counts them all. + Sources []string + // SourceCount is how many writes recorded ids in this transaction, + // including the ones past recheckSourceLimit that Sources does not name. + SourceCount int +} + +// recheckSourceLimit is how many names a bounded recheck line prints in full — +// the writes a commit message names, and the dependents a failure message +// names — before the rest reach it as a count. One transaction can close, +// update or delete an unbounded number of issues (batch_closer.go closes N in +// one), so an uncapped line would grow with N — the same unbounded growth +// deleteRecheckLabel already bounds within a single delete. +const recheckSourceLimit = 3 + +// Empty reports whether there is nothing to recheck. +func (r BlockedRecheck) Empty() bool { + return len(r.IssueIDs) == 0 && len(r.WispIDs) == 0 +} + +// CommitMessage is the Dolt commit message a recheck that changed a row mints. +// +// It always names the writes Sources kept and only counts the ones past +// recheckSourceLimit: this commit is the sole record of which write triggered a +// repair, so dropping the retained names for a bare count would discard exactly +// what they are kept for. The count is of the unnamed writes, not of all of +// them, which also keeps the line honest when Sources deduplicated two writes +// that rendered the same label (two equal-sized bulk deletes) and SourceCount +// therefore runs ahead of it. +func (r BlockedRecheck) CommitMessage() string { + message := "bd: recheck blocked after " + strings.Join(r.Sources, ", ") + if unnamed := r.SourceCount - len(r.Sources); unnamed > 0 { + message += fmt.Sprintf(" and %d more", unnamed) + } + return message +} + +// blockedRecheckTransactions holds the open scopes, one per transaction. It is +// keyed by the DBTX interface value: a write records into a scope only when +// the very same *sql.Tx the store scoped reaches noteBlockedRecheck. A future +// DBTX wrapper around that tx would compare unequal and silently record +// nothing — the same shape, and the same caveat, as the events-journal scope +// in journal.go. +var blockedRecheckTransactions sync.Map // map[DBTX]*BlockedRecheck; entries live for one transaction + +// ErrBlockedRecheckFailed marks a failure of the post-commit recheck of +// dependents' blocked state. The write that preceded it is committed and +// durable; only the recheck failed, so at worst a dependent carries the stale +// is_blocked flag that `bd doctor` and `bd recompute-blocked` already repair. +// +// A store never returns it to the caller of the write. An error out of a +// store call means the write did not land, and every caller reads it that +// way: surfacing a post-commit failure would make automated callers retry an +// applied mutation and double-apply it (the contract at +// internal/storage/dolt/issue_operations_tx.go). Stores wrap the failure with +// this sentinel and log it instead, which is what makes the two +// distinguishable in a log line and keeps the cause reachable for anything +// that inspects one. +var ErrBlockedRecheckFailed = errors.New("blocked-state recheck after a committed write failed") + +// BlockedRecheckFailed wraps a recheck failure so both ErrBlockedRecheckFailed +// and the underlying cause stay reachable through errors.Is and errors.As. +func BlockedRecheckFailed(err error) error { + return fmt.Errorf("%w: %w", ErrBlockedRecheckFailed, err) +} + +// BlockedRecheckFailureMessage is what a store says about a failed recheck. +// Both stores log the same sentence (each through its own package's sink) so +// this text lives next to the sentinel it reports rather than twice in two +// store packages. +// +// It names the rows left unrechecked and the command that repairs them because +// the failure reaches nobody else: the write committed, its caller was handed +// nil, and the only other signal is the counter beside this line. An operator +// who cannot tell which dependents went stale cannot act on it, and the whole +// defect the recheck repairs is invisible staleness. +func BlockedRecheckFailureMessage(pending BlockedRecheck, err error) string { + return fmt.Sprintf("%v; left unrechecked and possibly stale: %s — repair with `bd recompute-blocked`", + err, pending.unrecheckedNames()) +} + +// unrecheckedNames names the recorded dependents, bounded by +// recheckSourceLimit: one transaction can record an unbounded number of them. +func (r BlockedRecheck) unrecheckedNames() string { + ids := make([]string, 0, len(r.IssueIDs)+len(r.WispIDs)) + ids = append(ids, r.IssueIDs...) + ids = append(ids, r.WispIDs...) + if len(ids) <= recheckSourceLimit { + return strings.Join(ids, ", ") + } + return fmt.Sprintf("%s and %d more", strings.Join(ids[:recheckSourceLimit], ", "), len(ids)-recheckSourceLimit) +} + +type blockedRecheckContextKey struct{} + +// BlockedRecheckTimeout bounds one post-commit recheck. It sits above the +// stores' own write-retry budget so the retry policy, not this ceiling, is +// what normally ends a struggling recheck; it exists so a wedged connection +// cannot hold the caller of an already-committed write indefinitely. +const BlockedRecheckTimeout = 30 * time.Second + +// BlockedRecheckContext returns the context a post-commit recheck runs on. +// +// It is detached from the caller's cancellation: the write being repaired has +// already committed, so a context canceled between the commit and the recheck +// (a CLI deadline, a disconnected HTTP client, shutdown) must not skip the +// repair and leave exactly the stale flag the recheck exists to settle. It +// carries its own deadline instead, and it is marked so that InBlockedRecheck +// reports true inside it. +func BlockedRecheckContext(ctx context.Context) (context.Context, context.CancelFunc) { + detached := context.WithValue(context.WithoutCancel(ctx), blockedRecheckContextKey{}, struct{}{}) + return context.WithTimeout(detached, BlockedRecheckTimeout) +} + +// InBlockedRecheck reports whether ctx is already running a post-commit +// recheck, so a store can refuse to start another one from inside it. +// +// A recheck runs on the store's ordinary write-transaction runner, which +// scopes its transaction and rechecks whatever that transaction recorded. It +// terminates today only because the recompute the recheck runs +// (RecomputeIsBlockedInTxWithResult) records nothing — an invariant no +// signature enforces. A change that made the recompute record would otherwise +// recurse once per level until the context died. This is the guard rail: +// one write, at most one recheck. +func InBlockedRecheck(ctx context.Context) bool { + _, ok := ctx.Value(blockedRecheckContextKey{}).(struct{}) + return ok +} + +// ScopeBlockedRecheckTransaction lets the unblocking writes in tx record the +// dependents they recomputed, for TakeBlockedRecheck once tx has committed. +// Store implementations call it right after BeginTx and run the returned +// cleanup when the transaction ends. An unscoped transaction records nothing. +func ScopeBlockedRecheckTransaction(tx DBTX) func() { + if tx == nil { + return func() {} + } + blockedRecheckTransactions.Store(tx, &BlockedRecheck{}) + return func() { blockedRecheckTransactions.Delete(tx) } +} + +// TakeBlockedRecheck returns the dependents recorded in tx and clears them. +func TakeBlockedRecheck(tx DBTX) BlockedRecheck { + scope, ok := blockedRecheckTransactions.Load(tx) + if !ok { + return BlockedRecheck{} + } + pending := scope.(*BlockedRecheck) + taken := *pending + *pending = BlockedRecheck{} + return taken +} + +// noteBlockedRecheck records the dependents a write recomputed in tx. source +// labels the write for the recheck's commit message. exclude names ids this +// call leaves out: for a close or status update, the issue whose row this +// transaction wrote (a concurrent writer conflicts on that row instead of +// racing past it); for a delete, the rows that no longer exist. A dependency +// removal excludes nothing — its dependent is exactly the row that needs +// rechecking. exclude filters this call's ids only: an id an earlier write in +// the same transaction recorded stays pending, and rechecking a row that a +// later write deleted is a no-op, so that is harmless. +// +// A call that contributes no new id contributes no source either: the recheck +// it would name has nothing of its own to recheck, and naming it would put +// writes in the commit message that the recheck never touched. +func noteBlockedRecheck(tx DBTX, source string, exclude []string, issueIDs, wispIDs []string) { + scope, ok := blockedRecheckTransactions.Load(tx) + if !ok { + return + } + pending := scope.(*BlockedRecheck) + issues, addedIssues := appendRecheckIDs(pending.IssueIDs, issueIDs, exclude) + wisps, addedWisps := appendRecheckIDs(pending.WispIDs, wispIDs, exclude) + pending.IssueIDs, pending.WispIDs = issues, wisps + if !addedIssues && !addedWisps { + return + } + pending.SourceCount++ + if len(pending.Sources) < recheckSourceLimit { + pending.Sources, _ = appendRecheckIDs(pending.Sources, []string{source}, nil) + } +} + +// appendRecheckIDs adds the ids of one write that are neither excluded nor +// already pending, and reports whether it added any. +func appendRecheckIDs(pending, ids, exclude []string) ([]string, bool) { + seen := make(map[string]bool, len(pending)+len(exclude)) + for _, id := range pending { + seen[id] = true + } + for _, id := range exclude { + seen[id] = true + } + added := false + for _, id := range ids { + if !seen[id] { + seen[id] = true + pending = append(pending, id) + added = true + } + } + return pending, added +} + +// deleteRecheckLabel is the Sources label for a delete of ids. Up to three +// ids are named; beyond that only the count, so a bulk delete cannot grow a +// commit message without bound. scope, when set, says where the ids came +// from ("from "). +func deleteRecheckLabel(ids []string, scope string) string { + var label string + if len(ids) <= 3 { + label = "delete of " + strings.Join(ids, ", ") + } else { + label = fmt.Sprintf("delete of %d issues", len(ids)) + } + if scope != "" { + label += " " + scope + } + return label +} + +// statusChangeRecheckLabel is the Sources label for a status change of id to +// the inactive newStatus. A close through update reads the same as a close +// through CloseIssue, so the two paths mint the same commit message. +func statusChangeRecheckLabel(id, newStatus string) string { + if newStatus == string(types.StatusClosed) { + return "close of " + id + } + return fmt.Sprintf("status change of %s to %s", id, newStatus) +} diff --git a/internal/storage/issueops/blocked_recheck_test.go b/internal/storage/issueops/blocked_recheck_test.go new file mode 100644 index 000000000..9206a9f8a --- /dev/null +++ b/internal/storage/issueops/blocked_recheck_test.go @@ -0,0 +1,213 @@ +package issueops + +import ( + "context" + "errors" + "slices" + "strings" + "testing" + + sqlmock "github.com/DATA-DOG/go-sqlmock" +) + +// scopedRecheckTx opens a sqlmock transaction scoped for recheck recording. +func scopedRecheckTx(t *testing.T) DBTX { + t.Helper() + db, mock, err := sqlmock.New() + if err != nil { + t.Fatalf("sqlmock.New: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + mock.ExpectBegin() + tx, err := db.BeginTx(t.Context(), nil) + if err != nil { + t.Fatalf("BeginTx: %v", err) + } + t.Cleanup(ScopeBlockedRecheckTransaction(tx)) + return tx +} + +// TestBlockedRecheck_RecordsEveryUnblockingWrite pins the recording contract +// the stores rely on: ids are deduplicated across writes, an excluded id never +// reaches the recheck even when a later write names it, sources are labels +// joined in order, and Take clears the scope. +func TestBlockedRecheck_RecordsEveryUnblockingWrite(t *testing.T) { + tx := scopedRecheckTx(t) + + noteBlockedRecheck(tx, "close of rb-a", []string{"rb-a"}, []string{"rb-a", "rb-c"}, nil) + noteBlockedRecheck(tx, "dependency removal rb-c -> rb-b", nil, []string{"rb-c"}, []string{"rb-w"}) + noteBlockedRecheck(tx, deleteRecheckLabel([]string{"rb-b"}, ""), []string{"rb-b"}, []string{"rb-b", "rb-d"}, nil) + noteBlockedRecheck(tx, "close of rb-a", []string{"rb-a"}, []string{"rb-c"}, nil) + + pending := TakeBlockedRecheck(tx) + if want := []string{"rb-c", "rb-d"}; !slices.Equal(pending.IssueIDs, want) { + t.Fatalf("IssueIDs = %v, want %v", pending.IssueIDs, want) + } + if want := []string{"rb-w"}; !slices.Equal(pending.WispIDs, want) { + t.Fatalf("WispIDs = %v, want %v", pending.WispIDs, want) + } + if want := "bd: recheck blocked after close of rb-a, dependency removal rb-c -> rb-b, delete of rb-b"; pending.CommitMessage() != want { + t.Fatalf("CommitMessage() = %q, want %q", pending.CommitMessage(), want) + } + if again := TakeBlockedRecheck(tx); !again.Empty() || len(again.Sources) != 0 { + t.Fatalf("second Take returned %+v, want an empty scope", again) + } +} + +// TestBlockedRecheck_BoundsTheCommitMessage: one transaction can close or +// delete an unbounded number of issues, so the recheck's commit message names +// at most recheckSourceLimit writes and counts the rest — it keeps naming the +// ones it retained, because this commit is the only record of which write +// triggered the repair. A write that contributed no id to recheck is not named +// at all, since the recheck never touches anything on its behalf. +func TestBlockedRecheck_BoundsTheCommitMessage(t *testing.T) { + tx := scopedRecheckTx(t) + + // A write whose only recomputed id is its own excluded row adds nothing. + noteBlockedRecheck(tx, "close of rb-lonely", []string{"rb-lonely"}, []string{"rb-lonely"}, nil) + if pending := TakeBlockedRecheck(tx); !pending.Empty() || len(pending.Sources) != 0 || pending.SourceCount != 0 { + t.Fatalf("a write that recorded no id left %+v, want an untouched scope", pending) + } + + for _, id := range []string{"rb-1", "rb-2", "rb-3", "rb-4"} { + noteBlockedRecheck(tx, "close of blocker-"+id, nil, []string{id}, nil) + } + pending := TakeBlockedRecheck(tx) + if pending.SourceCount != 4 || len(pending.Sources) != recheckSourceLimit { + t.Fatalf("SourceCount = %d, Sources = %v, want 4 recorded writes and %d names", pending.SourceCount, pending.Sources, recheckSourceLimit) + } + want := "bd: recheck blocked after close of blocker-rb-1, close of blocker-rb-2, close of blocker-rb-3 and 1 more" + if pending.CommitMessage() != want { + t.Fatalf("CommitMessage() = %q, want %q", pending.CommitMessage(), want) + } +} + +// TestBlockedRecheck_CountsOnlyTheWritesItDidNotName: Sources is deduplicated +// and SourceCount is not, so two writes rendering one label (two equal-sized +// bulk deletes) leave the count ahead of the names. The message counts the +// writes it did not name rather than all of them, so that skew cannot make it +// claim more unnamed writes than exist — or drop the one name it has. +func TestBlockedRecheck_CountsOnlyTheWritesItDidNotName(t *testing.T) { + tx := scopedRecheckTx(t) + + label := deleteRecheckLabel([]string{"rb-1", "rb-2", "rb-3", "rb-4"}, "") + noteBlockedRecheck(tx, label, nil, []string{"rb-c"}, nil) + noteBlockedRecheck(tx, label, nil, []string{"rb-d"}, nil) + + pending := TakeBlockedRecheck(tx) + if pending.SourceCount != 2 || len(pending.Sources) != 1 { + t.Fatalf("SourceCount = %d, Sources = %v, want 2 recorded writes deduplicated to 1 name", pending.SourceCount, pending.Sources) + } + if want := "bd: recheck blocked after delete of 4 issues and 1 more"; pending.CommitMessage() != want { + t.Fatalf("CommitMessage() = %q, want %q", pending.CommitMessage(), want) + } +} + +// TestBlockedRecheck_UnscopedTransactionRecordsNothing: a transaction the +// store never scoped (or whose scope already ended) records and returns +// nothing, so the stores' Tx surfaces keep their pre-recheck behaviour. +func TestBlockedRecheck_UnscopedTransactionRecordsNothing(t *testing.T) { + db, mock, err := sqlmock.New() + if err != nil { + t.Fatalf("sqlmock.New: %v", err) + } + t.Cleanup(func() { _ = db.Close() }) + mock.ExpectBegin() + tx, err := db.BeginTx(t.Context(), nil) + if err != nil { + t.Fatalf("BeginTx: %v", err) + } + noteBlockedRecheck(tx, "close of rb-a", []string{"rb-a"}, []string{"rb-c"}, nil) + if pending := TakeBlockedRecheck(tx); !pending.Empty() { + t.Fatalf("unscoped tx recorded %+v, want nothing", pending) + } + ScopeBlockedRecheckTransaction(nil)() // a nil tx is a no-op scope +} + +func TestBlockedRecheck_Labels(t *testing.T) { + cases := []struct { + name string + got string + want string + }{ + {"single delete", deleteRecheckLabel([]string{"rb-b"}, ""), "delete of rb-b"}, + {"three named", deleteRecheckLabel([]string{"a", "b", "c"}, ""), "delete of a, b, c"}, + {"count beyond three", deleteRecheckLabel([]string{"a", "b", "c", "d"}, ""), "delete of 4 issues"}, + {"source repo scope", deleteRecheckLabel(make([]string, 40), "from github.com/x/y"), "delete of 40 issues from github.com/x/y"}, + {"close through update", statusChangeRecheckLabel("rb-b", "closed"), "close of rb-b"}, + {"pin through update", statusChangeRecheckLabel("rb-b", "pinned"), "status change of rb-b to pinned"}, + } + for _, tc := range cases { + if tc.got != tc.want { + t.Errorf("%s: got %q, want %q", tc.name, tc.got, tc.want) + } + } +} + +// TestBlockedRecheckContext_SurvivesItsCallerAndGuardsRecursion pins the two +// properties the stores' post-commit recheck depends on: it keeps running +// after the write's own context is cancelled (the write is already durable, +// so a cancelled caller must not skip the repair), and it marks itself so a +// recheck cannot start another recheck. +func TestBlockedRecheckContext_SurvivesItsCallerAndGuardsRecursion(t *testing.T) { + caller, cancelCaller := context.WithCancel(t.Context()) + if InBlockedRecheck(caller) { + t.Fatal("an ordinary write context reads as a recheck") + } + + ctx, cancel := BlockedRecheckContext(caller) + defer cancel() + cancelCaller() + + if err := ctx.Err(); err != nil { + t.Fatalf("recheck context died with its caller: %v", err) + } + if _, ok := ctx.Deadline(); !ok { + t.Fatal("recheck context has no deadline of its own") + } + if !InBlockedRecheck(ctx) { + t.Fatal("a recheck context does not report itself, so a recheck could start another") + } +} + +// TestBlockedRecheckFailed_KeepsSentinelAndCause: a store wraps a recheck +// failure so its log line tells a committed write whose recheck failed from a +// write that never landed, without losing the underlying cause. +func TestBlockedRecheckFailed_KeepsSentinelAndCause(t *testing.T) { + cause := errors.New("dolt: connection reset") + err := BlockedRecheckFailed(cause) + if !errors.Is(err, ErrBlockedRecheckFailed) { + t.Fatalf("errors.Is(err, ErrBlockedRecheckFailed) = false for %v", err) + } + if !errors.Is(err, cause) { + t.Fatalf("errors.Is(err, cause) = false for %v", err) + } + if want := "blocked-state recheck after a committed write failed: dolt: connection reset"; err.Error() != want { + t.Fatalf("err = %q, want %q", err.Error(), want) + } + if errors.Is(cause, ErrBlockedRecheckFailed) { + t.Fatal("a bare cause must not read as a recheck failure") + } +} + +// TestBlockedRecheckFailureMessage_NamesTheStaleRowsAndTheRepair: the failure +// never reaches the write's caller, so this line plus the Dolt store's counter +// are its only trace. An operator reading it has to learn which rows went stale +// and what repairs them, and a bulk write must not turn one warning into an +// unbounded line. +func TestBlockedRecheckFailureMessage_NamesTheStaleRowsAndTheRepair(t *testing.T) { + cause := BlockedRecheckFailed(errors.New("dolt: connection reset")) + + got := BlockedRecheckFailureMessage(BlockedRecheck{IssueIDs: []string{"rb-c"}, WispIDs: []string{"rb-w"}}, cause) + want := "blocked-state recheck after a committed write failed: dolt: connection reset; " + + "left unrechecked and possibly stale: rb-c, rb-w — repair with `bd recompute-blocked`" + if got != want { + t.Fatalf("message = %q, want %q", got, want) + } + + bulk := BlockedRecheck{IssueIDs: []string{"rb-1", "rb-2", "rb-3", "rb-4", "rb-5"}} + got = BlockedRecheckFailureMessage(bulk, cause) + if want := "left unrechecked and possibly stale: rb-1, rb-2, rb-3 and 2 more — repair with `bd recompute-blocked`"; !strings.HasSuffix(got, want) { + t.Fatalf("message for %d ids = %q, want it to end with %q", len(bulk.IssueIDs), got, want) + } +} diff --git a/internal/storage/issueops/bulk_ops.go b/internal/storage/issueops/bulk_ops.go index 0077d0463..e96d5b513 100644 --- a/internal/storage/issueops/bulk_ops.go +++ b/internal/storage/issueops/bulk_ops.go @@ -234,6 +234,7 @@ func DeleteIssuesBySourceRepoInTx(ctx context.Context, tx *sql.Tx, sourceRepo st if err := RecomputeIsBlockedInTx(ctx, tx, affectedIssues, affectedWisps); err != nil { return int(rowsAffected), fmt.Errorf("recompute is_blocked after source-repo delete: %w", err) } + noteBlockedRecheck(tx, deleteRecheckLabel(issueIDs, "from "+sourceRepo), issueIDs, affectedIssues, affectedWisps) return int(rowsAffected), nil } diff --git a/internal/storage/issueops/close.go b/internal/storage/issueops/close.go index 0c70107f0..6b8242996 100644 --- a/internal/storage/issueops/close.go +++ b/internal/storage/issueops/close.go @@ -381,6 +381,7 @@ func closeIssueInTx(ctx context.Context, tx DBTX, id string, reason, actor, sess if err != nil { return nil, fmt.Errorf("recompute is_blocked after close for %s: %w", id, err) } + noteBlockedRecheck(tx, "close of "+id, []string{id}, affectedIssues, affectedWisps) // Snapshot only after all derived blocked-state maintenance has completed. // recordEvent gates the human audit event, never the journal. diff --git a/internal/storage/issueops/delete.go b/internal/storage/issueops/delete.go index f8ed23ad5..7ed67f634 100644 --- a/internal/storage/issueops/delete.go +++ b/internal/storage/issueops/delete.go @@ -51,6 +51,7 @@ func DeleteIssueInTx(ctx context.Context, tx *sql.Tx, id string, actor string) e if err := RecomputeIsBlockedInTx(ctx, tx, affectedIssues, affectedWisps); err != nil { return fmt.Errorf("recompute is_blocked after delete for %s: %w", id, err) } + noteBlockedRecheck(tx, deleteRecheckLabel([]string{id}, ""), []string{id}, affectedIssues, affectedWisps) return nil } @@ -354,6 +355,7 @@ func DeleteResolvedSetInTx(ctx context.Context, tx *sql.Tx, set DeletionSet, dry if err := RecomputeIsBlockedInTx(ctx, tx, affectedIssues, affectedWisps); err != nil { return nil, fmt.Errorf("recompute is_blocked after batch delete: %w", err) } + noteBlockedRecheck(tx, deleteRecheckLabel(set.All, ""), set.All, affectedIssues, affectedWisps) return result, nil } diff --git a/internal/storage/issueops/dependencies.go b/internal/storage/issueops/dependencies.go index db7f2943b..88e151d2d 100644 --- a/internal/storage/issueops/dependencies.go +++ b/internal/storage/issueops/dependencies.go @@ -1035,6 +1035,9 @@ func removeDependencyInTx(ctx context.Context, tx *sql.Tx, issueID, dependsOnID, return false, fmt.Errorf("recompute is_blocked after remove dependency %s -> %s: %w", issueID, dependsOnID, err) } mergeRecomputeIsBlockedResult(recomputeResult, recomputed) + // The dependent is exactly the row a racing unblocking write can leave + // stale, so nothing is excluded from the post-commit recheck. + noteBlockedRecheck(tx, fmt.Sprintf("dependency removal %s -> %s", issueID, dependsOnID), nil, affectedIssues, affectedWisps) // Snapshot only after all derived blocked-state maintenance has completed. // Never gated on emitEvent — a structural removal is as real to a replaying // consumer as one from an explicit dep verb. diff --git a/internal/storage/issueops/update.go b/internal/storage/issueops/update.go index 1ef27ce6c..10fda9f09 100644 --- a/internal/storage/issueops/update.go +++ b/internal/storage/issueops/update.go @@ -534,6 +534,9 @@ func updateIssueInTx(ctx context.Context, tx DBTX, id string, updates map[string } updateResult.IssueRowsChanged = !isWisp || recompute.IssueRowsChanged updateResult.WispRowsChanged = isWisp || recompute.WispRowsChanged + if !newActive { + noteBlockedRecheck(tx, statusChangeRecheckLabel(id, newStatus), []string{id}, affectedIssues, affectedWisps) + } } } From d79ef480488f63bf4e0be8784c586b291d6e4364 Mon Sep 17 00:00:00 2001 From: Gas City Adopt-PR Date: Tue, 29 Sep 2026 03:48:16 +0000 Subject: [PATCH 17/45] test(release-1.3.x): black-box regression tests for the #6716 blocked-flag stall [NOT a cherry-pick] Release-only test addition for the #6719 hand port. The tests #6719 ships reference symbols the fix introduces, so on the pre-fix release line they fail only to compile. These two use only store entry points that exist on both sides of the fix, so they fail behaviourally on hotfix/1.3.1 @ f5a9402790 and pass with the port: - dolt (server tier): a close through the store's own write runner (withWriteTx and withRetryTx) pins its snapshot, a sibling close commits on a second connection, the store's close commits last. Pre-fix rb-c stays is_blocked=1 and absent from GetReadyWork; post-fix it is ready. - embeddeddolt: embedded transactions serialize, so the contract is pinned instead: a dependent a committed close recomputed is settled on a fresh snapshot after commit. Proxied (uow/domain-db) mode is not covered: #6719 on main does not scope that route either (see blocked_recheck.go's doc comment and the CHANGELOG entry), so there is no proxied behaviour to test on this line. Co-Authored-By: Claude Opus 5.5 --- .../close_recheck_blocked_regression_test.go | 110 ++++++++++++++++++ ...locked_recheck_regression_internal_test.go | 73 ++++++++++++ 2 files changed, 183 insertions(+) create mode 100644 internal/storage/dolt/close_recheck_blocked_regression_test.go create mode 100644 internal/storage/embeddeddolt/blocked_recheck_regression_internal_test.go diff --git a/internal/storage/dolt/close_recheck_blocked_regression_test.go b/internal/storage/dolt/close_recheck_blocked_regression_test.go new file mode 100644 index 000000000..d453507cc --- /dev/null +++ b/internal/storage/dolt/close_recheck_blocked_regression_test.go @@ -0,0 +1,110 @@ +package dolt + +import ( + "context" + "database/sql" + "slices" + "testing" + + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// TestCloseRecheckBlocked_StoreWriteCommittingLastUnblocks is the release-line +// regression test for gastownhall/beads#6716 (formula fan-in stall). It uses +// only store entry points that exist before the fix, so it compiles on both +// sides of it and fails without it. +// +// A close runs through the store's own write runner and pins its snapshot +// while both blockers are open. A sibling close then commits on another +// connection; its in-transaction recompute cannot see the store's uncommitted +// close, so it leaves rb-c blocked. The store's close commits last, and its +// in-transaction recompute likewise saw the sibling open. Without the +// post-commit recheck rb-c stays is_blocked=1 and missing from ready work on +// every committed snapshot. +func TestCloseRecheckBlocked_StoreWriteCommittingLastUnblocks(t *testing.T) { + runners := []struct { + name string + run func(*DoltStore, context.Context, func(*sql.Tx) error) error + }{ + {"withWriteTx", (*DoltStore).withWriteTx}, + {"withRetryTx", (*DoltStore).withRetryTx}, + } + for _, runner := range runners { + t.Run(runner.name, func(t *testing.T) { + store, cleanup := setupConcurrentTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + for _, id := range []string{"rb-a", "rb-b", "rb-c"} { + createPerm(t, ctx, store, id) + } + addDependency(t, ctx, store, "rb-c", "rb-a", types.DepBlocks) + addDependency(t, ctx, store, "rb-c", "rb-b", types.DepBlocks) + assertIsBlocked(t, ctx, store, "issues", "rb-c", true) + + siblingCommitted := false + if err := runner.run(store, ctx, func(tx *sql.Tx) error { + var open int + if err := tx.QueryRowContext(ctx, + "SELECT COUNT(*) FROM issues WHERE id IN ('rb-a', 'rb-b') AND status <> 'closed'").Scan(&open); err != nil { + return err + } + if open != 2 { + t.Fatalf("store close snapshot sees %d open blockers, want 2", open) + } + if _, err := issueops.CloseIssueInTx(ctx, tx, "rb-b", "done", "tester", ""); err != nil { + return err + } + if siblingCommitted { + return nil + } + closeSiblingOnOwnConnection(t, ctx, store, "rb-a") + siblingCommitted = true + return nil + }); err != nil { + t.Fatalf("store close of rb-b: %v", err) + } + + assertIsBlocked(t, ctx, store, "issues", "rb-c", false) + ready, err := store.GetReadyWork(ctx, types.WorkFilter{}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + if !slices.ContainsFunc(ready, func(i *types.Issue) bool { return i.ID == "rb-c" }) { + t.Fatal("rb-c has no open blockers but is missing from ready work") + } + }) + } +} + +// closeSiblingOnOwnConnection closes id in a transaction on a second pooled +// connection and commits it, with the ordinary in-transaction recompute, and +// checks that it left rb-c blocked: the stale state #6716 starts from. +func closeSiblingOnOwnConnection(t *testing.T, ctx context.Context, store *DoltStore, id string) { + t.Helper() + conn, err := store.db.Conn(ctx) + if err != nil { + t.Fatalf("acquire sibling connection: %v", err) + } + defer func() { _ = conn.Close() }() + tx, err := conn.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("begin sibling tx: %v", err) + } + if _, err := issueops.CloseIssueInTx(ctx, tx, id, "done", "tester", ""); err != nil { + _ = tx.Rollback() + t.Fatalf("sibling close of %s: %v", id, err) + } + if err := tx.Commit(); err != nil { + t.Fatalf("commit sibling close of %s: %v", id, err) + } + // Read on this connection: the store's runner holds the pool's other one. + var blocked bool + if err := conn.QueryRowContext(ctx, "SELECT is_blocked FROM issues WHERE id = 'rb-c'").Scan(&blocked); err != nil { + t.Fatalf("read rb-c after sibling close: %v", err) + } + if !blocked { + t.Fatal("sibling close unblocked rb-c although the store's close of the other blocker has not committed") + } +} diff --git a/internal/storage/embeddeddolt/blocked_recheck_regression_internal_test.go b/internal/storage/embeddeddolt/blocked_recheck_regression_internal_test.go new file mode 100644 index 000000000..774b68c54 --- /dev/null +++ b/internal/storage/embeddeddolt/blocked_recheck_regression_internal_test.go @@ -0,0 +1,73 @@ +//go:build cgo + +package embeddeddolt + +import ( + "context" + "database/sql" + "path/filepath" + "slices" + "testing" + + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// TestEmbeddedBlockedRecheckSettlesDependentAfterCommit is the release-line +// regression test for the embedded half of gastownhall/beads#6716. It uses +// only store entry points that exist before the fix, so it compiles on both +// sides of it and fails without it. +// +// Embedded transactions serialize, so the two-session race cannot be staged +// here. What the fix promises on this tier is the contract: a dependent a +// committed close recomputed is recomputed again on a fresh snapshot. The +// test stands in for the unseen sibling with a write inside the same +// transaction that bypasses the recompute, after the close's own recompute +// has run and left rb-c blocked. +func TestEmbeddedBlockedRecheckSettlesDependentAfterCommit(t *testing.T) { + ctx := context.Background() + store, err := Open(ctx, filepath.Join(t.TempDir(), ".beads"), "recheckreg", "main") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + if err := store.SetConfig(ctx, "issue_prefix", "rb"); err != nil { + t.Fatal(err) + } + for _, id := range []string{"rb-a", "rb-b", "rb-c"} { + iss := &types.Issue{ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := store.CreateIssue(ctx, iss, "tester"); err != nil { + t.Fatalf("create %s: %v", id, err) + } + } + for _, blocker := range []string{"rb-a", "rb-b"} { + if err := store.AddDependency(ctx, &types.Dependency{IssueID: "rb-c", DependsOnID: blocker, Type: types.DepBlocks}, "tester"); err != nil { + t.Fatalf("add dependency rb-c -> %s: %v", blocker, err) + } + } + + if err := store.withConn(ctx, true, func(tx *sql.Tx) error { + if _, err := issueops.CloseIssueInTx(ctx, tx, "rb-a", "done", "tester", ""); err != nil { + return err + } + var blocked bool + if err := tx.QueryRowContext(ctx, "SELECT is_blocked FROM issues WHERE id = 'rb-c'").Scan(&blocked); err != nil { + return err + } + if !blocked { + t.Fatal("rb-c unblocked in-transaction while rb-b was still open") + } + _, err := tx.ExecContext(ctx, "UPDATE issues SET status = 'closed', closed_at = NOW() WHERE id = 'rb-b'") + return err + }); err != nil { + t.Fatalf("close rb-a: %v", err) + } + + ready, err := store.GetReadyWork(ctx, types.WorkFilter{}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + if !slices.ContainsFunc(ready, func(i *types.Issue) bool { return i.ID == "rb-c" }) { + t.Fatal("rb-c has no open blockers but is missing from ready work (stale is_blocked)") + } +} From c296fc9205681122c600414e9ca301998f0df76a Mon Sep 17 00:00:00 2001 From: Vishnu J Date: Sat, 26 Sep 2026 13:55:44 -0700 Subject: [PATCH 18/45] Beads #6648: report partial batch-close failure (#6740) * fix(close): fail partial batch closes Fixes #6648 * review: exit nonzero on proxied partial close, update tests and changelog Co-Authored-By: Claude Sonnet 5 * review: print close failure summary only for multi-id batches Co-Authored-By: Claude Sonnet 5 * review: machine-readable close failures, and adjudicate --claim-next Maintainer fixup for the two major findings on #6740 (Beads #6648). 1. --claim-next x partial failure (major). --claim-next rides inside the batch transaction and fires whenever something LANDED, so a mixed batch claimed a fresh issue and then exited 1, with nothing in the diff, the CHANGELOG or the tests saying so. Adjudicated as "the claim stands and the report names it", not "the claim is suppressed". Suppressing it at cmd/bd -- the site the review named -- cannot un-claim anything: ExecuteCloseBatch (and the uow provider's twin) commit the claim alongside the survivors before the CLI ever sees a refusal, so gating the report would only leave an issue assigned to the actor with nothing on stdout or stderr saying so. Strictly worse than announcing it. The claim is now named in the text summary and in the JSON failure payload, so a caller that exits on the nonzero status can still see what it owns. The review's "unbounded claim leak -- another ready issue on every attempt" does not hold, and the new test pins why: on a retry the survivor is already closed, so `landed` is 0 and an empty batch earns no claim (close_batch.go:146, uow/batch_closer.go:71). The leak is bounded at the one claim the first run earned. The defect that remains -- a durable claim the caller is never told about -- is what this fixes. 2. --json partial failure carried no machine-readable signal (major). The new exit 1 left stdout as the success-shaped array of CLOSED issues and put only human refusal prose on stderr, so a JSON consumer could not learn WHICH ids failed without scraping it -- the exact fragility #6648 was filed about, left in place for the callers most likely to batch. Both routes now report through one reportCloseFailures helper modelled on bd update's reportUpdateFailures: a compact {"error","failed":[{"id","error"}]} line on stderr, stdout keeping the closed-issues array. That also removes the summary hand-rolled twice with two denominators (the architectural nit). Also taken, all non-gating and adjacent to the above: - CHANGELOG narrowed to the refusals it actually covers, noting that an UNRESOLVABLE id still aborts the direct route before any close runs while the proxied route refuses it per-argument (the route-parity overclaim). - Tests for the three deliberately-chosen behaviours that had no pin: the --json partial failure (proxied), the --claim-next interaction above (embedded, both failure channels), and the single-id summary suppression that commit 97151ab8e exists solely to add -- asserted in TestParityCloseNothingSettledExits1, which checked stdout but not stderr. Contributor commits preserved; behaviour of the original fix is unchanged on every route it already covered. Single-id stderr stays byte-identical. Validated: go build ./..., go vet ./cmd/bd/..., TestParityClose family, cmd/bd/protocol, TestEmbeddedClose close/claim-next subtests (BEADS_TEST_EMBEDDED_DOLT=1), TestProxiedServerClose partial-failure subtests (BEADS_TEST_PROXIED_SERVER=1), TestMultiIDUpdatePartialFailure. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Sonnet 5 Co-authored-by: Julian Knutsen (cherry picked from commit 730fb9867e654a7cde030a41e10b1d3dcf1fe141) Hand-resolved for hotfix/1.3.1: CHANGELOG.md conflicted only because main's Unreleased/Fixed list does not exist on the release line. The release CHANGELOG is kept and this entry is added verbatim under Unreleased "### Fixed". No code hunk needed changes. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 17 ++++ cmd/bd/close.go | 103 +++++++++++++++++++++++ cmd/bd/close_embedded_test.go | 78 +++++++++++++++++ cmd/bd/close_proxied_integration_test.go | 88 +++++++++++++++++++ cmd/bd/close_proxied_server.go | 9 +- cmd/bd/protocol/exit_codes_test.go | 15 ++-- cmd/bd/write_verbs_parity_test.go | 27 +++--- 7 files changed, 319 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4a025b100..cf4dc89f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -119,6 +119,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `bd mol burn`, `bd duplicates --merge`, and the wisp writes — closes, updates, deletes and demote-to-wisp. +- **`bd close` now exits non-zero when any issue in a batch fails to close** + ([#6648](https://github.com/gastownhall/beads/issues/6648)). A batch with one + refused id used to exit 0 as long as another id closed, so scripts could not + tell that part of the batch was left open. The closable ids still close, the + refusals are still printed, and a final `Error: N of M issues failed to close` + line is added; ids that were already closed still count as success. Both the + direct and proxied-server routes behave this way for ids the close policy or + the engine refuses. (An id that cannot be *resolved* at all is unchanged and + still differs by route: the direct route aborts the whole command before any + close runs, while the proxied route refuses that argument and closes the + rest.) In `--json` mode the summary is instead a compact JSON line on stderr + naming the failed ids, matching `bd update`'s partial-failure report, while + stdout keeps the usual closed-issues array. `--claim-next` still claims when + part of the batch closed — the claim commits inside the batch's own + transaction and a sibling's refusal does not roll it back — so the summary + names the claimed id rather than leaving it silently assigned. + ## [1.3.1] - 2026-09-16 ### Changed diff --git a/cmd/bd/close.go b/cmd/bd/close.go index dee1ddc3e..d4ceda333 100644 --- a/cmd/bd/close.go +++ b/cmd/bd/close.go @@ -2,6 +2,7 @@ package main import ( "context" + "encoding/json" "fmt" "os" "strings" @@ -145,6 +146,7 @@ the flags appear in the command line.`, closedIssues := []*types.Issue{} closedCount := 0 alreadyClosed := 0 + var failures []closeIDFailure firstSettledID := "" for i, id := range resolvedIDs { @@ -153,10 +155,15 @@ the flags appear in the command line.`, // The CLI's own close policy refused this argument, so the // batch never saw it. fmt.Fprintln(os.Stderr, plan.refusals[i]) + failures = append(failures, closeIDFailure{ID: id, Error: plan.refusals[i]}) continue } if res.Err != nil { fmt.Fprintln(os.Stderr, closeDirectRefusal(id, res.Err)) + // The typed error, not the decorated display line: the --force + // hint closeDirectRefusal appends is advice for a human reader, + // and the id already has its own field. + failures = append(failures, closeIDFailure{ID: id, Error: res.Err.Error()}) continue } @@ -318,6 +325,17 @@ the flags appear in the command line.`, // claimed store anyway: the batch committed the claim, but the sweep // below still has to name it if a molecule auto-close made the store // dirty again. + // + // A batch that also REFUSED an id still reports its claim here, and + // that is deliberate (#6648). The claim landed inside the batch's + // transaction alongside the survivors and a sibling's refusal does not + // roll it back, so suppressing the report would not un-claim anything — + // it would only leave an issue assigned to this actor with nothing on + // stdout or stderr saying so. The claim is announced here and named + // again in reportCloseFailures' summary, so a caller that exits on the + // nonzero status can still see what it owns. Retrying the same command + // does not stack claims: the survivor is already closed the second time + // round, nothing lands, and an empty batch earns no claim. var claimedNextIssue *types.Issue if claimNext && closedForCommand && !continueFlag { if claimedNext != nil { @@ -370,6 +388,9 @@ the flags appear in the command line.`, } totalAttempted := len(resolvedIDs) + if len(failures) > 0 { + return reportCloseFailures(failures, totalAttempted, closeClaimedID(claimedNextIssue), jsonOutput) + } if totalAttempted > 0 && closedCount == 0 && alreadyClosed == 0 { return SilentExit() } @@ -377,6 +398,88 @@ the flags appear in the command line.`, }, } +// closeIDFailure records one issue id a batch close refused, and why. It +// mirrors updateIDFailure (update.go) on purpose: `bd close` and `bd update` +// are the two write verbs that take a list of ids, so a caller that learned to +// read one partial-failure report should not have to learn a second shape. +type closeIDFailure struct { + ID string `json:"id"` + Error string `json:"error"` +} + +// closeClaimedID names the issue --claim-next claimed on this run, or "" when +// the run claimed nothing. +func closeClaimedID(claimed *types.Issue) string { + if claimed == nil { + return "" + } + return claimed.ID +} + +// reportCloseFailures emits the partial-failure summary both close routes end +// on and returns the nonzero exit that #6648 added. Every refusal has already +// been printed inline, in the order the ids were typed; this is the summary +// that says how much of the batch was left open. +// +// Both routes report through here so the wording and the JSON shape are stated +// once instead of hand-rolled per route. Each route still passes its own +// denominator, because only it knows what it attempted — the direct route +// counts resolved ids, the proxied one counts arguments. +// +// The summary is suppressed for a single-id batch deliberately: one inline +// refusal is already unambiguous about which id failed, and suppressing it +// keeps single-id stderr byte-identical to what it was before close learned to +// fail a partial batch. +// +// In --json mode the summary is one compact JSON line on stderr — the shape +// reportUpdateFailures established — so a consumer that sees exit 1 can read +// WHICH ids failed instead of scraping English prose, while stdout keeps the +// plain closed-issues array a successful close emits. claimedNextID names the +// issue --claim-next claimed on this run, if any: that claim is committed +// inside the batch's own transaction and a sibling's refusal does not roll it +// back, so the report names it rather than pretending it did not happen. +func reportCloseFailures(failures []closeIDFailure, total int, claimedNextID string, jsonOut bool) error { + if total <= 1 { + return SilentExit() + } + msg := fmt.Sprintf("%d of %d issues failed to close", len(failures), total) + + if !jsonOut { + fmt.Fprintf(os.Stderr, "Error: %s\n", msg) + if claimedNextID != "" { + fmt.Fprintf(os.Stderr, "Note: --claim-next already claimed %s; it stays assigned to you.\n", claimedNextID) + } + return SilentExit() + } + + inner := map[string]interface{}{ + "error": msg, + "failed": failures, + } + if claimedNextID != "" { + inner["claimed"] = claimedNextID + } + var payload interface{} + if jsonEnvelopeEnabled() { + payload = map[string]interface{}{ + "schema_version": JSONSchemaVersion, + "data": inner, + } + } else { + inner["schema_version"] = JSONSchemaVersion + payload = inner + } + data, err := json.Marshal(payload) + if err != nil { + // Marshaling flat strings cannot realistically fail; fall back to the + // text summary rather than exiting with nothing said at all. + fmt.Fprintf(os.Stderr, "Error: %s\n", msg) + } else { + fmt.Fprintln(os.Stderr, string(data)) + } + return SilentExit() +} + func init() { registerCloseReasonFlag(closeCmd) closeCmd.Flags().String("resolution", "", "Alias for --reason (Jira CLI convention)") diff --git a/cmd/bd/close_embedded_test.go b/cmd/bd/close_embedded_test.go index 4f5614338..f5fc349dd 100644 --- a/cmd/bd/close_embedded_test.go +++ b/cmd/bd/close_embedded_test.go @@ -113,6 +113,84 @@ func TestEmbeddedClose(t *testing.T) { } }) + // A batch still closes the IDs it can close, but a caller must receive a + // nonzero exit when any sibling was refused. Otherwise an orchestrator can + // record the whole batch as complete while a blocked issue remains open. + t.Run("mixed_batch_refusal_exits_nonzero_after_closing_survivor", func(t *testing.T) { + blocker := bdCreate(t, bd, dir, "Mixed batch blocker", "--type", "task") + blocked := bdCreate(t, bd, dir, "Mixed batch blocked", "--type", "task") + closable := bdCreate(t, bd, dir, "Mixed batch closable", "--type", "task") + bdDepAdd(t, bd, dir, blocked.ID, blocker.ID) + + out := bdCloseFail(t, bd, dir, closable.ID, blocked.ID) + if !strings.Contains(out, "1 of 2 issues failed to close") { + t.Errorf("expected partial-close summary, got: %s", out) + } + if got := bdShow(t, bd, dir, closable.ID); got.Status != types.StatusClosed { + t.Errorf("closable issue status = %s, want closed", got.Status) + } + if got := bdShow(t, bd, dir, blocked.ID); got.Status != types.StatusOpen { + t.Errorf("blocked issue status = %s, want open", got.Status) + } + }) + + // The --claim-next × partial-failure interaction, adjudicated rather than + // left implicit (#6648). --claim-next rides inside the batch transaction + // and fires whenever something LANDED, so a mixed batch claims and then + // exits 1. The claim is durable — the refused sibling does not roll it back + // — so the contract pinned here is "the claim stands and the failure + // summary names it", NOT "the claim is silently suppressed": suppressing + // only the report would leave an issue assigned to this actor with nothing + // saying so, which is strictly worse than announcing it. + // + // Its own project, because the assertion depends on exactly which issue is + // ready at claim time: once `closable` closes, `blocker` is the only ready + // issue left (`blocked` still depends on it), so the claim is deterministic. + t.Run("mixed_batch_claim_next_names_the_claim_it_kept", func(t *testing.T) { + kdir, _, _ := bdInit(t, bd, "--prefix", "kn") + closable := bdCreate(t, bd, kdir, "Claim-next batch closable", "--type", "task") + blocker := bdCreate(t, bd, kdir, "Claim-next batch blocker", "--type", "task") + blocked := bdCreate(t, bd, kdir, "Claim-next batch blocked", "--type", "task") + bdDepAdd(t, bd, kdir, blocked.ID, blocker.ID) + + out := bdCloseFail(t, bd, kdir, closable.ID, blocked.ID, "--claim-next") + if !strings.Contains(out, "1 of 2 issues failed to close") { + t.Errorf("expected the partial-close summary, got: %s", out) + } + if !strings.Contains(out, "already claimed "+blocker.ID) { + t.Errorf("expected the failure summary to name the claim it kept (%s), got: %s", blocker.ID, out) + } + if got := bdShow(t, bd, kdir, closable.ID); got.Status != types.StatusClosed { + t.Errorf("closable issue status = %s, want closed", got.Status) + } + if got := bdShow(t, bd, kdir, blocked.ID); got.Status != types.StatusOpen { + t.Errorf("blocked issue status = %s, want open", got.Status) + } + // The claim really landed: the report is not describing a claim that + // was rolled back with the refusal. + if got := bdShow(t, bd, kdir, blocker.ID); got.Assignee == "" { + t.Errorf("claimed issue %s assignee = %q, want it assigned: the claim commits inside the batch transaction and the refused sibling does not undo it", + blocker.ID, got.Assignee) + } + + // And retrying does NOT stack a second claim, which is what makes + // keeping the first one safe. `closable` is already closed on the way + // back through, so the batch lands nothing and an empty batch earns no + // claim (see the already-closed re-close subtest below). `spare` is + // created only now, so it is the one ready issue a repeat claim could + // take — without it this assertion would pass vacuously, since the + // first run left nothing else ready. + spare := bdCreate(t, bd, kdir, "Claim-next batch spare", "--type", "task") + out = bdCloseFail(t, bd, kdir, closable.ID, blocked.ID, "--claim-next") + if strings.Contains(out, "already claimed "+spare.ID) { + t.Errorf("a retry claimed %s as well, got: %s", spare.ID, out) + } + if got := bdShow(t, bd, kdir, spare.ID); got.Assignee != "" { + t.Errorf("retrying the same failed batch claimed a second issue (%s assignee = %q); the leak must be bounded at the one claim the first run earned", + spare.ID, got.Assignee) + } + }) + // Proves the S7 delegation: `bd close` on a blocked issue now surfaces the // engine's atomic guard (storage.ErrCloseBlocked) rather than a duplicated // CLI pre-check. The refusal must be atomic — the issue stays open because the diff --git a/cmd/bd/close_proxied_integration_test.go b/cmd/bd/close_proxied_integration_test.go index 6791a0ec9..74038e62b 100644 --- a/cmd/bd/close_proxied_integration_test.go +++ b/cmd/bd/close_proxied_integration_test.go @@ -42,6 +42,19 @@ func bdProxiedCloseFail(t *testing.T, bd, dir string, args ...string) string { return stdout + stderr } +// lastJSONObjectLine returns the last line of s that looks like a compact JSON +// object, or "". Partial-failure stderr is the per-id refusal prose followed by +// the machine-readable summary, so the report is the last such line. +func lastJSONObjectLine(s string) string { + lines := strings.Split(strings.TrimSpace(s), "\n") + for i := len(lines) - 1; i >= 0; i-- { + if line := strings.TrimSpace(lines[i]); strings.HasPrefix(line, "{") { + return line + } + } + return "" +} + func bdProxiedCloseJSON(t *testing.T, bd, dir string, args ...string) []*types.Issue { t.Helper() fullArgs := append([]string{"close", "--json"}, args...) @@ -363,6 +376,81 @@ func TestProxiedServerClose(t *testing.T) { } }) + t.Run("close_partial_failure_exits_nonzero", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "cpf") + closable := bdProxiedCreate(t, bd, p.dir, "Closable") + blocker := bdProxiedCreate(t, bd, p.dir, "Blocker") + blocked := bdProxiedCreate(t, bd, p.dir, "Blocked", "--deps", "depends-on:"+blocker.ID) + out := bdProxiedCloseFail(t, bd, p.dir, closable.ID, blocked.ID) + if !strings.Contains(out, "1 of 2 issues failed to close") { + t.Errorf("expected partial-failure summary, got: %s", out) + } + db := openProxiedDB(t, p) + if got := readStatus(t, db, closable.ID); got != types.StatusClosed { + t.Errorf("closable issue status = %q, want closed despite the refused sibling", got) + } + if got := readStatus(t, db, blocked.ID); got == types.StatusClosed { + t.Error("blocked issue should remain open without --force") + } + }) + + // The machine-readable half of the #6648 contract. A JSON consumer is + // exactly the caller the issue was filed for, so exit 1 must come with a + // payload naming the failed ids instead of prose it would have to scrape — + // while stdout keeps the success-shaped closed-issues array, the way + // bd update's reportUpdateFailures already splits them. + t.Run("close_partial_failure_json_names_the_failed_ids", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "cpj") + closable := bdProxiedCreate(t, bd, p.dir, "JSON closable") + blocker := bdProxiedCreate(t, bd, p.dir, "JSON blocker") + blocked := bdProxiedCreate(t, bd, p.dir, "JSON blocked", "--deps", "depends-on:"+blocker.ID) + + stdout, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, "close", "--json", closable.ID, blocked.ID) + if err == nil { + t.Fatalf("expected a partial batch close to exit nonzero\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + + // stdout keeps the success shape: just the survivor. + start := strings.Index(stdout, "[") + if start < 0 { + t.Fatalf("expected the closed-issues array on stdout, got:\n%s", stdout) + } + var closed []*types.Issue + if jsonErr := json.Unmarshal([]byte(stdout[start:]), &closed); jsonErr != nil { + t.Fatalf("parse closed array: %v\nraw: %s", jsonErr, stdout[start:]) + } + if len(closed) != 1 || closed[0].ID != closable.ID { + t.Fatalf("stdout closed array = %d issue(s) %v, want only the survivor %s", len(closed), closed, closable.ID) + } + + // stderr's last line is the failure report. + line := lastJSONObjectLine(stderr) + if line == "" { + t.Fatalf("expected a compact JSON failure line on stderr, got:\n%s", stderr) + } + var report struct { + Error string `json:"error"` + Failed []struct { + ID string `json:"id"` + Error string `json:"error"` + } `json:"failed"` + } + if jsonErr := json.Unmarshal([]byte(line), &report); jsonErr != nil { + t.Fatalf("parse failure report: %v\nraw: %s", jsonErr, line) + } + if report.Error != "1 of 2 issues failed to close" { + t.Errorf("failure report error = %q, want the N of M summary", report.Error) + } + if len(report.Failed) != 1 || report.Failed[0].ID != blocked.ID { + t.Fatalf("failure report named %+v, want exactly the refused id %s", report.Failed, blocked.ID) + } + if report.Failed[0].Error == "" { + t.Errorf("failure report for %s carried no reason", blocked.ID) + } + }) + t.Run("close_pinned_refuses_without_force", func(t *testing.T) { t.Parallel() p := newSharedProxiedProject(t, bd, "cpr") diff --git a/cmd/bd/close_proxied_server.go b/cmd/bd/close_proxied_server.go index 5fc9d268c..5ead731e8 100644 --- a/cmd/bd/close_proxied_server.go +++ b/cmd/bd/close_proxied_server.go @@ -122,9 +122,13 @@ func runCloseProxiedServer(cmd *cobra.Command, ctx context.Context, args []strin outcomes, closeReasons := closeProxiedOutcomes(&pre, result) post := closeProxiedRunPostClose(ctx, args, in, outcomes) - for _, e := range pre.errors { + // pre.errors is indexed by argument position, so args[i] is the id this + // refusal belongs to. + var failures []closeIDFailure + for i, e := range pre.errors { if e != "" { fmt.Fprintln(os.Stderr, e) + failures = append(failures, closeIDFailure{ID: args[i], Error: e}) } } for _, w := range post.warnings { @@ -182,6 +186,9 @@ func runCloseProxiedServer(cmd *cobra.Command, ctx context.Context, args []strin } } + if len(failures) > 0 { + return reportCloseFailures(failures, len(args), closeClaimedID(claimedNextIssue), in.jsonOut) + } if len(args) > 0 && len(outcomes) == 0 { return SilentExit() } diff --git a/cmd/bd/protocol/exit_codes_test.go b/cmd/bd/protocol/exit_codes_test.go index b07870ead..abaf6bddd 100644 --- a/cmd/bd/protocol/exit_codes_test.go +++ b/cmd/bd/protocol/exit_codes_test.go @@ -41,11 +41,10 @@ func TestProtocol_UpdateNonexistentExitsNonZero(t *testing.T) { } } -// TestProtocol_ClosePartialFailureExitsZero verifies that when closing +// TestProtocol_ClosePartialFailureExitsNonZero verifies that when closing // multiple issues where some succeed and some fail (e.g., blocked), the -// command exits zero (partial success counts as success) and still closes -// the closeable ones. -func TestProtocol_ClosePartialFailureExitsZero(t *testing.T) { +// command exits non-zero and still closes the closeable ones. +func TestProtocol_ClosePartialFailureExitsNonZero(t *testing.T) { t.Parallel() w := newWorkspace(t) @@ -54,9 +53,11 @@ func TestProtocol_ClosePartialFailureExitsZero(t *testing.T) { blocked := w.create("Blocked issue") w.run("dep", "add", blocked, blocker, "--type=blocks") - // Close both: closeable should succeed, blocked should fail. - // Partial success (closedCount > 0) exits 0. - w.run("close", closeable, blocked) + // Close both: closeable should succeed, blocked should fail, and the + // refusal makes the batch exit 1. + if _, code := w.runExpectError("close", closeable, blocked); code != 1 { + t.Errorf("expected exit code 1, got %d", code) + } // Verify the closeable one was actually closed despite partial failure out := w.run("show", closeable, "--json") diff --git a/cmd/bd/write_verbs_parity_test.go b/cmd/bd/write_verbs_parity_test.go index 6a339cb15..7404070b0 100644 --- a/cmd/bd/write_verbs_parity_test.go +++ b/cmd/bd/write_verbs_parity_test.go @@ -28,7 +28,9 @@ // flips one, the assertion is updated IN THAT COMMIT with a comment naming the // ruling. Any OTHER assertion in this file changing is a regression, not a // refactor. All four have now landed; every remaining assertion is unchanged -// from the pre-rewire CLI and must stay that way. +// from the pre-rewire CLI and must stay that way. Waiver: the close exit-code +// contract (#6648: partial batch failure exits nonzero) intentionally flips +// the TestParityClose* assertions that pinned exit 0. // // Harness: the commands' RunE functions are invoked in-process against a real // storage.DoltStorage, with stdout/stderr captured and the returned error @@ -1584,11 +1586,10 @@ func TestParityCloseNoIDAndNoLastTouchedExits1(t *testing.T) { } } -// TestParityClosePartialFailureExitsZero pins the close exit contract's -// permissive half (cmd/bd/close.go:377-380): as long as ONE id settled as -// closed, a batch with refused ids still exits 0. The refusal is reported on -// stderr only. -func TestParityClosePartialFailureExitsZero(t *testing.T) { +// TestParityClosePartialFailureExitsOne pins that a batch with a refused id +// exits 1 even when other ids closed: the refusal and an `N of M issues failed +// to close` summary go to stderr, and the closable ids still close. +func TestParityClosePartialFailureExitsOne(t *testing.T) { env := newParityEnv(t) env.seed("test-cls4", "Closable", nil) env.seed("test-cls5", "Owned by another actor", func(i *types.Issue) { @@ -1598,11 +1599,11 @@ func TestParityClosePartialFailureExitsZero(t *testing.T) { env.setFlags(closeCmd, nil) res := env.run(closeCmd, "test-cls4", "test-cls5") - if res.exitCode != 0 { - t.Fatalf("exit = %d, want 0 (partial failure is still success today)\nstderr:\n%s", res.exitCode, res.stderr) + if res.exitCode != 1 { + t.Fatalf("exit = %d, want 1 (a refused id fails the batch)\nstderr:\n%s", res.exitCode, res.stderr) } wantErr := fmt.Sprintf("cannot close %s: assignee is %q, actor is %q; reclaim or use --force to override\n", - "test-cls5", "someone-else", actor) + "test-cls5", "someone-else", actor) + "Error: 1 of 2 issues failed to close\n" if res.stderr != wantErr { t.Errorf("stderr = %q, want %q", res.stderr, wantErr) } @@ -1620,7 +1621,7 @@ func TestParityClosePartialFailureExitsZero(t *testing.T) { // TestParityCloseNothingSettledExits1 pins the strict half of the same // contract: when NO id settled as closed, close returns SilentExit() — exit 1 -// with no extra stdout. Source: cmd/bd/close.go:377-380 and +// with no extra stdout. Source: cmd/bd/close.go:382-384 and // cmd/bd/errors.go:119-121. func TestParityCloseNothingSettledExits1(t *testing.T) { env := newParityEnv(t) @@ -1640,6 +1641,12 @@ func TestParityCloseNothingSettledExits1(t *testing.T) { if res.stdout != "" { t.Errorf("stdout = %q, want empty (SilentExit prints nothing itself)", res.stdout) } + // The batch summary is multi-id ONLY. A single refused id already names + // itself on stderr, so adding `N of M issues failed to close` to it would + // be noise — and this is the sole assertion pinning that suppression. + if strings.Contains(res.stderr, "issues failed to close") { + t.Errorf("stderr = %q, want no batch summary for a single id", res.stderr) + } if got := env.lastTouched(); got != "" { t.Errorf("last-touched = %q, want it untouched when nothing settled", got) } From 0144dbe76b4c60f63787115f76eeb1f87fd97749 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Sun, 27 Sep 2026 01:51:19 -0700 Subject: [PATCH 19/45] fix(close): make --json failed[].error route-independent (post-merge #6740) (#6771) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Post-merge review of #6740 (landed range c25059ada..730fb9867) found one major: the machine-readable `failed[].error` field that PR added carried a DIFFERENT string for the identical refusal depending on whether the caller was on the embedded or the proxied-server route — the exact class of route drift the close rewire exists to remove, shipped in the same commit that states the principle in a comment. No test pinned the text, so CI was blind to it. Two causes, both on the proxied route: 1. The harvest loop copied `pre.errors[i]` — closeProxiedRefusal's DECORATED display line — straight into the JSON field, so a blocked id gained " (use --force to override)" that the direct route deliberately keeps out of the same field. Fixed by carrying the typed error in a parallel per-argument slot (`failureErrors`), mirroring the direct route's split, and reading that slot in the new closeProxiedFailures. 2. Stripping the decoration was not sufficient: a refusal reaching this route has also been wrapped by the layers it passed through, so the undecorated typed error still read close x-1: db: IssueSQLRepository.CloseChecked x-1: cannot close blocked … against the direct route's cannot close blocked issue: x-1 is blocked by [x-2] closeProxiedTypedRefusal unwraps to the sentence the engine actually wrote for the refusal classes the CLI already names by sentinel. Anything else keeps its whole chain, where the wrapping is the only thing that says where an unexpected failure came from. Tests: the split and the unwrap are pinned in an untagged file so they run in the CGO_ENABLED=0 lane too, and the shared assertion is applied by BOTH integration suites — a new direct-route twin of close_partial_failure_json_names_the_failed_ids, which also gives the default route its first pin on reportCloseFailures' jsonOut branch. Every pin was mutation-proved: reading the display line reds the three decorated classes while the undecorated open-children control stays green; converging by stripping the hint from stderr instead reds the human-line half; recording the decorated line on the direct route reds the twin. Scope: this is the only entry under the review's Required Changes. The five non-gating minors (single-id --json report, closeCmd.Long paragraph, text-mode per-id reprint, the `claimed` JSON type, the CHANGELOG not-found parenthetical) are deliberately left for separate follow-ups. Co-authored-by: Claude Opus 5 (cherry picked from commit 01bd1f1e54dbf2ba1f51a3fc86c6ee05ce4cfe41) Hand-resolved for hotfix/1.3.1: CHANGELOG.md conflicted only on context (main's Unreleased/Fixed list has entries the release line lacks). The release CHANGELOG is kept and the same wording change is applied to the #6648 entry carried by the previous pick. No code hunk needed changes. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 5 +- cmd/bd/close_embedded_test.go | 70 ++++++ cmd/bd/close_failure_report_test.go | 281 +++++++++++++++++++++++ cmd/bd/close_proxied_integration_test.go | 9 + cmd/bd/close_proxied_server.go | 115 +++++++++- 5 files changed, 468 insertions(+), 12 deletions(-) create mode 100644 cmd/bd/close_failure_report_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index cf4dc89f9..eae47e4d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -131,7 +131,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 close runs, while the proxied route refuses that argument and closes the rest.) In `--json` mode the summary is instead a compact JSON line on stderr naming the failed ids, matching `bd update`'s partial-failure report, while - stdout keeps the usual closed-issues array. `--claim-next` still claims when + stdout keeps the usual closed-issues array. Each `failed[]` entry carries the + refusal as the engine worded it, identically on both routes; the `--force` + hint and the route's own framing stay on the human-readable stderr line, which + is unchanged. `--claim-next` still claims when part of the batch closed — the claim commits inside the batch's own transaction and a sibling's refusal does not roll it back — so the summary names the claimed id rather than leaving it silently assigned. diff --git a/cmd/bd/close_embedded_test.go b/cmd/bd/close_embedded_test.go index f5fc349dd..0a0d945c2 100644 --- a/cmd/bd/close_embedded_test.go +++ b/cmd/bd/close_embedded_test.go @@ -134,6 +134,76 @@ func TestEmbeddedClose(t *testing.T) { } }) + // The direct-route twin of TestProxiedClose/ + // close_partial_failure_json_names_the_failed_ids. Two things were asserted + // on the proxied route only: that exit 1 comes with a payload naming the + // failed ids while stdout keeps the success-shaped closed-issues array, and + // that failed[].error is the TYPED error rather than the decorated line + // stderr shows a person. This is the default route — what most scripts and + // the parity harness exercise — so a regression in reportCloseFailures' + // jsonOut branch was previously invisible to CI on the route most callers + // are on, and the two routes could drift on the field's contents with + // nothing failing. + t.Run("close_partial_failure_json_names_the_failed_ids", func(t *testing.T) { + jdir, _, _ := bdInit(t, bd, "--prefix", "jf") + closable := bdCreate(t, bd, jdir, "JSON closable", "--type", "task") + blocker := bdCreate(t, bd, jdir, "JSON blocker", "--type", "task") + blocked := bdCreate(t, bd, jdir, "JSON blocked", "--type", "task") + bdDepAdd(t, bd, jdir, blocked.ID, blocker.ID) + + cmd := exec.Command(bd, "close", "--json", closable.ID, blocked.ID) + cmd.Dir = jdir + cmd.Env = bdEnv(jdir) + stdoutBuf, stderrBuf, err := runCommandBuffers(t, cmd) + stdout, stderr := stdoutBuf.String(), stderrBuf.String() + if err == nil { + t.Fatalf("expected a partial batch close to exit nonzero\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + + // stdout keeps the success shape: just the survivor. + start := strings.Index(stdout, "[") + if start < 0 { + t.Fatalf("expected the closed-issues array on stdout, got:\n%s", stdout) + } + var closed []*types.Issue + if jsonErr := json.Unmarshal([]byte(stdout[start:]), &closed); jsonErr != nil { + t.Fatalf("parse closed array: %v\nraw: %s", jsonErr, stdout[start:]) + } + if len(closed) != 1 || closed[0].ID != closable.ID { + t.Fatalf("stdout closed array = %d issue(s), want only the survivor %s\nraw: %s", len(closed), closable.ID, stdout[start:]) + } + + // stderr's last line is the failure report. + line := lastJSONObjectLine(stderr) + if line == "" { + t.Fatalf("expected a compact JSON failure line on stderr, got:\n%s", stderr) + } + var report struct { + Error string `json:"error"` + Failed []struct { + ID string `json:"id"` + Error string `json:"error"` + } `json:"failed"` + } + if jsonErr := json.Unmarshal([]byte(line), &report); jsonErr != nil { + t.Fatalf("parse failure report: %v\nraw: %s", jsonErr, line) + } + if report.Error != "1 of 2 issues failed to close" { + t.Errorf("failure report error = %q, want the N of M summary", report.Error) + } + if len(report.Failed) != 1 || report.Failed[0].ID != blocked.ID { + t.Fatalf("failure report named %+v, want exactly the refused id %s", report.Failed, blocked.ID) + } + assertCloseFailedErrorIsTyped(t, report.Failed[0].Error, stderr) + + if got := bdShow(t, bd, jdir, closable.ID); got.Status != types.StatusClosed { + t.Errorf("closable issue status = %s, want closed despite the refused sibling", got.Status) + } + if got := bdShow(t, bd, jdir, blocked.ID); got.Status != types.StatusOpen { + t.Errorf("blocked issue status = %s, want open", got.Status) + } + }) + // The --claim-next × partial-failure interaction, adjudicated rather than // left implicit (#6648). --claim-next rides inside the batch transaction // and fires whenever something LANDED, so a mixed batch claims and then diff --git a/cmd/bd/close_failure_report_test.go b/cmd/bd/close_failure_report_test.go new file mode 100644 index 000000000..cb05c024e --- /dev/null +++ b/cmd/bd/close_failure_report_test.go @@ -0,0 +1,281 @@ +package main + +import ( + "fmt" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/issueops" +) + +// assertCloseFailedErrorIsTyped checks one failed[].error entry against the +// route-parity contract both close routes owe a --json consumer: the field +// carries the typed error, while the line a person reads keeps the --force hint. +// +// The two integration suites share it, which is the point — a contract that is +// supposed to be identical per route should be asserted by identical code. It is +// written against the RELATIONSHIP between the two spellings rather than a +// transcribed sentence so neither suite has to hardcode how the engine renders a +// blocker list: closeDirectRefusal and closeProxiedRefusal both produce exactly +// the typed error plus the hint, so the typed error must appear on stderr with +// the hint appended and must not contain the hint itself. +// +// Lives in an untagged file so it is compiled in the CGO_ENABLED=0 lane too, +// alongside the pins below; its callers are the cgo-gated suites. +func assertCloseFailedErrorIsTyped(t *testing.T, failedError, stderr string) { + t.Helper() + const hint = " (use --force to override)" + if !strings.HasPrefix(failedError, "cannot close blocked issue:") { + t.Errorf("failed[].error = %q, want the engine's typed ErrCloseBlocked vocabulary: both routes must spell one refusal one way", + failedError) + } + if strings.Contains(failedError, hint) { + t.Errorf("failed[].error = %q carries the --force hint; that decoration is advice for a human reader and belongs on the stderr line only", + failedError) + } + if want := failedError + hint; !strings.Contains(stderr, want) { + t.Errorf("stderr is missing the decorated refusal %q; the hint must still reach the person reading it\nstderr:\n%s", + want, stderr) + } +} + +// The route-parity contract for the machine-readable half of the partial-failure +// report: `failed[].error` carries the TYPED error on both routes, and the +// decorated display line stays on stderr where a human reads it. +// +// The direct route already splits the two (close.go records res.Err.Error() in +// closeIDFailure while printing closeDirectRefusal's decorated line). The proxied +// route used to copy pre.errors — the display line — into the JSON field, so the +// same blocked-id refusal answered "cannot close blocked issue: X is blocked by +// [Y]" embedded and "... (use --force to override)" proxied. These live here +// rather than in the proxied integration suite because nothing in the split needs +// a server: the refusal slots are filled by closeProxiedOutcomes and read by +// closeProxiedFailures, both pure. +func TestCloseProxiedFailuresRecordTheTypedErrorNotTheDisplayLine(t *testing.T) { + blockedErr := fmt.Errorf("%w: pb-2 is blocked by [pb-1]", storage.ErrCloseBlocked) + + // arg 0 closes, arg 1 is refused by the engine. + args := []string{"pb-3", "pb-2"} + pre := closeProxiedPreflight{ + items: []issueops.BatchCloseItem{ + {IssueID: "pb-3", Reason: "done"}, + {IssueID: "pb-2", Reason: "done"}, + }, + itemArgs: []int{0, 1}, + before: map[string]*types.Issue{"pb-3": {ID: "pb-3"}}, + errors: make([]string, len(args)), + failureErrors: make([]string, len(args)), + } + outcomes, _ := closeProxiedOutcomes(&pre, issueops.CloseBatchResult{ + Outcomes: []issueops.CloseOutcome{ + {IssueID: "pb-3", Issue: &types.Issue{ID: "pb-3"}, Changed: true}, + {IssueID: "pb-2", Err: blockedErr}, + }, + }) + if len(outcomes) != 1 || outcomes[0].id != "pb-3" { + t.Fatalf("outcomes = %+v, want only the survivor pb-3", outcomes) + } + + failures := closeProxiedFailures(&pre, args) + if len(failures) != 1 || failures[0].ID != "pb-2" { + t.Fatalf("failures = %+v, want exactly the refused id pb-2", failures) + } + + // The machine field is the typed error, byte for byte. + if failures[0].Error != blockedErr.Error() { + t.Errorf("failed[].error = %q, want the typed error %q: a --json consumer keying off this field must see the same string on both routes", + failures[0].Error, blockedErr.Error()) + } + if strings.Contains(failures[0].Error, "use --force to override") { + t.Errorf("failed[].error = %q, must not carry the --force hint: that is advice for a human reader, and the direct route records res.Err.Error() here", + failures[0].Error) + } + + // ...and the human line keeps the decoration. Without this half the fix + // could have passed by stripping the hint from stderr too, which is a + // different regression. + if want := closeProxiedRefusal("pb-2", blockedErr); pre.errors[1] != want { + t.Errorf("stderr line = %q, want the decorated refusal %q", pre.errors[1], want) + } + if !strings.Contains(pre.errors[1], "use --force to override") { + t.Errorf("stderr line = %q, want it to keep the --force hint for the person reading it", pre.errors[1]) + } +} + +// Every engine refusal class closeProxiedRefusal decorates, checked against the +// typed error it decorates. ErrCloseOpenChildren is in the table on purpose: it +// passes through undecorated, so it is the row that would still pass if the fix +// had been written as "special-case the blocked hint" instead of "record the +// typed error". +func TestCloseProxiedFailuresTypedErrorPerRefusalClass(t *testing.T) { + tests := []struct { + name string + err error + }{ + { + name: "blocked gains the force hint on stderr", + err: fmt.Errorf("%w: rc-1 is blocked by [rc-9]", storage.ErrCloseBlocked), + }, + { + name: "not found is rewritten on stderr", + err: fmt.Errorf("%w: rc-1", storage.ErrNotFound), + }, + { + name: "default class is wrapped on stderr", + err: fmt.Errorf("dolt: connection reset"), + }, + { + name: "open children passes through undecorated", + err: &storage.CloseOpenChildrenError{IssueID: "rc-1", OpenChildren: 2}, + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + args := []string{"rc-1"} + pre := closeProxiedPreflight{ + items: []issueops.BatchCloseItem{{IssueID: "rc-1", Reason: "done"}}, + itemArgs: []int{0}, + before: map[string]*types.Issue{}, + errors: make([]string, len(args)), + failureErrors: make([]string, len(args)), + } + closeProxiedOutcomes(&pre, issueops.CloseBatchResult{ + Outcomes: []issueops.CloseOutcome{{IssueID: "rc-1", Err: tc.err}}, + }) + + failures := closeProxiedFailures(&pre, args) + if len(failures) != 1 { + t.Fatalf("failures = %+v, want one", failures) + } + if failures[0].Error != tc.err.Error() { + t.Errorf("failed[].error = %q, want the typed error %q", failures[0].Error, tc.err.Error()) + } + if pre.errors[0] != closeProxiedRefusal("rc-1", tc.err) { + t.Errorf("stderr line = %q, want closeProxiedRefusal's spelling %q", pre.errors[0], closeProxiedRefusal("rc-1", tc.err)) + } + }) + } +} + +// The second half of the route-parity contract, and the one the decoration fix +// alone does not deliver: a refusal that reached the proxied route arrives +// wrapped by the layers it travelled through, so even the undecorated typed +// error read differently per route until closeProxiedTypedRefusal unwrapped it. +// +// The wrapper strings here are transcribed from the two production sites named +// in that function's comment, and the "want" column is the byte-identical +// string the direct route records for the same refusal. +func TestCloseProxiedTypedRefusalUnwrapsTheRoleWrappers(t *testing.T) { + blocked := fmt.Errorf("%w: wr-1 is blocked by [wr-2]", storage.ErrCloseBlocked) + openChildren := &storage.CloseOpenChildrenError{IssueID: "wr-1", OpenChildren: 3} + unexpected := fmt.Errorf("dolt: connection reset") + + // What the role actually hands back: db/issue.go wraps, then issue.go wraps. + roleWrapped := func(inner error) error { + return fmt.Errorf("close wr-1: %w", fmt.Errorf("db: IssueSQLRepository.CloseChecked wr-1: %w", inner)) + } + + tests := []struct { + name string + err error + want string + }{ + { + name: "blocked, unwrapped", + err: blocked, + want: blocked.Error(), + }, + { + name: "blocked, wrapped by the role", + err: roleWrapped(blocked), + want: blocked.Error(), + }, + { + name: "open children, wrapped by the role", + err: roleWrapped(openChildren), + want: openChildren.Error(), + }, + { + // No sentinel to converge on, and the chain is the only thing that + // says where the failure came from, so it is kept whole. + name: "unexpected error keeps its whole chain", + err: roleWrapped(unexpected), + want: "close wr-1: db: IssueSQLRepository.CloseChecked wr-1: dolt: connection reset", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + if got := closeProxiedTypedRefusal(tc.err); got != tc.want { + t.Errorf("closeProxiedTypedRefusal = %q, want %q", got, tc.want) + } + }) + } + + // The walk must stop AT the refusal, not at the bare sentinel underneath + // it: unwrapping one link too far would drop the subject and leave every + // blocked id reporting the same reasonless sentence. + if got := closeProxiedTypedRefusal(roleWrapped(blocked)); got == storage.ErrCloseBlocked.Error() { + t.Errorf("closeProxiedTypedRefusal = %q, which is the bare sentinel: the id and its blockers were unwrapped away", got) + } +} + +// A refusal decided by the CLI's own close policy has no separate display form, +// so both fields hold the one sentence closeProxiedCheckOne returned — and the +// failures still come back in the order the caller typed the ids, across a mix +// of policy and engine refusals. errors is what decides WHICH arguments failed, +// so a policy slot with an empty failureErrors twin would be a silent +// reason-less entry; this pins that it is not. +func TestCloseProxiedFailuresKeepPolicyRefusalsAndTypedOrder(t *testing.T) { + policyRefusal := "cannot close mx-1: gate not satisfied" + engineErr := fmt.Errorf("%w: mx-3 is blocked by [mx-9]", storage.ErrCloseBlocked) + + // arg 0 refused by policy, arg 1 closes, arg 2 refused by the engine. + args := []string{"mx-1", "mx-2", "mx-3"} + pre := closeProxiedPreflight{ + items: []issueops.BatchCloseItem{ + {IssueID: "mx-2", Reason: "done"}, + {IssueID: "mx-3", Reason: "done"}, + }, + itemArgs: []int{1, 2}, + before: map[string]*types.Issue{"mx-2": {ID: "mx-2"}}, + errors: make([]string, len(args)), + failureErrors: make([]string, len(args)), + } + // closeProxiedRunPreflight fills both slots for a policy refusal. + pre.errors[0] = policyRefusal + pre.failureErrors[0] = policyRefusal + + closeProxiedOutcomes(&pre, issueops.CloseBatchResult{ + Outcomes: []issueops.CloseOutcome{ + {IssueID: "mx-2", Issue: &types.Issue{ID: "mx-2"}, Changed: true}, + {IssueID: "mx-3", Err: engineErr}, + }, + }) + + failures := closeProxiedFailures(&pre, args) + want := []closeIDFailure{ + {ID: "mx-1", Error: policyRefusal}, + {ID: "mx-3", Error: engineErr.Error()}, + } + if len(failures) != len(want) { + t.Fatalf("failures = %+v, want %+v", failures, want) + } + for i := range want { + if failures[i] != want[i] { + t.Errorf("failures[%d] = %+v, want %+v", i, failures[i], want[i]) + } + } + + // An entry never goes out reason-less. No production writer can reach this + // state — both fill the twin — but the degradation is defined rather than + // accidental, because an empty reason is the one answer a caller asking why + // an id failed cannot use. + pre.failureErrors[0] = "" + if got := closeProxiedFailures(&pre, args); got[0].Error != policyRefusal { + t.Errorf("failures[0].Error = %q, want the display line %q as the fallback", got[0].Error, policyRefusal) + } +} diff --git a/cmd/bd/close_proxied_integration_test.go b/cmd/bd/close_proxied_integration_test.go index 74038e62b..53df108c9 100644 --- a/cmd/bd/close_proxied_integration_test.go +++ b/cmd/bd/close_proxied_integration_test.go @@ -449,6 +449,15 @@ func TestProxiedServerClose(t *testing.T) { if report.Failed[0].Error == "" { t.Errorf("failure report for %s carried no reason", blocked.ID) } + + // failed[].error is the TYPED error, the same on this route as on the + // direct one — see the twin assertions in TestEmbeddedClose/ + // close_partial_failure_json_names_the_failed_ids. The --force hint is + // advice for a human reader, so it belongs on the stderr line above and + // not in the field a --json consumer keys off; recording the decorated + // display line here gave the identical refusal two spellings depending + // on which route the caller was on. + assertCloseFailedErrorIsTyped(t, report.Failed[0].Error, stderr) }) t.Run("close_pinned_refuses_without_force", func(t *testing.T) { diff --git a/cmd/bd/close_proxied_server.go b/cmd/bd/close_proxied_server.go index 5ead731e8..cfff01077 100644 --- a/cmd/bd/close_proxied_server.go +++ b/cmd/bd/close_proxied_server.go @@ -39,11 +39,22 @@ type closeProxiedInput struct { // errors is indexed by ARGUMENT position, not by item, so a refusal reported // here and one reported by the batch still print in the order the caller typed // the ids. +// +// failureErrors is the same list at the same indices, but carries what a --json +// consumer reads instead of what a person reads: the TYPED error for an engine +// refusal, where errors holds closeProxiedRefusal's decorated display line. The +// direct route makes exactly this split (close.go records res.Err.Error() in +// closeIDFailure while printing closeDirectRefusal), and keeping the decorated +// line in the machine-readable field made the same refusal answer differently +// per route — which is the drift this whole close rewire exists to remove. A +// policy refusal decided here has no separate display form, so both slots hold +// the one sentence closeProxiedCheckOne returned. type closeProxiedPreflight struct { - items []issueops.BatchCloseItem - itemArgs []int - before map[string]*types.Issue - errors []string + items []issueops.BatchCloseItem + itemArgs []int + before map[string]*types.Issue + errors []string + failureErrors []string } type closeProxiedOutcome struct { @@ -122,15 +133,12 @@ func runCloseProxiedServer(cmd *cobra.Command, ctx context.Context, args []strin outcomes, closeReasons := closeProxiedOutcomes(&pre, result) post := closeProxiedRunPostClose(ctx, args, in, outcomes) - // pre.errors is indexed by argument position, so args[i] is the id this - // refusal belongs to. - var failures []closeIDFailure - for i, e := range pre.errors { + for _, e := range pre.errors { if e != "" { fmt.Fprintln(os.Stderr, e) - failures = append(failures, closeIDFailure{ID: args[i], Error: e}) } } + failures := closeProxiedFailures(&pre, args) for _, w := range post.warnings { fmt.Fprintf(os.Stderr, "Warning: %s\n", w) } @@ -230,14 +238,19 @@ func proxiedBatchCloser() (issueops.BatchCloser, error) { // close policy to it, in one read-only unit of work. func closeProxiedRunPreflight(ctx context.Context, args, reasons []string, in closeProxiedInput) (closeProxiedPreflight, error) { pre := closeProxiedPreflight{ - errors: make([]string, len(args)), - before: make(map[string]*types.Issue, len(args)), + errors: make([]string, len(args)), + failureErrors: make([]string, len(args)), + before: make(map[string]*types.Issue, len(args)), } _, err := uow.RunTxRead(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (struct{}, error) { for i, id := range args { refusal, current := closeProxiedCheckOne(ctx, uw, id, in) if refusal != "" { pre.errors[i] = refusal + // A policy refusal is already the bare sentence, so the human + // and machine fields agree; only the engine refusals below have + // a decorated display form to strip. + pre.failureErrors[i] = refusal continue } pre.before[id] = current @@ -299,6 +312,10 @@ func closeProxiedOutcomes(pre *closeProxiedPreflight, result issueops.CloseBatch item := pre.items[j] if outcome.Err != nil { pre.errors[pre.itemArgs[j]] = closeProxiedRefusal(item.IssueID, outcome.Err) + // The typed error, not the decorated display line: the --force hint + // closeProxiedRefusal appends is advice for a human reader, and the + // id already has its own field. Same split as the direct route. + pre.failureErrors[pre.itemArgs[j]] = closeProxiedTypedRefusal(outcome.Err) continue } before := pre.before[item.IssueID] @@ -326,6 +343,82 @@ func closeProxiedOutcomes(pre *closeProxiedPreflight, result issueops.CloseBatch return outcomes, reasons } +// closeProxiedFailures pairs every refused argument with the id it belongs to, +// for the machine-readable half of the partial-failure report. +// +// Both slices are indexed by ARGUMENT position, so args[i] is the id that earned +// pre.errors[i], and the failures come back in the order the caller typed the +// ids. errors decides WHICH arguments failed — it is the slot the stderr print +// consults too, so the two halves of the report can never disagree about that — +// while failureErrors supplies the reason, because that is the field where the +// typed error belongs. +// +// Every writer of an errors slot fills its failureErrors twin, so the fallback +// below is unreachable today. It is here because the cost of getting that wrong +// is asymmetric: a caller that asked why an id failed is better served by the +// decorated line than by an empty string, so a future writer that forgets the +// twin degrades the wording instead of dropping the reason. +func closeProxiedFailures(pre *closeProxiedPreflight, args []string) []closeIDFailure { + var failures []closeIDFailure + for i, e := range pre.errors { + if e == "" { + continue + } + reason := pre.failureErrors[i] + if reason == "" { + reason = e + } + failures = append(failures, closeIDFailure{ID: args[i], Error: reason}) + } + return failures +} + +// closeProxiedTypedRefusal is the reason a --json consumer reads for one +// refused id: the typed error, spelled the way the direct route spells it. +// +// Stripping closeProxiedRefusal's decoration is not enough on its own to make +// the two routes agree, because a refusal reaching this route has also been +// wrapped by the layers it traveled through — the use case adds "close : " +// (storage/domain/issue.go) and the SQL repository adds +// "db: IssueSQLRepository.CloseChecked : " (storage/domain/db/issue.go) — +// while the direct route hands back the refusal exactly as the engine minted +// it. So the same blocked id reads +// +// cannot close blocked issue: x-1 is blocked by [x-2] (direct) +// close x-1: db: IssueSQLRepository.CloseChecked x-1: cannot close blocked … (proxied) +// +// For the refusal classes the CLI already names by sentinel, we report the +// outermost error that the sentinel itself explains: that is the sentence the +// engine wrote, and it is byte-identical to the direct route's. Neither wrap +// tells a caller anything it does not know — it asked to close that id — and +// one of them leaks an internal repository type name into an operator-facing +// field. +// +// Anything outside those classes keeps its whole chain. There the wrapping is +// not noise, it is the only thing that says where an unexpected failure came +// from, and there is no engine sentence to converge on. +func closeProxiedTypedRefusal(err error) string { + for _, sentinel := range []error{storage.ErrCloseBlocked, storage.ErrCloseOpenChildren} { + if !errors.Is(err, sentinel) { + continue + } + // Walk in until the next link IS the bare sentinel: that link's parent + // is the refusal with its subject filled in ("… x-1 is blocked by + // [x-2]"), where the sentinel alone would say only "cannot close + // blocked issue". + refusal := err + for { + next := errors.Unwrap(refusal) + if next == nil || next == sentinel || !errors.Is(next, sentinel) { + break + } + refusal = next + } + return refusal.Error() + } + return err.Error() +} + // closeProxiedRefusal spells one item's typed refusal the way this route has // always spelled it. The vocabulary is matched with errors.Is rather than by // reading the message, which is the point of the outcome carrying a typed From 26467a8d02b4e470dbb41c0769616e3db6f82c62 Mon Sep 17 00:00:00 2001 From: Matthew Doka Date: Sun, 27 Sep 2026 01:02:26 -0700 Subject: [PATCH 20/45] fix(storage): cascade wisp deletes into wisp_dependencies (#6563) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(storage): cascade wisp deletes into wisp_dependencies DeleteWispFromDependenciesInTx and DeleteWispsFromDependenciesInTx only cleaned the dependencies table, never wisp_dependencies — even though DeleteCascadeTables(isWisp=true) declares wisp_dependencies as part of the wisp deletion set. On stores whose wisp_dependencies table lacks the migration-0047 FK constraints (like a live production store observed with only the ck_wisp_dep_one_target CHECK), every wisp deletion orphaned that wisp's wisp_dependencies rows on both sides, and reaper scans flagged the leftovers as dangling_parent_ref anomalies that recurred after each cleanup (~100+ rows/hour under a busy daemon). Remove the deleted wisp's wisp_dependencies edges as child (issue_id) and as parent (depends_on_wisp_id) in both the single and batch delete paths. Two targeted DELETEs per direction rather than one OR query, so each hits its own index — the ff-tqm pattern; a single OR across both columns previously caused Dolt i/o timeouts. Adds TestWispDeleteCascade_CleansUpWispDependencies to the standalone- server integration harness (the existing wisp_gc_test.go coverage requires a Docker testcontainer and silently skips without one). The test drops the fresh-bootstrap FKs first so it exercises the delete path's own cascade, matching the production schema; it fails without this fix and covers the single-delete path the Docker tests omit. Co-Authored-By: Claude Opus 4.6 * fixup(review): pin over-deletion and correct two cascade comments Maintainer review fixes for PR #6563. No production behavior change. The new subtests asserted only that the deleted wisp's wisp_dependencies rows were gone, so an unguarded "DELETE FROM wisp_dependencies" with no WHERE passed both of them — verified by mutating both helpers and watching the test stay green. Each subtest now also creates an edge between two wisps it never deletes (a bystander pair for the single delete, an out-of-batch pair for the IN (...) form) and asserts that edge survives. Under the same mutation the test now fails three times with expected-vs-got. This also replaces the comment that claimed the surviving wisps' edge set was checked, which it was not: every edge the subtest created involved the deleted wisp. dropWispDependencyFKs attributed the FK constraints to migration 0047. They come from 0021, which creates all three inline in CREATE TABLE; 0058 re-adds two when healing a legacy split store, and 0047's ADD CONSTRAINT statements are gated on that same legacy path, so neither runs on the fresh bootstrap this harness uses. wispAuxCascadeTables exists so the cascade set cannot drift between the two delete paths, so note there why wisp_dependencies is handled in issueops instead: its rows are journaled as dependency removals. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 4.6 Co-authored-by: Julian Knutsen (cherry picked from commit 4213c229a4ec72aed8b1875be14a428fa8e150a3) --- .../wisp_delete_cascade_integration_test.go | 112 ++++++++++++++++++ internal/storage/dolt/wisps.go | 2 +- internal/storage/issueops/dependencies.go | 27 +++++ 3 files changed, 140 insertions(+), 1 deletion(-) diff --git a/internal/storage/dolt/wisp_delete_cascade_integration_test.go b/internal/storage/dolt/wisp_delete_cascade_integration_test.go index c57cdaecc..34a885c5d 100644 --- a/internal/storage/dolt/wisp_delete_cascade_integration_test.go +++ b/internal/storage/dolt/wisp_delete_cascade_integration_test.go @@ -101,6 +101,28 @@ func dropWispAuxFKs(t *testing.T, ctx context.Context, db *sql.DB) { } } +// dropWispDependencyFKs removes the wisp_dependencies FK constraints so the +// test store's schema matches the live hq store, whose wisp_dependencies +// table carries only the ck_wisp_dep_one_target CHECK constraint and no FKs. +// A fresh bootstrap gets all three from migration 0021, which creates them +// inline in CREATE TABLE; 0058 re-adds two of them when healing a legacy +// split store, and 0047's ADD CONSTRAINT statements are gated on that same +// legacy path, so neither runs here. Without this drop, the fresh store's +// ON DELETE CASCADE would clean wisp_dependencies for free and the test +// could not fail against a delete path that leaks. +func dropWispDependencyFKs(t *testing.T, ctx context.Context, db *sql.DB) { + t.Helper() + for _, stmt := range []string{ + "ALTER TABLE wisp_dependencies DROP FOREIGN KEY fk_wisp_dep_issue", + "ALTER TABLE wisp_dependencies DROP FOREIGN KEY fk_wisp_dep_wisp_target", + "ALTER TABLE wisp_dependencies DROP FOREIGN KEY fk_wisp_dep_issue_target", + } { + if _, err := db.ExecContext(ctx, stmt); err != nil { + t.Fatalf("drop wisp_dependencies FK (%s): %v", stmt, err) + } + } +} + // wispAuxTables lists the four wisp auxiliary tables a wisp delete must // cascade into. This mirrors issueops.DeleteCascadeTables(true), which // already documents this exact set as belonging to a wisp delete — separate @@ -254,3 +276,93 @@ func TestWispDeleteCascade_RepeatedCyclesLeaveNoOrphans(t *testing.T) { assertWispAuxTablesTotalZero(t, ctx, store.db, i) } } + +// TestWispDeleteCascade_CleansUpWispDependencies is the regression test for +// the hq dangling_parent_ref reaper anomaly: DeleteWispFromDependenciesInTx +// and DeleteWispsFromDependenciesInTx cleaned only the dependencies table, +// never wisp_dependencies — even though DeleteCascadeTables(true) declares +// wisp_dependencies as part of the wisp deletion set. On stores without the +// migration-0047 FKs (like live hq), every wisp deletion orphaned that +// wisp's wisp_dependencies rows on both sides (issue_id and +// depends_on_wisp_id). The package's existing regression tests for this +// behavior (TestDeleteWispBatch_CleansUpDependencies, +// TestDeleteWispBatch_BothDirectionsCleared in wisp_gc_test.go) require a +// Docker-testcontainer Dolt server; this covers the same contract — plus +// the single-delete path they omit — on the standalone-server harness. +func TestWispDeleteCascade_CleansUpWispDependencies(t *testing.T) { + store := setupWispCascadeStore(t) + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + + dropWispDependencyFKs(t, ctx, store.db) + + t.Run("single delete clears both directions", func(t *testing.T) { + // mid appears as both child (issue_id, via mid->parent) and parent + // (depends_on_wisp_id, via child->mid) in wisp_dependencies. + parent := createTestWisp(t, ctx, store, "wisp-dep parent") + mid := createTestWisp(t, ctx, store, "wisp-dep mid") + child := createTestWisp(t, ctx, store, "wisp-dep child") + mustAddWispDep(t, ctx, store, mid.ID, parent.ID) + mustAddWispDep(t, ctx, store, child.ID, mid.ID) + + // An edge between two wisps that are never deleted here. Neither of + // its endpoints is mid, so it must still be there afterwards: without + // it the subtest only asserts that rows disappeared, which an + // over-broad DELETE (missing or wrong WHERE) satisfies just as well. + bystanderA := createTestWisp(t, ctx, store, "wisp-dep bystander a") + bystanderB := createTestWisp(t, ctx, store, "wisp-dep bystander b") + mustAddWispDep(t, ctx, store, bystanderA.ID, bystanderB.ID) + + if err := store.deleteWisp(ctx, mid.ID); err != nil { + t.Fatalf("deleteWisp: %v", err) + } + + if n := countWispDependencyRows(t, ctx, store.db, mid.ID); n != 0 { + t.Errorf("expected 0 wisp_dependencies rows referencing deleted wisp %s, got %d", mid.ID, n) + } + if n := countWispDependencyRows(t, ctx, store.db, bystanderA.ID, bystanderB.ID); n != 1 { + t.Errorf("expected the bystander edge to survive deleteWisp(%s), got %d rows", mid.ID, n) + } + + // parent and child are left holding no edges once mid's rows go, so + // deleting them must succeed and must still not touch the bystander. + if err := store.deleteWisp(ctx, parent.ID); err != nil { + t.Fatalf("deleteWisp parent: %v", err) + } + if err := store.deleteWisp(ctx, child.ID); err != nil { + t.Fatalf("deleteWisp child: %v", err) + } + if n := countWispDependencyRows(t, ctx, store.db, bystanderA.ID, bystanderB.ID); n != 1 { + t.Errorf("expected the bystander edge to survive the parent/child deletes, got %d rows", n) + } + }) + + t.Run("batch delete clears both directions", func(t *testing.T) { + root := createTestWisp(t, ctx, store, "wisp-dep batch root") + stepA := createTestWisp(t, ctx, store, "wisp-dep batch step-a") + stepB := createTestWisp(t, ctx, store, "wisp-dep batch step-b") + mustAddWispDep(t, ctx, store, stepA.ID, root.ID) + mustAddWispDep(t, ctx, store, stepB.ID, stepA.ID) + + // Same over-deletion guard as the single-delete subtest, against the + // IN (...) form: both endpoints are outside the deleted batch. + outsideA := createTestWisp(t, ctx, store, "wisp-dep outside-batch a") + outsideB := createTestWisp(t, ctx, store, "wisp-dep outside-batch b") + mustAddWispDep(t, ctx, store, outsideA.ID, outsideB.ID) + + deleted, err := store.deleteWispBatch(ctx, []string{root.ID, stepA.ID, stepB.ID}) + if err != nil { + t.Fatalf("deleteWispBatch: %v", err) + } + if deleted != 3 { + t.Fatalf("expected 3 deleted, got %d", deleted) + } + + if n := countWispDependencyRows(t, ctx, store.db, root.ID, stepA.ID, stepB.ID); n != 0 { + t.Errorf("expected 0 wisp_dependencies rows after batch delete, got %d", n) + } + if n := countWispDependencyRows(t, ctx, store.db, outsideA.ID, outsideB.ID); n != 1 { + t.Errorf("expected the out-of-batch edge to survive deleteWispBatch, got %d rows", n) + } + }) +} diff --git a/internal/storage/dolt/wisps.go b/internal/storage/dolt/wisps.go index 3cd08ab0a..cf92a428f 100644 --- a/internal/storage/dolt/wisps.go +++ b/internal/storage/dolt/wisps.go @@ -420,7 +420,7 @@ func (s *DoltStore) deleteWispBatch(ctx context.Context, ids []string) (int, err } // deleteWispBatchTx deletes one batch of wisps inside its own transaction. -// Keeping each transaction to ≤200 wisps (6 DELETE statements) ensures it +// Keeping each transaction to ≤200 wisps (8 DELETE statements) ensures it // completes well within Dolt's 10 s write timeout. func (s *DoltStore) deleteWispBatchTx(ctx context.Context, ids []string) (int, error) { tx, err := s.db.BeginTx(ctx, nil) diff --git a/internal/storage/issueops/dependencies.go b/internal/storage/issueops/dependencies.go index 88e151d2d..755ec8da8 100644 --- a/internal/storage/issueops/dependencies.go +++ b/internal/storage/issueops/dependencies.go @@ -580,6 +580,20 @@ func DeleteWispFromDependenciesInTx(ctx context.Context, tx *sql.Tx, wispID stri "DELETE FROM dependencies WHERE depends_on_wisp_id = ?", wispID); err != nil { return fmt.Errorf("delete wisp %s from dependencies: %w", wispID, err) } + // wisp_dependencies is part of the wisp deletion set (DeleteCascadeTables), + // but no delete path cleaned it: every wisp deletion orphaned its + // wisp_dependencies rows on both sides, accumulating dangling parent/child + // refs that reaper scans flag as anomalies. Remove the wisp's edges as + // child (issue_id) and as parent (depends_on_wisp_id). Two targeted + // DELETEs, not one OR query, so each hits its own index (ff-tqm). + if _, err := tx.ExecContext(ctx, + "DELETE FROM wisp_dependencies WHERE issue_id = ?", wispID); err != nil { + return fmt.Errorf("delete wisp %s child rows from wisp_dependencies: %w", wispID, err) + } + if _, err := tx.ExecContext(ctx, + "DELETE FROM wisp_dependencies WHERE depends_on_wisp_id = ?", wispID); err != nil { + return fmt.Errorf("delete wisp %s parent rows from wisp_dependencies: %w", wispID, err) + } return nil } @@ -594,6 +608,19 @@ func DeleteWispsFromDependenciesInTx(ctx context.Context, tx *sql.Tx, wispIDs [] args...); err != nil { return fmt.Errorf("delete wisps from dependencies: %w", err) } + // See DeleteWispFromDependenciesInTx: wisp_dependencies rows must go with + // the wisps, on both the child and parent side. Two targeted DELETEs, not + // one OR query, so each hits its own index (ff-tqm). + if _, err := tx.ExecContext(ctx, + fmt.Sprintf("DELETE FROM wisp_dependencies WHERE issue_id IN (%s)", inClause), + args...); err != nil { + return fmt.Errorf("delete wisps child rows from wisp_dependencies: %w", err) + } + if _, err := tx.ExecContext(ctx, + fmt.Sprintf("DELETE FROM wisp_dependencies WHERE depends_on_wisp_id IN (%s)", inClause), + args...); err != nil { + return fmt.Errorf("delete wisps parent rows from wisp_dependencies: %w", err) + } return nil } From cf08d4e3f9b6bdac92780dd57e103bc62b0c20e7 Mon Sep 17 00:00:00 2001 From: Vishnu J Date: Sat, 26 Sep 2026 13:58:55 -0700 Subject: [PATCH 21/45] fix(storage): cascade wisp aux rows in the shared wisp delete path (#6487) (#6627) deleteIssueRowInTx, behind Transaction.DeleteIssue (bd mol burn) and DeleteResolvedSetInTx, removed only the wisps row and the dependencies edges, so on stores lacking the wisp aux FKs every deleted wisp orphaned its wisp_labels, wisp_events, wisp_comments and wisp_child_counters rows. #5343 fixed only the DoltStore wisp GC paths. Move the aux table list and delete into issueops.DeleteWispAuxRowsInTx, call it from deleteIssueRowInTx, and have the DoltStore paths use the same helper so the table set cannot drift. Forward-fix only: cleaning existing orphans needs a batched backfill and is left to a follow-up. Sibling of #6563 (wisp_dependencies). Co-authored-by: Claude Sonnet 5 (cherry picked from commit 3ba08dba1cd3e23e55f70109a2b3c703cb1e76d4) --- ...wisp_delete_cascade_tx_integration_test.go | 61 +++++++++++++++++++ internal/storage/dolt/wisps.go | 38 +----------- internal/storage/issueops/delete.go | 37 +++++++++++ .../issueops/journal_completeness_test.go | 1 + 4 files changed, 101 insertions(+), 36 deletions(-) create mode 100644 internal/storage/dolt/wisp_delete_cascade_tx_integration_test.go diff --git a/internal/storage/dolt/wisp_delete_cascade_tx_integration_test.go b/internal/storage/dolt/wisp_delete_cascade_tx_integration_test.go new file mode 100644 index 000000000..6a51bf8ae --- /dev/null +++ b/internal/storage/dolt/wisp_delete_cascade_tx_integration_test.go @@ -0,0 +1,61 @@ +//go:build integration && !windows + +package dolt + +import ( + "context" + "database/sql" + "fmt" + "testing" + "time" + + "github.com/steveyegge/beads/internal/storage" +) + +// TestWispDeleteCascade_TransactionDeleteIssue covers the delete path #5343 +// left uncovered: Transaction.DeleteIssue (what `bd mol burn` uses) goes +// through issueops.DeleteIssueInTx -> deleteIssueRowInTx rather than +// deleteWisp/deleteWispBatchTx, and removed only the wisps row and the +// dependencies edges. On stores without the wisp aux FKs (dropped by +// setupWispCascadeStore) every burned wisp orphaned its wisp_labels, +// wisp_events, wisp_comments and wisp_child_counters rows (#6487). +func TestWispDeleteCascade_TransactionDeleteIssue(t *testing.T) { + store := setupWispCascadeStore(t) + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + + wisp := createTestWisp(t, ctx, store, "tx-delete wisp") + other := createTestWisp(t, ctx, store, "tx-delete bystander wisp") + seedWispAuxRows(t, ctx, store.db, wisp.ID) + seedWispAuxRows(t, ctx, store.db, other.ID) + bystanderBefore := make([]int, len(wispAuxTables)) + for i, tc := range wispAuxTables { + bystanderBefore[i] = countWispAuxRows(t, ctx, store.db, tc.table, tc.column, other.ID) + } + + if err := store.RunInTransaction(ctx, "test: burn wisp", func(tx storage.Transaction) error { + return tx.DeleteIssue(ctx, wisp.ID) + }); err != nil { + t.Fatalf("Transaction.DeleteIssue: %v", err) + } + + assertWispAuxRowsGone(t, ctx, store.db, wisp.ID) + + // The delete must be scoped to the deleted wisp: a bystander's rows stay. + for i, tc := range wispAuxTables { + if got := countWispAuxRows(t, ctx, store.db, tc.table, tc.column, other.ID); got != bystanderBefore[i] { + t.Errorf("bystander wisp %s: %s rows changed from %d to %d", other.ID, tc.table, bystanderBefore[i], got) + } + } +} + +func countWispAuxRows(t *testing.T, ctx context.Context, db *sql.DB, table, column, id string) int { + t.Helper() + var count int + //nolint:gosec // G201: table/column come from the fixed wispAuxTables literal, not input. + q := fmt.Sprintf("SELECT COUNT(*) FROM %s WHERE %s = ?", table, column) + if err := db.QueryRowContext(ctx, q, id).Scan(&count); err != nil { + t.Fatalf("count %s: %v", table, err) + } + return count +} diff --git a/internal/storage/dolt/wisps.go b/internal/storage/dolt/wisps.go index cf92a428f..36b6bee66 100644 --- a/internal/storage/dolt/wisps.go +++ b/internal/storage/dolt/wisps.go @@ -294,40 +294,6 @@ func (s *DoltStore) closeWispChecked(ctx context.Context, id string, actor strin return storage.CloseIssueResult{Unchanged: res.AlreadyClosed, OpenChildren: res.OpenChildren}, nil } -// wispAuxCascadeTables lists the wisp auxiliary tables a wisp delete must -// also clean up, mirroring internal/storage/schema/cli_migrations.go:300-315. -// wisp_child_counters is keyed on parent_id (a wisp can be a parent whose -// children hold the counter row); the other three are keyed on issue_id. -// Some deployed stores enforce this via FK ON DELETE CASCADE and some do not -// (be-zdqyl: the migration adding those FKs was never promoted out of -// migrations/ignored/), so the delete paths below must not rely on the -// database to do it for them. -var wispAuxCascadeTables = []struct{ table, column string }{ - {"wisp_labels", "issue_id"}, - {"wisp_events", "issue_id"}, - {"wisp_comments", "issue_id"}, - {"wisp_child_counters", "parent_id"}, -} - -// deleteWispAuxRowsInTx removes every row the given wisp ids own across -// wispAuxCascadeTables. Shared by deleteWisp and deleteWispBatchTx so the -// table set cannot drift between the two paths. -func deleteWispAuxRowsInTx(ctx context.Context, tx *sql.Tx, ids []string) error { - if len(ids) == 0 { - return nil - } - inClause, args := doltBuildSQLInClause(ids) - for _, aux := range wispAuxCascadeTables { - //nolint:gosec // G201: aux.table/aux.column come from the fixed wispAuxCascadeTables literal; inClause contains only ? markers - if _, err := tx.ExecContext(ctx, - fmt.Sprintf("DELETE FROM %s WHERE %s IN (%s)", aux.table, aux.column, inClause), - args...); err != nil { - return fmt.Errorf("delete wisp aux rows from %s: %w", aux.table, err) - } - } - return nil -} - // deleteWisp permanently removes a wisp and its related data. func (s *DoltStore) deleteWisp(ctx context.Context, id string) error { tx, err := s.db.BeginTx(ctx, nil) @@ -374,7 +340,7 @@ func (s *DoltStore) deleteWisp(ctx context.Context, id string) error { return err } - if err := deleteWispAuxRowsInTx(ctx, tx, []string{id}); err != nil { + if err := issueops.DeleteWispAuxRowsInTx(ctx, tx, []string{id}); err != nil { return fmt.Errorf("delete wisp aux rows for %s: %w", id, err) } @@ -475,7 +441,7 @@ func (s *DoltStore) deleteWispBatchTx(ctx context.Context, ids []string) (int, e return 0, err } - if err := deleteWispAuxRowsInTx(ctx, tx, ids); err != nil { + if err := issueops.DeleteWispAuxRowsInTx(ctx, tx, ids); err != nil { return 0, fmt.Errorf("delete wisp aux rows: %w", err) } diff --git a/internal/storage/issueops/delete.go b/internal/storage/issueops/delete.go index 7ed67f634..e0406c8f2 100644 --- a/internal/storage/issueops/delete.go +++ b/internal/storage/issueops/delete.go @@ -85,6 +85,9 @@ func deleteIssueRowInTx(ctx context.Context, tx *sql.Tx, id string, isWisp bool, if err := DeleteWispFromDependenciesInTx(ctx, tx, id); err != nil { return err } + if err := DeleteWispAuxRowsInTx(ctx, tx, []string{id}); err != nil { + return err + } } else if err := DeleteLeaseInTx(ctx, tx, id); err != nil { // A deleted issue holds no lease. return err @@ -92,6 +95,40 @@ func deleteIssueRowInTx(ctx context.Context, tx *sql.Tx, id string, isWisp bool, return nil } +// wispAuxCascadeTables lists the wisp auxiliary tables a wisp delete must also +// clean up, mirroring internal/storage/schema/cli_migrations.go:300-315. +// wisp_child_counters is keyed on parent_id (a wisp can be a parent whose +// children hold the counter row); the other three are keyed on issue_id. +// Some deployed stores enforce this via FK ON DELETE CASCADE and some do not +// (be-zdqyl: the migration adding those FKs was never promoted out of +// migrations/ignored/), so the delete paths must not rely on the database to +// do it for them. wisp_dependencies is cleaned by DeleteWisp(s)FromDependenciesInTx. +var wispAuxCascadeTables = []struct{ table, column string }{ + {"wisp_labels", "issue_id"}, + {"wisp_events", "issue_id"}, + {"wisp_comments", "issue_id"}, + {"wisp_child_counters", "parent_id"}, +} + +// DeleteWispAuxRowsInTx removes every row the given wisp ids own across +// wispAuxCascadeTables. Shared by every wisp delete path (the DoltStore wisp +// GC paths and deleteIssueRowInTx) so the table set cannot drift between them. +func DeleteWispAuxRowsInTx(ctx context.Context, tx *sql.Tx, wispIDs []string) error { + if len(wispIDs) == 0 { + return nil + } + inClause, args := buildSQLInClause(wispIDs) + for _, aux := range wispAuxCascadeTables { + //nolint:gosec // G201: aux.table/aux.column come from the fixed wispAuxCascadeTables literal; inClause contains only ? placeholders + if _, err := tx.ExecContext(ctx, + fmt.Sprintf("DELETE FROM %s WHERE %s IN (%s)", aux.table, aux.column, inClause), + args...); err != nil { + return fmt.Errorf("delete wisp aux rows from %s: %w", aux.table, err) + } + } + return nil +} + // DeletionSet is the EXACT set of rows one delete removes, split by the plane // each row lives in. // diff --git a/internal/storage/issueops/journal_completeness_test.go b/internal/storage/issueops/journal_completeness_test.go index 9e28558cd..d14458524 100644 --- a/internal/storage/issueops/journal_completeness_test.go +++ b/internal/storage/issueops/journal_completeness_test.go @@ -170,6 +170,7 @@ var beadDMLExemptions = map[string]string{ "RetargetInboundDependenciesToIssueInTx": "rewrites dep rows during promote; PromoteFromEphemeralInTx journals it", "DeleteWispFromDependenciesInTx": "cleans up dep rows during a delete that journals the delete", "DeleteWispsFromDependenciesInTx": "cleans up dep rows during a delete that journals the delete", + "DeleteWispAuxRowsInTx": "cleans up wisp label/event/comment/counter rows during a delete that journals the delete", // (4) compaction maintenance — a lossy content rewrite outside the // create/update/close/delete/dep/label op vocabulary. Carried from the From 98d9675f5e2f22e3f20c5e36e2448d6420993f2b Mon Sep 17 00:00:00 2001 From: Vishnu J Date: Sun, 27 Sep 2026 01:31:11 -0700 Subject: [PATCH 22/45] fix: honor dolt.auto-commit in proxied server mode (#6499) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(dolt): honor dolt.auto-commit=batch/off on the proxied-server write path (#4995) In proxied-server mode every write minted a Dolt history commit whatever dolt.auto-commit said. Two causes: - cmd/bd/main.go never applied issueOpsContext(rootCtx) in the proxiedServerMode branch of the root pre-run, so the context carried no deferred-commit marker. - internal/storage/uow/doltserver_tx.go: doltServerTx.Commit ignored issueops.VersionCommitDeferred(ctx) and ran CALL DOLT_COMMIT('-Am', ?) whenever a message was supplied. Apply issueOpsContext to rootCtx in the proxied setup, and have doltServerTx.Commit blank its message under a deferred context so the existing plain-COMMIT branch persists the write to the working set without advancing Dolt history. dolt.auto-commit=on is unchanged. This commit alone strands batch/off writes in the working set; the explicit flush point (`bd dolt commit`) follows in the later commits of this series. (Originally recorded as "witness: salvage uncommitted work (ocb-ani)"; the change itself is unmodified.) * test(uow,cmd/bd): add regression tests for proxied dolt.auto-commit batch/off (#4995) Add focused regression tests verifying that proxied dolt.auto-commit batch/off defer Dolt version commits: - cmd/bd: TestIssueOpsContextDefersByModeInProxiedServerMode verifies that issueOpsContext defers version commits in proxied-server mode when dolt.auto-commit is batch or off. - internal/storage/uow: TestDoltServerTxCommitDefersDoltCommitUnderDeferredContext and TestDoltServerTxRunTxWithDeferredContextSkipsDoltCommit verify that doltServerTx.Commit blanks the commit message under a deferred context, bypassing DOLT_COMMIT and status checks, and executing plain COMMIT to persist writes into the working set without advancing Dolt history. * fix(dolt): add the proxied flush point dolt.auto-commit=batch/off defers to (#4995) The deferral half of this PR makes dolt.auto-commit=batch/off actually take effect on the proxied-server write path. On its own that turns batch/off into "never commit": proxied mode returns from the root pre-run before newDoltStore runs, so getStore() is nil and `bd dolt commit` has no store to commit through. Writes pile up in the sql-server working set where `dolt log`, `dolt push` and backups never see them. Add the missing half. cmd/bd/dolt_commit_proxied_server.go is the proxied dual of `bd dolt commit`: it takes the uow.MaintenanceProvider route (RunNonTx) and runs CALL DOLT_COMMIT('-Am', ?) on a pinned connection, gated on issueops.HasPendingChanges. It deliberately does not go through doltServerTx.Commit, which blanks its own message under the deferred context; routing the flush through the deferral it exists to discharge would make the flush a no-op. doltCommitCmd dispatches to it on usesProxiedServer(); the message is built before the branch so both routes name the same default, and the "Committed." / "Nothing to commit." contract is unchanged. --dolt-auto-commit's help no longer claims proxied-server routes are unaffected: it names the proxied flush point and keeps the SIGTERM/SIGHUP flush marked embedded-only. docs/reference/configuration.md says the same. `bd vc commit`'s proxied refusal now points at `bd dolt commit`. Tests: - cmd/bd/dolt_commit_proxied_integration_test.go: three batch writes leave dolt_log unchanged and stay readable, `bd dolt commit` advances it by exactly one, a second `bd dolt commit` reports "Nothing to commit." with dolt_log unchanged, and one write under --dolt-auto-commit on advances it by one. On the PR's original base it failed at the `bd dolt commit` step with "no store available". On current main the front door refuses the command before that code is reached, so this test is only green together with the next commit, which moves `dolt commit` to supported in the proxy capability registry. - TestIssueOpsContextDefersByModeInProxiedServerMode renamed to TestIssueOpsContextDefersByModePolicyTable and the proxiedServerMode assignment dropped: issueOpsContext branches only on writesCommitNow(), never on the storage mode, so the old name over-promised. - doltserver_tx_test.go: "bd: update bd-4995" -> "bd: update GH#4995", which no longer reads as a bead id. Co-Authored-By: Claude Opus 5 (1M context) * fix(proxy): move `dolt commit` from refused to supported in the capability registry Main's proxy capability registry (added by #6269, 24a9da6cc) declares `dolt commit` direct-only: it has a "proxy.dolt_commit.unsupported" refusal in proxyMaintenanceRefusals (cmd/bd/proxy_capability.go) and is classed HistoryDirectOnly in historyCapabilityMatrix (cmd/bd/history_capability.go). The front door therefore refuses `bd dolt commit` in proxied-server mode before dispatch, so the proxied dual added earlier in this series is unreachable, and dolt.auto-commit=batch/off has no way to flush. the history/VC front-door contract, and neither the registry, its tests nor the docs explain why `dolt commit` was listed. Nothing on main implemented `dolt commit` for the proxied route at the time, which is consistent with the entry being a statement of what existed rather than a design decision. This flips a capability main had declared unsupported, deliberately: the proxied route now has an implementation (runDoltCommitProxiedServer), and batch/off need it as their flush point (GH#4995). Declared the same way as the other proxied-supported command, `dolt remote remove`: HistoryProxySupported in the history matrix, and no row in proxyMaintenanceRefusals. Tests follow: history_capability_test.go moves `dolt commit` from the direct-only assertions and refusal contract to the proxy-supported ones (including the nested front-door pass-through), and history_remote_frontdoor_matrix_test.go drops the refusal row and adds a `dolt commit` pass-through check to TestHistoryRemoteSupportedFrontDoorParity. Co-Authored-By: Claude Sonnet 5 * fix(proxy): exempt explicit commit points from the proxied deferral (#4995) Maintainer review fixup for PR #6499, adopt-pr iterations 1-2. Major (Contract & Interface Fidelity). The deferral is applied once, to rootCtx, so it blanketed BOTH commit classes the direct route distinguishes by which wrapper a verb picks: policy-honoring writes (transactHonoringAutoCommit, per-verb issueOpsContext) and explicit commit points (transact, which passes the caller's message whatever the policy says). Under dolt.auto-commit=batch a proxied `bd batch -m "release batch"` therefore persisted the rows, discarded the message and minted nothing, while the identical command and config on the direct route committed it — and batch_proxied_server.go's own contract comment rejects exactly that ("would persist these writes while silently bypassing Dolt history"), as does batch.go's "the contract is the same on both backends". issueops.WithImmediateVersionCommit clears the marker and cmd/bd's explicitCommitPointContext applies it in the duals whose direct twin calls transact. Scope is the intersection of the transact call sites with proxyPermittedPaths: batch, mol bond/pour/squash, mol wisp create, and the WISP half of mol burn (cook and migrate issues also call transact but are refused in proxied mode, so they have no dual to exempt). The review sized the radius at batch + 4 mol verbs; mol wisp create is the same class — its direct twin reaches cloneSubgraph's transact through spawnMoleculeWithOptions — so it is exempted with them. Membership is per commit site, not per command (iteration 2). `bd mol burn` is two classes on the direct route: wisps go through burnWisps -> transact, while persistent molecules go through deleteBatch -> issueOpsContext -> dolt/deleter.go, which returns before DOLT_COMMIT when the context defers. Exempting the whole command minted one Dolt commit per persistent molecule under batch/off — on a destructive verb, where the identical direct-route command defers them all to `bd dolt commit`. runMolBurnProxiedServer now applies the exemption to the wisp RunTxResult only, and the flag help, configuration.md and the helper's own radius comment say so. Minors: - The flush passes '--author' like DoltStorage.CommitAll, so the one commit batch mode keeps is attributed to the bd actor rather than the SQL session user (without it dolt_log reads `root `). - runDoltCommitProxiedServer calls the shared runProxiedNonTx instead of re-implementing its provider checks, so `bd --json dolt commit` keeps the JSON error envelope; `dolt commit`'s RunE no longer re-wraps an already-reported failure, and runProxiedNonTx's message gains the provider %T the open-coded version had. - The flag help and configuration.md drop two false claims: that the deferral covers "every write" in proxied mode (the bd serve plane over the same provider never touches rootCtx, so daemon writes still commit per write) and that the SIGTERM/SIGHUP flush is "embedded-only" (flushBatchCommitOnShutdown gates on an open store, which direct SQL-server mode has; only proxied mode lacks one). Both surfaces now name the explicit-commit-point exemption. - Both user-facing enumerations of that exemption are now derived from the explicitCommitPointContext call sites rather than restated (iteration 2): they had omitted mol wisp create, listed mol burn wholesale, and implied a message-less `bd batch` defers when batch.go synthesizes a default message and the dual exempts unconditionally, so every proxied batch commits. - doltserver_tx.go's ephemeral-form comment names its second entry point — RunTx/RunTxResult under a deferred context — where its ONLY-dolt_ignored-state rationale stops applying. Not done: regenerating docs/CLI_REFERENCE.md. Its --dolt-auto-commit drift is base-preexisting, and the generated CLI docs are built from the release tag pinned in docs/cli-docs.pin, not from this source, so touching them converts the drift gate's inherited-drift warn-pass into a failure. Tests. The uow blanking table gains the immediate-commit arm (tagless, so it runs in every lane); the proxied integration test gains an explicit-batch-commit step with a deferred plain create as its control, plus the flush attribution assertions. Mutation-proved: reverting explicitCommitPointContext to a no-op reds the new step (dolt_log 80 -> 80, no commit minted), and dropping '--author' reds the attribution assertion with committer=root. Iteration 2 adds the WIRING pin those left open — the mechanism was covered, but deleting `ctx = explicitCommitPointContext(ctx)` from a dual was green in every lane, which is why the mol burn over-application above was invisible. dolt_autocommit_exemption_wiring_test.go is tagless and needs no server: a uow.UnitOfWorkProvider that records the context RunTxResult hands NewUOW and then refuses observes exactly the context each dual gives its transaction. Mutation-proved per arm: neutering the exemption at batch/bond/squash reds those three rows and only those; dropping it from the wisp RunTxResult reds the wisp arm alone; and restoring the command-wide reassignment reds the persistent arm alone, reproducing the defect this iteration fixes. Validation: go build ./..., go vet (cgo and CGO_ENABLED=0), gofmt clean; internal/storage/issueops; internal/storage/uow -run TestDoltServerTx; the cmd/bd registry / history / policy / batch / mol / wisp / errors battery; and the gated real-server lane BEADS_TEST_PROXIED_SERVER=1 green for both TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce (41s) and TestProxiedServerMolBurn (33s, all four subtests including the mixed wisp+persistent burn this iteration changed). Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 (1M context) Co-authored-by: Julian Knutsen (cherry picked from commit edb86e34702c02067a57cee961ed18e5fc020991) Hand-resolved for hotfix/1.3.1: cmd/bd/history_remote_frontdoor_matrix_test.go conflicted because main's context around TestHistoryRemoteSupportedFrontDoorParity includes direct-topology test helpers (snapshotHistoryState, newDirectHistoryProject, historyCleanEnv, runDirectHistory, openDirectHistoryDB, startHistoryUnixBridge) that the release line does not have. Those main-only helpers are NOT brought over; only this commit's own change to that region (the updated doc comment) is applied. The refusal-row removal and the new dolt-commit parity assertion applied cleanly. All other files applied unchanged. Co-Authored-By: Claude Opus 5.5 --- cmd/bd/batch_proxied_server.go | 3 + cmd/bd/capability_registry.go | 19 +- cmd/bd/capability_registry_history_test.go | 15 +- cmd/bd/dolt.go | 49 ++-- cmd/bd/dolt_autocommit.go | 29 +++ .../dolt_autocommit_exemption_wiring_test.go | 238 ++++++++++++++++++ cmd/bd/dolt_autocommit_server_test.go | 34 +++ .../dolt_commit_proxied_integration_test.go | 191 ++++++++++++++ cmd/bd/dolt_commit_proxied_server.go | 88 +++++++ cmd/bd/dolt_commit_proxied_server_test.go | 29 +++ cmd/bd/errors.go | 9 + .../history_remote_frontdoor_matrix_test.go | 14 +- cmd/bd/main.go | 12 +- cmd/bd/mol_proxied_server.go | 25 +- cmd/bd/proxied_maintenance.go | 5 +- cmd/bd/vc.go | 5 +- cmd/bd/wisp_proxied_server.go | 6 + docs/reference/configuration.md | 22 +- internal/storage/issueops/version_commit.go | 12 + internal/storage/uow/doltserver_tx.go | 25 +- internal/storage/uow/doltserver_tx_test.go | 83 ++++++ 21 files changed, 873 insertions(+), 40 deletions(-) create mode 100644 cmd/bd/dolt_autocommit_exemption_wiring_test.go create mode 100644 cmd/bd/dolt_commit_proxied_integration_test.go create mode 100644 cmd/bd/dolt_commit_proxied_server.go create mode 100644 cmd/bd/dolt_commit_proxied_server_test.go diff --git a/cmd/bd/batch_proxied_server.go b/cmd/bd/batch_proxied_server.go index 52fd7eb14..99ccb890e 100644 --- a/cmd/bd/batch_proxied_server.go +++ b/cmd/bd/batch_proxied_server.go @@ -38,6 +38,9 @@ func runBatchProxiedServer(ctx context.Context, ops []batchOp, commitMsg string) if uowProvider == nil { return nil, HandleErrorRespectJSON("proxied-server UOW provider not initialized") } + // Explicit commit point, like `transact` on the direct route: the commit + // carries the caller's message whatever dolt.auto-commit says (GH#4995). + ctx = explicitCommitPointContext(ctx) return uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) ([]batchOpResult, string, error) { // Built per attempt, not per call: a retried batch must report the // results of the attempt that actually committed. diff --git a/cmd/bd/capability_registry.go b/cmd/bd/capability_registry.go index 5ee3b2aa5..ab18b06d1 100644 --- a/cmd/bd/capability_registry.go +++ b/cmd/bd/capability_registry.go @@ -324,12 +324,13 @@ var proxyCapabilityRegistry = []capabilityRow{ refusedPath("repo sync", "proxy.repo.unsupported", ProxyReasonDesign, ""), // --- dolt version-control verbs ----------------------------------------- - // CALL DOLT_PUSH/DOLT_PULL/DOLT_COMMIT are SQL procedures and the generic - // repository that wraps them is already written; these need a use case and - // a route, not proxy work. `dolt remote remove` is already routed, which is - // the precedent. - refusedPath("dolt commit", "proxy.dolt_commit.unsupported", ProxyReasonUnimplemented, trackVersionCtl). - withHistory(HistoryDirectOnly), + // CALL DOLT_PUSH/DOLT_PULL are SQL procedures and the generic repository + // that wraps them is already written; these need a use case and a route, + // not proxy work. `dolt remote remove` is already routed, which is the + // precedent — and `dolt commit` has now followed it: GH#4995 supplied the + // use case (it is the flush point dolt.auto-commit=batch/off defers to) and + // runDoltCommitProxiedServer supplied the route, so its row moved to the + // routed-history block below. refusedPath("dolt push", "proxy.dolt_push.unsupported", ProxyReasonUnimplemented, trackVersionCtl). withHistory(HistoryDirectOnly), refusedPath("dolt pull", "proxy.dolt_pull.unsupported", ProxyReasonUnimplemented, trackVersionCtl). @@ -437,10 +438,14 @@ var proxyCapabilityRegistry = []capabilityRow{ {Path: "duplicates", ArgSet: "--auto-merge --dry-run", Rule: proxyCapabilityRule{Outcome: ProxyOutcomeHonored}}, // --- routed history ----------------------------------------------------- - // The two paths that already have a proxied route, recorded so the class is + // The paths that already have a proxied route, recorded so the class is // asserted rather than remembered. permitted("history").withHistory(HistoryProxySupported), permitted("dolt remote remove").withHistory(HistoryProxySupported), + // `dolt commit` is routed by runDoltCommitProxiedServer. It is also the + // flush point dolt.auto-commit=batch/off defers to (GH#4995), so refusing it + // at the front door would leave those modes with no way to flush. + permitted("dolt commit").withHistory(HistoryProxySupported), // --- untyped refusals inside RunE --------------------------------------- // The gate permits these paths and the command refuses itself with a bare diff --git a/cmd/bd/capability_registry_history_test.go b/cmd/bd/capability_registry_history_test.go index 87d1fd44d..3884b6765 100644 --- a/cmd/bd/capability_registry_history_test.go +++ b/cmd/bd/capability_registry_history_test.go @@ -12,13 +12,17 @@ func TestHistoryCapabilityMatrixExactPaths(t *testing.T) { if got, ok := LookupHistoryCapability("history"); !ok || got != HistoryProxySupported { t.Fatalf("history = %q, %v; want proxy-supported", got, ok) } - for _, path := range []string{"branch", "conflicts", "repo", "federation", "vc", "flatten", "dolt push", "dolt pull", "dolt commit", "dolt remote", "sync"} { + for _, path := range []string{"branch", "conflicts", "repo", "federation", "vc", "flatten", "dolt push", "dolt pull", "dolt remote", "sync"} { if got, ok := LookupHistoryCapability(path); !ok || got != HistoryDirectOnly { t.Errorf("%q = %q, %v; want direct-only", path, got, ok) } } - if got, ok := LookupHistoryCapability("dolt remote remove"); !ok || got != HistoryProxySupported { - t.Fatalf("dolt remote remove = %q, %v; want proxy-supported", got, ok) + // dolt commit is the flush point dolt.auto-commit=batch/off defers to + // (GH#4995), so it must stay reachable on the proxied route. + for _, path := range []string{"dolt remote remove", "dolt commit"} { + if got, ok := LookupHistoryCapability(path); !ok || got != HistoryProxySupported { + t.Fatalf("%s = %q, %v; want proxy-supported", path, got, ok) + } } for _, path := range []string{"dolt remote add", "dolt remote list", "dolt remote reset-data"} { if got, ok := LookupHistoryCapability(path); !ok || got != HistoryDirectOnly { @@ -50,7 +54,6 @@ func TestHistoryDirectOnlyRefusalContract(t *testing.T) { "flatten": {"proxy.flatten.unsupported", "flatten is not supported in proxied-server mode"}, "dolt push": {"proxy.dolt_push.unsupported", "dolt push is not supported in proxied-server mode"}, "dolt pull": {"proxy.dolt_pull.unsupported", "dolt pull is not supported in proxied-server mode"}, - "dolt commit": {"proxy.dolt_commit.unsupported", "dolt commit is not supported in proxied-server mode"}, "dolt remote add": {"proxy.dolt_remote.unsupported", "dolt remote add is not supported in proxied-server mode"}, "dolt remote list": {"proxy.dolt_remote.unsupported", "dolt remote list is not supported in proxied-server mode"}, "dolt remote reset-data": {"proxy.dolt_remote.unsupported", "dolt remote reset-data is not supported in proxied-server mode"}, @@ -94,7 +97,7 @@ func TestHistoryNestedFrontDoorsRefuseAndSupportedPathsPass(t *testing.T) { "conflicts resolve": {"proxy.conflicts.unsupported", "conflicts resolve is not supported in proxied-server mode"}, "federation sync": {"proxy.federation.unsupported", "federation sync is not supported in proxied-server mode"}, } - for _, path := range []string{"vc merge", "vc commit", "repo add", "conflicts resolve", "federation sync", "dolt remote remove"} { + for _, path := range []string{"vc merge", "vc commit", "repo add", "conflicts resolve", "federation sync", "dolt remote remove", "dolt commit"} { parts := strings.Split(path, " ") root := &cobra.Command{Use: "bd"} cmd := &cobra.Command{Use: parts[0]} @@ -105,7 +108,7 @@ func TestHistoryNestedFrontDoorsRefuseAndSupportedPathsPass(t *testing.T) { cmd = child } err := validateProxyRegistryBeforeProvider(cmd, ProxyTopologyManagedLocal) - if path == "dolt remote remove" { + if path == "dolt remote remove" || path == "dolt commit" { if err != nil { t.Fatalf("supported %s refused: %v", path, err) } diff --git a/cmd/bd/dolt.go b/cmd/bd/dolt.go index 009b15bb6..3fc18fc07 100644 --- a/cmd/bd/dolt.go +++ b/cmd/bd/dolt.go @@ -728,26 +728,47 @@ For more options (--stdin, custom messages), see: bd vc commit`, SilenceErrors: true, RunE: func(cmd *cobra.Command, args []string) error { ctx := context.Background() - st := getStore() - if st == nil { - return HandleError("no store available") - } msg, _ := cmd.Flags().GetString("message") if msg == "" { msg = fmt.Sprintf("bd: dolt commit (auto-commit) by %s", getActor()) } - // CommitAll, not Commit: this command's contract is "any uncommitted - // changes in the working set", including changes made externally and - // the config table — which server-mode Commit excludes (GH#2455), so - // out-of-band config dirt used to survive this command forever. Its - // committed bool also replaces the HEAD-before/HEAD-after comparison - // this command used to detect tolerated no-ops with, which cost two - // extra HEAD reads and raced against concurrent writers. - committed, err := st.CommitAll(ctx, msg) + + var ( + committed bool + err error + ) + if usesProxiedServer() { + // Proxied mode never opens a store — the root pre-run returns + // before newDoltStore — so getStore() is nil here and the UOW + // provider is the only handle on the server. This is the flush + // point dolt.auto-commit=batch/off defers to on that route + // (GH#4995). The message is built above so both routes name the + // same default. + committed, err = runDoltCommitProxiedServer(ctx, msg) + } else { + st := getStore() + if st == nil { + return HandleError("no store available") + } + // CommitAll, not Commit: this command's contract is "any uncommitted + // changes in the working set", including changes made externally and + // the config table — which server-mode Commit excludes (GH#2455), so + // out-of-band config dirt used to survive this command forever. Its + // committed bool also replaces the HEAD-before/HEAD-after comparison + // this command used to detect tolerated no-ops with, which cost two + // extra HEAD reads and raced against concurrent writers. + committed, err = st.CommitAll(ctx, msg) + } if err != nil { - if isDoltNothingToCommit(err) { + switch { + case isDoltNothingToCommit(err): committed = false - } else { + case isReportedExit(err): + // The proxied route reports provider failures itself, through + // HandleErrorRespectJSON, so the JSON envelope is already on + // stdout; re-wrapping would print a second, plain Error line. + return err + default: return HandleError("%v", err) } } diff --git a/cmd/bd/dolt_autocommit.go b/cmd/bd/dolt_autocommit.go index 186c9a76f..a32205e85 100644 --- a/cmd/bd/dolt_autocommit.go +++ b/cmd/bd/dolt_autocommit.go @@ -91,6 +91,35 @@ func issueOpsContext(ctx context.Context) (context.Context, error) { return issueops.WithDeferredVersionCommit(ctx), nil } +// explicitCommitPointContext clears the dolt.auto-commit deferral for a proxied +// dual whose direct-route twin is an explicit commit point. +// +// The direct route encodes two commit classes, and it encodes them by which +// wrapper a verb picks: writes whose Dolt commit is auto-commit policy go +// through transactHonoringAutoCommit or issueOpsContext above, while the +// explicit commit points go through transact and mint a Dolt commit carrying +// the caller's message whatever the policy says. The proxied route has no such +// choice to make per verb — it applies the policy ONCE, to rootCtx in the root +// pre-run (GH#4995) — so the second class has to opt back out here, or +// doltServerTx.Commit blanks the message it was given: a proxied +// `bd batch -m "release batch"` under dolt.auto-commit=batch would persist the +// rows, discard the message and mint nothing, while the identical command and +// config on the direct route commits it. +// +// Scope is the intersection of two in-tree lists: the transact() call sites, +// and the paths the proxied front door permits (proxyPermittedPaths in +// capability_registry.go). That is batch, mol bond/pour/squash, mol wisp create +// and the WISP half of mol burn; cook and migrate issues also call transact but +// are refused in proxied mode, so they have no dual to exempt. +// +// Membership is per commit site, not per command: mol burn's persistent half +// reaches deleteBatch -> issueOpsContext on the direct route, which is the +// policy-honoring class, so runMolBurnProxiedServer exempts only the wisp +// transaction. TestProxiedDualsExemptTheirExplicitCommitPoints pins both arms. +func explicitCommitPointContext(ctx context.Context) context.Context { + return issueops.WithImmediateVersionCommit(ctx) +} + type doltAutoCommitParams struct { // Command is the top-level bd command name (e.g., "create", "update"). Command string diff --git a/cmd/bd/dolt_autocommit_exemption_wiring_test.go b/cmd/bd/dolt_autocommit_exemption_wiring_test.go new file mode 100644 index 000000000..0dde74c32 --- /dev/null +++ b/cmd/bd/dolt_autocommit_exemption_wiring_test.go @@ -0,0 +1,238 @@ +package main + +import ( + "context" + "errors" + "testing" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/storage/uow" + "github.com/steveyegge/beads/internal/types" +) + +// The tests below pin the WIRING half of the GH#4995 exemption. The mechanism +// half — that WithImmediateVersionCommit defeats the route-wide deferral — +// is pinned in internal/storage/uow; what is pinned here is that each proxied +// dual whose direct twin is an explicit commit point actually hands that +// exemption to its transaction, and that the duals whose twin is NOT one do +// not. Without this, deleting explicitCommitPointContext from a dual is green +// in every lane that does not have a real proxied server. +// +// The seam is uow.RunTxResult's own first act: it asks the provider for a unit +// of work, passing the caller's context. A provider that records that context +// and then refuses observes exactly "the context this dual hands its +// transaction", with no server, no store and no fixture beyond the reads the +// dual makes to decide what to write. + +var errStopBeforeProbeTx = errors.New("exemption wiring probe: stop before the transaction") + +// exemptionProbeProvider serves readUOW to the first `reads` NewUOW calls (the +// dual's own read phase) and refuses the rest, recording every context it was +// handed. The refusal is not a serialization error, so RunTxResult treats it as +// permanent and makes exactly one NewUOW call per transaction. +type exemptionProbeProvider struct { + readUOW uow.UnitOfWork + reads int + ctxs []context.Context +} + +func (p *exemptionProbeProvider) NewUOW(ctx context.Context) (uow.UnitOfWork, error) { + p.ctxs = append(p.ctxs, ctx) + if len(p.ctxs) <= p.reads { + return p.readUOW, nil + } + return nil, errStopBeforeProbeTx +} + +func (p *exemptionProbeProvider) Close(context.Context) error { return nil } + +// txCtxs is the contexts handed to RunTx*, in call order: everything after the +// leading read opens. +func (p *exemptionProbeProvider) txCtxs() []context.Context { + if len(p.ctxs) <= p.reads { + return nil + } + return p.ctxs[p.reads:] +} + +// installExemptionProbe points the package's proxied provider at the probe for +// the duration of one test. +func installExemptionProbe(t *testing.T, p *exemptionProbeProvider) { + t.Helper() + prev := uowProvider + uowProvider = p + t.Cleanup(func() { uowProvider = prev }) +} + +// exemptionProbeUOW embeds the interface without implementing it: anything a +// dual's read phase needs beyond the three use cases below panics and names +// itself, rather than being silently satisfied. +type exemptionProbeUOW struct { + uow.UnitOfWork + issues domain.IssueUseCase + labels domain.LabelUseCase + deps domain.DependencyUseCase +} + +func (u exemptionProbeUOW) Close(context.Context) {} +func (u exemptionProbeUOW) IssueUseCase() domain.IssueUseCase { return u.issues } +func (u exemptionProbeUOW) LabelUseCase() domain.LabelUseCase { return u.labels } +func (u exemptionProbeUOW) DependencyUseCase() domain.DependencyUseCase { return u.deps } + +// CommentUseCase is collected by workapi.NewUOWDetailSource when the read +// phase resolves an id; nothing on that path calls it. +func (u exemptionProbeUOW) CommentUseCase() domain.CommentUseCase { return nil } + +type exemptionProbeIssues struct { + domain.IssueUseCase + byID map[string]*types.Issue +} + +func (s exemptionProbeIssues) GetIssue(_ context.Context, id string) (*types.Issue, error) { + if issue, ok := s.byID[id]; ok && !issue.Ephemeral { + return issue, nil + } + return nil, storage.ErrNotFound +} + +func (s exemptionProbeIssues) GetWisp(_ context.Context, id string) (*types.Issue, error) { + if issue, ok := s.byID[id]; ok && issue.Ephemeral { + return issue, nil + } + return nil, storage.ErrNotFound +} + +func (s exemptionProbeIssues) SearchIssues(_ context.Context, _ string, filter types.IssueFilter) (domain.SearchPage, error) { + var page domain.SearchPage + for _, id := range filter.IDs { + if issue, ok := s.byID[id]; ok { + page.Items = append(page.Items, issue) + } + } + return page, nil +} + +type exemptionProbeLabels struct { + domain.LabelUseCase +} + +func (exemptionProbeLabels) GetLabels(context.Context, string) ([]string, error) { return nil, nil } +func (exemptionProbeLabels) GetWispLabels(context.Context, string) ([]string, error) { return nil, nil } + +type exemptionProbeDeps struct { + domain.DependencyUseCase +} + +func (exemptionProbeDeps) ListWithIssueMetadata(context.Context, string, domain.DepListFilter) ([]*types.IssueWithDependencyMetadata, error) { + return nil, nil +} + +func (exemptionProbeDeps) GetForIssueIDs(context.Context, []string) (map[string][]*types.Dependency, error) { + return nil, nil +} + +func newExemptionProbeUOW(issues ...*types.Issue) uow.UnitOfWork { + byID := make(map[string]*types.Issue, len(issues)) + for _, issue := range issues { + byID[issue.ID] = issue + } + return exemptionProbeUOW{ + issues: exemptionProbeIssues{byID: byID}, + labels: exemptionProbeLabels{}, + deps: exemptionProbeDeps{}, + } +} + +// deferredRouteContext is what the proxied root pre-run hands every dispatch +// under dolt.auto-commit=batch/off. +func deferredRouteContext() context.Context { + return issueops.WithDeferredVersionCommit(context.Background()) +} + +// TestProxiedDualsExemptTheirExplicitCommitPoints pins the duals that reach +// their transaction directly. Deleting explicitCommitPointContext from any of +// them reds its row here. +func TestProxiedDualsExemptTheirExplicitCommitPoints(t *testing.T) { + cases := []struct { + name string + call func(context.Context) + }{ + { + name: "batch", + call: func(ctx context.Context) { + _, _ = runBatchProxiedServer(ctx, nil, "bd: batch probe") + }, + }, + { + name: "mol bond", + call: func(ctx context.Context) { + _ = runMolBondProxiedServer(ctx, molBondInput{argA: "bd-a", argB: "bd-b"}) + }, + }, + { + name: "mol squash", + call: func(ctx context.Context) { + _ = runMolSquashProxiedServer(ctx, molSquashInput{moleculeArg: "bd-mol"}) + }, + }, + } + + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + probe := &exemptionProbeProvider{} + installExemptionProbe(t, probe) + + tc.call(deferredRouteContext()) + + txCtxs := probe.txCtxs() + if len(txCtxs) != 1 { + t.Fatalf("%s: got %d transaction context(s), want 1 — the dual never reached RunTx*", tc.name, len(txCtxs)) + } + if issueops.VersionCommitDeferred(txCtxs[0]) { + t.Errorf("%s: the context handed to RunTx* still defers the version commit; "+ + "its direct twin is an explicit commit point, so this dual must clear the deferral "+ + "(doltServerTx.Commit blanks the message and mints nothing)", tc.name) + } + }) + } +} + +// TestProxiedMolBurnExemptsOnlyTheWispHalf pins the split. bd mol burn is two +// commit classes on the direct route: wisps go through burnWisps -> `transact` +// (explicit), persistent molecules go through deleteBatch -> issueOpsContext, +// whose deferral dolt's deleter honors by returning before DOLT_COMMIT. A +// command-wide exemption mints one commit per persistent molecule under +// batch/off, where the identical direct-route command defers them all. +func TestProxiedMolBurnExemptsOnlyTheWispHalf(t *testing.T) { + wisp := &types.Issue{ID: "bd-wisp1", Ephemeral: true} + persistent := &types.Issue{ID: "bd-mol1"} + + probe := &exemptionProbeProvider{ + reads: 1, // runMolBurnProxiedServer's classification read + readUOW: newExemptionProbeUOW(wisp, persistent), + } + installExemptionProbe(t, probe) + + // force=true skips the interactive confirmation; dryRun=false so both + // halves reach their transaction. + _ = runMolBurnProxiedServer(deferredRouteContext(), []string{wisp.ID, persistent.ID}, false, true) + + txCtxs := probe.txCtxs() + if len(txCtxs) != 2 { + t.Fatalf("got %d transaction context(s), want 2 (the wisp half then the persistent half); "+ + "the classification read did not produce one of each", len(txCtxs)) + } + + if issueops.VersionCommitDeferred(txCtxs[0]) { + t.Errorf("wisp half: the context handed to RunTxResult still defers the version commit, " + + "but its direct twin (burnWisps -> transact) commits whatever dolt.auto-commit says") + } + if !issueops.VersionCommitDeferred(txCtxs[1]) { + t.Errorf("persistent half: the context handed to RunTxResult no longer defers the version commit, " + + "so a proxied `bd mol burn` mints one Dolt commit per persistent molecule under " + + "dolt.auto-commit=batch/off — its direct twin (deleteBatch -> issueOpsContext) defers them " + + "all to `bd dolt commit`") + } +} diff --git a/cmd/bd/dolt_autocommit_server_test.go b/cmd/bd/dolt_autocommit_server_test.go index 64ea2d6ff..9476f5e46 100644 --- a/cmd/bd/dolt_autocommit_server_test.go +++ b/cmd/bd/dolt_autocommit_server_test.go @@ -284,3 +284,37 @@ func TestTransactHonoringAutoCommitBlanksMessageInServerBatch(t *testing.T) { }) } } + +// TestIssueOpsContextDefersByModePolicyTable pins the policy half of GH#4995: +// issueOpsContext defers the Dolt version commit for batch/off and does not for +// on/unset. It deliberately does not claim to cover the proxied-server wiring — +// issueOpsContext branches only on writesCommitNow(), never on the storage +// mode, so setting proxiedServerMode here would prove nothing. The proxied +// wiring is pinned by TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce +// and the deferral itself by the doltserver_tx tests. +func TestIssueOpsContextDefersByModePolicyTable(t *testing.T) { + for _, tc := range []struct { + mode string + wantDeferred bool + }{ + {mode: string(doltAutoCommitOn), wantDeferred: false}, + {mode: string(doltAutoCommitBatch), wantDeferred: true}, + {mode: string(doltAutoCommitOff), wantDeferred: true}, + {mode: "", wantDeferred: false}, + } { + t.Run("mode="+tc.mode, func(t *testing.T) { + saveStorageMode(t) + serverMode = false + proxiedServerMode = false + doltAutoCommit = tc.mode + + ctx, err := issueOpsContext(context.Background()) + if err != nil { + t.Fatalf("issueOpsContext: %v", err) + } + if got := storageissueops.VersionCommitDeferred(ctx); got != tc.wantDeferred { + t.Fatalf("VersionCommitDeferred = %v, want %v for mode %q", got, tc.wantDeferred, tc.mode) + } + }) + } +} diff --git a/cmd/bd/dolt_commit_proxied_integration_test.go b/cmd/bd/dolt_commit_proxied_integration_test.go new file mode 100644 index 000000000..8f7f8c894 --- /dev/null +++ b/cmd/bd/dolt_commit_proxied_integration_test.go @@ -0,0 +1,191 @@ +//go:build cgo + +package main + +import ( + "fmt" + "os" + "path/filepath" + "strconv" + "strings" + "testing" +) + +// sqlCountValue extracts a COUNT(*) cell from bdProxiedSQLJSON. sqlValueEquals +// only answers "is it this number"; the batch-flush assertions need the number +// itself so a failure can report how far HEAD actually moved. +func sqlCountValue(v any) (float64, bool) { + switch x := v.(type) { + case float64: + return x, true + case string: + f, err := strconv.ParseFloat(x, 64) + return f, err == nil + default: + return 0, false + } +} + +// TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce pins every half of +// GH#4995 on the proxied route: dolt.auto-commit=batch must leave policy writes +// in the working set (no Dolt commit per write), `bd dolt commit` must mint +// exactly one commit for the whole batch and attribute it to the actor, and the +// explicit commit points must keep minting their own commit with the caller's +// message despite the route-wide deferral. +// +// Before the proxied flush point existed, step 3 below failed with +// "no store available": proxied mode returns from the root pre-run before +// newDoltStore runs, so getStore() is nil and `bd dolt commit` had nothing to +// commit through. batch/off therefore meant "never commit" on this route. +func TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce(t *testing.T) { + requireProxiedServerEnv(t) + + bd := buildEmbeddedBD(t) + p := bdProxiedInit(t, bd, "batchflush") + + doltLogCount := func(t *testing.T) float64 { + t.Helper() + rows := bdProxiedSQLJSON(t, bd, p.dir, "SELECT COUNT(*) AS count FROM dolt_log") + if len(rows) != 1 { + t.Fatalf("expected one row from dolt_log count, got %d: %v", len(rows), rows) + } + n, ok := sqlCountValue(rows[0]["count"]) + if !ok { + t.Fatalf("dolt_log count is not numeric: %#v", rows[0]["count"]) + } + return n + } + + headCommit := func(t *testing.T) map[string]interface{} { + t.Helper() + rows := bdProxiedSQLJSON(t, bd, p.dir, + "SELECT message, committer, email FROM dolt_log ORDER BY date DESC LIMIT 1") + if len(rows) != 1 { + t.Fatalf("expected one dolt_log row for HEAD, got %d: %v", len(rows), rows) + } + return rows[0] + } + + head0 := doltLogCount(t) + + // 1. Three writes under the batch policy must not mint any Dolt commit. + for i := 1; i <= 3; i++ { + title := fmt.Sprintf("batch write %d", i) + stdout, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, + "--dolt-auto-commit", "batch", "create", title, "-p", "1") + if err != nil { + t.Fatalf("bd create %q failed: %v\nstdout:\n%s\nstderr:\n%s", title, err, stdout, stderr) + } + } + + if got := doltLogCount(t); got != head0 { + t.Fatalf("batch writes minted Dolt commits: dolt_log went %v -> %v (want unchanged)", head0, got) + } + + // 2. The writes are in the working set, not rolled back. + listOut, listErr, err := bdProxiedRunBuffers(t, bd, p.dir, "list", "--json") + if err != nil { + t.Fatalf("bd list --json failed: %v\nstdout:\n%s\nstderr:\n%s", err, listOut, listErr) + } + for i := 1; i <= 3; i++ { + title := fmt.Sprintf("batch write %d", i) + if !strings.Contains(listOut, title) { + t.Fatalf("deferred write %q is not readable after batch create:\n%s", title, listOut) + } + } + + // 3. The explicit flush point mints exactly one commit for the batch. + stdout, stderr, err := bdProxiedRunBuffers(t, bd, p.dir, + "--actor", "flush-actor", "dolt", "commit", "-m", "batch flush") + if err != nil { + t.Fatalf("bd dolt commit failed in proxied mode: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + if !strings.Contains(stdout, "Committed.") { + t.Fatalf("bd dolt commit did not report a commit:\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + if got, want := doltLogCount(t), head0+1; got != want { + t.Fatalf("bd dolt commit did not advance HEAD exactly once: dolt_log %v -> %v (want %v)", head0, got, want) + } + + // 3b. The flush is attributed to the bd actor, not the SQL session user. + // Under batch mode this is the only commit a whole batch of writes leaves + // behind, so `dolt log` has to be able to say whose batch it was — the + // '--author' DoltStorage.CommitAll passes on the direct route. + head := headCommit(t) + if got := head["message"]; got != "batch flush" { + t.Fatalf("flush commit message = %v, want %q (row: %v)", got, "batch flush", head) + } + if got := head["committer"]; got != "flush-actor" { + t.Fatalf("flush commit is not attributed to the actor: committer = %v, want %q (row: %v)", got, "flush-actor", head) + } + if got := head["email"]; got != "test@test.com" { + t.Fatalf("flush commit email = %v, want the git identity %q (row: %v)", got, "test@test.com", head) + } + + // 4. A second flush with nothing pending is a no-op, not a second commit. + stdout, stderr, err = bdProxiedRunBuffers(t, bd, p.dir, "dolt", "commit", "-m", "second flush") + if err != nil { + t.Fatalf("second bd dolt commit failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + if !strings.Contains(stdout, "Nothing to commit.") { + t.Fatalf("second bd dolt commit did not report an empty working set:\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + if got, want := doltLogCount(t), head0+1; got != want { + t.Fatalf("second bd dolt commit advanced HEAD: dolt_log = %v (want %v)", got, want) + } + + // 5. The deferral is policy-driven, not unconditional: with the default + // "on" policy a single create advances dolt_log by one on its own. + before := doltLogCount(t) + stdout, stderr, err = bdProxiedRunBuffers(t, bd, p.dir, + "--dolt-auto-commit", "on", "create", "immediate write", "-p", "1") + if err != nil { + t.Fatalf("bd create with --dolt-auto-commit on failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + if got, want := doltLogCount(t), before+1; got != want { + t.Fatalf("auto-commit=on did not commit per write: dolt_log %v -> %v (want %v)", before, got, want) + } + + // 6. The deferral is per commit CLASS, not per route: `bd batch -m` is an + // explicit commit point (its direct-route twin calls transact, which mints + // regardless of policy), so under batch it must still commit, carrying the + // caller's message. A plain create in the same policy is the control. + before = doltLogCount(t) + stdout, stderr, err = bdProxiedRunBuffers(t, bd, p.dir, + "--dolt-auto-commit", "batch", "create", "deferred control write", "-p", "1") + if err != nil { + t.Fatalf("control create under batch failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + if got := doltLogCount(t); got != before { + t.Fatalf("control create minted a commit under batch: dolt_log %v -> %v (want unchanged)", before, got) + } + + script := filepath.Join(p.dir, "explicit-batch.txt") + if err := os.WriteFile(script, []byte("create task 1 explicit batch write\n"), 0o600); err != nil { + t.Fatalf("write batch script: %v", err) + } + stdout, stderr, err = bdProxiedRunBuffers(t, bd, p.dir, + "--dolt-auto-commit", "batch", "batch", "-f", script, "-m", "explicit batch commit") + if err != nil { + t.Fatalf("bd batch under batch policy failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + if got, want := doltLogCount(t), before+1; got != want { + t.Fatalf("explicit batch commit did not mint exactly one commit under batch: dolt_log %v -> %v (want %v)", before, got, want) + } + // The message is the half a route-wide deferral silently drops: blanking it + // selects the plain-COMMIT form, which persists the rows and records nothing. + if got := headCommit(t)["message"]; got != "explicit batch commit" { + t.Fatalf("explicit batch commit lost the caller's message: %v (want %q)", got, "explicit batch commit") + } + + // 7. That commit swept the deferred control write with it (DOLT_COMMIT + // '-Am'), so the working set is clean and a flush now has nothing to do — + // the exemption does not strand the writes the policy deferred. + stdout, stderr, err = bdProxiedRunBuffers(t, bd, p.dir, "dolt", "commit", "-m", "post-batch flush") + if err != nil { + t.Fatalf("flush after explicit batch commit failed: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + if !strings.Contains(stdout, "Nothing to commit.") { + t.Fatalf("flush after explicit batch commit found pending work:\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } +} diff --git a/cmd/bd/dolt_commit_proxied_server.go b/cmd/bd/dolt_commit_proxied_server.go new file mode 100644 index 000000000..55cd82f13 --- /dev/null +++ b/cmd/bd/dolt_commit_proxied_server.go @@ -0,0 +1,88 @@ +package main + +import ( + "context" + "database/sql" + "fmt" + "strings" + + "github.com/steveyegge/beads/internal/storage/issueops" +) + +// runDoltCommitProxiedServer is the proxied dual of `bd dolt commit`: the +// explicit commit point that discharges dolt.auto-commit=batch/off on this +// route (GH#4995). Proxied mode returns from the root pre-run before +// newDoltStore ever runs, so getStore() is nil here and the UOW provider is the +// only handle on the server — without this dual, batch/off on a proxied +// checkout meant "never commit" and writes piled up in the working set forever. +// +// It deliberately does not open a unit of work: doltServerTx.Commit blanks its +// message under a deferred context (that is the deferral this command exists to +// discharge), so routing the flush through it would make the flush itself a +// no-op. It runs DOLT_COMMIT('-Am') on a pinned maintenance connection instead, +// which is the uow-side equivalent of DoltStorage.CommitAll: "everything in the +// working set", including out-of-band and config-table dirt, attributed to the +// same actor with '--author'. +func runDoltCommitProxiedServer(ctx context.Context, message string) (bool, error) { + committed := false + // runProxiedNonTx, not an open-coded provider assertion: it is the shared + // maintenance escape hatch (backup, compact, clean-databases) and it reports + // provider failures through HandleErrorRespectJSON, so `bd --json dolt + // commit` keeps emitting the JSON error envelope the rest of the proxied + // surface emits. + err := runProxiedNonTx(ctx, func(ctx context.Context, conn *sql.Conn) error { + // The same gate doltServerTx.Commit uses, and for the same reason: an + // empty DOLT_COMMIT is rejected server-side with "nothing to commit" + // and floods the server log. + pending, err := issueops.HasPendingChanges(ctx, conn) + if err != nil { + return err + } + if !pending { + return nil + } + if _, err := conn.ExecContext(ctx, "CALL DOLT_COMMIT('-Am', ?, '--author', ?);", message, proxiedCommitAuthor()); err != nil { + if isDoltNothingToCommit(err) { + return nil + } + return err + } + committed = true + return nil + }) + return committed, err +} + +// proxiedCommitAuthor is the '--author' identity for the proxied flush, the +// counterpart of DoltStorage.commitAuthorString on the direct route. Without it +// the commit is attributed to the SQL session user, and under batch mode this is +// the ONE commit that survives a whole batch of writes — so on a multi-agent +// deployment `dolt log` could not say whose batch was flushed. +// +// The actor half is the same getActor() the default commit message names; the +// address half comes from git, as it does for the committer identity the store +// resolves for its own commits. +func proxiedCommitAuthor() string { + name, email := proxiedServerCommitter() + if a := strings.TrimSpace(getActor()); a != "" { + name = a + } + return renderDoltCommitAuthor(name, email) +} + +// renderDoltCommitAuthor formats a Dolt '--author' argument. Dolt parses the +// git "Name " grammar and rejects anything else, so the angle brackets +// and newlines an actor name could carry are stripped rather than passed +// through to fail the flush. +func renderDoltCommitAuthor(name, email string) string { + clean := strings.NewReplacer("<", "", ">", "", "\n", " ", "\r", " ") + name = strings.TrimSpace(clean.Replace(name)) + email = strings.TrimSpace(clean.Replace(email)) + if name == "" { + name = "beads" + } + if email == "" { + email = "beads@localhost" + } + return fmt.Sprintf("%s <%s>", name, email) +} diff --git a/cmd/bd/dolt_commit_proxied_server_test.go b/cmd/bd/dolt_commit_proxied_server_test.go new file mode 100644 index 000000000..032bcb17b --- /dev/null +++ b/cmd/bd/dolt_commit_proxied_server_test.go @@ -0,0 +1,29 @@ +package main + +import "testing" + +// TestRenderDoltCommitAuthor pins the '--author' argument the proxied flush +// passes to DOLT_COMMIT (GH#4995). Dolt parses the git "Name " grammar +// and rejects anything else, and the name half is an actor string the caller +// controls, so a stray bracket or newline must be stripped rather than turned +// into a failed flush — the flush is the only commit batch mode keeps. +func TestRenderDoltCommitAuthor(t *testing.T) { + for _, tc := range []struct { + name string + actor string + email string + want string + }{ + {name: "actor and email", actor: "agent-7", email: "agent7@example.com", want: "agent-7 "}, + {name: "trims surrounding space", actor: " agent-7 ", email: " agent7@example.com\n", want: "agent-7 "}, + {name: "strips brackets the grammar owns", actor: "agent <7>", email: "a@b", want: "agent 7 "}, + {name: "strips embedded newline", actor: "agent\n7", email: "a@b", want: "agent 7 "}, + {name: "defaults an empty identity", actor: "", email: "", want: "beads "}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := renderDoltCommitAuthor(tc.actor, tc.email); got != tc.want { + t.Fatalf("renderDoltCommitAuthor(%q, %q) = %q, want %q", tc.actor, tc.email, got, tc.want) + } + }) + } +} diff --git a/cmd/bd/errors.go b/cmd/bd/errors.go index bf753777b..4c1422b2f 100644 --- a/cmd/bd/errors.go +++ b/cmd/bd/errors.go @@ -30,6 +30,15 @@ func exitCodeFromError(err error) (int, bool) { return 0, false } +// isReportedExit reports whether err is an already-rendered failure — the +// *exitError every Handle*Error returns after writing the message (or the JSON +// error envelope) itself. A caller that re-wraps one prints a second Error line +// and, in --json mode, a plain line after the envelope. +func isReportedExit(err error) bool { + _, reported := exitCodeFromError(err) + return reported +} + func activeWorkspaceNotFoundError() string { return "no active beads workspace found" } diff --git a/cmd/bd/history_remote_frontdoor_matrix_test.go b/cmd/bd/history_remote_frontdoor_matrix_test.go index 41223bd21..cb7b3fdc5 100644 --- a/cmd/bd/history_remote_frontdoor_matrix_test.go +++ b/cmd/bd/history_remote_frontdoor_matrix_test.go @@ -61,7 +61,6 @@ func TestHistoryRemoteRefusalFrontDoorMatrix(t *testing.T) { {"flatten", "proxy.flatten.unsupported", "flatten is not supported in proxied-server mode", []string{"flatten"}}, {"dolt push", "proxy.dolt_push.unsupported", "dolt push is not supported in proxied-server mode", []string{"dolt", "push"}}, {"dolt pull", "proxy.dolt_pull.unsupported", "dolt pull is not supported in proxied-server mode", []string{"dolt", "pull"}}, - {"dolt commit", "proxy.dolt_commit.unsupported", "dolt commit is not supported in proxied-server mode", []string{"dolt", "commit"}}, {"dolt remote add", "proxy.dolt_remote.unsupported", "dolt remote add is not supported in proxied-server mode", []string{"dolt", "remote", "add", "backup", "https://example.invalid/backup"}}, {"dolt remote list", "proxy.dolt_remote.unsupported", "dolt remote list is not supported in proxied-server mode", []string{"dolt", "remote", "list"}}, {"dolt remote reset-data", "proxy.dolt_remote.unsupported", "dolt remote reset-data is not supported in proxied-server mode", []string{"dolt", "remote", "reset-data", "backup"}}, @@ -126,8 +125,10 @@ func snapshotHistoryArtifacts(t *testing.T, p proxiedProject) []byte { return out } -// TestHistoryRemoteSupportedFrontDoorParity checks the two operations that -// are intentionally shared by direct and proxied providers. +// TestHistoryRemoteSupportedFrontDoorParity checks the operations that are +// intentionally shared by direct and proxied providers: history --events, +// dolt remote remove, and dolt commit (the flush point dolt.auto-commit=batch/off +// defers to, GH#4995). func TestHistoryRemoteSupportedFrontDoorParity(t *testing.T) { if os.Getenv("BEADS_TEST_PROXIED_SERVER") != "1" && os.Getenv(managedLocalProxiedEnvVar) != "1" { t.Skip("set a proxied test lane to run history parity") @@ -162,6 +163,13 @@ func TestHistoryRemoteSupportedFrontDoorParity(t *testing.T) { if err != nil || !strings.Contains(string(out), "backup") { t.Fatalf("remote remove failed: %v\n%s", err, out) } + // dolt commit must pass the front door rather than be refused. Exact + // commit semantics are pinned by + // TestProxiedServerBatchDefersThenDoltCommitAdvancesHeadOnce. + out, err = bdProxiedRun(t, bd, p.dir, "--json", "dolt", "commit") + if err != nil || strings.Contains(string(out), "proxy.dolt_commit.unsupported") { + t.Fatalf("dolt commit refused or failed: %v\n%s", err, out) + } }) } } diff --git a/cmd/bd/main.go b/cmd/bd/main.go index e93b6dd54..345639a35 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -879,7 +879,7 @@ func init() { rootCmd.PersistentFlags().BoolVar(&sandboxMode, "sandbox", false, "Sandbox mode: disables Dolt auto-push") rootCmd.PersistentFlags().BoolVar(&readonlyMode, "readonly", false, "Read-only mode: block write operations (for worker sandboxes)") rootCmd.PersistentFlags().BoolVar(&globalFlag, "global", false, "Use the global shared-server database (beads_global)") - rootCmd.PersistentFlags().StringVar(&doltAutoCommit, "dolt-auto-commit", "", "Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP). Applies to embedded and direct SQL-server modes; proxied-server routes are unaffected. Default: on. Override via config key dolt.auto-commit") + rootCmd.PersistentFlags().StringVar(&doltAutoCommit, "dolt-auto-commit", "", "Dolt auto-commit policy (off|on|batch). 'on': commit after each write. 'batch': defer commits to bd dolt commit; uncommitted changes persist in the working set until then (a live batch-mode bd process also flushes on SIGTERM/SIGHUP, except in proxied-server mode, where bd dolt commit is the only flush point). In proxied-server mode the deferral covers the writes the CLI makes on that route, including config and version metadata; explicit commit points (bd batch, bd mol bond/pour/squash, bd mol wisp create, and the wisp half of bd mol burn) still commit, and a bd serve process on the same database is unaffected. Default: on. Override via config key dolt.auto-commit") rootCmd.PersistentFlags().BoolVar(&cpuProfileEnabled, "cpu-profile", false, "Generate CPU profile for performance analysis") rootCmd.PersistentFlags().StringVar(&memProfilePath, "mem-profile", "", "Write heap profile to FILE on exit (also respects BEADS_MEM_PROFILE)") rootCmd.PersistentFlags().BoolVarP(&verboseFlag, "verbose", "v", false, "Enable verbose/debug output") @@ -1807,6 +1807,16 @@ var rootCmd = &cobra.Command{ } uowProvider = uow.NewNotifyingProvider(p, uowSinks) + // Honor dolt.auto-commit for proxied writes the same way + // issueOpsContext already does for the direct/SQL-server routes + // (bd-4wamg): batch/off defer the Dolt version commit rather than + // minting one per write (GH#4995). uow.issueOperations reads this + // off the context for every Create/Update/Close/Reopen it runs. + rootCtx, err = issueOpsContext(rootCtx) + if err != nil { + return HandleError("failed to resolve dolt auto-commit policy: %v", err) + } + if !previewMode { reconcileVersionProxiedServer(rootCtx) } diff --git a/cmd/bd/mol_proxied_server.go b/cmd/bd/mol_proxied_server.go index bb28a2d74..efb2e1cd0 100644 --- a/cmd/bd/mol_proxied_server.go +++ b/cmd/bd/mol_proxied_server.go @@ -27,6 +27,9 @@ func runPourProxiedServer(ctx context.Context, in pourInput) error { if uowProvider == nil { return HandleError("proxied-server UOW provider not initialized") } + // Explicit commit point, like the `transact` in cloneSubgraph the direct + // route reaches through spawnMolecule (GH#4995). + ctx = explicitCommitPointContext(ctx) vars, err := parseVarFlags(in.varFlags) if err != nil { @@ -372,6 +375,9 @@ func runMolBondProxiedServer(ctx context.Context, in molBondInput) error { if uowProvider == nil { return HandleErrorRespectJSON("proxied-server UOW provider not initialized") } + // Explicit commit point, like `transact` in bondProtoMol/bondMolecules on + // the direct route (GH#4995). + ctx = explicitCommitPointContext(ctx) if in.dryRun { uw, err := proxiedOpenReadUOW(ctx) @@ -446,6 +452,9 @@ func runMolSquashProxiedServer(ctx context.Context, in molSquashInput) error { if uowProvider == nil { return HandleErrorRespectJSON("proxied-server UOW provider not initialized") } + // Explicit commit point, like `transact` in runMolSquash on the direct + // route (GH#4995). + ctx = explicitCommitPointContext(ctx) if in.dryRun { uw, err := proxiedOpenReadUOW(ctx) @@ -516,6 +525,15 @@ func runMolBurnProxiedServer(ctx context.Context, args []string, dryRun, force b if uowProvider == nil { return HandleErrorRespectJSON("proxied-server UOW provider not initialized") } + // Only the wisp half is an explicit commit point, so the exemption is + // applied per commit site below rather than to this ctx (GH#4995). + // runMolBurn splits on the direct route: wisps go through burnWisps -> + // `transact`, which mints the commit whatever dolt.auto-commit says, while + // persistent molecules go through deleteBatch -> issueOpsContext, the + // policy-honoring class — dolt's deleter returns before DOLT_COMMIT when + // that context defers. Exempting the whole command would mint one commit + // per persistent molecule under batch/off, on a destructive verb, where the + // identical direct-route command defers them all to `bd dolt commit`. uw, err := proxiedOpenReadUOW(ctx) if err != nil { @@ -617,7 +635,8 @@ func runMolBurnProxiedServer(ctx context.Context, args []string, dryRun, force b } if len(wispIDs) > 0 { - result, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (*BurnResult, string, error) { + // The wisp half's direct twin is burnWisps -> `transact`. + result, err := uow.RunTxResult(explicitCommitPointContext(ctx), uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (*BurnResult, string, error) { r, err := burnWispsInto(ctx, newUOWMolWriter(uw), wispIDs, actor) if err != nil { return nil, "", err @@ -635,6 +654,10 @@ func runMolBurnProxiedServer(ctx context.Context, args []string, dryRun, force b } for _, id := range persistentIDs { + // No exemption: the persistent half's direct twin is deleteBatch -> + // issueOpsContext, so this stays on the inherited policy context and + // the message is blanked under batch/off, exactly as the direct route + // leaves these deletions in the working set for `bd dolt commit`. result, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (BurnResult, string, error) { subgraph, err := loadTemplateSubgraph(ctx, newUOWMolWriter(uw), id) if err != nil { diff --git a/cmd/bd/proxied_maintenance.go b/cmd/bd/proxied_maintenance.go index d41bc8ba0..867f703ea 100644 --- a/cmd/bd/proxied_maintenance.go +++ b/cmd/bd/proxied_maintenance.go @@ -13,7 +13,10 @@ func runProxiedNonTx(ctx context.Context, fn func(ctx context.Context, conn *sql } mp, ok := uowProvider.(uow.MaintenanceProvider) if !ok { - return HandleErrorRespectJSON("proxied-server provider does not support maintenance operations") + // The %T is the whole diagnostic: every wrapper in the proxied chain + // forwards RunNonTx with a compile-time assertion, so reaching this + // means a new decorator dropped it, and its type names the culprit. + return HandleErrorRespectJSON("proxied-server provider %T does not support maintenance operations", uowProvider) } return mp.RunNonTx(ctx, fn) } diff --git a/cmd/bd/vc.go b/cmd/bd/vc.go index a10a81cfe..360b2c2cf 100644 --- a/cmd/bd/vc.go +++ b/cmd/bd/vc.go @@ -179,7 +179,10 @@ Examples: SilenceErrors: true, RunE: func(cmd *cobra.Command, args []string) error { if usesProxiedServer() { - return HandleErrorRespectJSON("vc commit is not supported in proxied-server mode") + // There IS a commit point on this route now — `bd dolt commit` + // is the proxied flush point dolt.auto-commit=batch/off defers + // to (GH#4995) — so name it rather than leaving a dead end. + return HandleErrorRespectJSON("vc commit is not supported in proxied-server mode; use 'bd dolt commit' instead") } evt := metrics.NewCommandEvent("vc-commit") defer func() { diff --git a/cmd/bd/wisp_proxied_server.go b/cmd/bd/wisp_proxied_server.go index 25562faad..b05792e69 100644 --- a/cmd/bd/wisp_proxied_server.go +++ b/cmd/bd/wisp_proxied_server.go @@ -20,6 +20,12 @@ func runWispCreateProxiedServer(ctx context.Context, in wispCreateInput) error { if uowProvider == nil { return HandleError("proxied-server UOW provider not initialized") } + // Explicit commit point, like the `transact` in cloneSubgraph the direct + // route reaches through spawnMoleculeWithOptions (GH#4995). The wisp tables + // are dolt_ignored, so the commit this preserves is whatever tracked rows + // the spawn touches — the class membership is what matters here, not the + // size of the commit. + ctx = explicitCommitPointContext(ctx) vars, err := parseVarFlags(in.varFlags) if err != nil { diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index bd3995a26..8d2dfed68 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -106,7 +106,7 @@ Any key whose name contains `api_key`, `api-key`, `secret`, `token`, or `passwor | `agent.profile` | — | `BD_AGENT_PROFILE` | `conservative` | Policy profile `bd prime` uses for git/commit authority: `conservative`, `minimal`, `team-maintainer`; invalid values fall back to `conservative` | | `prime.max-memories` | `--max-memories` | `BD_PRIME_MAX_MEMORIES` | `0` | Max persistent memories injected by `bd prime` (0 = unlimited) | | `prime.max-memory-chars` | `--max-memory-chars` | `BD_PRIME_MAX_MEMORY_CHARS` | `0` | Max total bytes of memory entries injected by `bd prime`, at whole-memory boundaries (0 = unlimited) | -| `dolt.auto-commit` | `--dolt-auto-commit` | `BD_DOLT_AUTO_COMMIT` | `on` | Create a Dolt history commit after each successful write (see [below](#auto-commit-sql-commits-vs-dolt-commits)) | +| `dolt.auto-commit` | `--dolt-auto-commit` | `BD_DOLT_AUTO_COMMIT` | `on` | `off\|on\|batch`: `on` creates a Dolt history commit after each successful write; `batch`/`off` defer it to `bd dolt commit` (see [below](#auto-commit-sql-commits-vs-dolt-commits)) | | `dolt.auto-push` | — | `BD_DOLT_AUTO_PUSH` | `false` | Auto-push to Dolt remote after writes (opt-in; see [below](#auto-push)) | | `dolt.auto-push-interval` | — | `BD_DOLT_AUTO_PUSH_INTERVAL` | `5m` | Minimum time between auto-pushes | | `dolt.auto-push-timeout` | — | `BD_DOLT_AUTO_PUSH_TIMEOUT` | `30s` | Timeout for a single auto-push attempt | @@ -195,6 +195,26 @@ dolt: auto-commit: off ``` +`batch` and `off` defer the Dolt history commit instead of skipping the write: the change is +still durable in the working set, and `bd dolt commit` records the accumulated batch as a single +Dolt commit. This applies on every storage mode — embedded, direct SQL-server, and +proxied-server. In proxied-server mode the deferral covers the writes the CLI makes on that +route, including workspace config and version metadata, and `bd dolt commit` is the only flush +point there: the SIGTERM/SIGHUP flush of a live batch-mode process needs an open store, which +proxied mode never has (it does run in embedded and direct SQL-server modes). + +Two kinds of write stay outside the deferral, in every mode: + +- **Explicit commit points** — `bd batch` (whose commit message defaults to a synthesized one + when `-m` is absent, so every batch commits), `bd mol bond`, `bd mol pour`, `bd mol squash`, + `bd mol wisp create`, and the wisp half of `bd mol burn` — commit what they wrote, with the + message they were given: the commit is that command's contract rather than auto-commit + policy. Burning a *persistent* molecule is an ordinary delete, so it honors the policy and + defers with everything else. +- **Writes made by a `bd serve` process**, whose HTTP API requests carry their own context + instead of the CLI's, so a daemon keeps committing per write even while the CLI on the same + database defers. + ### Auto-backup Periodic Dolt-native backup to `.beads/backup/` provides a recovery path independent of the live database. Local Dolt commits (via `dolt.auto-commit`) remain the primary safety net; backup is a secondary layer. Unlike `bd export` or `.beads/issues.jsonl`, this is a full database backup: it preserves tables, branches, commit history, and working-set data. diff --git a/internal/storage/issueops/version_commit.go b/internal/storage/issueops/version_commit.go index dc17814ed..1abd8a4ff 100644 --- a/internal/storage/issueops/version_commit.go +++ b/internal/storage/issueops/version_commit.go @@ -19,6 +19,18 @@ func WithDeferredVersionCommit(ctx context.Context) context.Context { return context.WithValue(ctx, deferVersionCommitKey{}, true) } +// WithImmediateVersionCommit returns a context that creates its Dolt version +// commit even when an ancestor context deferred it. +// +// A route that applies the deferral once, for every command it dispatches — +// the proxied-server CLI sets it on the root context (GH#4995) — needs this to +// re-express the exemption the per-verb routes get by not being wrapped at all: +// an explicit commit point ("commit these writes now, with this message") is +// not a write whose versioning is policy, so it must not inherit the deferral. +func WithImmediateVersionCommit(ctx context.Context) context.Context { + return context.WithValue(ctx, deferVersionCommitKey{}, false) +} + // VersionCommitDeferred reports whether ctx defers version commits. func VersionCommitDeferred(ctx context.Context) bool { deferred, _ := ctx.Value(deferVersionCommitKey{}).(bool) diff --git a/internal/storage/uow/doltserver_tx.go b/internal/storage/uow/doltserver_tx.go index 12762d972..ee22e35d6 100644 --- a/internal/storage/uow/doltserver_tx.go +++ b/internal/storage/uow/doltserver_tx.go @@ -30,13 +30,28 @@ func (t *doltServerTx) Commit(ctx context.Context, message string) error { if t.done { return errors.New("uow: commit: already done") } + // dolt.auto-commit=batch/off (GH#4995): defer the Dolt version commit to a + // later explicit commit point (bd dolt commit), the same as the embedded + // and direct SQL-server backends already do at their own commit sites + // (bd-4wamg) — persist the write into the working set via a plain SQL + // COMMIT below, without minting Dolt history, regardless of what message + // the caller named. issueOpsContext sets this on the context before the + // proxied CLI dispatch opens its unit of work. + if message != "" && issueops.VersionCommitDeferred(ctx) { + message = "" + } // An empty message selects the EPHEMERAL commit form (bd-aq0ql): a plain // SQL COMMIT persists the transaction's writes into the working set - // without minting a Dolt commit or history. This exists for work that - // touches ONLY dolt_ignored state — today the leases table (bd-lrgn1), - // whose heartbeats must never create commits — and is only reachable via - // uow.RunTxEphemeral: RunTx/RunTxResult treat an empty commitMsg as - // "nothing to commit" and never call Commit at all. + // without minting a Dolt commit or history. It has two entry points: + // + // - uow.RunTxEphemeral, for work that touches ONLY dolt_ignored state — + // today the leases table (bd-lrgn1), whose heartbeats must never create + // commits. RunTx/RunTxResult treat an empty commitMsg as "nothing to + // commit" and never call Commit at all, so they cannot reach it that way. + // - RunTx/RunTxResult under a deferred-version-commit context, via the + // blanking just above (GH#4995). There the writes are ordinary versioned + // rows and the ONLY-dolt_ignored-state rationale does not apply: the + // working set is meant to hold them until the explicit flush point. stmt, args := "CALL DOLT_COMMIT('-Am', ?);", []interface{}{message} if message == "" { stmt, args = "COMMIT;", nil diff --git a/internal/storage/uow/doltserver_tx_test.go b/internal/storage/uow/doltserver_tx_test.go index 464a53494..036291ba8 100644 --- a/internal/storage/uow/doltserver_tx_test.go +++ b/internal/storage/uow/doltserver_tx_test.go @@ -20,6 +20,7 @@ import ( "testing" "github.com/DATA-DOG/go-sqlmock" + "github.com/steveyegge/beads/internal/storage/issueops" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -202,3 +203,85 @@ func TestDoltServerTxEphemeralCommitSkipsPendingCheck(t *testing.T) { require.NoError(t, err) require.NoError(t, mock.ExpectationsWereMet()) } + +// TestDoltServerTxCommitDefersDoltCommitUnderDeferredContext pins GH#4995: +// dolt.auto-commit=batch/off marks the context with WithDeferredVersionCommit. +// doltServerTx.Commit must blank the commit message, bypassing DOLT_COMMIT +// and the HasPendingChanges dolt_status check, and executing a plain COMMIT +// to persist writes into the working set without advancing Dolt history. +func TestDoltServerTxCommitDefersDoltCommitUnderDeferredContext(t *testing.T) { + for _, tc := range []struct { + name string + deferred bool + }{ + {name: "on (not deferred)", deferred: false}, + {name: "batch or off (deferred)", deferred: true}, + } { + t.Run(tc.name, func(t *testing.T) { + p, mock := newMockTxProvider(t) + mock.ExpectExec("START TRANSACTION").WillReturnResult(sqlmock.NewResult(0, 0)) + if tc.deferred { + // With deferred version commit (GH#4995), the non-empty message is blanked, + // skipping DOLT_COMMIT and executing plain COMMIT without status check. + mock.ExpectExec(matchPlainCommit).WillReturnResult(sqlmock.NewResult(0, 0)) + } else { + expectPendingChanges(mock, 1) + mock.ExpectExec("DOLT_COMMIT").WithArgs("bd: real write").WillReturnResult(sqlmock.NewResult(0, 1)) + } + + tx, err := p.BeginTx(context.Background()) + require.NoError(t, err) + + ctx := context.Background() + if tc.deferred { + ctx = issueops.WithDeferredVersionCommit(ctx) + } + + err = tx.Commit(ctx, "bd: real write") + require.NoError(t, err) + require.NoError(t, mock.ExpectationsWereMet()) + assert.Equal(t, 1, p.db.Stats().OpenConnections, "session must cleanly return to pool") + }) + } +} + +// TestDoltServerTxCommitHonorsImmediateVersionCommit pins the other half of the +// GH#4995 policy: the proxied route applies the deferral ONCE, to the root +// context, so the explicit commit points — the duals whose direct-route twin +// calls transact and mints a Dolt commit whatever dolt.auto-commit says — opt +// back out with issueops.WithImmediateVersionCommit. The caller's message must +// then survive all the way to DOLT_COMMIT; blanking it would persist the rows +// and record nothing, which is what `bd batch -m` must never do. +func TestDoltServerTxCommitHonorsImmediateVersionCommit(t *testing.T) { + p, mock := newMockTxProvider(t) + mock.ExpectExec("START TRANSACTION").WillReturnResult(sqlmock.NewResult(0, 0)) + expectPendingChanges(mock, 1) + mock.ExpectExec("DOLT_COMMIT").WithArgs("bd: batch 3 ops by tester").WillReturnResult(sqlmock.NewResult(0, 1)) + + tx, err := p.BeginTx(context.Background()) + require.NoError(t, err) + + // Composed in the order the proxied CLI composes them: the route-wide + // deferral from the pre-run, then the exemption from the dual. + ctx := issueops.WithImmediateVersionCommit(issueops.WithDeferredVersionCommit(context.Background())) + require.NoError(t, tx.Commit(ctx, "bd: batch 3 ops by tester")) + require.NoError(t, mock.ExpectationsWereMet()) + assert.Equal(t, 1, p.db.Stats().OpenConnections, "session must cleanly return to pool") +} + +// TestDoltServerTxRunTxWithDeferredContextSkipsDoltCommit verifies that a complete +// unit of work executed via RunTx under deferred version commit commits via plain +// COMMIT and succeeds without advancing Dolt history (GH#4995). +func TestDoltServerTxRunTxWithDeferredContextSkipsDoltCommit(t *testing.T) { + p, mock := newMockTxProvider(t) + mock.ExpectExec("START TRANSACTION").WillReturnResult(sqlmock.NewResult(0, 0)) + mock.ExpectExec(matchPlainCommit).WillReturnResult(sqlmock.NewResult(0, 0)) + + ctx := issueops.WithDeferredVersionCommit(context.Background()) + err := RunTx(ctx, p, func(ctx context.Context, uw UnitOfWork) (string, error) { + return "bd: update GH#4995", nil + }) + require.NoError(t, err) + require.NoError(t, mock.ExpectationsWereMet()) + assert.Equal(t, 1, p.db.Stats().OpenConnections) +} From 4d512abbc2ef3e7c2d4ac0a99f577e65f21568fc Mon Sep 17 00:00:00 2001 From: Harry Miller Date: Sat, 26 Sep 2026 02:50:09 -0700 Subject: [PATCH 23/45] fix(query): apply bd list's piped-output limit policy to bd query (#6229) (#6744) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(query): apply bd list's piped-output limit policy to bd query (#6229) Root cause: bd list resolves an unflagged --limit through a policy that treats piped stdout as unlimited (GH#4094), but bd query read its --limit flag straight through, so its default of workapi.DefaultQueryLimit (50) applied to piped output too. `bd query '...' | consumer` silently dropped every row past 50; the truncation hint did not fire either, because it only prints when stderr is a terminal. Fix: factor the unflagged tail of bd list's policy into unflaggedLimit / resolveUnflaggedLimit in list_input.go and call it from both commands, so they cannot drift. bd list's switch is unchanged in behavior; bd query's change is a single localized branch in gatherQueryInput. Precedence for bd query, mirroring bd list: 1. explicit --limit (including 0) wins 2. piped stdout -> 0 (unlimited) 3. agent mode on a terminal -> 20 4. otherwise the 50 default bd list keeps its own --all and list.limit branches ahead of step 2; bd query has neither (its --all means "include closed", and there is no query.limit config key), so none was invented. Agent-Signature: claude-code-opus-5-5-unknown-reasoning on behalf of Harry Miller Co-Authored-By: Claude Opus 5.5 (1M context) * review: document the query path's absent defensive cap; unstale a reachability comment Maintainer review fixes for PR #6744 (GH#6229). Both are comment-only; no behavior changes. issueops.QueryRequest.Limit now states that nothing backstops it. ListRequest pairs its Limit with MaxRows, a circuit breaker for an oversized answer, and neither knob that sets that one (--max-rows, BEADS_MAX_ROWS) is read anywhere on the query path, so an unlimited query is unbounded in the plain sense. That was already reachable via `--limit 0`; this PR makes a piped `bd query` reach it without asking, which is what makes it worth writing down. Recording the absence is the documented half of the reviewers' choice — threading a MaxRows twin onto QueryRequest stays available as a follow-up. unionCountsGoSideSortKeepsTheRightSubset's comment justified its case with "DefaultQueryLimit means a limit is always set", which this PR falsified for the CLI half: an unflagged piped `bd query '' --sort id` now arrives on the IsGoSideSort path with Limit 0 and no bound. The HTTP surface still brings a bound, because an absent `limit` parameter resolves to DefaultQueryLimit, so the clause is requalified rather than dropped. The assertions were never affected — they pass an explicit Limit: 5. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5.5 (1M context) Co-authored-by: Julian Knutsen (cherry picked from commit 0e317c4fc0e54764dbfb05a388cd32b2b53fdd2d) Hand-resolved for hotfix/1.3.1: CHANGELOG.md conflicted only on context (main's surrounding Unreleased entries do not exist on the release line). The release CHANGELOG is kept and this entry is added verbatim under Unreleased "### Fixed". No code hunk needed changes. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 8 ++ cmd/bd/list_input.go | 31 +++++- cmd/bd/query.go | 5 + cmd/bd/query_limit_policy_test.go | 95 +++++++++++++++++++ .../db/issue_search_union_window_test.go | 12 ++- issueops/querier.go | 11 +++ 6 files changed, 153 insertions(+), 9 deletions(-) create mode 100644 cmd/bd/query_limit_policy_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index eae47e4d3..5101a7e50 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -139,6 +139,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 transaction and a sibling's refusal does not roll it back — so the summary names the claimed id rather than leaving it silently assigned. +- **`bd query` no longer silently stops at 50 rows when its output is piped** + ([#6229](https://github.com/gastownhall/beads/issues/6229)). `bd list` has + treated piped stdout as unlimited since GH#4094, but `bd query` read its + `--limit` default of 50 straight through, so `bd query '...' | consumer` + dropped every row past 50 with no hint. An unflagged `bd query` now resolves + through the same policy as `bd list`: an explicit `--limit` wins, piped + stdout is unlimited, agent mode on a terminal gets 20, and a terminal gets 50. + ## [1.3.1] - 2026-09-16 ### Changed diff --git a/cmd/bd/list_input.go b/cmd/bd/list_input.go index 7d12e7c63..5dd69799c 100644 --- a/cmd/bd/list_input.go +++ b/cmd/bd/list_input.go @@ -331,10 +331,8 @@ func gatherListInput(cmd *cobra.Command) (listInput, error) { in.effectiveLimit = 0 case listLimitConfigured: in.effectiveLimit = limit - case !ui.IsTerminal(): - in.effectiveLimit = 0 // Piped stdout should not truncate (GH#4094) - case ui.IsAgentMode(): - in.effectiveLimit = 20 + default: + in.effectiveLimit = unflaggedLimit(limit) } // The request carries the limit the caller receives. Which row limit that // implies for the query - a sort SQL cannot express fetches everything and @@ -389,3 +387,28 @@ func parseListTimeFlag(cmd *cobra.Command, name string) (*time.Time, error) { } return &t, nil } + +// agentModeLimit is the page size an unflagged listing gets in agent mode on a +// terminal: ultra-compact output for an LLM context window. +const agentModeLimit = 20 + +// unflaggedLimit is the limit a listing command uses when the caller named +// none: the part of the policy `bd list` and `bd query` share, so the two +// cannot drift (GH#6229). Each command resolves its own earlier branches first +// (an explicit --limit, `bd list`'s --all and list.limit) and hands its own +// default here as fallback. +func unflaggedLimit(fallback int) int { + return resolveUnflaggedLimit(fallback, ui.IsTerminal(), ui.IsAgentMode()) +} + +// resolveUnflaggedLimit is unflaggedLimit with the environment passed in, so +// the terminal and agent-mode branches are testable from a piped `go test`. +func resolveUnflaggedLimit(fallback int, stdoutIsTerminal, agentMode bool) int { + switch { + case !stdoutIsTerminal: + return 0 // Piped stdout should not truncate (GH#4094) + case agentMode: + return agentModeLimit + } + return fallback +} diff --git a/cmd/bd/query.go b/cmd/bd/query.go index 67c689778..bdca76532 100644 --- a/cmd/bd/query.go +++ b/cmd/bd/query.go @@ -151,6 +151,11 @@ func gatherQueryInput(cmd *cobra.Command, args []string) (queryInput, error) { in := queryInput{expression: strings.Join(args, " ")} in.limit, _ = cmd.Flags().GetInt("limit") + if !cmd.Flags().Changed("limit") { + // bd list's policy for an unflagged limit: piped stdout is unlimited, + // agent mode is compact, a terminal gets the default (GH#6229). + in.limit = unflaggedLimit(in.limit) + } in.longFormat, _ = cmd.Flags().GetBool("long") in.parseOnly, _ = cmd.Flags().GetBool("parse-only") in.offset, _ = cmd.Flags().GetInt("offset") diff --git a/cmd/bd/query_limit_policy_test.go b/cmd/bd/query_limit_policy_test.go new file mode 100644 index 000000000..2d6c73e2c --- /dev/null +++ b/cmd/bd/query_limit_policy_test.go @@ -0,0 +1,95 @@ +package main + +import ( + "testing" + + "github.com/spf13/cobra" + + "github.com/steveyegge/beads/internal/ui" + "github.com/steveyegge/beads/internal/workapi" +) + +// newQueryLimitCommand registers the flags gatherQueryInput reads, with +// queryCmd's own --limit default. +func newQueryLimitCommand(t *testing.T, args ...string) *cobra.Command { + t.Helper() + cmd := &cobra.Command{Use: "query"} + cmd.Flags().IntP("limit", "n", workapi.DefaultQueryLimit, "") + cmd.Flags().Int("offset", 0, "") + cmd.Flags().BoolP("all", "a", false, "") + cmd.Flags().Bool("long", false, "") + cmd.Flags().String("sort", "", "") + cmd.Flags().BoolP("reverse", "r", false, "") + cmd.Flags().Bool("parse-only", false, "") + if err := cmd.ParseFlags(args); err != nil { + t.Fatalf("parse %v: %v", args, err) + } + return cmd +} + +// TestQueryLimitPolicyMatchesList pins GH#6229: `bd query` read --limit +// straight through, so `bd query ... | consumer` silently stopped at +// workapi.DefaultQueryLimit rows where `bd list | consumer` got every row +// (GH#4094). An unflagged query now resolves through the same policy as list. +func TestQueryLimitPolicyMatchesList(t *testing.T) { + if ui.IsTerminal() { + t.Skip("this test asserts the piped-stdout branch; go test's stdout is a pipe, but this run's is not") + } + for _, tc := range []struct { + name string + args []string + want int + }{ + {"an explicit limit wins when piped", []string{"--limit", "7"}, 7}, + {"an explicit default-sized limit is still honored when piped", []string{"--limit", "50"}, 50}, + {"an explicit zero stays unlimited", []string{"--limit", "0"}, 0}, + {"piped stdout is unlimited, not the query default", nil, 0}, + // query's --all means "include closed", not "no limit"; piping is + // what lifts the cap here. + {"--all does not change the piped policy", []string{"--all"}, 0}, + } { + t.Run(tc.name, func(t *testing.T) { + in, err := gatherQueryInput(newQueryLimitCommand(t, tc.args...), []string{"status=open"}) + if err != nil { + t.Fatalf("gatherQueryInput(%v): %v", tc.args, err) + } + if in.limit != tc.want { + t.Errorf("limit = %d, want %d", in.limit, tc.want) + } + if in.request.Limit == nil { + t.Fatal("QueryRequest.Limit is nil; `bd query` resolves its own limit before the request exists") + } + if *in.request.Limit != tc.want { + t.Errorf("QueryRequest.Limit = %d, want %d", *in.request.Limit, tc.want) + } + }) + } +} + +// TestUnflaggedLimitBranches covers the terminal and agent-mode branches the +// piped `go test` process cannot reach through gatherQueryInput or +// gatherListInput, for both commands' defaults. +func TestUnflaggedLimitBranches(t *testing.T) { + for _, fallback := range []int{workapi.DefaultListLimit, workapi.DefaultQueryLimit} { + for _, tc := range []struct { + name string + terminal bool + agent bool + want int + }{ + {"terminal gets the command default", true, false, fallback}, + {"agent mode on a terminal is compact", true, true, agentModeLimit}, + {"piped stdout is unlimited", false, false, 0}, + // Piping outranks agent mode, as it always has for bd list. + {"piped stdout is unlimited in agent mode too", false, true, 0}, + } { + if got := resolveUnflaggedLimit(fallback, tc.terminal, tc.agent); got != tc.want { + t.Errorf("%s: resolveUnflaggedLimit(%d, terminal=%v, agent=%v) = %d, want %d", + tc.name, fallback, tc.terminal, tc.agent, got, tc.want) + } + } + } + if agentModeLimit != 20 { + t.Errorf("agentModeLimit = %d, want 20 (bd list's historical agent-mode cap)", agentModeLimit) + } +} diff --git a/internal/storage/domain/db/issue_search_union_window_test.go b/internal/storage/domain/db/issue_search_union_window_test.go index 4809ae167..2e83166fb 100644 --- a/internal/storage/domain/db/issue_search_union_window_test.go +++ b/internal/storage/domain/db/issue_search_union_window_test.go @@ -68,11 +68,13 @@ func (s *testSuite) unionGoSideSortKeepsTheRightSubset() { // unionCountsGoSideSortKeepsTheRightSubset is the same case on the counts twin, // and that is the twin the live callers reach. workapi.BuildQueryPlan pushes // SortBy and Limit straight onto the filter for any filter-expressible query — -// no SQLLimit guard, and DefaultQueryLimit means a limit is always set — so -// `bd query '' --sort id` and GET /v0/beads/issues/query?sort=id both -// arrived here with SortBy "id" and a bound. Its own comment says the pushdown -// makes "the page the first N in the requested order on every backend", which -// was true of every key except the one IsGoSideSort names. +// no SQLLimit guard — so `bd query '' --sort id` and +// GET /v0/beads/issues/query?sort=id both arrive here with SortBy "id". The +// HTTP surface still brings a bound with it, because an absent `limit` +// parameter resolves to DefaultQueryLimit; the CLI need not, since GH#6229 +// gave a piped `bd query` the unflagged limit 0. Its own comment says the +// pushdown makes "the page the first N in the requested order on every +// backend", which was true of every key except the one IsGoSideSort names. func (s *testSuite) unionCountsGoSideSortKeepsTheRightSubset() { const prefix = "bd-uw-cgos" ids := s.seedTwoPlanes(prefix, 12) diff --git a/issueops/querier.go b/issueops/querier.go index ca6212a03..edd3b22b1 100644 --- a/issueops/querier.go +++ b/issueops/querier.go @@ -60,6 +60,17 @@ type QueryRequest struct { // constant serve both surfaces. A negative Limit is ErrValidation rather // than a synonym for unlimited: two spellings of "no bound" is one more // than a caller can check for. + // + // NOTHING BACKSTOPS IT. ListRequest pairs its Limit with MaxRows, a + // circuit breaker that refuses an oversized answer outright; this request + // has no twin to it by design, and neither knob that sets that one + // (`--max-rows`, BEADS_MAX_ROWS) is read anywhere on the query path. So an + // unlimited query is unbounded in the plain sense: it reads every row the + // expression matches, however many that is, and a caller who asked for one + // gets it rather than an ErrTooManyRows. The ceiling here is Limit itself. + // This is reachable without asking: a piped `bd query` resolves its + // unflagged limit to 0 before the request, the same policy `bd list` has + // applied to piped output since GH#4094. Limit *int // Offset skips the first N MATCHING rows, on EVERY implementation and for From 48be70fb7e918d10186d6dec5ae5c22f326bf05e Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 28 Sep 2026 02:45:58 -0700 Subject: [PATCH 24/45] feat(backup): proxied auto-backup on managed-local, serialized with bd backup sync (#6879) * feat(backup): run opted-in auto-backup on a managed-local proxied server managed-local, and #6694 made `bd backup status` admit the remaining gap: the proxied arm of PersistentPostRunE never called runPostRunAutoBackup, so backup.enabled=true was inert on every proxied workspace. Close it. The proxied arm now calls the auto-backup hook before it closes the provider, under the direct arm's maintenance gate (strict --readonly, `bd serve`, a migration freeze) plus previews. maybeAutoBackup picks a backend for the command: the direct store as before, or the proxied provider's RunNonTx seam, which `bd backup sync` already uses. The proxied backend is honored exactly where `bd backup sync` is: the "backup sync" capability row's rule for this workspace's topology, so managed-local only. Auto-backup registers a server-side backup remote, which on a server bd does not own is global to every client; an implicit trigger must not get further than the explicit verb. It never opens a provider of its own: a command that opened none (`bd dolt stop`) or closed it (restore) backs nothing up rather than relaunching the server. The default stays OFF, as on every server-mode workspace. The register/sync-to-a-local-dir sequence that DoltStore and EmbeddedDoltStore each carried a copy of moves to versioncontrolops.BackupToDir (and HEAD lookup to CurrentCommit), so the proxied backend reuses it instead of adding a third copy. Tests: TestManagedLocalProxiedAutoBackup (default off, opt-in writes a restorable backup, change detection, `bd dolt stop` stays quiescent, restore round trip with issue-set equality; fails against origin/main); TestManagedLocalProxiedBackupRestoreRefusedWhileAttached pins the restore design (refused while another bd client holds the workspace, serving child untouched, succeeds once it detaches); unit tests for the backend locality gate, the post-run wiring, the status note and BackupToDir. Both managed-local tests, and the existing round trip, are added to the proxied-local smoke lane's compiled-in list. Co-Authored-By: Claude Opus 5.5 * fix(backup): serialize auto-backup and bd backup sync with a workspace backup lock Review follow-up on #6879. Auto-backup and an explicit bd backup sync both run under the command's SHARED workspace gate, so nothing stopped them overlapping, and auto-backup's remove -> add -> sync on one fixed remote name races a concurrent copy of itself. A per-workspace flock (.beads/backup.lock, already gitignored by *.lock) now serializes them in every mode. Auto-backup takes it non-blocking and skips when held (the throttle is left alone, so a later command retries). bd backup sync waits up to exclusiveGateWait - the bound restore's exclusive gate acquisition uses - then fails with "another backup is running for this workspace" before touching the store. Restore needs no part in it: its exclusive workspace gate already excludes every command that could hold the lock. Tests: lock semantics (refusal while held, bounded wait, release), auto-backup skipping while held and running once released, and sync failing loudly without reaching the store. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 (cherry picked from commit 4e4ebdb443eefef1e6456e8b57bc69d5df7d85f2) Hand-resolved for hotfix/1.3.1: - cmd/bd/BUILD.bazel: does not exist on the release line (no Bazel build there); the file is dropped, as in #6905. - .github/workflows/proxied-local-smoke.yml: the release lane list differs from main (it has CapabilityMatrix and BackupRoundTrip, lacks TrackerUOWConformance). Kept the release list and appended this commit's two new tests, AutoBackup and BackupRestoreRefusedWhileAttached (BackupRoundTrip was already listed). The backup*.go path-filter hunks applied cleanly. - CHANGELOG.md: context-only conflict. The release CHANGELOG is kept and this commit's two Unreleased/Added entries are inserted verbatim before the bd purge entry, where main places them. No code hunk needed changes. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/proxied-local-smoke.yml | 4 +- CHANGELOG.md | 21 ++ cmd/bd/backup.go | 8 +- cmd/bd/backup_auto.go | 74 ++++-- ...kup_auto_proxied_local_integration_test.go | 240 ++++++++++++++++++ cmd/bd/backup_auto_proxied_test.go | 142 +++++++++++ cmd/bd/backup_auto_test.go | 22 +- cmd/bd/backup_dolt.go | 16 +- cmd/bd/backup_export.go | 39 ++- cmd/bd/backup_export_failure_test.go | 5 +- cmd/bd/backup_lock.go | 71 ++++++ cmd/bd/backup_lock_test.go | 133 ++++++++++ cmd/bd/backup_proxied_server.go | 71 +++++- cmd/bd/main.go | 9 + internal/storage/dolt/store.go | 34 +-- internal/storage/dolt/versioned.go | 7 +- .../storage/embeddeddolt/version_control.go | 33 +-- internal/storage/versioncontrolops/backup.go | 55 ++++ .../storage/versioncontrolops/backup_test.go | 29 +++ 19 files changed, 885 insertions(+), 128 deletions(-) create mode 100644 cmd/bd/backup_auto_proxied_local_integration_test.go create mode 100644 cmd/bd/backup_auto_proxied_test.go create mode 100644 cmd/bd/backup_lock.go create mode 100644 cmd/bd/backup_lock_test.go diff --git a/.github/workflows/proxied-local-smoke.yml b/.github/workflows/proxied-local-smoke.yml index eee9b7b19..7033f3faa 100644 --- a/.github/workflows/proxied-local-smoke.yml +++ b/.github/workflows/proxied-local-smoke.yml @@ -117,7 +117,9 @@ jobs: TestManagedLocalProxiedDoltLifecycleLeavesOtherTopologiesAlone \ TestManagedLocalProxiedPurgeWispsPlaneRetention \ TestManagedLocalProxiedCapabilityMatrix \ - TestManagedLocalProxiedBackupRoundTrip + TestManagedLocalProxiedBackupRoundTrip \ + TestManagedLocalProxiedAutoBackup \ + TestManagedLocalProxiedBackupRestoreRefusedWhileAttached do if ! grep -qx "$test_name" <<<"$listed"; then echo "managed-local lifecycle test $test_name is missing; got: '$listed'" >&2 diff --git a/CHANGELOG.md b/CHANGELOG.md index 5101a7e50..95333335f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -39,6 +39,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 server topology. Wiring an auto-backup hook through the new routes is left to a later slice. +- **Auto-backup runs on a managed-local proxied-server workspace.** The + proxied arm of the post-command hook now calls auto-backup, so an explicit + `backup.enabled=true` (or `BD_BACKUP_ENABLED=1`) takes the same throttled, + change-detected Dolt-native backup into `.beads/backup` it takes in direct + mode, through the proxied provider's non-transactional seam. Before this the + opt-in was inert on every proxied workspace and `bd backup status` said so. + The default stays OFF, as on every server-mode workspace: many bd clients + share one server. It is honored exactly where `bd backup sync` is — a + proxied server bd started itself — and stays inert on an external or + team-server topology, strict `--readonly`, `bd serve`, a preview + (`--dry-run`/`--inspect`) and a migration freeze. A command that opened no + provider (e.g. `bd dolt stop`) backs nothing up rather than relaunching the + server to do so. `bd backup status` now reports the proxied default as + `auto: off in proxied-server mode; set backup.enabled=true to opt in`. + +- **Auto-backup and `bd backup sync` no longer overlap.** A per-workspace + backup lock (`.beads/backup.lock`) serializes them in every mode: + auto-backup skips when another backup holds it, and `bd backup sync` waits + up to 5 seconds (the bound restore uses for the workspace gate) and then + fails with "another backup is running for this workspace". + - **`bd purge` covers an orchestrator's wisp retention sweep.** Three changes that together let `bd purge --wisps-plane --older-than 168h --force` replace a raw `DELETE FROM wisps` retention step: diff --git a/cmd/bd/backup.go b/cmd/bd/backup.go index 270536bcf..0e8e02fcc 100644 --- a/cmd/bd/backup.go +++ b/cmd/bd/backup.go @@ -42,10 +42,10 @@ Auto-backup default: coordinate destinations). 'bd config get backup.enabled' shows the effective value and its source. - On a proxied-server workspace that opt-in does not apply: auto-backup has - no post-command hook on that path, so backup.enabled=true is inert there. - The five backup verbs above are honored on a managed-local proxied server — - run 'bd backup sync' explicitly to back one up.`, + A proxied-server workspace is server mode too: auto-backup is OFF by + default and backup.enabled=true opts in. Both the opt-in and the five + backup verbs above are honored only on a managed-local proxied server — + one bd started itself; on a Dolt server bd does not own they are refused.`, GroupID: "sync", } diff --git a/cmd/bd/backup_auto.go b/cmd/bd/backup_auto.go index 8974439e2..988c0dcfb 100644 --- a/cmd/bd/backup_auto.go +++ b/cmd/bd/backup_auto.go @@ -2,6 +2,7 @@ package main import ( "context" + "errors" "fmt" "os" "strings" @@ -46,28 +47,21 @@ func isBackupAutoEnabled() bool { // already computed so the note cannot disagree with the number beside // it — and so status does not re-run the git-remote probe. // -// The note narrates the reason the decision actually used. Proxied-server -// is checked first and ignores the config source, because on that topology -// auto-backup cannot run at all whatever backup.enabled says: the proxied -// arm of PersistentPostRunE never calls runPostRunAutoBackup (main.go), so -// an explicit backup.enabled=true is inert and reporting a bare -// "enabled=true" would send the operator away believing backups happen. -// The five backup verbs are honored on managed-local (the only proxied -// shape that reaches this line — every other one is refused by -// requireLocalProxiedBackup), so `bd backup sync` is the real answer there. -// -// Every other sql-server shape keeps its post-run hook, so there the -// default is merely off — the reason `bd config get backup.enabled` -// already reports for the same condition. Before this arm existed, status -// attributed that OFF to a missing git remote, which is untrue: server -// mode short-circuits isBackupAutoEnabled whether or not a remote exists. +// The note narrates the reason the decision actually used. An explicit +// backup.enabled needs no note: the source explains it, and every shape +// that reaches `bd backup status` honors it — including a managed-local +// proxied server, whose post-run arm runs auto-backup (every other proxied +// shape is refused by requireLocalProxiedBackup before status renders). +// A server-mode default is OFF whether or not a git remote exists, so it +// must not be attributed to the remote; before these arms existed, status +// did exactly that. func backupAutoStatusNote(enabled bool) string { - if usesProxiedServer() { - return "auto-backup does not run on proxied-server; use 'bd backup sync'" - } if config.GetValueSource("backup.enabled") != config.SourceDefault { return "" } + if usesProxiedServer() { + return "auto: off in proxied-server mode; set backup.enabled=true to opt in" + } if usesSQLServer() { return "auto: off in sql-server mode" } @@ -125,6 +119,21 @@ func clientServerShareFilesystem() bool { // bd sessions don't need a chatty repeat on every command. var autoBackupSkipNoticeOnce sync.Once +// autoBackupBackendForCommand returns the storage this command's auto-backup +// runs against, or ok=false when there is nothing it may back up. +func autoBackupBackendForCommand() (localBackupBackend, bool) { + if usesProxiedServer() { + return proxiedAutoBackupBackend() + } + if store == nil { + return nil, false + } + if lm, ok := storage.UnwrapStore(store).(storage.LifecycleManager); ok && lm.IsClosed() { + return nil, false + } + return directLocalBackup{store: store}, true +} + // maybeAutoBackup runs a Dolt-native backup if enabled and the throttle interval has passed. // Called from PersistentPostRun after auto-commit. func maybeAutoBackup(ctx context.Context) { @@ -139,10 +148,8 @@ func maybeAutoBackup(ctx context.Context) { if !isBackupAutoEnabled() { return } - if store == nil { - return - } - if lm, ok := storage.UnwrapStore(store).(storage.LifecycleManager); ok && lm.IsClosed() { + backend, ok := autoBackupBackendForCommand() + if !ok { return } @@ -152,7 +159,11 @@ func maybeAutoBackup(ctx context.Context) { // constructs is meaningless to the server — register fails on // every command. Skip cleanly with a one-time INFO so operators // know auto-backup is silent on purpose. - if !clientServerShareFilesystem() { + // + // Proxied workspaces do not consult the host: proxiedAutoBackupBackend + // has already required managed-local, where bd spawned the server on + // this filesystem itself. + if !usesProxiedServer() && !clientServerShareFilesystem() { autoBackupSkipNoticeOnce.Do(func() { if !isQuiet() && !jsonOutput { fmt.Fprintln(os.Stderr, @@ -190,7 +201,7 @@ func maybeAutoBackup(ctx context.Context) { } // Change detection: skip if nothing changed - currentCommit, err := store.GetCurrentCommit(ctx) + currentCommit, err := backend.CurrentCommit(ctx) if err != nil { fmt.Fprintf(os.Stderr, "Warning: auto-backup skipped: failed to get current commit: %v\n", err) return @@ -200,8 +211,21 @@ func maybeAutoBackup(ctx context.Context) { return } + // One backup at a time per workspace (backup_lock.go). Auto-backup never + // waits: a held lock means a backup is already running. + release, err := acquireBackupLock(0) + if err != nil { + if errors.Is(err, errBackupBusy) { + debug.Logf("backup: skipping — another backup is running\n") + return + } + fmt.Fprintf(os.Stderr, "Warning: auto-backup skipped: %v\n", err) + return + } + defer release() + // Run the backup (force=true since we already checked change detection above) - if _, err := runBackupExport(ctx, true); err != nil { + if _, err := runBackupExport(ctx, backend, true); err != nil { if !isQuiet() && !jsonOutput { fmt.Fprintf(os.Stderr, "Warning: auto-backup failed: %v\n", err) } diff --git a/cmd/bd/backup_auto_proxied_local_integration_test.go b/cmd/bd/backup_auto_proxied_local_integration_test.go new file mode 100644 index 000000000..c87f5389e --- /dev/null +++ b/cmd/bd/backup_auto_proxied_local_integration_test.go @@ -0,0 +1,240 @@ +//go:build cgo && unix + +package main + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "slices" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" + "github.com/steveyegge/beads/internal/storage/dbproxy/server" + "github.com/steveyegge/beads/internal/workspacegate" +) + +// autoBackupOptIn is the explicit opt-in a server-mode workspace needs, with +// the throttle collapsed so every command in a test is due. +var autoBackupOptIn = []string{"BD_BACKUP_ENABLED=true", "BD_BACKUP_INTERVAL=1ms"} + +// TestManagedLocalProxiedAutoBackup is the claim this slice makes: on a +// managed-local proxied workspace, an explicit backup.enabled=true makes the +// post-command hook take a Dolt-native backup into .beads/backup, exactly as +// it does in direct mode — and that backup restores. +// +// Like the explicit-verb round trip next door, the proof is a restore, not an +// exit code: an auto-backup that wrote a manifest nobody can restore from would +// pass every other assertion here. +// +// Named TestManagedLocalProxied* so the proxied-local smoke lane runs it. +func TestManagedLocalProxiedAutoBackup(t *testing.T) { + requireManagedLocalProxiedEnv(t) + + bd := buildEmbeddedBD(t) + p := bdManagedLocalInit(t, bd, "bkauto", 5*time.Minute) + autoDir := filepath.Join(p.beadsDir, "backup") + manifest := filepath.Join(autoDir, "manifest") + + // Server mode defaults auto-backup OFF: many clients share one server. + // Nothing may be written without the opt-in. + bdProxiedCreate(t, bd, p.dir, "no opt-in, no backup", "-p", "2") + if _, err := os.Stat(manifest); err == nil { + t.Fatalf("auto-backup ran without an opt-in: %s exists", manifest) + } + + // With the opt-in, one ordinary write is enough. + created := proxiedAutoBackupCreate(t, bd, p, "captured by auto-backup") + if _, err := os.Stat(manifest); err != nil { + t.Fatalf("opted-in auto-backup produced no manifest at %s: %v", manifest, err) + } + state := readProxiedAutoBackupState(t, bd, p) + if state.LastDoltCommit == "" || state.Timestamp.IsZero() { + t.Fatalf("auto-backup did not record its watermark: %+v", state) + } + wantIDs := proxiedIssueIDs(t, bd, p) + if !slices.Contains(wantIDs, created) { + t.Fatalf("precondition: %s missing from %v", created, wantIDs) + } + + // A read-only command with nothing new to capture must not advance the + // watermark: change detection runs before the backup, as in direct mode. + if _, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, autoBackupOptIn, "list", "--json"); err != nil { + t.Fatalf("bd list with auto-backup opted in: %v\n%s", err, stderr) + } + if again := readProxiedAutoBackupState(t, bd, p); again.LastDoltCommit != state.LastDoltCommit { + t.Fatalf("auto-backup advanced with no new commit: %q -> %q", state.LastDoltCommit, again.LastDoltCommit) + } + + // `bd dolt stop` opens no provider, so its post-run hook has nothing to + // back up — and must not relaunch the server it just stopped to find out. + if _, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, autoBackupOptIn, "dolt", "stop"); err != nil { + t.Fatalf("bd dolt stop: %v\n%s", err, stderr) + } + assertManagedProxiedQuiescent(t, p, "bd dolt stop with auto-backup opted in") + + // Diverge from the backup, then restore it and compare. + after := bdProxiedCreate(t, bd, p.dir, "created after the auto-backup", "-p", "2") + out, err := bdProxiedRun(t, bd, p.dir, "backup", "restore", autoDir, "--force", "--json") + if err != nil { + t.Fatalf("bd backup restore %s --force: %v\n%s", autoDir, err, out) + } + gotIDs := proxiedIssueIDs(t, bd, p) + if slices.Contains(gotIDs, after.ID) { + t.Fatalf("restore kept %s, created after the auto-backup; got %v", after.ID, gotIDs) + } + if !slices.Equal(gotIDs, wantIDs) { + t.Fatalf("restored issues differ from the auto-backed-up set:\n got %v\n want %v", gotIDs, wantIDs) + } +} + +// TestManagedLocalProxiedBackupRestoreRefusedWhileAttached pins the restore +// design choice: restore REPLACES the database, so it refuses while any other +// bd command is attached to the workspace, and in that case touches nothing — +// the serving child keeps running and the data stays as it was. +// +// "Attached" is what another bd process holds for its lifetime: a SHARED +// workspace gate. The test holds exactly that, in-process, rather than racing +// a second process into position. +func TestManagedLocalProxiedBackupRestoreRefusedWhileAttached(t *testing.T) { + requireManagedLocalProxiedEnv(t) + + bd := buildEmbeddedBD(t) + p := bdManagedLocalInit(t, bd, "bkatt", 5*time.Minute) + + bdProxiedCreate(t, bd, p.dir, "in the backup", "-p", "1") + dest := filepath.Join(t.TempDir(), "dolt-backup") + if out, err := bdProxiedRun(t, bd, p.dir, "backup", "init", dest); err != nil { + t.Fatalf("bd backup init: %v\n%s", err, out) + } + if out, err := bdProxiedRun(t, bd, p.dir, "backup", "sync"); err != nil { + t.Fatalf("bd backup sync: %v\n%s", err, out) + } + after := bdProxiedCreate(t, bd, p.dir, "not in the backup", "-p", "1") + before := proxiedIssueIDs(t, bd, p) + + proxyPID := readManagedProxyPidFile(t, p) + backendPID := readManagedBackendPidFile(t, p) + if proxyPID == nil || backendPID == nil { + t.Fatal("precondition: the managed topology should be serving") + } + + gates, err := buildWorkspaceGateSet(p.beadsDir) + if err != nil { + t.Fatalf("resolve workspace gates: %v", err) + } + held, err := workspacegate.AcquireAll(context.Background(), workspacegate.Shared, + workspacegate.Options{Reason: "test: attached bd client"}, gates...) + if err != nil { + t.Fatalf("hold the workspace gate: %v", err) + } + released := false + release := func() { + if !released { + released = true + if err := held.Release(); err != nil { + t.Errorf("release workspace gate: %v", err) + } + } + } + t.Cleanup(release) + + stdout, _, err := bdProxiedRunBuffers(t, bd, p.dir, "backup", "restore", dest, "--force", "--json") + if err == nil { + t.Fatalf("bd backup restore succeeded while another bd client was attached:\n%s", stdout) + } + var refusal struct { + Error string `json:"error"` + } + if jerr := json.Unmarshal([]byte(stdout[strings.Index(stdout, "{"):]), &refusal); jerr != nil { + t.Fatalf("restore refusal is not JSON on stdout: %v\n%s", jerr, stdout) + } + if !strings.Contains(refusal.Error, "other bd commands are using this workspace") { + t.Fatalf("restore refusal does not say why: %q", refusal.Error) + } + + // Refused means untouched: the serving topology is the same processes. + if pf := readManagedProxyPidFile(t, p); pf == nil || pf.Pid != proxyPID.Pid || !processAlive(pf.Pid) { + t.Fatalf("refused restore disturbed the proxy: before %+v, now %+v", proxyPID, pf) + } + if pf := readManagedBackendPidFile(t, p); pf == nil || pf.Pid != backendPID.Pid || !processAlive(pf.Pid) { + t.Fatalf("refused restore disturbed the dolt child: before %+v, now %+v", backendPID, pf) + } + + release() + if got := proxiedIssueIDs(t, bd, p); !slices.Equal(got, before) || !slices.Contains(got, after.ID) { + t.Fatalf("refused restore changed the data:\n got %v\n want %v", got, before) + } + + // Once the other client detaches, the same restore goes through. + if out, err := bdProxiedRun(t, bd, p.dir, "backup", "restore", dest, "--force", "--json"); err != nil { + t.Fatalf("bd backup restore after the client detached: %v\n%s", err, out) + } + if slices.Contains(proxiedIssueIDs(t, bd, p), after.ID) { + t.Fatalf("restore after detach did not replace the database: %s is still here", after.ID) + } +} + +// proxiedAutoBackupCreate creates an issue with auto-backup opted in and +// returns its ID. +func proxiedAutoBackupCreate(t *testing.T, bd string, p proxiedProject, title string) string { + t.Helper() + stdout, stderr, err := bdProxiedRunBuffersWithEnv(t, bd, p.dir, autoBackupOptIn, "create", title, "-p", "2", "--silent") + if err != nil { + t.Fatalf("bd create with auto-backup opted in: %v\n%s", err, stderr) + } + if strings.Contains(stderr, "auto-backup") { + t.Fatalf("auto-backup reported a problem:\n%s", stderr) + } + id := strings.TrimSpace(stdout) + if id == "" { + t.Fatalf("bd create --silent printed no ID; stderr:\n%s", stderr) + } + return id +} + +// readProxiedAutoBackupState reads the auto-backup watermark through +// `bd backup status --json`, the surface an operator uses. +func readProxiedAutoBackupState(t *testing.T, bd string, p proxiedProject) backupState { + t.Helper() + out, err := bdProxiedRun(t, bd, p.dir, "backup", "status", "--json") + if err != nil { + t.Fatalf("bd backup status --json: %v\n%s", err, out) + } + start := strings.Index(string(out), "{") + if start < 0 { + t.Fatalf("bd backup status --json produced no JSON:\n%s", out) + } + var status struct { + Backup backupState `json:"backup"` + } + if err := json.Unmarshal(out[start:], &status); err != nil { + t.Fatalf("parse backup status JSON: %v\n%s", err, out) + } + return status.Backup +} + +// proxiedIssueIDs lists every issue ID in the workspace, sorted. +func proxiedIssueIDs(t *testing.T, bd string, p proxiedProject) []string { + t.Helper() + var ids []string + for _, issue := range bdProxiedListJSON(t, bd, p, "--all") { + ids = append(ids, issue.ID) + } + slices.Sort(ids) + return ids +} + +// assertManagedProxiedQuiescent fails when either managed-local pidfile is +// present, i.e. something relaunched the topology. +func assertManagedProxiedQuiescent(t *testing.T, p proxiedProject, after string) { + t.Helper() + for _, name := range []string{proxy.PIDFileName, server.PIDFileName} { + if _, err := os.Stat(filepath.Join(p.proxyRoot, name)); err == nil { + t.Fatalf("after %s, %s is present: the topology was relaunched", after, name) + } + } +} diff --git a/cmd/bd/backup_auto_proxied_test.go b/cmd/bd/backup_auto_proxied_test.go new file mode 100644 index 000000000..87c9ffc11 --- /dev/null +++ b/cmd/bd/backup_auto_proxied_test.go @@ -0,0 +1,142 @@ +package main + +import ( + "context" + "database/sql" + "os" + "testing" + + "github.com/spf13/cobra" + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/storage/uow" +) + +// fakeMaintenanceProvider is a proxied provider stand-in exposing the +// non-transactional seam proxied auto-backup runs on. It never opens a +// connection; the tests here only assert whether it would be used. +type fakeMaintenanceProvider struct { + uow.UnitOfWorkProvider + runs int +} + +func (f *fakeMaintenanceProvider) RunNonTx(context.Context, func(context.Context, *sql.Conn) error) error { + f.runs++ + return nil +} + +func (f *fakeMaintenanceProvider) Close(context.Context) error { return nil } + +// TestProxiedAutoBackupBackendFollowsLocality pins where auto-backup may run on +// a proxied workspace: exactly where `bd backup sync` is honored. Auto-backup +// registers a server-side backup remote, which on a server bd does not own is +// global to every client of it — the storm the explicit verbs refuse by design +// — so an implicit trigger must not get further than the explicit one. An +// unreadable sidecar resolves to the unknown topology and fails closed. +// +// Cannot be parallel: mutates the proxiedServerMode and uowProvider globals. +func TestProxiedAutoBackupBackendFollowsLocality(t *testing.T) { + for _, tc := range []struct { + name string + sidecar string + noProvider bool + wantBackend bool + }{ + {name: "managed-local with an open provider", sidecar: `{}`, wantBackend: true}, + {name: "managed-local with no provider open", sidecar: `{}`, noProvider: true}, + {name: "external tcp", sidecar: `{"external":{"host":"db.example.com","port":3306}}`}, + {name: "external unix socket", sidecar: `{"external":{"socket":"/tmp/dolt.sock"}}`}, + {name: "unreadable sidecar", sidecar: `{not json`}, + } { + t.Run(tc.name, func(t *testing.T) { + beadsDir := prepareBackupStatusTest(t) + proxiedServerMode = true + if err := os.WriteFile(configfile.ProxiedServerClientInfoPath(beadsDir), []byte(tc.sidecar), 0o600); err != nil { + t.Fatalf("write proxied sidecar: %v", err) + } + + oldProvider := uowProvider + t.Cleanup(func() { uowProvider = oldProvider }) + fake := &fakeMaintenanceProvider{} + uowProvider = fake + if tc.noProvider { + uowProvider = nil + } + + backend, ok := autoBackupBackendForCommand() + if ok != tc.wantBackend { + t.Fatalf("autoBackupBackendForCommand() ok = %v, want %v", ok, tc.wantBackend) + } + if !ok { + return + } + if _, isProxied := backend.(proxiedLocalBackup); !isProxied { + t.Fatalf("backend = %T, want proxiedLocalBackup", backend) + } + if fake.runs != 0 { + t.Fatalf("resolving the backend ran %d maintenance call(s); it must not touch the server", fake.runs) + } + }) + } +} + +// TestPersistentPostRunProxiedRunsAutoBackup pins the wiring: the proxied arm +// of PersistentPostRunE calls the auto-backup hook, under the same gate the +// direct arm's maintenance net uses, plus previews. +// +// Before this hook existed the proxied arm only pruned the journal and closed +// the provider, so backup.enabled=true was inert on every proxied workspace. +// +// Cannot be parallel: mutates PersistentPostRunE's package globals. +func TestPersistentPostRunProxiedRunsAutoBackup(t *testing.T) { + for _, tc := range []struct { + name string + cmd func() *cobra.Command + readonly bool + want int + }{ + {name: "ordinary command", cmd: func() *cobra.Command { return &cobra.Command{Use: "list"} }, want: 1}, + {name: "strict readonly", cmd: func() *cobra.Command { return &cobra.Command{Use: "list"} }, readonly: true}, + {name: "bd serve", cmd: func() *cobra.Command { return &cobra.Command{Use: serveCmdName} }}, + {name: "preview", cmd: func() *cobra.Command { + c := &cobra.Command{Use: "list"} + c.Flags().Bool("dry-run", false, "") + if err := c.Flags().Set("dry-run", "true"); err != nil { + t.Fatalf("set --dry-run: %v", err) + } + return c + }}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Chdir(t.TempDir()) + t.Setenv("BEADS_DIR", "") + + oldStore, oldProvider := store, uowProvider + oldReadonly, oldProxied := readonlyMode, proxiedServerMode + oldRootCtx, oldRootCancel := rootCtx, rootCancel + oldSpan, oldProfile, oldTrace := commandSpan, profileFile, traceFile + oldBackup := runPostRunAutoBackup + t.Cleanup(func() { + store, uowProvider = oldStore, oldProvider + readonlyMode, proxiedServerMode = oldReadonly, oldProxied + rootCtx, rootCancel = oldRootCtx, oldRootCancel + commandSpan, profileFile, traceFile = oldSpan, oldProfile, oldTrace + runPostRunAutoBackup = oldBackup + }) + + calls := 0 + runPostRunAutoBackup = func(context.Context) { calls++ } + store, uowProvider = nil, nil + readonlyMode = tc.readonly + proxiedServerMode = true + rootCtx, rootCancel = context.Background(), nil + commandSpan, profileFile, traceFile = nil, nil, nil + + if err := rootCmd.PersistentPostRunE(tc.cmd(), nil); err != nil { + t.Fatalf("PersistentPostRunE: %v", err) + } + if calls != tc.want { + t.Fatalf("auto-backup hook calls = %d, want %d", calls, tc.want) + } + }) + } +} diff --git a/cmd/bd/backup_auto_test.go b/cmd/bd/backup_auto_test.go index 5e64195db..f2a10803b 100644 --- a/cmd/bd/backup_auto_test.go +++ b/cmd/bd/backup_auto_test.go @@ -133,9 +133,9 @@ func TestIsBackupAutoEnabled(t *testing.T) { // status` reachable on a proxied server for the first time, where it rendered // the embedded-mode footnote "(auto: no git remote)". On any sql-server shape // the value is false because usesSQLServer() short-circuits, whether or not a -// remote exists, so that reason was simply untrue; and on proxied specifically -// even an explicit backup.enabled=true changes nothing, because the proxied -// arm of PersistentPostRunE never calls runPostRunAutoBackup. +// remote exists, so that reason was simply untrue. Since the proxied post-run +// arm gained its auto-backup hook, an explicit opt-in on proxied is honored +// and needs no caveat. // // Cannot be parallel: mutates the proxiedServerMode global, primeHasGitRemote // and env vars. @@ -191,23 +191,25 @@ func TestBackupAutoStatusNote(t *testing.T) { wantNote: "", }, { - name: "proxied default → names the missing hook, not the remote", + // Proxied is server mode: the default is off whether or not a + // remote exists, and the note names the opt-in that now works. + name: "proxied default → off, names the opt-in, not the remote", envVal: "\x00", hasRemote: true, proxied: true, wantEnabled: false, - wantNote: "auto-backup does not run on proxied-server; use 'bd backup sync'", + wantNote: "auto: off in proxied-server mode; set backup.enabled=true to opt in", }, { - // The dead opt-in: enabled=true with nothing to run it. The note - // must survive an explicit source here, or status reports a bare - // "enabled=true" and the operator believes backups are happening. - name: "proxied explicit opt-in → still inert, still annotated", + // The proxied post-run arm runs auto-backup on managed-local (the + // only proxied shape status renders on), so an explicit opt-in is + // real there and needs no caveat. + name: "proxied explicit opt-in → honored, no note", envVal: "true", hasRemote: false, proxied: true, wantEnabled: true, - wantNote: "auto-backup does not run on proxied-server; use 'bd backup sync'", + wantNote: "", }, } diff --git a/cmd/bd/backup_dolt.go b/cmd/bd/backup_dolt.go index 5ee084260..3e2aebbd7 100644 --- a/cmd/bd/backup_dolt.go +++ b/cmd/bd/backup_dolt.go @@ -142,6 +142,20 @@ Run 'bd backup init ' first to configure a destination.`, }() ctx := rootCtx + if usesProxiedServer() { + if err := requireLocalProxiedBackup("backup sync"); err != nil { + return err + } + } + release, err := acquireBackupLock(backupLockWait) + if err != nil { + if errors.Is(err, errBackupBusy) { + return HandleErrorRespectJSON("%v; retry when it completes", err) + } + return HandleErrorRespectJSON("%v", err) + } + defer release() + if usesProxiedServer() { return runBackupSyncProxied(ctx) } @@ -453,7 +467,7 @@ backup configuration. The backup data at the destination is not deleted.`, } // Also remove backup_export if it exists (auto-export may have created it at same URL) - _ = bs.BackupRemove(ctx, "backup_export") + _ = bs.BackupRemove(ctx, versioncontrolops.ExportBackupName) // Remove local config if path, err := doltBackupConfigPath(); err == nil { diff --git a/cmd/bd/backup_export.go b/cmd/bd/backup_export.go index 25b7ac945..654ba7c47 100644 --- a/cmd/bd/backup_export.go +++ b/cmd/bd/backup_export.go @@ -122,9 +122,35 @@ func atomicWriteFile(path string, data []byte) error { return nil } +// localBackupBackend is what a Dolt-native backup into a local directory needs +// from storage: the commit it would capture, and the backup itself. The direct +// store supplies one (directLocalBackup) and so does the proxied provider +// (proxiedLocalBackup, backup_proxied_server.go). +type localBackupBackend interface { + CurrentCommit(ctx context.Context) (string, error) + BackupToDir(ctx context.Context, dir string) error +} + +// directLocalBackup is localBackupBackend over an embedded or sql-server store. +type directLocalBackup struct { + store storage.DoltStorage +} + +func (b directLocalBackup) CurrentCommit(ctx context.Context) (string, error) { + return b.store.GetCurrentCommit(ctx) +} + +func (b directLocalBackup) BackupToDir(ctx context.Context, dir string) error { + bs, ok := storage.UnwrapStore(b.store).(storage.BackupStore) + if !ok { + return fmt.Errorf("storage backend does not support backup operations") + } + return bs.BackupDatabase(ctx, dir) +} + // runBackupExport performs a Dolt-native backup to .beads/backup/. // Returns the updated state. -func runBackupExport(ctx context.Context, force bool) (*backupState, error) { +func runBackupExport(ctx context.Context, backend localBackupBackend, force bool) (*backupState, error) { dir, err := backupDir() if err != nil { return nil, err @@ -137,7 +163,7 @@ func runBackupExport(ctx context.Context, force bool) (*backupState, error) { // Change detection: skip if nothing changed (unless forced) if !force { - currentCommit, err := store.GetCurrentCommit(ctx) + currentCommit, err := backend.CurrentCommit(ctx) if err != nil { return nil, fmt.Errorf("failed to get current commit: %w", err) } @@ -147,12 +173,7 @@ func runBackupExport(ctx context.Context, force bool) (*backupState, error) { } } - bs, ok := storage.UnwrapStore(store).(storage.BackupStore) - if !ok { - return nil, fmt.Errorf("storage backend does not support backup operations") - } - - if err := bs.BackupDatabase(ctx, dir); err != nil { + if err := backend.BackupToDir(ctx, dir); err != nil { // Persist the attempt time even on failure so the throttle // interval (checked by maybeAutoBackup via state.Timestamp) // applies to the next command. Without this, a sync that keeps @@ -170,7 +191,7 @@ func runBackupExport(ctx context.Context, force bool) (*backupState, error) { } // Update watermarks - currentCommit, err := store.GetCurrentCommit(ctx) + currentCommit, err := backend.CurrentCommit(ctx) if err != nil { return nil, fmt.Errorf("failed to get current commit for state: %w", err) } diff --git a/cmd/bd/backup_export_failure_test.go b/cmd/bd/backup_export_failure_test.go index 174756bf8..5fd22c081 100644 --- a/cmd/bd/backup_export_failure_test.go +++ b/cmd/bd/backup_export_failure_test.go @@ -63,13 +63,10 @@ func TestRunBackupExport_PersistsThrottleOnFailure(t *testing.T) { t.Fatalf("precondition: expected zero timestamp, got %v", st.Timestamp) } - oldStore := store fake := &failingBackupStore{commit: "deadbeef", backupErr: errors.New("sync to backup: server too busy")} - store = fake - t.Cleanup(func() { store = oldStore }) before := time.Now().UTC() - if _, err := runBackupExport(context.Background(), true); err == nil { + if _, err := runBackupExport(context.Background(), directLocalBackup{store: fake}, true); err == nil { t.Fatal("expected runBackupExport to return the sync error, got nil") } if fake.backupCalls != 1 { diff --git a/cmd/bd/backup_lock.go b/cmd/bd/backup_lock.go new file mode 100644 index 000000000..832159ef2 --- /dev/null +++ b/cmd/bd/backup_lock.go @@ -0,0 +1,71 @@ +package main + +import ( + "errors" + "fmt" + "os" + "path/filepath" + "time" + + "github.com/steveyegge/beads/internal/beads" + "github.com/steveyegge/beads/internal/lockfile" +) + +// The backup lock serializes the two writers of a workspace's Dolt backup: +// post-command auto-backup and an explicit `bd backup sync`. Both run under the +// command's SHARED workspace gate, so the gate alone lets them overlap, and the +// auto-backup body does remove → add → sync on one fixed remote name — two +// concurrent runs race each other's registration and full-sync the database +// twice. Restore needs no part in it: it holds the workspace gate EXCLUSIVELY, +// which already excludes every command that could take this lock. +// +// The two callers wait differently, and deliberately: +// - auto-backup does not wait. Another backup running means one is already +// happening; it skips, and the throttle picks it up on a later command. +// - `bd backup sync` waits up to backupLockWait — the same bound an exclusive +// workspace-gate acquisition (restore) uses — and then fails with an error +// naming the busy lock, so an operator is never told it synced when it +// did not. +const backupLockFileName = "backup.lock" + +// backupLockWait bounds how long `bd backup sync` waits for another backup. +// A var so tests can shorten it. +var backupLockWait = exclusiveGateWait + +// errBackupBusy is returned when another backup holds the lock past the wait. +var errBackupBusy = errors.New("another backup is running for this workspace") + +const backupLockPoll = 50 * time.Millisecond + +// acquireBackupLock takes the workspace's backup lock, polling up to wait +// (zero means one non-blocking attempt). The returned release must be called. +func acquireBackupLock(wait time.Duration) (release func(), err error) { + beadsDir := beads.FindBeadsDir() + if beadsDir == "" { + return nil, fmt.Errorf("backup lock: %s", activeWorkspaceNotFoundError()) + } + path := filepath.Join(beadsDir, backupLockFileName) + f, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0o600) //nolint:gosec // path is the workspace's own lock file + if err != nil { + return nil, fmt.Errorf("backup lock: open %s: %w", path, err) + } + deadline := time.Now().Add(wait) + for { + err := lockfile.FlockExclusiveNonBlocking(f) + if err == nil { + return func() { + _ = lockfile.FlockUnlock(f) + _ = f.Close() + }, nil + } + if !lockfile.IsLocked(err) { + _ = f.Close() + return nil, fmt.Errorf("backup lock: %s: %w", path, err) + } + if !time.Now().Before(deadline) { + _ = f.Close() + return nil, errBackupBusy + } + time.Sleep(backupLockPoll) + } +} diff --git a/cmd/bd/backup_lock_test.go b/cmd/bd/backup_lock_test.go new file mode 100644 index 000000000..2248b45ad --- /dev/null +++ b/cmd/bd/backup_lock_test.go @@ -0,0 +1,133 @@ +package main + +import ( + "context" + "errors" + "os" + "strings" + "testing" + "time" +) + +// TestBackupLockSerializesBackups pins the lock itself: a second holder is +// refused while the first holds it, a bounded wait gives up with errBackupBusy, +// and releasing makes it available again. +// +// Cannot be parallel: prepareBackupStatusTest mutates package globals. +func TestBackupLockSerializesBackups(t *testing.T) { + prepareBackupStatusTest(t) + + release, err := acquireBackupLock(0) + if err != nil { + t.Fatalf("first acquisition: %v", err) + } + if _, err := acquireBackupLock(0); !errors.Is(err, errBackupBusy) { + t.Fatalf("non-blocking acquisition while held = %v, want errBackupBusy", err) + } + start := time.Now() + if _, err := acquireBackupLock(150 * time.Millisecond); !errors.Is(err, errBackupBusy) { + t.Fatalf("bounded wait while held = %v, want errBackupBusy", err) + } + if waited := time.Since(start); waited < 100*time.Millisecond { + t.Fatalf("bounded wait returned after %s; it must poll for the wait it was given", waited) + } + release() + + again, err := acquireBackupLock(0) + if err != nil { + t.Fatalf("acquisition after release: %v", err) + } + again() +} + +// TestAutoBackupSkipsWhileAnotherBackupRuns: auto-backup never waits on the +// lock. With it held the backup does not run and the throttle state is left +// alone, so the next command retries; once released, the same call backs up. +func TestAutoBackupSkipsWhileAnotherBackupRuns(t *testing.T) { + prepareBackupStatusTest(t) + t.Setenv("BD_BACKUP_ENABLED", "true") + t.Setenv("BD_BACKUP_INTERVAL", "1ms") + t.Setenv("BD_GIT_HOOK", "") + initConfigForTest(t) + + oldStore := store + fake := &failingBackupStore{commit: "c1"} + store = fake + t.Cleanup(func() { store = oldStore }) + + release, err := acquireBackupLock(0) + if err != nil { + t.Fatalf("hold the backup lock: %v", err) + } + maybeAutoBackup(context.Background()) + if fake.backupCalls != 0 { + t.Fatalf("auto-backup ran %d backup(s) while another backup held the lock", fake.backupCalls) + } + dir, err := backupDir() + if err != nil { + t.Fatal(err) + } + if st, _ := loadBackupState(dir); !st.Timestamp.IsZero() { + t.Fatalf("a skipped auto-backup advanced the throttle to %v", st.Timestamp) + } + + release() + maybeAutoBackup(context.Background()) + if fake.backupCalls != 1 { + t.Fatalf("auto-backup ran %d backup(s) after the lock was released, want 1", fake.backupCalls) + } +} + +// TestBackupSyncFailsWhenAnotherBackupHoldsTheLock: an explicit sync waits +// backupLockWait for another backup and then fails loudly, before touching the +// store — it never reports a sync it did not do. +func TestBackupSyncFailsWhenAnotherBackupHoldsTheLock(t *testing.T) { + prepareBackupStatusTest(t) + oldWait := backupLockWait + backupLockWait = 100 * time.Millisecond + t.Cleanup(func() { backupLockWait = oldWait }) + + oldStore := store + fake := &failingBackupStore{commit: "c1"} + store = fake + t.Cleanup(func() { store = oldStore }) + + release, err := acquireBackupLock(0) + if err != nil { + t.Fatalf("hold the backup lock: %v", err) + } + defer release() + + var runErr error + stderr := captureStderrForBackupLockTest(t, func() { + runErr = backupSyncCmd.RunE(backupSyncCmd, nil) + }) + if runErr == nil { + t.Fatal("bd backup sync succeeded while another backup held the lock") + } + if !strings.Contains(stderr, "another backup is running") { + t.Fatalf("sync refusal does not say why; stderr:\n%s", stderr) + } + if fake.backupCalls != 0 { + t.Fatalf("refused sync still reached the store (%d backup calls)", fake.backupCalls) + } +} + +func captureStderrForBackupLockTest(t *testing.T, fn func()) string { + t.Helper() + stdioMutex.Lock() + defer stdioMutex.Unlock() + r, w, err := os.Pipe() + if err != nil { + t.Fatal(err) + } + old := os.Stderr + os.Stderr = w + fn() + os.Stderr = old + _ = w.Close() + buf := make([]byte, 1<<16) + n, _ := r.Read(buf) + _ = r.Close() + return string(buf[:n]) +} diff --git a/cmd/bd/backup_proxied_server.go b/cmd/bd/backup_proxied_server.go index 13bfaacac..0cce7e50c 100644 --- a/cmd/bd/backup_proxied_server.go +++ b/cmd/bd/backup_proxied_server.go @@ -12,6 +12,7 @@ import ( "github.com/steveyegge/beads/internal/beads" "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/debug" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" "github.com/steveyegge/beads/internal/storage/issueops" @@ -47,19 +48,83 @@ const proxiedBackupTargetName = defaultDoltBackupName // Defense in depth, not a second policy: the rule and its wording come from the // same registry row the pre-provider gate consults, so the two cannot drift. func requireLocalProxiedBackup(path string) error { - row, ok := LookupCapabilityRow(path, "") + rule, ok := proxiedBackupRule(path) if !ok { // Unreachable: TestProxyCapabilityRegistryCoversCommandTree fails the // build for a backup path with no row. Refuse rather than guess. return HandleErrorRespectJSON("%s has no proxied-server capability row", path) } - rule := row.ruleFor(resolveProxiedTopology(beads.FindBeadsDir())) if rule.Outcome == ProxyOutcomeHonored { return nil } return HandleProxyCapabilityError(proxyCapabilityErrorFor(rule)) } +// proxiedBackupRule resolves the registry rule for a backup path on this +// workspace's proxied topology. ok is false only for a path with no row. +func proxiedBackupRule(path string) (proxyCapabilityRule, bool) { + row, ok := LookupCapabilityRow(path, "") + if !ok { + return proxyCapabilityRule{}, false + } + return row.ruleFor(resolveProxiedTopology(beads.FindBeadsDir())), true +} + +// proxiedAutoBackupBackend returns the backend post-command auto-backup runs +// against on a proxied-server workspace, or ok=false when it must not run. +// +// Auto-backup is a `bd backup sync` nobody typed, so it is honored exactly +// where that verb is — the "backup sync" registry row, i.e. managed-local +// only. Anywhere else the backup remote it registers would land on a server bd +// does not own, where it is global to every client; that is the storm the +// explicit verbs refuse by design, and an implicit trigger must not reopen it. +// +// It never opens a provider of its own. A command that did not open one (or +// closed it, as restore does) has no live topology to back up, and opening one +// here would relaunch a server the command may just have stopped. +func proxiedAutoBackupBackend() (localBackupBackend, bool) { + if rule, ok := proxiedBackupRule("backup sync"); !ok || rule.Outcome != ProxyOutcomeHonored { + debug.Logf("backup: skipping auto-backup — proxied topology does not honor backup sync\n") + return nil, false + } + if uowProvider == nil { + return nil, false + } + mp, ok := uowProvider.(uow.MaintenanceProvider) + if !ok { + debug.Logf("backup: skipping auto-backup — provider %T has no maintenance seam\n", uowProvider) + return nil, false + } + return proxiedLocalBackup{provider: mp}, true +} + +// proxiedLocalBackup is localBackupBackend over the proxied provider's +// non-transactional seam, the one `bd backup sync` uses. +// +// No pre-backup commit: change detection compares HEAD, and every proxied +// write transaction already ends in DOLT_COMMIT('-Am') (uow doltServerTx), so +// HEAD is where the data is — the invariant direct mode gets from running +// auto-commit just before auto-backup. +type proxiedLocalBackup struct { + provider uow.MaintenanceProvider +} + +func (b proxiedLocalBackup) CurrentCommit(ctx context.Context) (string, error) { + var hash string + err := b.provider.RunNonTx(ctx, func(ctx context.Context, conn *sql.Conn) error { + var err error + hash, err = versioncontrolops.CurrentCommit(ctx, conn) + return err + }) + return hash, err +} + +func (b proxiedLocalBackup) BackupToDir(ctx context.Context, dir string) error { + return b.provider.RunNonTx(ctx, func(ctx context.Context, conn *sql.Conn) error { + return versioncontrolops.BackupToDir(ctx, conn, conn, dir) + }) +} + // proxiedActiveDatabase asks the server which database the provider opened, // rather than re-deriving it from the workspace config. The provider already // resolved that question (metadata.json, then --database/--db), and a second @@ -203,7 +268,7 @@ func runBackupRemoveProxied(ctx context.Context) error { return err } // auto-export may have registered a second remote at the same URL. - _ = versioncontrolops.BackupRemove(ctx, conn, "backup_export") + _ = versioncontrolops.BackupRemove(ctx, conn, versioncontrolops.ExportBackupName) return nil }) if err != nil { diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 345639a35..0943da305 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -2000,6 +2000,15 @@ var rootCmd = &cobra.Command{ if shouldAutoPruneEventsJournal(cmd) { maybeAutoPruneEventsJournal(rootCtx, beads.FindBeadsDir()) } + // Auto-backup, through the provider this command opened. Same + // gate as the direct arm's maintenance net below (strict + // --readonly, `bd serve` and a migration freeze all skip it), + // plus previews: a --dry-run must not register a backup remote + // or write backup state. proxiedAutoBackupBackend decides the + // topology, so this stays inert off managed-local. + if runsPostCommandMaintenance(cmd.Name(), readonlyMode) && !isPreviewCommand(cmd) && !commandFreeze.Frozen() { + runPostRunAutoBackup(rootCtx) + } if uowProvider != nil { _ = uowProvider.Close(rootCtx) uowProvider = nil diff --git a/internal/storage/dolt/store.go b/internal/storage/dolt/store.go index d077cff5e..1c5a922ff 100644 --- a/internal/storage/dolt/store.go +++ b/internal/storage/dolt/store.go @@ -1392,44 +1392,12 @@ func (s *DoltStore) BackupRemove(ctx context.Context, name string) error { // BackupDatabase registers dir as a file:// Dolt backup remote and syncs // the full database to it, preserving complete commit history. func (s *DoltStore) BackupDatabase(ctx context.Context, dir string) error { - info, err := os.Stat(dir) - if err != nil { - return fmt.Errorf("backup destination does not exist: %w", err) - } - if !info.IsDir() { - return fmt.Errorf("backup destination is not a directory: %s", dir) - } - - backupURL, err := versioncontrolops.DirToFileURL(dir) - if err != nil { - return err - } - backupName := "backup_export" - syncDB, err := s.oneShotConn(0) if err != nil { return err } defer syncDB.Close() - - // Register as a backup remote (idempotent — remove first if exists). - _ = versioncontrolops.BackupRemove(ctx, s.db, backupName) - if err := versioncontrolops.BackupAdd(ctx, s.db, backupName, backupURL); err != nil { - // Another backup (e.g. "default" registered by `bd backup init`) may - // already point to this URL. In that case, sync using the existing - // remote name rather than failing. - if conflict := versioncontrolops.ExtractAddressConflictName(err); conflict != "" { - if syncErr := versioncontrolops.BackupSync(ctx, syncDB, conflict); syncErr != nil { - return fmt.Errorf("sync to backup: %w", syncErr) - } - return nil - } - return fmt.Errorf("register backup remote: %w", err) - } - if err := versioncontrolops.BackupSync(ctx, syncDB, backupName); err != nil { - return fmt.Errorf("sync to backup: %w", err) - } - return nil + return versioncontrolops.BackupToDir(ctx, s.db, syncDB, dir) } // RestoreDatabase restores the database from a Dolt backup at dir. diff --git a/internal/storage/dolt/versioned.go b/internal/storage/dolt/versioned.go index 7c1678355..ad0c4a7a6 100644 --- a/internal/storage/dolt/versioned.go +++ b/internal/storage/dolt/versioned.go @@ -188,12 +188,7 @@ func (s *DoltStore) ListBranches(ctx context.Context) ([]string, error) { // GetCurrentCommit returns the hash of the current HEAD commit. // Implements storage.VersionedStorage. func (s *DoltStore) GetCurrentCommit(ctx context.Context) (string, error) { - var hash string - err := s.db.QueryRowContext(ctx, "SELECT DOLT_HASHOF('HEAD')").Scan(&hash) - if err != nil { - return "", fmt.Errorf("failed to get current commit: %w", err) - } - return hash, nil + return versioncontrolops.CurrentCommit(ctx, s.db) } // GetStateHash returns a hash of the entire database including the working diff --git a/internal/storage/embeddeddolt/version_control.go b/internal/storage/embeddeddolt/version_control.go index 47f55cc87..130cf8894 100644 --- a/internal/storage/embeddeddolt/version_control.go +++ b/internal/storage/embeddeddolt/version_control.go @@ -749,39 +749,8 @@ func (s *EmbeddedDoltStore) BackupRemove(ctx context.Context, name string) error // the database to it. The dir must exist locally. This preserves full Dolt // commit history. func (s *EmbeddedDoltStore) BackupDatabase(ctx context.Context, dir string) error { - info, err := os.Stat(dir) - if err != nil { - return fmt.Errorf("backup destination does not exist: %w", err) - } - if !info.IsDir() { - return fmt.Errorf("backup destination is not a directory: %s", dir) - } - - backupURL, err := versioncontrolops.DirToFileURL(dir) - if err != nil { - return err - } - backupName := "backup_export" - return s.withMutatingDBConn(ctx, func(db versioncontrolops.DBConn) error { - // Register as a backup remote (idempotent — remove first if exists). - _ = versioncontrolops.BackupRemove(ctx, db, backupName) - if err := versioncontrolops.BackupAdd(ctx, db, backupName, backupURL); err != nil { - // Another backup (e.g. "default" registered by `bd backup init`) may - // already point to this URL. In that case, sync using the existing - // remote name rather than failing. - if conflict := versioncontrolops.ExtractAddressConflictName(err); conflict != "" { - if syncErr := versioncontrolops.BackupSync(ctx, db, conflict); syncErr != nil { - return fmt.Errorf("sync to backup: %w", syncErr) - } - return nil - } - return fmt.Errorf("register backup remote: %w", err) - } - if err := versioncontrolops.BackupSync(ctx, db, backupName); err != nil { - return fmt.Errorf("sync to backup: %w", err) - } - return nil + return versioncontrolops.BackupToDir(ctx, db, db, dir) }) } diff --git a/internal/storage/versioncontrolops/backup.go b/internal/storage/versioncontrolops/backup.go index 56acacf27..d8dd44a4d 100644 --- a/internal/storage/versioncontrolops/backup.go +++ b/internal/storage/versioncontrolops/backup.go @@ -3,10 +3,16 @@ package versioncontrolops import ( "context" "fmt" + "os" "path/filepath" "strings" ) +// ExportBackupName is the Dolt backup remote the auto-backup path registers +// for its local backup directory. `bd backup init` registers a different +// name, so the two can coexist until they point at the same address. +const ExportBackupName = "backup_export" + // BackupAdd registers a Dolt backup destination. func BackupAdd(ctx context.Context, db DBConn, name, url string) error { if _, err := db.ExecContext(ctx, "CALL DOLT_BACKUP('add', ?, ?)", name, url); err != nil { @@ -47,6 +53,55 @@ func BackupRestore(ctx context.Context, db DBConn, url, dbName string, force boo return nil } +// CurrentCommit returns the hash of the current HEAD commit. +func CurrentCommit(ctx context.Context, db DBConn) (string, error) { + var hash string + if err := db.QueryRowContext(ctx, "SELECT DOLT_HASHOF('HEAD')").Scan(&hash); err != nil { + return "", fmt.Errorf("failed to get current commit: %w", err) + } + return hash, nil +} + +// BackupToDir registers dir as the ExportBackupName file:// backup remote and +// syncs the full database to it, preserving complete commit history. dir must +// already exist on this machine. +// +// register carries the remote add/remove; sync carries DOLT_BACKUP('sync'), +// which streams the database and can outlive a pooled connection's read +// deadline, so a caller that has one hands in a long-timeout connection there. +// A caller with a single connection passes it for both. +// +// Registration is idempotent (remove, then add). When another remote — e.g. +// the one `bd backup init` registered — already holds the same address, Dolt +// refuses the add; the sync then goes through that remote's name instead. +func BackupToDir(ctx context.Context, register, sync DBConn, dir string) error { + info, err := os.Stat(dir) + if err != nil { + return fmt.Errorf("backup destination does not exist: %w", err) + } + if !info.IsDir() { + return fmt.Errorf("backup destination is not a directory: %s", dir) + } + backupURL, err := DirToFileURL(dir) + if err != nil { + return err + } + + name := ExportBackupName + _ = BackupRemove(ctx, register, name) + if err := BackupAdd(ctx, register, name, backupURL); err != nil { + conflict := ExtractAddressConflictName(err) + if conflict == "" { + return fmt.Errorf("register backup remote: %w", err) + } + name = conflict + } + if err := BackupSync(ctx, sync, name); err != nil { + return fmt.Errorf("sync to backup: %w", err) + } + return nil +} + // DirToFileURL resolves dir to an absolute path and returns a file:// URL. // // An input that already carries a scheme is rejected rather than prefixed. diff --git a/internal/storage/versioncontrolops/backup_test.go b/internal/storage/versioncontrolops/backup_test.go index d58df7660..907b0c29c 100644 --- a/internal/storage/versioncontrolops/backup_test.go +++ b/internal/storage/versioncontrolops/backup_test.go @@ -1,7 +1,10 @@ package versioncontrolops import ( + "context" "fmt" + "os" + "path/filepath" "strings" "testing" ) @@ -74,3 +77,29 @@ func TestExtractAddressConflictName(t *testing.T) { }) } } + +// TestBackupToDirRefusesMissingOrFileDestination pins the local-directory +// precondition every auto-backup caller (embedded, sql-server and proxied) +// now shares. It fails before any SQL is issued, so nil connections are safe: +// a regression that reached the server first would panic here. +func TestBackupToDirRefusesMissingOrFileDestination(t *testing.T) { + dir := t.TempDir() + file := filepath.Join(dir, "not-a-dir") + if err := os.WriteFile(file, nil, 0o600); err != nil { + t.Fatal(err) + } + + for _, tc := range []struct { + name, dir, want string + }{ + {name: "missing", dir: filepath.Join(dir, "absent"), want: "does not exist"}, + {name: "file", dir: file, want: "is not a directory"}, + } { + t.Run(tc.name, func(t *testing.T) { + err := BackupToDir(context.Background(), nil, nil, tc.dir) + if err == nil || !strings.Contains(err.Error(), tc.want) { + t.Fatalf("BackupToDir(%q) error = %v, want one containing %q", tc.dir, err, tc.want) + } + }) + } +} From d15081d60e65f95752fa9cd821cd80498716d8d1 Mon Sep 17 00:00:00 2001 From: Gas City Adopt-PR Date: Tue, 29 Sep 2026 04:10:59 +0000 Subject: [PATCH 25/45] test(release-1.3.x): capability matrix expects proxied bd dolt commit to be honored after #6499 [NOT a cherry-pick] Release-only test adaptation. cmd/bd/topology_capability_matrix_test.go exists only on the release line (#6579 [release/1.3.0]); main has no copy, so the #6499 pick could not update it. Its "dolt commit" row still expected the proxy.dolt_commit.unsupported refusal on every proxied topology, which #6499 removes (capability_registry.go: permitted("dolt commit")). The row now expects HONORED on proxied topologies, matching the registry. Before this change TestManagedLocalProxiedCapabilityMatrix failed on the dolt commit cell, and every later refusal cell then reported "proxy started for a refused command" because the honored commit had started the proxy. No other expectation changes. Co-Authored-By: Claude Opus 5.5 --- cmd/bd/topology_capability_matrix_test.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cmd/bd/topology_capability_matrix_test.go b/cmd/bd/topology_capability_matrix_test.go index 5d79a2bd0..b7d11c5f3 100644 --- a/cmd/bd/topology_capability_matrix_test.go +++ b/cmd/bd/topology_capability_matrix_test.go @@ -312,10 +312,10 @@ var capabilityProbes = []probe{ direct: expectation{ outcome: outcomeHonored, reason: reasonNA, }, + // Routed on every proxied topology by runDoltCommitProxiedServer + // (#6499): it is the flush point dolt.auto-commit=batch/off defers to. proxied: expectation{ - outcome: outcomeRefusedTyped, code: "proxy.dolt_commit.unsupported", - substr: "dolt commit is not supported in proxied-server mode", - reason: reasonUnimplemented, + outcome: outcomeHonored, reason: reasonNA, }, }, { From 70867888ffee312ad9df6d15d0c0c0b0164c4f5d Mon Sep 17 00:00:00 2001 From: Gas City Adopt-PR Date: Tue, 29 Sep 2026 05:20:09 +0000 Subject: [PATCH 26/45] test(release-1.3.x): drop dolt commit from the release-only direct-only refusal loop after #6499 [NOT a cherry-pick] Release-only test adaptation. On main, TestHistoryDirectOnlyRefusalContract iterates the keys of its expected map, so #6499 only had to delete the "dolt commit" entry. The release copy (adapted in 2b27259cb4) iterates an explicit path list instead, which the pick could not see. That list still had "dolt commit", which #6499 now permits on the proxied route, so the test read an empty refusal and failed: dolt commit refusal = "" (unexpected end of JSON input) Seen in the nightly full-test and pr-linux runs dispatched on hotfix/1.3.1-batch2. The only change is removing "dolt commit" from that list. The refusal contract for every other path is unchanged. Co-Authored-By: Claude Opus 5.5 --- cmd/bd/capability_registry_history_test.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmd/bd/capability_registry_history_test.go b/cmd/bd/capability_registry_history_test.go index 3884b6765..ad212600c 100644 --- a/cmd/bd/capability_registry_history_test.go +++ b/cmd/bd/capability_registry_history_test.go @@ -62,7 +62,7 @@ func TestHistoryDirectOnlyRefusalContract(t *testing.T) { oldProvider := uowProvider oldJSON := jsonOutput t.Cleanup(func() { uowProvider = oldProvider; jsonOutput = oldJSON }) - for _, path := range []string{"branch", "conflicts", "repo", "federation", "vc", "flatten", "dolt push", "dolt pull", "dolt commit", "dolt remote add", "sync"} { + for _, path := range []string{"branch", "conflicts", "repo", "federation", "vc", "flatten", "dolt push", "dolt pull", "dolt remote add", "sync"} { parts := strings.Split(path, " ") root := &cobra.Command{Use: "bd"} cmd := &cobra.Command{Use: parts[0]} From 84e6faaa2d86a94a1482beede8fb7da434d0911a Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 29 Sep 2026 01:08:10 -0700 Subject: [PATCH 27/45] fix(sql): classify bd sql statements with the SQL parser; never drop result sets (#6932) Proxied bd sql misclassified WITH [RECURSIVE] ... SELECT as a write and silently returned no rows; CALL result sets were dropped on both paths. Classify statements with the dolthub/vitess parser (internal/storage/sqlclass: Read/Write/Mixed); Mixed commits and renders any rows, so no path silently drops a result set. Co-Authored-By: Claude Opus 5.5 (cherry picked from commit c4b1cb699df20a6ed41b07e3a0b05dec22c534d7) hand-resolved: CHANGELOG entry placed under the hotfix's [Unreleased] ### Fixed; MODULE.bazel and BUILD.bazel files dropped (no Bazel on hotfix/1.3.1). Go code applied unchanged. --- CHANGELOG.md | 18 +++ cmd/bd/sql.go | 28 +++-- cmd/bd/sql_proxied_integration_test.go | 77 +++++++++++++ cmd/bd/sql_proxied_server.go | 124 +++++++++------------ go.mod | 2 +- internal/storage/sqlclass/sqlclass.go | 88 +++++++++++++++ internal/storage/sqlclass/sqlclass_test.go | 67 +++++++++++ 7 files changed, 320 insertions(+), 84 deletions(-) create mode 100644 internal/storage/sqlclass/sqlclass.go create mode 100644 internal/storage/sqlclass/sqlclass_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 95333335f..dae3513ba 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,6 +126,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **`bd sql` no longer drops the rows of CTE queries or CALL result sets** + ([#6932](https://github.com/gastownhall/beads/pull/6932)). In proxied-server + mode, `WITH name(cols) AS (...) SELECT ...` and `WITH RECURSIVE ...` queries + were misread as writes and printed `OK, 0 rows affected` with no error; they + now return their rows. Statements are now classified with the Dolt SQL + parser (shared by the proxied and direct paths): reads return rows, plain + writes (including CTE-prefixed `UPDATE`/`DELETE`) commit and report + `rows_affected`, and anything that may both write and return rows (`CALL`, + `EXPLAIN ANALYZE` of a write, `RETURNING`) or that the parser cannot + classify is committed and prints whatever rows it returns, so + `CALL DOLT_BRANCH(...)`-style result sets are rendered instead of discarded. + Visible changes for scripts: in direct server mode a multi-statement write, + or a single write statement the parser cannot classify, now prints `OK` / + `{"status":"ok"}` (as proxied mode already did) instead of + `OK, N rows affected` / `{"rows_affected":N}`; and `--readonly` now refuses + statements it cannot parse (for example `PRAGMA`) instead of treating them + as reads. + - Unblocking two blockers of one dependent at the same time — closing both, or a close racing a `bd dep remove` or a delete of the other — no longer leaves the dependent stuck as blocked and hidden from `bd ready` until diff --git a/cmd/bd/sql.go b/cmd/bd/sql.go index 366072bc6..34dedfe09 100644 --- a/cmd/bd/sql.go +++ b/cmd/bd/sql.go @@ -9,6 +9,7 @@ import ( "github.com/spf13/cobra" "github.com/steveyegge/beads/internal/metrics" "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/sqlclass" ) var sqlCmd = &cobra.Command{ @@ -25,9 +26,11 @@ Examples: bd sql 'DELETE FROM dirty_issues WHERE issue_id = "bd-abc123"' bd sql --csv 'SELECT id, title, status FROM issues' -The query is passed directly to the database. SELECT queries return results as a -table (or JSON/CSV with --json/--csv). Non-SELECT queries (INSERT, UPDATE, DELETE) -report the number of rows affected. +The query is passed directly to the database. Reads (SELECT, WITH ... SELECT, +SHOW, EXPLAIN) return results as a table (or JSON/CSV with --json/--csv). Writes +(INSERT, UPDATE, DELETE, DDL) report the number of rows affected. Statements +that may both write and return rows, such as CALL, are committed and print any +rows they return (or "OK"). In proxied-server mode, multiple statements separated by ';' run as a single committed batch and report "OK", and --database runs the query against a @@ -71,15 +74,13 @@ WARNING: Direct database access bypasses the storage layer. Use with caution.`, ctx := rootCtx - trimmed := strings.TrimSpace(strings.ToUpper(query)) - isRead := strings.HasPrefix(trimmed, "SELECT") || - strings.HasPrefix(trimmed, "EXPLAIN") || - strings.HasPrefix(trimmed, "PRAGMA") || - strings.HasPrefix(trimmed, "SHOW") || - strings.HasPrefix(trimmed, "DESCRIBE") || - strings.HasPrefix(trimmed, "WITH") - - if isRead { + // Reads and statements that may return rows go through Query so a + // result set is always rendered; only plain writes use Exec. + kind := sqlclass.Classify(query) + if kind != sqlclass.Write { + if kind == sqlclass.Mixed { + CheckReadonly("sql") + } rows, err := db.QueryContext(ctx, query) if err != nil { return HandleErrorRespectJSON("query error: %v", err) @@ -90,6 +91,9 @@ WARNING: Direct database access bypasses the storage layer. Use with caution.`, if err != nil { return HandleErrorRespectJSON("getting columns: %v", err) } + if kind == sqlclass.Mixed && len(columns) == 0 { + return printSQLStatusOK() + } allRows := make([]map[string]interface{}, 0) for rows.Next() { diff --git a/cmd/bd/sql_proxied_integration_test.go b/cmd/bd/sql_proxied_integration_test.go index 89bb6ec0d..6dec9e7f0 100644 --- a/cmd/bd/sql_proxied_integration_test.go +++ b/cmd/bd/sql_proxied_integration_test.go @@ -179,3 +179,80 @@ func sqlValueEquals(v any, want float64) bool { return false } } + +// TestProxiedServerSQLStatementRouting pins how proxied `bd sql` routes +// statements that a prefix scan misreads: CTE reads must return their rows, +// CTE writes must commit, and a statement that writes and returns a result set +// must render that result set instead of reporting "0 rows affected". +func TestProxiedServerSQLStatementRouting(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + p := newSharedProxiedProject(t, bd, "sqlcte") + + one := bdProxiedCreate(t, bd, p.dir, "CTE issue one", "--type", "task", "--priority", "1") + two := bdProxiedCreate(t, bd, p.dir, "CTE issue two", "--type", "task", "--priority", "2") + + t.Run("cte_column_list_select_returns_rows", func(t *testing.T) { + rows := bdProxiedSQLJSON(t, bd, p.dir, + "WITH t(issue_id) AS (SELECT id FROM issues WHERE id = '"+one.ID+"') SELECT issue_id FROM t") + if len(rows) != 1 { + t.Fatalf("expected 1 row, got %d: %v", len(rows), rows) + } + if rows[0]["issue_id"] != one.ID { + t.Errorf("expected issue_id=%q, got %v", one.ID, rows[0]["issue_id"]) + } + }) + + t.Run("recursive_cte_select_returns_rows", func(t *testing.T) { + query := "WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < 3) SELECT i FROM n" + rows := bdProxiedSQLJSON(t, bd, p.dir, query) + if len(rows) != 3 { + t.Fatalf("expected 3 rows, got %d: %v", len(rows), rows) + } + stdout, _ := bdProxiedSQL(t, bd, p.dir, query) + if !strings.Contains(stdout, "(3 rows)") { + t.Errorf("expected '(3 rows)' in table output, got:\n%s", stdout) + } + }) + + t.Run("cte_write_commits", func(t *testing.T) { + stdout, _ := bdProxiedSQL(t, bd, p.dir, "--json", + "WITH t(issue_id) AS (SELECT id FROM issues WHERE id = '"+two.ID+"') "+ + "UPDATE issues SET priority = 0 WHERE id IN (SELECT issue_id FROM t)") + start := strings.Index(stdout, "{") + if start < 0 { + t.Fatalf("no JSON object in exec output:\n%s", stdout) + } + var res map[string]interface{} + if err := json.Unmarshal([]byte(stdout[start:]), &res); err != nil { + t.Fatalf("parse exec JSON: %v\nraw: %s", err, stdout[start:]) + } + if !sqlValueEquals(res["rows_affected"], 1) { + t.Errorf("expected rows_affected=1, got %v", res["rows_affected"]) + } + + db := openProxiedDB(t, p) + var priority int + if err := db.QueryRow("SELECT priority FROM issues WHERE id = ?", two.ID).Scan(&priority); err != nil { + t.Fatalf("query priority for %s: %v", two.ID, err) + } + if priority != 0 { + t.Errorf("expected committed priority=0, got %d", priority) + } + }) + + t.Run("call_renders_result_set", func(t *testing.T) { + rows := bdProxiedSQLJSON(t, bd, p.dir, "CALL DOLT_BRANCH('sql-routing-probe')") + if len(rows) != 1 { + t.Fatalf("expected the procedure's 1 status row, got %d: %v", len(rows), rows) + } + if _, ok := rows[0]["status"]; !ok { + t.Errorf("expected a status column, got %v", rows[0]) + } + branches := bdProxiedSQLJSON(t, bd, p.dir, "SELECT name FROM dolt_branches WHERE name = 'sql-routing-probe'") + if len(branches) != 1 { + t.Errorf("expected the CALL to have created branch sql-routing-probe, got %v", branches) + } + }) +} diff --git a/cmd/bd/sql_proxied_server.go b/cmd/bd/sql_proxied_server.go index 5e43622df..894c7e37c 100644 --- a/cmd/bd/sql_proxied_server.go +++ b/cmd/bd/sql_proxied_server.go @@ -8,6 +8,7 @@ import ( "strings" "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/storage/sqlclass" "github.com/steveyegge/beads/internal/storage/uow" ) @@ -16,9 +17,16 @@ func runSQLProxiedServer(ctx context.Context, query string, csvOutput bool) erro return HandleError("proxied-server UOW provider not initialized") } - multiStatement := isMultiStatementSQL(query) + if isMultiStatementSQL(query) { + CheckReadonly("sql") + if _, err := runSQLProxiedExec(ctx, query); err != nil { + return err + } + return printSQLStatusOK() + } - if !multiStatement && sqlQueryIsRead(query) { + switch sqlclass.Classify(query) { + case sqlclass.Read: result, err := uow.RunTxRead(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (*domain.RawSQLResult, error) { return uw.RawSQLUseCase().Query(ctx, query) }) @@ -26,10 +34,45 @@ func runSQLProxiedServer(ctx context.Context, query string, csvOutput bool) erro return HandleErrorRespectJSON("query error: %v", err) } return renderRawSQLResult(result, csvOutput) + + case sqlclass.Write: + CheckReadonly("sql") + affected, err := runSQLProxiedExec(ctx, query) + if err != nil { + return err + } + if jsonOutput { + return outputJSON(map[string]interface{}{ + "rows_affected": affected, + }) + } + fmt.Printf("OK, %d rows affected\n", affected) + return nil + + default: + // The statement may write and may return rows. Run it through Query + // inside a committing transaction so a result set is rendered rather + // than silently discarded, and any write is committed rather than + // rolled back with a read transaction. + CheckReadonly("sql") + result, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (*domain.RawSQLResult, string, error) { + result, err := uw.RawSQLUseCase().Query(ctx, query) + if err != nil { + return nil, "", err + } + return result, "bd sql: " + query, nil + }) + if err != nil { + return HandleErrorRespectJSON("query error: %v", err) + } + if len(result.Columns) == 0 { + return printSQLStatusOK() + } + return renderRawSQLResult(result, csvOutput) } +} - CheckReadonly("sql") - +func runSQLProxiedExec(ctx context.Context, query string) (int64, error) { affected, err := uow.RunTxResult(ctx, uowProvider, func(ctx context.Context, uw uow.UnitOfWork) (int64, string, error) { affected, err := uw.RawSQLUseCase().Exec(ctx, query) if err != nil { @@ -38,26 +81,18 @@ func runSQLProxiedServer(ctx context.Context, query string, csvOutput bool) erro return affected, "bd sql: " + query, nil }) if err != nil { - return HandleErrorRespectJSON("exec error: %v", err) - } - - if multiStatement { - if jsonOutput { - return outputJSON(map[string]interface{}{ - "status": "ok", - }) - } - fmt.Println("OK") - return nil + return 0, HandleErrorRespectJSON("exec error: %v", err) } + return affected, nil +} +func printSQLStatusOK() error { if jsonOutput { return outputJSON(map[string]interface{}{ - "rows_affected": affected, + "status": "ok", }) } - - fmt.Printf("OK, %d rows affected\n", affected) + fmt.Println("OK") return nil } @@ -134,59 +169,6 @@ func topLevelStatementCount(query string) int { return count } -func sqlQueryIsRead(query string) bool { - trimmed := strings.TrimSpace(strings.ToUpper(query)) - switch { - case strings.HasPrefix(trimmed, "SELECT"), - strings.HasPrefix(trimmed, "EXPLAIN"), - strings.HasPrefix(trimmed, "PRAGMA"), - strings.HasPrefix(trimmed, "SHOW"), - strings.HasPrefix(trimmed, "DESCRIBE"): - return true - case strings.HasPrefix(trimmed, "WITH"): - return withOuterStatementIsRead(trimmed) - default: - return false - } -} - -func withOuterStatementIsRead(upperTrimmed string) bool { - depth := 0 - var quote byte - closedCTE := false - for i := 0; i < len(upperTrimmed); i++ { - c := upperTrimmed[i] - if quote != 0 { - if c == quote { - quote = 0 - } - continue - } - switch c { - case '\'', '"', '`': - quote = c - case '(': - depth++ - case ')': - depth-- - if depth == 0 { - closedCTE = true - } - case ' ', '\t', '\n', '\r', ',': - if c == ',' && depth == 0 { - closedCTE = false - } - default: - if depth == 0 && closedCTE { - rest := strings.TrimLeft(upperTrimmed[i:], " \t\n\r") - return strings.HasPrefix(rest, "SELECT") || - strings.HasPrefix(rest, "EXPLAIN") - } - } - } - return true -} - func renderRawSQLResult(result *domain.RawSQLResult, csvOutput bool) error { columns := result.Columns diff --git a/go.mod b/go.mod index 85f9a81ad..43870b367 100644 --- a/go.mod +++ b/go.mod @@ -155,7 +155,7 @@ require ( github.com/dolthub/gozstd v0.0.0-20240423170813-23a2903bca63 // indirect github.com/dolthub/ishell v0.0.0-20260414231531-5f031e3e9037 // indirect github.com/dolthub/jsonpath v0.0.2-0.20240227200619-19675ab05c71 // indirect - github.com/dolthub/vitess v0.0.0-20260728212736-0542037326d7 // indirect + github.com/dolthub/vitess v0.0.0-20260728212736-0542037326d7 github.com/dustin/go-humanize v1.0.1 // indirect github.com/ebitengine/purego v0.10.0 // indirect github.com/edsrzf/mmap-go v1.2.0 // indirect diff --git a/internal/storage/sqlclass/sqlclass.go b/internal/storage/sqlclass/sqlclass.go new file mode 100644 index 000000000..1c677e33d --- /dev/null +++ b/internal/storage/sqlclass/sqlclass.go @@ -0,0 +1,88 @@ +// Package sqlclass classifies raw SQL statements for `bd sql` with the same +// MySQL grammar the Dolt server parses, so callers can route a statement to a +// read transaction, a committing exec, or a committing query without prefix +// scanning. +package sqlclass + +import ( + "github.com/dolthub/vitess/go/vt/sqlparser" +) + +// Kind says how a single SQL statement must be executed so that writes are +// committed and result sets are never dropped. +type Kind int + +const ( + // Read changes nothing and returns a result set: run it in a read + // transaction and render its rows. + Read Kind = iota + // Write changes data and returns no result set: execute it in a + // committing transaction and report the rows affected. + Write + // Mixed may change data and may also return a result set (CALL, EXPLAIN + // ANALYZE of a write, RETURNING, or anything the parser cannot classify): + // run it in a committing transaction and render any rows it returns. + Mixed +) + +// String returns the kind's name for test and diagnostic output. +func (k Kind) String() string { + switch k { + case Read: + return "read" + case Write: + return "write" + default: + return "mixed" + } +} + +// Classify classifies a single SQL statement by its main statement, so CTEs +// (WITH [RECURSIVE] name(cols) AS (...)), comments, and string literals that +// contain keywords do not affect the result. Input the parser rejects is +// Mixed: it may still be valid for the server, and Mixed never discards rows +// or rolls back writes. +func Classify(query string) Kind { + stmt, err := sqlparser.Parse(query) + if err != nil || stmt == nil { + return Mixed + } + return classify(stmt) +} + +func classify(stmt sqlparser.Statement) Kind { + switch s := stmt.(type) { + case *sqlparser.Select: + if s.Into != nil { + return Write + } + return Read + case *sqlparser.SetOp, *sqlparser.Show, *sqlparser.OtherRead: + return Read + case *sqlparser.Explain: + if !s.Analyze { + return Read + } + if s.Statement != nil && classify(s.Statement) == Read { + return Read + } + return Mixed + case *sqlparser.Insert: + return writeUnlessReturning(len(s.Returning)) + case *sqlparser.Update: + return writeUnlessReturning(len(s.Returning)) + case *sqlparser.Delete: + return writeUnlessReturning(len(s.Returning)) + case *sqlparser.DDL, *sqlparser.AlterTable, *sqlparser.DBDDL, *sqlparser.Set, *sqlparser.Use: + return Write + default: + return Mixed + } +} + +func writeUnlessReturning(returning int) Kind { + if returning > 0 { + return Mixed + } + return Write +} diff --git a/internal/storage/sqlclass/sqlclass_test.go b/internal/storage/sqlclass/sqlclass_test.go new file mode 100644 index 000000000..4d275c8d8 --- /dev/null +++ b/internal/storage/sqlclass/sqlclass_test.go @@ -0,0 +1,67 @@ +package sqlclass + +import "testing" + +func TestClassify(t *testing.T) { + cases := []struct { + name string + query string + want Kind + }{ + {"select", "SELECT 1", Read}, + {"lowercase select", " select id from issues", Read}, + {"union", "SELECT 1 UNION SELECT 2", Read}, + {"parenthesized union", "(SELECT 1) UNION (SELECT 2)", Read}, + {"show", "SHOW TABLES", Read}, + {"describe", "DESCRIBE issues", Read}, + {"explain select", "EXPLAIN SELECT * FROM issues", Read}, + {"leading block comment", "/* note */ SELECT 1", Read}, + {"leading line comment", "-- note\nSELECT 1", Read}, + + // CTE reads: a prefix scanner that treats the column list after the CTE + // name as the CTE body sees "AS" as the main verb and calls it a write. + {"cte select", "WITH t AS (SELECT 1 AS n) SELECT n FROM t", Read}, + {"cte column list", "WITH t(n) AS (SELECT 1) SELECT n FROM t", Read}, + {"recursive cte column list", "WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < 3) SELECT i FROM n", Read}, + {"recursive cte", "WITH RECURSIVE t AS (SELECT 1 AS i UNION ALL SELECT i + 1 FROM t WHERE i < 3) SELECT * FROM t", Read}, + {"multiple ctes with column lists", "WITH a(x) AS (SELECT 1), b(y) AS (SELECT x FROM a) SELECT y FROM b", Read}, + {"cte nested parens", "WITH t AS (SELECT (1 + (2 * 3)) AS n) SELECT n FROM t", Read}, + {"cte keywords in literal", "WITH t AS (SELECT ') DELETE FROM issues' AS s) SELECT s FROM t", Read}, + {"cte escaped quote in literal", `WITH t AS (SELECT 'a\'(' AS s) SELECT s FROM t`, Read}, + {"cte comment with keywords", "WITH t AS (SELECT 1 AS n /* ) DELETE */) SELECT n FROM t", Read}, + {"cte union main", "WITH t(n) AS (SELECT 1) SELECT n FROM t UNION SELECT 2", Read}, + + // CTE writes must stay writes: a read transaction is rolled back. + {"cte delete", "WITH t AS (SELECT id FROM issues) DELETE FROM issues WHERE id IN (SELECT id FROM t)", Write}, + {"cte column list update", "WITH t(i) AS (SELECT id FROM issues) UPDATE issues SET priority = 1 WHERE id IN (SELECT i FROM t)", Write}, + {"recursive cte delete", "WITH RECURSIVE t(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM t WHERE i < 3) DELETE FROM issues WHERE priority IN (SELECT i FROM t)", Write}, + {"cte escaped quote hides delete", `WITH t AS (SELECT 'a\'(' AS x) DELETE FROM issues`, Write}, + {"insert select with cte", "INSERT INTO labels (issue_id, label) WITH t AS (SELECT id FROM issues) SELECT id, 'x' FROM t", Write}, + + {"insert", "INSERT INTO labels (issue_id, label) VALUES ('a', 'b')", Write}, + {"replace", "REPLACE INTO labels (issue_id, label) VALUES ('a', 'b')", Write}, + {"update", "UPDATE issues SET priority = 1", Write}, + {"delete", "DELETE FROM issues", Write}, + {"create table", "CREATE TABLE x (id INT PRIMARY KEY)", Write}, + {"alter table", "ALTER TABLE issues ADD COLUMN x INT", Write}, + {"drop table", "DROP TABLE x", Write}, + {"set", "SET @x = 1", Write}, + {"select into variable", "SELECT 1 INTO @x", Write}, + + // Statements that may write AND return rows, or that the parser + // cannot classify, must run in a committing transaction that still + // renders any result set. + {"call", "CALL DOLT_COMMIT('-Am', 'msg')", Mixed}, + {"explain analyze delete", "EXPLAIN ANALYZE DELETE FROM issues", Mixed}, + {"unparseable", "PRAGMA table_info(issues)", Mixed}, + {"garbage", "NOT SQL AT ALL (", Mixed}, + {"empty", "", Mixed}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := Classify(tc.query); got != tc.want { + t.Errorf("Classify(%q) = %v, want %v", tc.query, got, tc.want) + } + }) + } +} From 66c6a0c9b398e199b70a6c3d2dd290fc37096ffb Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 29 Sep 2026 02:10:46 -0700 Subject: [PATCH 28/45] fix(types): one source of truth for issue types across bd types and create (#6934) bd types and create validation now both resolve custom types through issueops.ComposeCustomTypes in every mode (direct, server, proxied). Server-mode bd types now includes config.yaml types, proxied config set storage-class accepts DB types, and bd types lists the system types (message, molecule, gate, event); JSON gains additive system_types. Custom-type read errors now fail instead of being swallowed. Co-Authored-By: Claude Opus 5.5 (cherry picked from commit 2bb5337355244880125c7a2bec7f5680053b0a76) hand-resolved: CHANGELOG entry placed under the hotfix's [Unreleased] ### Fixed; BUILD.bazel files dropped (no Bazel on hotfix/1.3.1). Go code applied unchanged. --- CHANGELOG.md | 16 ++ cmd/bd/config.go | 6 +- cmd/bd/create_proxied_server.go | 10 +- cmd/bd/create_proxied_server_test.go | 55 ------ cmd/bd/graph_apply.go | 16 +- cmd/bd/types.go | 94 ++++++++-- cmd/bd/types_builtin_test.go | 40 ++++ cmd/bd/types_create_agreement_test.go | 130 +++++++++++++ cmd/bd/types_proxied_server.go | 20 -- internal/storage/domain/db/config.go | 48 +---- internal/storage/issueops/config_helpers.go | 56 +++--- .../issueops/custom_types_parity_test.go | 173 ++++++++++++++++++ 12 files changed, 490 insertions(+), 174 deletions(-) create mode 100644 cmd/bd/types_builtin_test.go create mode 100644 cmd/bd/types_create_agreement_test.go delete mode 100644 cmd/bd/types_proxied_server.go create mode 100644 internal/storage/issueops/custom_types_parity_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index dae3513ba..64c9223f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,6 +126,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **`bd types` now lists exactly the types `bd create --type` accepts** + ([#6934](https://github.com/gastownhall/beads/pull/6934)). The two resolved + the type set through different code and could disagree: a server-mode + workspace's `bd types` dropped `.beads/config.yaml` `types.custom` entries + once the database had any custom type, although create accepts them; in + proxied mode `bd config set storage-class.` checked `config.yaml` alone + and refused custom types registered in the database; and the built-in system + types `message`, `molecule`, `gate` and `event` were never listed. Every + mode now composes custom types by one rule (the `custom_types` table, else + the `types.custom` config row, always unioned with `config.yaml`). `bd types` + gains a "System types" section, and `bd types --json` an additive + `system_types` field; `core_types` and `custom_types` are unchanged. A + failure to read the custom types now fails `bd types`, `bd create --graph` + and `bd config set storage-class.*` instead of being silently treated as "no + custom types". + - **`bd sql` no longer drops the rows of CTE queries or CALL result sets** ([#6932](https://github.com/gastownhall/beads/pull/6932)). In proxied-server mode, `WITH name(cols) AS (...) SELECT ...` and `WITH RECURSIVE ...` queries diff --git a/cmd/bd/config.go b/cmd/bd/config.go index 3510473a5..1a6abf1eb 100644 --- a/cmd/bd/config.go +++ b/cmd/bd/config.go @@ -994,7 +994,11 @@ func validateStorageClassConfig(key, value string) error { if canonical := issueType.Normalize(); canonical != issueType { return fmt.Errorf("invalid key %q: %q is an alias of %q, and create-time lookup uses the canonical type; set storage-class.%s instead", key, suffix, canonical, canonical) } - if !issueType.IsValidWithCustom(loadEmbeddedCustomTypes()) { + customTypes, err := resolveWorkspaceCustomTypes(rootCtx) + if err != nil { + return err + } + if !issueType.IsValidWithCustom(customTypes) { return fmt.Errorf("invalid key %q: unknown issue type %q (use a built-in type, or add it to types.custom first)", key, suffix) } if _, err := types.ParseStorageClass(value); err != nil { diff --git a/cmd/bd/create_proxied_server.go b/cmd/bd/create_proxied_server.go index 45329b6c6..86899b0bb 100644 --- a/cmd/bd/create_proxied_server.go +++ b/cmd/bd/create_proxied_server.go @@ -11,7 +11,6 @@ import ( "github.com/spf13/cobra" - "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/storage/domain" "github.com/steveyegge/beads/internal/storage/uow" @@ -21,13 +20,6 @@ import ( "github.com/steveyegge/beads/issueops" ) -func resolveProxiedCustomTypes(dbTypes []string) []string { - if len(dbTypes) > 0 { - return dbTypes - } - return config.GetCustomTypesFromYAML() -} - func runCreateProxiedServer(cmd *cobra.Command, ctx context.Context, in createInput) error { if in.repoOverrideSet { return HandleProxyCapabilityError(AssertProxyCapability(ProxyModeProxied, ProxyCapRepo)) @@ -390,7 +382,7 @@ func runCreateProxiedGraph(_ *cobra.Command, ctx context.Context, in createInput // The returned useWisp is the plan-wide table routing decision. func validateProxiedGraphPlan(plan *GraphApplyPlan, in createInput, cctx domain.CreateContext, issueExists func(id string) (bool, error)) (useWisp bool, err error) { cfg := graphPlanConfig{ - customTypes: resolveProxiedCustomTypes(cctx.CustomTypes), + customTypes: cctx.CustomTypes, // No YAML fallback for statuses — the server database is authoritative // (that's where 'bd config set status.custom' writes) and statuses are // store-only everywhere (single-issue create, list filters), unlike diff --git a/cmd/bd/create_proxied_server_test.go b/cmd/bd/create_proxied_server_test.go index b06bd6b7f..a1a327920 100644 --- a/cmd/bd/create_proxied_server_test.go +++ b/cmd/bd/create_proxied_server_test.go @@ -3,13 +3,10 @@ package main import ( "encoding/json" "fmt" - "os" - "path/filepath" "reflect" "testing" "time" - "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/issueops" ) @@ -473,55 +470,3 @@ func TestParseMarkdownDependencies_DoesNotSwapBlocks(t *testing.T) { t.Errorf("got %#v, want %#v (no swap-direction)", got, want) } } - -func TestResolveProxiedCustomTypes_PrefersDBTypes(t *testing.T) { - got := resolveProxiedCustomTypes([]string{"db-a", "db-b"}) - if !reflect.DeepEqual(got, []string{"db-a", "db-b"}) { - t.Errorf("got %#v, want [db-a db-b] — DB types must win when non-empty", got) - } -} - -func TestResolveProxiedCustomTypes_FallsBackToYAML(t *testing.T) { - restore := withTestYAMLCustomTypes(t, "molecule,gate,convoy") - defer restore() - - for _, dbTypes := range [][]string{nil, {}} { - got := resolveProxiedCustomTypes(dbTypes) - if !reflect.DeepEqual(got, []string{"molecule", "gate", "convoy"}) { - t.Errorf("dbTypes=%v: got %#v, want YAML fallback [molecule gate convoy]", dbTypes, got) - } - } -} - -func TestResolveProxiedCustomTypes_EmptyEverywhere(t *testing.T) { - restore := withTestYAMLCustomTypes(t, "") - defer restore() - - got := resolveProxiedCustomTypes(nil) - if len(got) != 0 { - t.Errorf("expected empty result, got %#v", got) - } -} - -func withTestYAMLCustomTypes(t *testing.T, customCSV string) func() { - t.Helper() - tmpDir := t.TempDir() - beadsDir := filepath.Join(tmpDir, ".beads") - if err := os.MkdirAll(beadsDir, 0o755); err != nil { - t.Fatalf("mkdir .beads: %v", err) - } - var content string - if customCSV != "" { - content = "types:\n custom: \"" + customCSV + "\"\n" - } - if err := os.WriteFile(filepath.Join(beadsDir, "config.yaml"), []byte(content), 0o644); err != nil { - t.Fatalf("write config.yaml: %v", err) - } - t.Chdir(tmpDir) - - config.ResetForTesting() - if err := config.Initialize(); err != nil { - t.Fatalf("config.Initialize: %v", err) - } - return func() { config.ResetForTesting() } -} diff --git a/cmd/bd/graph_apply.go b/cmd/bd/graph_apply.go index 05750e30f..9f40f7e6d 100644 --- a/cmd/bd/graph_apply.go +++ b/cmd/bd/graph_apply.go @@ -12,7 +12,6 @@ import ( "strings" "time" - "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/storage" "github.com/steveyegge/beads/internal/types" "github.com/steveyegge/beads/internal/validation" @@ -305,15 +304,6 @@ func warnUnknownGraphFields(w io.Writer, unknown map[string][]string) []string { return hintFields } -func loadEmbeddedCustomTypes() []string { - if store != nil { - if ct, err := store.GetCustomTypes(rootCtx); err == nil && len(ct) > 0 { - return ct - } - } - return config.GetCustomTypesFromYAML() -} - // loadEmbeddedCustomStatuses reads custom statuses from the store only — no // YAML fallback, matching single-issue create and loadListFilterConfig // (custom types fall back to YAML; custom statuses deliberately do not). @@ -348,9 +338,13 @@ func createIssuesFromGraph(planFile string, dryRun bool, opts GraphApplyOptions) return HandleErrorRespectJSON("parsing graph plan: %v", err) } + customTypes, err := resolveWorkspaceCustomTypes(rootCtx) + if err != nil { + return HandleErrorRespectJSON("%v", err) + } dbPrefix, allowedPrefixes := loadEmbeddedIDPrefixes() cfg := graphPlanConfig{ - customTypes: loadEmbeddedCustomTypes(), + customTypes: customTypes, customStatuses: loadEmbeddedCustomStatuses(), dbPrefix: dbPrefix, allowedPrefixes: allowedPrefixes, diff --git a/cmd/bd/types.go b/cmd/bd/types.go index a10d65a52..727833260 100644 --- a/cmd/bd/types.go +++ b/cmd/bd/types.go @@ -6,7 +6,9 @@ import ( "strings" "github.com/spf13/cobra" + "github.com/steveyegge/beads/internal/config" "github.com/steveyegge/beads/internal/metrics" + "github.com/steveyegge/beads/internal/storage/issueops" "github.com/steveyegge/beads/internal/types" ) @@ -26,14 +28,45 @@ var coreWorkTypes = []struct { {types.TypeMilestone, "Marks completion of a set of related issues (contains no work itself)"}, } +// systemTypeDescriptions describes the built-in types that are not core work +// types. bd's own commands create them (mail, molecules, gates, set-state), and +// like the core types they validate without any types.custom entry. +var systemTypeDescriptions = map[types.IssueType]string{ + types.TypeMessage: "Message between agents or users", + types.TypeMolecule: "Molecule root for swarm coordination (bd mol)", + types.TypeGate: "Async coordination gate (bd gate, formula gates)", + types.TypeEvent: "Audit-trail event (bd set-state)", +} + +// systemWorkTypes lists every built-in type that is not a core work type, in +// the types package's declaration order. Together with coreWorkTypes it covers +// exactly the types IssueType.IsBuiltIn accepts. +func systemWorkTypes() []typeInfo { + core := make(map[types.IssueType]bool, len(coreWorkTypes)) + for _, t := range coreWorkTypes { + core[t.Type] = true + } + var out []typeInfo + for _, t := range append(append([]types.IssueType{}, types.AllIssueTypes...), types.TypeEvent) { + if core[t] { + continue + } + out = append(out, typeInfo{Name: string(t), Description: systemTypeDescriptions[t]}) + } + return out +} + var typesCmd = &cobra.Command{ Use: "types", GroupID: "views", Short: "List valid issue types", Long: `List all valid issue types that can be used with bd create --type. -Core work types (bug, task, feature, chore, epic, decision, spike, story, milestone) are always valid. -Additional types require configuration via types.custom in .beads/config.yaml. +Core work types (bug, task, feature, chore, epic, decision, spike, story, milestone) +and system types (message, molecule, gate, event) are always valid. +Custom types are registered with 'bd config set types.custom' or declared under +types.custom in .beads/config.yaml; the list shown is exactly the set that +bd create --type accepts. Examples: bd types # List all types with descriptions @@ -55,32 +88,56 @@ Examples: return printSections(jsonOutput) } - if usesProxiedServer() { - return runTypesProxiedServer(rootCtx) - } - - if err := ensureDirectMode("types command requires direct database access"); err != nil { - return HandleError("%v", err) - } - - var customTypes []string - ctx := context.Background() - if store != nil { - if ct, err := store.GetCustomTypes(ctx); err == nil { - customTypes = ct + if !usesProxiedServer() { + if err := ensureDirectMode("types command requires direct database access"); err != nil { + return HandleError("%v", err) } } + customTypes, err := resolveWorkspaceCustomTypes(rootCtx) + if err != nil { + return HandleError("%v", err) + } return renderTypes(customTypes) }, } +// resolveWorkspaceCustomTypes returns the workspace's registered custom issue +// types in either storage mode. Both routes end in issueops.ComposeCustomTypes, +// the rule issue create/update validation also resolves through, so what +// `bd types` lists is exactly what `bd create --type` accepts. +func resolveWorkspaceCustomTypes(ctx context.Context) ([]string, error) { + if usesProxiedServer() { + uw, err := openProxiedListUOW(ctx) + if err != nil { + return nil, err + } + defer uw.Close(ctx) + customTypes, err := uw.ConfigUseCase().GetCustomTypes(ctx) + if err != nil { + return nil, fmt.Errorf("reading custom types: %w", err) + } + return customTypes, nil + } + if store == nil { + // No database is open (e.g. a --dry-run plan check): the config.yaml + // layer is the only one available, composed by the same rule. + return issueops.ComposeCustomTypes(nil, "", config.GetCustomTypesFromYAML()), nil + } + customTypes, err := store.GetCustomTypes(ctx) + if err != nil { + return nil, fmt.Errorf("reading custom types: %w", err) + } + return customTypes, nil +} + func renderTypes(customTypes []string) error { if jsonOutput { result := struct { CoreTypes []typeInfo `json:"core_types"` + SystemTypes []typeInfo `json:"system_types"` CustomTypes []string `json:"custom_types,omitempty"` - }{} + }{SystemTypes: systemWorkTypes()} for _, t := range coreWorkTypes { result.CoreTypes = append(result.CoreTypes, typeInfo{ @@ -97,6 +154,11 @@ func renderTypes(customTypes []string) error { fmt.Printf(" %-14s %s\n", t.Type, t.Description) } + fmt.Println("\nSystem types (built-in, created by bd commands):") + for _, t := range systemWorkTypes() { + fmt.Printf(" %-14s %s\n", t.Name, t.Description) + } + if len(customTypes) > 0 { fmt.Println("\nConfigured custom types:") for _, t := range customTypes { diff --git a/cmd/bd/types_builtin_test.go b/cmd/bd/types_builtin_test.go new file mode 100644 index 000000000..bed351643 --- /dev/null +++ b/cmd/bd/types_builtin_test.go @@ -0,0 +1,40 @@ +package main + +import ( + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +// TestBdTypesListsEveryBuiltIn pins that `bd types` (core + system sections) +// lists exactly the types that validate without configuration, i.e. the set +// IssueType.IsBuiltIn accepts, each with a description. +func TestBdTypesListsEveryBuiltIn(t *testing.T) { + listed := map[types.IssueType]bool{} + for _, ct := range coreWorkTypes { + if listed[ct.Type] { + t.Errorf("core type %q listed twice", ct.Type) + } + listed[ct.Type] = true + } + for _, st := range systemWorkTypes() { + it := types.IssueType(st.Name) + if listed[it] { + t.Errorf("type %q listed as both core and system", it) + } + if st.Description == "" { + t.Errorf("system type %q has no description", it) + } + listed[it] = true + } + for it := range listed { + if !it.IsBuiltIn() { + t.Errorf("bd types lists %q as built-in, but IsBuiltIn rejects it", it) + } + } + for _, it := range append(append([]types.IssueType{}, types.AllIssueTypes...), types.TypeEvent) { + if !listed[it] { + t.Errorf("built-in type %q validates without configuration but bd types does not list it", it) + } + } +} diff --git a/cmd/bd/types_create_agreement_test.go b/cmd/bd/types_create_agreement_test.go new file mode 100644 index 000000000..a371dab96 --- /dev/null +++ b/cmd/bd/types_create_agreement_test.go @@ -0,0 +1,130 @@ +//go:build cgo + +package main + +import ( + "encoding/json" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" +) + +// typesAgreementRunner runs bd in a workspace and returns stdout (or stdout+stderr on +// failure) plus the exit error. +type typesAgreementRunner func(t *testing.T, args ...string) ([]byte, error) + +// assertTypesAgreeWithCreate is the `bd types` ⇔ `bd create --type` contract: +// every type `bd types --json` lists is accepted by create and by +// storage-class config validation, and an unlisted type is rejected. +// Custom types are seeded from both sources — the database (bd config set) +// and the project's .beads/config.yaml — because each storage mode resolves +// them; the listing and validation paths must compose them identically. +func assertTypesAgreeWithCreate(t *testing.T, run typesAgreementRunner, beadsDir string) { + t.Helper() + + if out, err := run(t, "config", "set", "types.custom", "dbtype"); err != nil { + t.Fatalf("config set types.custom: %v\n%s", err, out) + } + cfgPath := filepath.Join(beadsDir, "config.yaml") + cfg, err := os.ReadFile(cfgPath) + if err != nil { + t.Fatalf("read config.yaml: %v", err) + } + if strings.Contains(string(cfg), "types.custom") || strings.Contains(string(cfg), "\ntypes:") { + t.Fatalf("fixture config.yaml already declares types; test assumes it does not:\n%s", cfg) + } + cfg = append(cfg, []byte("\ntypes.custom: \"yamltype\"\n")...) + if err := os.WriteFile(cfgPath, cfg, 0o644); err != nil { + t.Fatalf("write config.yaml: %v", err) + } + + out, err := run(t, "types", "--json") + if err != nil { + t.Fatalf("types --json: %v\n%s", err, out) + } + var listed struct { + CoreTypes []typeInfo `json:"core_types"` + SystemTypes []typeInfo `json:"system_types"` + CustomTypes []string `json:"custom_types"` + } + if err := json.Unmarshal(out, &listed); err != nil { + t.Fatalf("unmarshal types --json: %v\n%s", err, out) + } + names := map[string]bool{} + for _, ti := range append(listed.CoreTypes, listed.SystemTypes...) { + names[ti.Name] = true + } + for _, c := range listed.CustomTypes { + names[c] = true + } + + // Types create is known to accept must be listed. + for _, want := range []string{"task", "message", "molecule", "gate", "event", "dbtype", "yamltype"} { + if !names[want] { + t.Errorf("bd types does not list %q, but bd create accepts it; listed=%v", want, names) + } + } + + // Every listed type must be accepted by create ... + for name := range names { + if out, err := run(t, "create", "--silent", "--type", name, "agreement "+name); err != nil { + t.Errorf("bd types lists %q but bd create --type %s failed: %v\n%s", name, name, err, out) + } + } + // ... and by storage-class validation, which names issue types too. + for _, name := range listed.CustomTypes { + if out, err := run(t, "config", "set", "storage-class."+name, "versioned"); err != nil { + t.Errorf("bd types lists %q but config set storage-class.%s failed: %v\n%s", name, name, err, out) + } + } + + // An unlisted type is rejected. + if out, err := run(t, "create", "--silent", "--type", "notregistered", "agreement unlisted"); err == nil { + t.Errorf("bd create accepted unlisted type notregistered: %s", out) + } +} + +func TestProxiedServerTypesAgreeWithCreate(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + p := newSharedProxiedProject(t, bd, "tac") + assertTypesAgreeWithCreate(t, func(t *testing.T, args ...string) ([]byte, error) { + return bdProxiedRun(t, bd, p.dir, args...) + }, p.beadsDir) +} + +func TestEmbeddedTypesAgreeWithCreate(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt integration tests") + } + t.Parallel() + bd := buildEmbeddedBD(t) + dir, beadsDir, _ := bdInit(t, bd, "--prefix", "tac") + assertTypesAgreeWithCreate(t, func(t *testing.T, args ...string) ([]byte, error) { + return bdRunWithFlockRetry(t, bd, dir, args...) + }, beadsDir) +} + +// Server mode (`bd init --server` against an external dolt sql-server) lists +// through DoltStore.GetCustomTypes, the resolver that used to drop +// config.yaml types once the database had any. +func TestServerModeTypesAgreeWithCreate(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + p := newServerModeProject(t, bd, "tas") + assertTypesAgreeWithCreate(t, func(t *testing.T, args ...string) ([]byte, error) { + t.Helper() + cmd := exec.Command(bd, args...) + cmd.Dir = p.dir + cmd.Env = p.env + stdout, stderr, err := runCommandBuffers(t, cmd) + if err != nil { + return append(stdout.Bytes(), stderr.Bytes()...), err + } + return stdout.Bytes(), nil + }, p.beadsDir) +} diff --git a/cmd/bd/types_proxied_server.go b/cmd/bd/types_proxied_server.go deleted file mode 100644 index cefb89af6..000000000 --- a/cmd/bd/types_proxied_server.go +++ /dev/null @@ -1,20 +0,0 @@ -package main - -import ( - "context" -) - -func runTypesProxiedServer(ctx context.Context) error { - uw, err := openProxiedListUOW(ctx) - if err != nil { - return HandleError("%v", err) - } - defer uw.Close(ctx) - - var customTypes []string - if ct, err := uw.ConfigUseCase().GetCustomTypes(ctx); err == nil { - customTypes = ct - } - - return renderTypes(customTypes) -} diff --git a/internal/storage/domain/db/config.go b/internal/storage/domain/db/config.go index 8436a09d5..19115ba8e 100644 --- a/internal/storage/domain/db/config.go +++ b/internal/storage/domain/db/config.go @@ -126,22 +126,25 @@ func (r *configSQLRepositoryImpl) GetAllConfig(ctx context.Context) (map[string] return out, nil } +// GetCustomTypes resolves the workspace's custom issue types through +// issueops.ComposeCustomTypes, the same rule the embedded and server-mode +// stores use, so proxied `bd types` and proxied create/update validation +// accept exactly the same set. func (r *configSQLRepositoryImpl) GetCustomTypes(ctx context.Context) ([]string, error) { fromTable, err := r.readCustomTypesTable(ctx) if err != nil { return nil, err } - fromDB := fromTable - if len(fromDB) == 0 { - fromConfig, err := r.readCustomTypesConfig(ctx) + var configValue string + if len(fromTable) == 0 { + configValue, err = r.GetConfig(ctx, "types.custom") if err != nil { - return nil, err + return nil, fmt.Errorf("db: GetCustomTypes: %w", err) } - fromDB = fromConfig } - return unionWithYAMLCustomTypes(fromDB, config.GetCustomTypesFromYAML()), nil + return issueops.ComposeCustomTypes(fromTable, configValue, config.GetCustomTypesFromYAML()), nil } func (r *configSQLRepositoryImpl) readCustomTypesTable(ctx context.Context) ([]string, error) { @@ -169,39 +172,6 @@ func (r *configSQLRepositoryImpl) readCustomTypesTable(ctx context.Context) ([]s return out, nil } -func (r *configSQLRepositoryImpl) readCustomTypesConfig(ctx context.Context) ([]string, error) { - value, err := r.GetConfig(ctx, "types.custom") - if err != nil { - return nil, fmt.Errorf("db: GetCustomTypes: %w", err) - } - return issueops.ParseTypesConfigValue(value), nil -} - -func unionWithYAMLCustomTypes(dbTypes, yamlTypes []string) []string { - if len(dbTypes) == 0 && len(yamlTypes) == 0 { - return nil - } - seen := make(map[string]struct{}, len(dbTypes)+len(yamlTypes)) - out := make([]string, 0, len(dbTypes)+len(yamlTypes)) - for _, src := range [][]string{dbTypes, yamlTypes} { - for _, t := range src { - t = strings.TrimSpace(t) - if t == "" { - continue - } - if _, ok := seen[t]; ok { - continue - } - seen[t] = struct{}{} - out = append(out, t) - } - } - if len(out) == 0 { - return nil - } - return out -} - func (r *configSQLRepositoryImpl) GetAllowedPrefixes(ctx context.Context) (string, error) { value, err := r.GetConfig(ctx, "allowed_prefixes") if err != nil { diff --git a/internal/storage/issueops/config_helpers.go b/internal/storage/issueops/config_helpers.go index d21de3b98..f17d858d4 100644 --- a/internal/storage/issueops/config_helpers.go +++ b/internal/storage/issueops/config_helpers.go @@ -56,7 +56,7 @@ func ResolveCustomConfigInTx(ctx context.Context, tx DBTX) (statuses []types.Cus return nil, nil, err } if statusesFromTable && typesFromTable { - return statuses, customTypes, nil + return statuses, ComposeCustomTypes(customTypes, "", config.GetCustomTypesFromYAML()), nil } cfg, err := getConfigKeysInTx(ctx, tx, "status.custom", "types.custom") @@ -66,12 +66,8 @@ func ResolveCustomConfigInTx(ctx context.Context, tx DBTX) (statuses []types.Cus statuses = ParseStatusFallback(yamlStatuses) } } - if !typesFromTable { - if yamlTypes := config.GetCustomTypesFromYAML(); len(yamlTypes) > 0 { - customTypes = yamlTypes - } - } - return statuses, customTypes, nil + // The types.custom row is unreadable; the remaining layers still compose. + return statuses, ComposeCustomTypes(customTypes, "", config.GetCustomTypesFromYAML()), nil } if !statusesFromTable { @@ -83,14 +79,7 @@ func ResolveCustomConfigInTx(ctx context.Context, tx DBTX) (statuses []types.Cus statuses = ParseStatusFallback(yamlStatuses) } } - if !typesFromTable { - if v := cfg["types.custom"]; v != "" { - customTypes = ParseTypesConfigValue(v) - } else if yamlTypes := config.GetCustomTypesFromYAML(); len(yamlTypes) > 0 { - customTypes = yamlTypes - } - } - return statuses, customTypes, nil + return statuses, ComposeCustomTypes(customTypes, cfg["types.custom"], config.GetCustomTypesFromYAML()), nil } func resolveCustomStatusesFromTableInTx(ctx context.Context, tx DBTX) ([]types.CustomStatus, bool, error) { @@ -280,21 +269,42 @@ func ResolveCustomTypesInTx(ctx context.Context, tx DBTX) ([]string, error) { // fallback. The table-empty case is intentional: schema migration creates the // table but may not have populated it from the existing types.custom config // string yet. + var configValue string if len(fromDB) == 0 { value, err := GetConfigInTx(ctx, tx, "types.custom") if err != nil { return customTypesYAMLFallback(config.GetCustomTypesFromYAML, err) } - if value != "" { - fromDB = ParseTypesConfigValue(value) - } + configValue = value } + return ComposeCustomTypes(fromDB, configValue, config.GetCustomTypesFromYAML()), nil +} - // Overlay-union with YAML types.custom regardless of the DB-side result. - // gastownhall/beads#4024: project-extension types declared in .beads/config.yaml - // must validate server-side even when the database side hasn't been migrated - // or populated with those types. - return mergeWithYAMLCustomTypes(fromDB, config.GetCustomTypesFromYAML), nil +// ComposeCustomTypes is the single rule for which custom issue types a +// workspace has. Every reader of the three sources — the resolvers here, the +// proxied-server config repository, and therefore both `bd types` and +// create/update validation in every mode — composes them through this +// function, so the listed set and the accepted set cannot drift apart: +// +// 1. the custom_types table, when it has rows; +// 2. otherwise the types.custom config row (CSV or JSON array), which covers +// databases whose table has not been populated yet; +// 3. always unioned with config.yaml types.custom, the project-extension +// overlay (gastownhall/beads#4024). +// +// Names are trimmed, empties dropped and duplicates removed; database types +// keep their order and YAML-only types are appended in declared order. +// Returns nil when no layer supplies a type. +func ComposeCustomTypes(fromTable []string, configValue string, yamlTypes []string) []string { + fromDB := dedupePreservingOrder(fromTable) + if len(fromDB) == 0 && configValue != "" { + fromDB = dedupePreservingOrder(ParseTypesConfigValue(configValue)) + } + merged := mergeWithYAMLCustomTypes(fromDB, func() []string { return yamlTypes }) + if len(merged) == 0 { + return nil + } + return merged } // mergeWithYAMLCustomTypes returns the union of dbTypes and the YAML-declared diff --git a/internal/storage/issueops/custom_types_parity_test.go b/internal/storage/issueops/custom_types_parity_test.go new file mode 100644 index 000000000..f4147ccc0 --- /dev/null +++ b/internal/storage/issueops/custom_types_parity_test.go @@ -0,0 +1,173 @@ +package issueops + +import ( + "context" + "os" + "path/filepath" + "reflect" + "regexp" + "testing" + + "github.com/DATA-DOG/go-sqlmock" + "github.com/steveyegge/beads/internal/config" +) + +// withYAMLCustomTypes points the process config at a throwaway +// .beads/config.yaml declaring types.custom, with HOME and XDG_CONFIG_HOME +// isolated so no user-level config can leak in. +func withYAMLCustomTypes(t *testing.T, customCSV string) { + t.Helper() + dir := t.TempDir() + t.Setenv("HOME", filepath.Join(dir, "home")) + t.Setenv("XDG_CONFIG_HOME", filepath.Join(dir, "xdg")) + t.Setenv("BEADS_DIR", "") + beadsDir := filepath.Join(dir, ".beads") + if err := os.MkdirAll(beadsDir, 0o755); err != nil { + t.Fatalf("mkdir .beads: %v", err) + } + content := "" + if customCSV != "" { + content = "types:\n custom: \"" + customCSV + "\"\n" + } + if err := os.WriteFile(filepath.Join(beadsDir, "config.yaml"), []byte(content), 0o644); err != nil { + t.Fatalf("write config.yaml: %v", err) + } + t.Chdir(dir) + config.ResetForTesting() + if err := config.Initialize(); err != nil { + t.Fatalf("config.Initialize: %v", err) + } + t.Cleanup(config.ResetForTesting) +} + +// customTypeSources describes one database + config.yaml state. +type customTypeSources struct { + table []string // custom_types rows + configRow string // config types.custom value ("" = unset) + yaml string // config.yaml types.custom CSV ("" = unset) +} + +func expectCustomTypesTable(mock sqlmock.Sqlmock, names []string) { + rows := sqlmock.NewRows([]string{"name"}) + for _, n := range names { + rows.AddRow(n) + } + mock.ExpectQuery(regexp.QuoteMeta("SELECT name FROM custom_types ORDER BY name")).WillReturnRows(rows) +} + +// resolveViaCustomTypesInTx is the resolver issue create/update validation uses. +func resolveViaCustomTypesInTx(t *testing.T, src customTypeSources) []string { + t.Helper() + db, mock, tx := beginMockTx(t) + defer db.Close() + expectCustomTypesTable(mock, src.table) + if len(src.table) == 0 { + rows := sqlmock.NewRows([]string{"value"}) + if src.configRow != "" { + rows.AddRow(src.configRow) + } + mock.ExpectQuery(regexp.QuoteMeta("SELECT value FROM config WHERE `key` = ?")). + WithArgs("types.custom").WillReturnRows(rows) + } + got, err := ResolveCustomTypesInTx(context.Background(), tx) + if err != nil { + t.Fatalf("ResolveCustomTypesInTx: %v", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } + return got +} + +// resolveViaCustomConfigInTx is the resolver the server-mode store's +// GetCustomTypes (and so direct-mode `bd types`) uses. +func resolveViaCustomConfigInTx(t *testing.T, src customTypeSources) []string { + t.Helper() + db, mock, tx := beginMockTx(t) + defer db.Close() + // One custom status so the status half resolves from its table and the + // config-row read below is driven by the types half alone. + mock.ExpectQuery(regexp.QuoteMeta("SELECT name, category FROM custom_statuses ORDER BY name")). + WillReturnRows(sqlmock.NewRows([]string{"name", "category"}).AddRow("review", "wip")) + expectCustomTypesTable(mock, src.table) + if len(src.table) == 0 { + rows := sqlmock.NewRows([]string{"key", "value"}) + if src.configRow != "" { + rows.AddRow("types.custom", src.configRow) + } + mock.ExpectQuery(regexp.QuoteMeta("SELECT `key`, value FROM config WHERE `key` IN (?,?)")). + WithArgs("status.custom", "types.custom").WillReturnRows(rows) + } + _, got, err := ResolveCustomConfigInTx(context.Background(), tx) + if err != nil { + t.Fatalf("ResolveCustomConfigInTx: %v", err) + } + if err := mock.ExpectationsWereMet(); err != nil { + t.Fatalf("unmet SQL expectations: %v", err) + } + return got +} + +// TestCustomTypeResolversAgree pins the single composition rule for custom +// issue types (custom_types table, else the types.custom config row, always +// unioned with config.yaml types.custom). The listing path (`bd types`) and +// the validation path (create/update) must resolve the same set, or `bd types` +// advertises types that create rejects or vice versa. +func TestCustomTypeResolversAgree(t *testing.T) { + cases := []struct { + name string + src customTypeSources + want []string + }{ + {"nothing configured", customTypeSources{}, nil}, + {"table only", customTypeSources{table: []string{"convoy", "rig"}}, []string{"convoy", "rig"}}, + {"config row only", customTypeSources{configRow: "convoy,rig"}, []string{"convoy", "rig"}}, + {"yaml only", customTypeSources{yaml: "gizmo"}, []string{"gizmo"}}, + {"table plus yaml", customTypeSources{table: []string{"convoy"}, yaml: "gizmo"}, []string{"convoy", "gizmo"}}, + {"config row plus yaml", customTypeSources{configRow: "convoy", yaml: "gizmo,convoy"}, []string{"convoy", "gizmo"}}, + {"table wins over config row", customTypeSources{table: []string{"convoy"}, configRow: "convoy,rig", yaml: "gizmo"}, []string{"convoy", "gizmo"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + withYAMLCustomTypes(t, tc.src.yaml) + validate := resolveViaCustomTypesInTx(t, tc.src) + list := resolveViaCustomConfigInTx(t, tc.src) + if len(validate) == 0 && len(tc.want) == 0 && len(list) == 0 { + return + } + if !reflect.DeepEqual(validate, tc.want) { + t.Errorf("ResolveCustomTypesInTx = %#v, want %#v", validate, tc.want) + } + if !reflect.DeepEqual(list, tc.want) { + t.Errorf("ResolveCustomConfigInTx custom types = %#v, want %#v (must match the validation resolver)", list, tc.want) + } + }) + } +} + +func TestComposeCustomTypes(t *testing.T) { + cases := []struct { + name string + table []string + configRow string + yaml []string + want []string + }{ + {"empty", nil, "", nil, nil}, + {"table trimmed and deduped", []string{" convoy ", "", "convoy", "rig"}, "", nil, []string{"convoy", "rig"}}, + {"config row json form", nil, `["convoy","rig"]`, nil, []string{"convoy", "rig"}}, + {"table ignores config row", []string{"convoy"}, "rig", nil, []string{"convoy"}}, + {"yaml appended after db", []string{"rig"}, "", []string{"gizmo", "rig"}, []string{"rig", "gizmo"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := ComposeCustomTypes(tc.table, tc.configRow, tc.yaml) + if len(got) == 0 && len(tc.want) == 0 { + return + } + if !reflect.DeepEqual(got, tc.want) { + t.Errorf("ComposeCustomTypes = %#v, want %#v", got, tc.want) + } + }) + } +} From a8e5ccfc10b30eaa55cf644db4102c17362e1a50 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 29 Sep 2026 02:11:21 -0700 Subject: [PATCH 29/45] fix(init): admit a server workspace whose .beads/dolt is empty so the version marker is written (#6935) A server-mode workspace with an empty .beads/dolt and no .local_version was refused as a legacy workspace, blocking every command including bd init. An empty directory holds no legacy data, so the guard now admits it (server mode only); bd init writes the marker and an existing workspace in that state self-heals on its next command. Refs #5682. Co-Authored-By: Claude Opus 5.5 (cherry picked from commit e3a12f4e5c813b57d64d6fdb8d435eccdcc18f1d) hand-resolved: CHANGELOG entry placed under the hotfix's [Unreleased] ### Fixed; cmd/bd/BUILD.bazel dropped (no Bazel on hotfix/1.3.1). Go code applied unchanged. --- CHANGELOG.md | 14 ++ cmd/bd/legacy_upgrade_guard.go | 30 ++++ .../legacy_upgrade_guard_config_yaml_test.go | 6 +- cmd/bd/legacy_upgrade_guard_e2e_test.go | 6 +- .../legacy_upgrade_guard_empty_root_test.go | 143 ++++++++++++++++++ ...gacy_upgrade_guard_external_server_test.go | 141 +++++++++++++++++ cmd/bd/legacy_upgrade_guard_test.go | 10 +- cmd/bd/legacy_upgrade_guard_witness_test.go | 7 +- 8 files changed, 338 insertions(+), 19 deletions(-) create mode 100644 cmd/bd/legacy_upgrade_guard_empty_root_test.go create mode 100644 cmd/bd/legacy_upgrade_guard_external_server_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 64c9223f8..eec177eb2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,6 +126,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **A server-mode workspace with an empty `.beads/dolt` is no longer refused + as legacy** + ([#5682](https://github.com/gastownhall/beads/issues/5682), + [#6935](https://github.com/gastownhall/beads/pull/6935)). bd creates `.beads/dolt` on use even when the data lives on an + external Dolt server, and leaves it empty. When the gitignored + `.local_version` witness was missing — after a fresh checkout, or when a + provisioner created the empty root before `bd init --server --external` — + the legacy-upgrade guard refused every command, `bd init` included, as a + "legacy Dolt server workspace", and no bd command could repair it. An empty + root holds nothing a legacy release could have left, so the guard now admits + it: `bd init` writes the witness, and an existing workspace in that state + re-seeds it on the next command. A pre-1.0 witness, or a `.beads/dolt` that + holds anything, is still refused. + - **`bd types` now lists exactly the types `bd create --type` accepts** ([#6934](https://github.com/gastownhall/beads/pull/6934)). The two resolved the type set through different code and could disagree: a server-mode diff --git a/cmd/bd/legacy_upgrade_guard.go b/cmd/bd/legacy_upgrade_guard.go index 02c8e36e0..738da3ac4 100644 --- a/cmd/bd/legacy_upgrade_guard.go +++ b/cmd/bd/legacy_upgrade_guard.go @@ -75,6 +75,20 @@ func guardLegacyUpgradeWorkspace(beadsDir string) error { return nil } if serverMode { + if !present && isEmptyDirectory(filepath.Join(beadsDir, "dolt")) { + // An empty .beads/dolt holds no Dolt data, so no legacy release + // could have left anything in it to protect: it carries no more + // evidence than a server workspace without that directory, which + // is admitted above. bd creates exactly this shape itself — the + // local Dolt root is made on use even when the data lives on an + // external server — so refusing it made `bd init --server + // --external` fail over a provisioner's empty root, and made a + // workspace that lost its gitignored witness unusable with no bd + // command able to repair it. Admitting it lets the command re-seed + // the witness. A present pre-1.0 witness was refused above, and a + // root holding anything at all still refuses below. + return nil + } if present && classifyVersionWitness(version) == witnessEraUnknown { // A witness that is present but unreadable — including one left // blank or whitespace-only by an interrupted or disk-full @@ -265,6 +279,22 @@ func hasLegacyDoltRoot(beadsDir string) bool { return err == nil && info.IsDir() && info.Mode()&os.ModeSymlink == 0 } +// isEmptyDirectory reports whether path is a real (non-symlink) directory with +// no entries. Any error reading it answers false, so callers fail closed. +func isEmptyDirectory(path string) bool { + info, err := os.Lstat(path) + if err != nil || !info.IsDir() || info.Mode()&os.ModeSymlink != 0 { + return false + } + dir, err := os.Open(path) // #nosec G304 -- bounded probe beneath selected workspace + if err != nil { + return false + } + defer func() { _ = dir.Close() }() + names, err := dir.Readdirnames(1) + return len(names) == 0 && err == io.EOF +} + func isNonEmptyRegularFile(path string) bool { info, err := os.Lstat(path) return err == nil && info.Mode().IsRegular() && info.Size() > 0 diff --git a/cmd/bd/legacy_upgrade_guard_config_yaml_test.go b/cmd/bd/legacy_upgrade_guard_config_yaml_test.go index 46640052d..37c89519a 100644 --- a/cmd/bd/legacy_upgrade_guard_config_yaml_test.go +++ b/cmd/bd/legacy_upgrade_guard_config_yaml_test.go @@ -31,7 +31,7 @@ func bindSelectedWorkspaceConfig(t *testing.T, beadsDir string) { } // writeDoltRootWorkspace lays down the ambiguous shape the guard classifies: a -// .beads directory that owns a local Dolt root, plus whatever metadata.json, +// .beads directory that owns a populated local Dolt root, plus whatever metadata.json, // config.yaml and version witness the case needs. An empty string means "do not // write this file at all". func writeDoltRootWorkspace(t *testing.T, metadata, configYaml, version string) string { @@ -52,9 +52,7 @@ func writeDoltRootWorkspace(t *testing.T, metadata, configYaml, version string) t.Fatal(err) } } - if err := os.Mkdir(filepath.Join(beadsDir, "dolt"), 0o700); err != nil { - t.Fatal(err) - } + populateLegacyDoltRoot(t, beadsDir) bindSelectedWorkspaceConfig(t, beadsDir) return beadsDir } diff --git a/cmd/bd/legacy_upgrade_guard_e2e_test.go b/cmd/bd/legacy_upgrade_guard_e2e_test.go index 619f2289a..0eb09ed87 100644 --- a/cmd/bd/legacy_upgrade_guard_e2e_test.go +++ b/cmd/bd/legacy_upgrade_guard_e2e_test.go @@ -490,7 +490,7 @@ func TestLegacyGuardUsesSelectedTargetSharedServerConfig(t *testing.T) { // TestLegacyGuardAdmitsConfigYamlServerWorkspace runs the real binary against // the workspace shape gc provisions — `dolt.mode: server` in -// .beads/config.yaml, a local .beads/dolt root, a metadata.json that names no +// .beads/config.yaml, a populated local .beads/dolt root, a metadata.json that names no // dolt_mode — and pins that the guard no longer calls it legacy. The in-process // guard tests bind the workspace config themselves; only a subprocess proves // that PersistentPreRunE's own binding puts the selected workspace's @@ -518,9 +518,7 @@ func TestLegacyGuardAdmitsConfigYamlServerWorkspace(t *testing.T) { repoDir := t.TempDir() initGitRepo(t, repoDir) beadsDir := filepath.Join(repoDir, ".beads") - if err := os.MkdirAll(filepath.Join(beadsDir, "dolt"), 0o700); err != nil { - t.Fatal(err) - } + populateLegacyDoltRoot(t, beadsDir) writeFile(t, filepath.Join(beadsDir, "metadata.json"), []byte(`{"backend":"dolt","dolt_database":"hq"}`)) writeFile(t, filepath.Join(beadsDir, "config.yaml"), []byte("dolt:\n mode: server\n auto-start: false\n")) if tt.version != "" { diff --git a/cmd/bd/legacy_upgrade_guard_empty_root_test.go b/cmd/bd/legacy_upgrade_guard_empty_root_test.go new file mode 100644 index 000000000..b1dac315b --- /dev/null +++ b/cmd/bd/legacy_upgrade_guard_empty_root_test.go @@ -0,0 +1,143 @@ +package main + +import ( + "os" + "path/filepath" + "testing" +) + +// populateLegacyDoltRoot lays down a .beads/dolt root that holds a Dolt +// database, the shape a historical local sql-server left behind. The guard has +// to treat such a root as evidence worth protecting. +func populateLegacyDoltRoot(t *testing.T, beadsDir string) { + t.Helper() + if err := os.MkdirAll(filepath.Join(beadsDir, "dolt", "beads", ".dolt"), 0o700); err != nil { + t.Fatal(err) + } +} + +// writeEmptyDoltRootServerWorkspace lays down the shape a current bd leaves +// behind after `bd init --server --external`: server mode selected, a +// .beads/dolt directory that bd created on use but that holds nothing because +// the data lives on the external server, and — once the gitignored witness has +// been lost, or when a provisioner created the empty root before bd init — +// no .local_version. +func writeEmptyDoltRootServerWorkspace(t *testing.T, metadata, configYaml, version string) string { + t.Helper() + beadsDir := t.TempDir() + for name, contents := range map[string]string{ + "metadata.json": metadata, + "config.yaml": configYaml, + } { + if contents == "" { + continue + } + if err := os.WriteFile(filepath.Join(beadsDir, name), []byte(contents), 0o600); err != nil { + t.Fatal(err) + } + } + if version != "" { + if err := writeLocalVersion(filepath.Join(beadsDir, localVersionFile), version); err != nil { + t.Fatal(err) + } + } + if err := os.Mkdir(filepath.Join(beadsDir, "dolt"), 0o700); err != nil { + t.Fatal(err) + } + bindSelectedWorkspaceConfig(t, beadsDir) + return beadsDir +} + +const externalServerMetadata = `{"backend":"dolt","dolt_mode":"server","dolt_server_host":"db.example.test","dolt_database":"ext_db"}` + +// TestLegacyUpgradeGuardAdmitsServerWorkspaceWithEmptyDoltRoot pins the +// external-host recovery: a server-mode workspace whose .beads/dolt is empty +// and whose witness is missing holds no local Dolt data a legacy release could +// have left there, so it carries no more evidence than a workspace with no +// .beads/dolt at all — which the guard already admits. Refusing it made the +// workspace unusable after `bd init --server --external` whenever the witness +// was absent, with no bd command able to repair it. +func TestLegacyUpgradeGuardAdmitsServerWorkspaceWithEmptyDoltRoot(t *testing.T) { + cases := []struct { + name string + metadata string + configYaml string + }{ + {name: "metadata.json external server", metadata: externalServerMetadata}, + {name: "metadata.json server", metadata: `{"backend":"dolt","dolt_mode":"server"}`}, + {name: "config.yaml server mode", configYaml: configYamlServerMode}, + // Proxied-server does not select server mode, so it never reaches the + // empty-root branch; it is admitted by the final non-embedded arm. Pin + // that so a later change to either arm cannot start refusing it. + {name: "metadata.json proxied server", metadata: `{"backend":"dolt","dolt_mode":"proxied-server"}`}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + warnings := captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, tc.metadata, tc.configYaml, "") + + if err := guardLegacyUpgradeWorkspace(beadsDir); err != nil { + t.Fatalf("guardLegacyUpgradeWorkspace() = %v, want nil", err) + } + if warnings.Len() != 0 { + t.Fatalf("guard warned about an absent witness: %q", warnings.String()) + } + }) + } +} + +// TestLegacyUpgradeGuardEmptyDoltRootAdmissionStaysNarrow proves the empty-root +// admission relaxes nothing else: pre-1.0 evidence still refuses, a root that +// holds anything still refuses, and outside server mode an empty root is still +// the legacy embedded shape. +func TestLegacyUpgradeGuardEmptyDoltRootAdmissionStaysNarrow(t *testing.T) { + t.Run("historical server witness", func(t *testing.T) { + captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, externalServerMetadata, "", "0.62.0") + + if err := guardLegacyUpgradeWorkspace(beadsDir); !isLegacyUpgradeRefusal(err) { + t.Fatalf("guardLegacyUpgradeWorkspace() = %v, want migration refusal", err) + } + }) + + t.Run("historical embedded witness", func(t *testing.T) { + captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, externalServerMetadata, "", "0.49.6") + + if err := guardLegacyUpgradeWorkspace(beadsDir); !isLegacyUpgradeRefusal(err) { + t.Fatalf("guardLegacyUpgradeWorkspace() = %v, want migration refusal", err) + } + }) + + for _, entry := range []string{"beads", ".dolt", "stray-file"} { + t.Run("root holding "+entry, func(t *testing.T) { + captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, externalServerMetadata, "", "") + path := filepath.Join(beadsDir, "dolt", entry) + var err error + if entry == "stray-file" { + err = os.WriteFile(path, nil, 0o600) + } else { + err = os.Mkdir(path, 0o700) + } + if err != nil { + t.Fatal(err) + } + + if err := guardLegacyUpgradeWorkspace(beadsDir); !isLegacyUpgradeRefusal(err) { + t.Fatalf("guardLegacyUpgradeWorkspace() = %v, want migration refusal", err) + } + }) + } + + for _, metadata := range []string{`{"backend":"dolt"}`, `{"backend":"dolt","dolt_mode":"embedded"}`} { + t.Run("embedded "+metadata, func(t *testing.T) { + captureLegacyUpgradeWarnings(t) + beadsDir := writeEmptyDoltRootServerWorkspace(t, metadata, "", "") + + if err := guardLegacyUpgradeWorkspace(beadsDir); !isLegacyUpgradeRefusal(err) { + t.Fatalf("guardLegacyUpgradeWorkspace() = %v, want migration refusal", err) + } + }) + } +} diff --git a/cmd/bd/legacy_upgrade_guard_external_server_test.go b/cmd/bd/legacy_upgrade_guard_external_server_test.go new file mode 100644 index 000000000..1f243fb50 --- /dev/null +++ b/cmd/bd/legacy_upgrade_guard_external_server_test.go @@ -0,0 +1,141 @@ +//go:build cgo + +package main + +import ( + "context" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" +) + +// runExternalServerBD runs the bd binary under test in repoDir with env and +// returns its combined output. +func runExternalServerBD(t *testing.T, repoDir string, env []string, args ...string) (string, error) { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, buildBDUnderTest(t), args...) + cmd.Dir = repoDir + cmd.Env = env + out, err := cmd.CombinedOutput() + if ctxErr := ctx.Err(); ctxErr != nil { + t.Fatalf("bd %s did not finish before the deadline: %v\n%s", strings.Join(args, " "), ctxErr, out) + } + return string(out), err +} + +// externalServerTestEnv isolates a bd child from the developer's home and +// config (a real ~/.beads may select a shared server) and from the ambient +// BEADS_DOLT_* endpoint this package's TestMain exports, so only the flags and +// files under test choose the target. +func externalServerTestEnv(t *testing.T) []string { + t.Helper() + home := t.TempDir() + return append(envWithoutBeadsStorageSettings(), + "HOME="+home, + "XDG_CONFIG_HOME="+filepath.Join(home, ".config"), + "BEADS_TEST_IGNORE_REPO_CONFIG=1", + "BD_DISABLE_METRICS=1", + "BEADS_DOLT_AUTO_START=0", + ) +} + +func externalServerInitArgs(database string, extra ...string) []string { + args := []string{ + "init", "--quiet", "--server", "--external", + "--server-host", "127.0.0.1", + "--server-port", fmt.Sprintf("%d", testDoltServerPort), + "--database", database, + "--prefix", "ext", + "--skip-hooks", "--skip-agents", + } + return append(args, extra...) +} + +func assertCurrentVersionWitness(t *testing.T, beadsDir string) { + t.Helper() + data, err := os.ReadFile(filepath.Join(beadsDir, localVersionFile)) + if err != nil { + t.Fatalf("expected %s after the command: %v", localVersionFile, err) + } + if got := strings.TrimSpace(string(data)); got != Version { + t.Fatalf("%s = %q, want %q", localVersionFile, got, Version) + } +} + +// TestExternalServerInitWritesWitnessOverEmptyDoltRoot reproduces the +// provisioner shape: config.yaml already selects server mode and an empty +// .beads/dolt exists before `bd init --server --external` runs. The legacy +// guard used to refuse that init as a "legacy Dolt server workspace", so the +// scope never got a witness and every later command was refused too. +func TestExternalServerInitWritesWitnessOverEmptyDoltRoot(t *testing.T) { + skipIfNoDolt(t) + env := externalServerTestEnv(t) + + repoDir := t.TempDir() + initGitRepo(t, repoDir) + beadsDir := filepath.Join(repoDir, ".beads") + if err := os.MkdirAll(filepath.Join(beadsDir, "dolt"), 0o700); err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(beadsDir, "config.yaml"), []byte("dolt.mode: server\n")) + + database := uniqueTestDBName(t) + t.Cleanup(func() { dropTestDatabase(database, testDoltServerPort) }) + + out, err := runExternalServerBD(t, repoDir, append(env, "BEADS_DIR="+beadsDir), + externalServerInitArgs(database, "--init-if-missing")...) + if err != nil { + t.Fatalf("bd init --server --external over an empty .beads/dolt failed: %v\n%s", err, out) + } + assertCurrentVersionWitness(t, beadsDir) +} + +// TestExternalServerWorkspaceRecoversMissingWitness covers a workspace already +// in the broken state: initialized against an external server, then left +// without .local_version (it is gitignored, so a fresh checkout or a cleanup +// drops it) while bd's own empty .beads/dolt remains. The next ordinary +// command must open the workspace and re-seed the witness with no manual +// steps. +func TestExternalServerWorkspaceRecoversMissingWitness(t *testing.T) { + skipIfNoDolt(t) + env := externalServerTestEnv(t) + + repoDir := t.TempDir() + initGitRepo(t, repoDir) + beadsDir := filepath.Join(repoDir, ".beads") + + database := uniqueTestDBName(t) + t.Cleanup(func() { dropTestDatabase(database, testDoltServerPort) }) + + out, err := runExternalServerBD(t, repoDir, env, externalServerInitArgs(database)...) + if err != nil { + t.Fatalf("bd init --server --external failed: %v\n%s", err, out) + } + assertCurrentVersionWitness(t, beadsDir) + + entries, err := os.ReadDir(filepath.Join(beadsDir, "dolt")) + if err != nil { + t.Fatalf("precondition: expected bd init to leave a .beads/dolt directory: %v", err) + } + if len(entries) != 0 { + t.Fatalf("precondition: expected an empty .beads/dolt for an external server, found %d entries", len(entries)) + } + if err := os.Remove(filepath.Join(beadsDir, localVersionFile)); err != nil { + t.Fatal(err) + } + + out, err = runExternalServerBD(t, repoDir, env, "list", "--json", "--all") + if err != nil { + t.Fatalf("bd list on an external-server workspace without a witness failed: %v\n%s", err, out) + } + if strings.Contains(out, "explicit migration is required") { + t.Fatalf("bd list reported a legacy-upgrade refusal:\n%s", out) + } + assertCurrentVersionWitness(t, beadsDir) +} diff --git a/cmd/bd/legacy_upgrade_guard_test.go b/cmd/bd/legacy_upgrade_guard_test.go index fe16d698c..cfae520a9 100644 --- a/cmd/bd/legacy_upgrade_guard_test.go +++ b/cmd/bd/legacy_upgrade_guard_test.go @@ -206,14 +206,12 @@ func TestLegacyUpgradeGuardMetadataLessSQLiteAndCurrentEmbeddedPrecedence(t *tes } }) - t.Run("explicit server metadata with missing version witness and local Dolt root is refused", func(t *testing.T) { + t.Run("explicit server metadata with missing version witness and populated local Dolt root is refused", func(t *testing.T) { beadsDir := t.TempDir() if err := os.WriteFile(filepath.Join(beadsDir, "metadata.json"), []byte(`{"backend":"dolt","dolt_mode":"server"}`), 0o600); err != nil { t.Fatal(err) } - if err := os.Mkdir(filepath.Join(beadsDir, "dolt"), 0o700); err != nil { - t.Fatal(err) - } + populateLegacyDoltRoot(t, beadsDir) if err := guardLegacyUpgradeWorkspace(beadsDir); !isLegacyUpgradeRefusal(err) { t.Fatalf("guardLegacyUpgradeWorkspace() = %v, want migration refusal", err) } @@ -290,9 +288,7 @@ func TestLegacyUpgradeGuardServerSelectionBeatsStaleEmbeddedRepository(t *testin t.Fatal(err) } } - if err := os.Mkdir(filepath.Join(beadsDir, "dolt"), 0o700); err != nil { - t.Fatal(err) - } + populateLegacyDoltRoot(t, beadsDir) staleRepo := filepath.Join(beadsDir, "embeddeddolt", "stale", ".dolt") if err := os.MkdirAll(staleRepo, 0o700); err != nil { t.Fatal(err) diff --git a/cmd/bd/legacy_upgrade_guard_witness_test.go b/cmd/bd/legacy_upgrade_guard_witness_test.go index 65044227f..a2f77d49c 100644 --- a/cmd/bd/legacy_upgrade_guard_witness_test.go +++ b/cmd/bd/legacy_upgrade_guard_witness_test.go @@ -84,7 +84,8 @@ func TestLegacyServerVersionSpansHistoricalServerEra(t *testing.T) { } // writeSelectedServerWorkspace lays down the ambiguous shape the guard has to -// classify: a selected Dolt server workspace that still owns a local Dolt root. +// classify: a selected Dolt server workspace that still owns a populated local +// Dolt root. func writeSelectedServerWorkspace(t *testing.T, version string) string { t.Helper() beadsDir := t.TempDir() @@ -97,9 +98,7 @@ func writeSelectedServerWorkspace(t *testing.T, version string) string { t.Fatal(err) } } - if err := os.Mkdir(filepath.Join(beadsDir, "dolt"), 0o700); err != nil { - t.Fatal(err) - } + populateLegacyDoltRoot(t, beadsDir) return beadsDir } From e25e5a71d3420ddbb8b5ea4ded08ae0fb5230333 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 29 Sep 2026 02:22:09 -0700 Subject: [PATCH 30/45] fix(show): support bd show --watch under --proxied-server (#6933) bd show --watch was refused in proxied-server mode (proxy.watch.unsupported). The direct and proxied routes now share one watch loop (2s poll, redraw on id:status:updated_at change, SIGINT/SIGTERM stop); the proxied route opens a short read per render/poll. A missing id now exits 1 on both routes, and failed polls keep the last render quietly. Co-Authored-By: Claude Opus 5.5 (cherry picked from commit 5ac483bc75fc6b6c872c076b8554c8569a46f693) hand-resolved: cmd/bd/proxy_capability.go: the hotfix front door keys on cmd.Name(), not commandRegistryPath; dropped its show --watch refusal block and kept the hotfix's ready block unchanged. cmd/bd/show_proxied_server.go: took main's side (in-runner watch refusal removed). cmd/bd/proxy_capability_test.go: took main's newProxyFrontDoorCommand and create/--repo front-door tests; kept both the hotfix's TestProxyCapabilityRefusalDoesNotNeedProvider and main's TestProxyFrontDoorAdmitsShowWatch. CHANGELOG entry placed under the hotfix's [Unreleased] ### Fixed; cmd/bd/BUILD.bazel dropped (no Bazel on hotfix/1.3.1). --- CHANGELOG.md | 12 + cmd/bd/proxy_capability.go | 13 +- cmd/bd/proxy_capability_test.go | 56 ++++- cmd/bd/show.go | 3 +- cmd/bd/show_display.go | 96 +++++--- cmd/bd/show_embedded_test.go | 25 ++ cmd/bd/show_proxied_integration_test.go | 11 - cmd/bd/show_proxied_server.go | 59 +++-- cmd/bd/show_proxied_watch.go | 57 +++++ cmd/bd/show_proxied_watch_integration_test.go | 216 ++++++++++++++++++ cmd/bd/show_watch_loop_test.go | 152 ++++++++++++ 11 files changed, 618 insertions(+), 82 deletions(-) create mode 100644 cmd/bd/show_proxied_watch.go create mode 100644 cmd/bd/show_proxied_watch_integration_test.go create mode 100644 cmd/bd/show_watch_loop_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index eec177eb2..827c55fe9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,6 +126,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **`bd show --watch` works under `--proxied-server`.** It was refused with + `proxy.watch.unsupported` although the provider can answer the poll the way + `bd list --watch` already does, so every proxied workspace — the default + transport for managed-local setups — had no way to watch a bead. The proxied + route now shares the direct route's loop: render once, re-read every 2s, + redraw only when the issue's status or `updated_at` changes, stop cleanly on + Ctrl+C/SIGTERM, and require exactly one id. Each poll opens its own short + unit of work, so a long watch never pins a transaction, and a poll that + fails (the issue was deleted, the backend blipped) keeps the last render + without printing, as on the direct route. On both routes a watch whose id + cannot be found now exits non-zero instead of 0. + - **A server-mode workspace with an empty `.beads/dolt` is no longer refused as legacy** ([#5682](https://github.com/gastownhall/beads/issues/5682), diff --git a/cmd/bd/proxy_capability.go b/cmd/bd/proxy_capability.go index 73869cdc7..f946c6c33 100644 --- a/cmd/bd/proxy_capability.go +++ b/cmd/bd/proxy_capability.go @@ -166,8 +166,10 @@ var ( // is threading it through the UOW reader the way `list` already does, so // the refusal itself is the gap. maxRowsRefusal = refused("proxy.max_rows.unsupported", "--max-rows / BEADS_MAX_ROWS is not supported in proxied-server mode", ProxyReasonUnimplemented, trackLongTail) - // `list --watch` is routed, so watching is possible over the provider; - // nothing else has been wired to it. + // The mode-wide default for a --watch that has not been wired to the + // provider. Both commands that register the flag (`list`, `show`) poll the + // provider and are honored per-command below, so this refuses only a + // future --watch until someone routes it. watchRefusal = refused("proxy.watch.unsupported", "watch mode not supported in proxied-server mode", ProxyReasonUnimplemented, trackLongTail) ) @@ -197,7 +199,7 @@ var proxyCapabilityMatrix = map[ProxyMode]map[ProxyCapability]proxyCapabilityRul } var proxyCommandCapabilities = map[string]map[ProxyMode]map[ProxyCapability]proxyCapabilityRule{ - "show": {ProxyModeProxied: {ProxyCapWatch: watchRefusal}}, + "show": {ProxyModeProxied: {ProxyCapWatch: honored()}}, "list": {ProxyModeProxied: {ProxyCapWatch: honored(), ProxyCapMaxRows: honored(), ProxyCapRepo: notApplicable()}}, "dep tree": {ProxyModeProxied: {ProxyCapMaxRows: honored()}}, "ready": {ProxyModeProxied: {ProxyCapMaxRows: maxRowsRefusal}}, @@ -305,11 +307,6 @@ func validateProxyCapabilitiesBeforeProvider(cmd *cobra.Command) error { if name == "create" && cmd.Flags().Changed("repo") { return HandleProxyCapabilityError(AssertProxyCapability(ProxyModeProxied, ProxyCapRepo)) } - if name == "show" { - if watch, _ := cmd.Flags().GetBool("watch"); watch { - return HandleProxyCapabilityError(AssertProxyCommandCapability("show", ProxyModeProxied, ProxyCapWatch)) - } - } if name == "ready" { maxRows, _, err := resolveMaxRows(cmd) if err != nil { diff --git a/cmd/bd/proxy_capability_test.go b/cmd/bd/proxy_capability_test.go index 4e104b1a5..c7b14ea33 100644 --- a/cmd/bd/proxy_capability_test.go +++ b/cmd/bd/proxy_capability_test.go @@ -126,7 +126,7 @@ func TestProxyCapabilityCommandRows(t *testing.T) { {"ready", "--max-rows", ProxyOutcomeRefused}, {"graph", "--max-rows", ProxyOutcomeRefused}, {"find-duplicates", "--max-rows", ProxyOutcomeRefused}, - {"show", "--watch", ProxyOutcomeRefused}, + {"show", "--watch", ProxyOutcomeHonored}, {"list", "--watch", ProxyOutcomeHonored}, } for _, topology := range []ProxyTopology{ProxyTopologyManagedLocal, ProxyTopologyExternalTCP, ProxyTopologyExternalUnix} { @@ -290,17 +290,32 @@ func TestProxyMaintenanceRefusalLeavesFilesUntouched(t *testing.T) { } } +// newProxyFrontDoorCommand builds `bd --` under a real root. The +// root matters: the front door keys on a command's path below its root, so a +// parentless command has no path and matches no rule — which is how a test +// tree can report success for a refusal that never fired. +func newProxyFrontDoorCommand(t *testing.T, use, flag string) *cobra.Command { + t.Helper() + root := &cobra.Command{Use: "bd"} + cmd := &cobra.Command{Use: use} + cmd.Flags().Bool(flag, false, "") + root.AddCommand(cmd) + if err := cmd.Flags().Set(flag, "true"); err != nil { + t.Fatal(err) + } + if got := commandRegistryPath(cmd); got != use { + t.Fatalf("test command path = %q, want %q", got, use) + } + return cmd +} + func TestProxyCapabilityRefusalFrontDoorTextBeforeProvider(t *testing.T) { oldJSON := jsonOutput jsonOutput = false t.Cleanup(func() { jsonOutput = oldJSON }) - cmd := &cobra.Command{Use: "show"} - cmd.Flags().Bool("watch", false, "") - if err := cmd.Flags().Set("watch", "true"); err != nil { - t.Fatal(err) - } + cmd := newProxyFrontDoorCommand(t, "create", "repo") got := captureStderr(t, func() { _ = validateProxyCapabilitiesBeforeProvider(cmd) }) - if !strings.Contains(got, "watch mode not supported in proxied-server mode") { + if !strings.Contains(got, "--repo is not supported with --proxied-server") { t.Fatalf("text refusal = %q", got) } } @@ -309,14 +324,12 @@ func TestProxyCapabilityRefusalFrontDoorJSONIncludesCode(t *testing.T) { oldJSON := jsonOutput jsonOutput = true t.Cleanup(func() { jsonOutput = oldJSON }) - cmd := &cobra.Command{Use: "show"} - cmd.Flags().Bool("watch", false, "") - _ = cmd.Flags().Set("watch", "true") + cmd := newProxyFrontDoorCommand(t, "create", "repo") out := captureStdout(t, func() error { _ = validateProxyCapabilitiesBeforeProvider(cmd) return nil }) - if !strings.Contains(out, `"code": "proxy.watch.unsupported"`) { + if !strings.Contains(out, `"code": "proxy.repo.unsupported"`) { t.Fatalf("JSON refusal = %q", out) } } @@ -337,6 +350,27 @@ func TestProxyCapabilityRefusalDoesNotNeedProvider(t *testing.T) { } } +// TestProxyFrontDoorAdmitsShowWatch pins that `bd show --watch` reaches the +// proxied provider. It used to be refused here with proxy.watch.unsupported, +// although the provider can answer the poll exactly as `list --watch` does. +func TestProxyFrontDoorAdmitsShowWatch(t *testing.T) { + oldJSON := jsonOutput + jsonOutput = true + t.Cleanup(func() { jsonOutput = oldJSON }) + cmd := newProxyFrontDoorCommand(t, "show", "watch") + var err error + out := captureStdout(t, func() error { + err = validateProxyCapabilitiesBeforeProvider(cmd) + return nil + }) + if err != nil || strings.Contains(out, "proxy.watch.unsupported") { + t.Fatalf("front door refused show --watch: err=%v out=%q", err, out) + } + if err := AssertProxyCommandCapability("show", ProxyModeProxied, ProxyCapWatch); err != nil { + t.Fatalf("show --watch capability refused: %v", err) + } +} + func TestProxyCapabilityDirectEscapeHatch(t *testing.T) { for _, cap := range []ProxyCapability{ProxyCapReadonly, ProxyCapMaxRows, ProxyCapWatch, ProxyCapRepo} { if err := AssertProxyCapability(ProxyModeDirect, cap); err != nil { diff --git a/cmd/bd/show.go b/cmd/bd/show.go index cfc0e3b35..8d0c96880 100644 --- a/cmd/bd/show.go +++ b/cmd/bd/show.go @@ -99,8 +99,7 @@ var showCmd = &cobra.Command{ if len(args) != 1 { return HandleErrorRespectJSON("watch mode requires exactly one issue ID") } - watchIssue(ctx, args[0]) - return nil + return watchIssue(ctx, args[0]) } if showThread { diff --git a/cmd/bd/show_display.go b/cmd/bd/show_display.go index a3e065604..8ff78de67 100644 --- a/cmd/bd/show_display.go +++ b/cmd/bd/show_display.go @@ -14,73 +14,109 @@ import ( "github.com/steveyegge/beads/internal/uimd" ) -// displayShowIssue displays a single issue (reusable for watch mode). -// Matches the full bd show output: header, metadata, content, labels, deps, comments. -func displayShowIssue(ctx context.Context, issueID string) { - displayShowIssueReturn(ctx, issueID) -} - // singleIssueSnapshot builds a comparable string from a single issue's state // so we can detect when the issue has changed between poll cycles. func singleIssueSnapshot(issue *types.Issue) string { return fmt.Sprintf("%s:%s:%d", issue.ID, issue.Status, issue.UpdatedAt.UnixNano()) } +// issueWatchSource is what a `bd show --watch` loop reads from. render draws +// the full view and returns the issue it drew, or nil when there was nothing to +// draw (it reports why itself); fetch re-reads the issue for the change check +// and must not print anything, since the loop owns what a failed poll means. +// The direct store and the proxied-server provider each supply one, so both +// routes share the snapshot, redraw and poll-failure rules below instead of +// copying them. +type issueWatchSource struct { + render func(ctx context.Context) *types.Issue + fetch func(ctx context.Context) (*types.Issue, error) +} + +// showWatchPollInterval is how often a watched issue is re-read. +const showWatchPollInterval = 2 * time.Second + // watchIssue polls for changes to an issue and auto-refreshes the display (GH#654). // Uses polling instead of fsnotify because Dolt stores data in a server-side // database, not files — file watchers never fire. -func watchIssue(ctx context.Context, issueID string) { - // Initial display and snapshot - issue := displayShowIssueReturn(ctx, issueID) +func watchIssue(ctx context.Context, issueID string) error { + return runIssueWatch(ctx, issueWatchSource{ + render: func(ctx context.Context) *types.Issue { return displayShowIssueReturn(ctx, issueID) }, + fetch: func(ctx context.Context) (*types.Issue, error) { return fetchIssue(ctx, issueID) }, + }) +} + +// runIssueWatch drives watchIssueLoop on a real ticker until Ctrl+C or SIGTERM. +// It fails when the initial render found nothing to watch; the render has +// already said why, so the failure is silent. +func runIssueWatch(ctx context.Context, src issueWatchSource) error { + // Deferred Stop prevents signal handler leak + sigChan := make(chan os.Signal, 1) + signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM) + defer signal.Stop(sigChan) + + ticker := time.NewTicker(showWatchPollInterval) + defer ticker.Stop() + + if !watchIssueLoop(ctx, src, ticker.C, sigChan) { + return SilentExit() + } + return nil +} + +// watchIssueLoop renders the issue once, then on every tick re-reads it and +// redraws only when singleIssueSnapshot changes. A poll that fails — the +// issue was deleted, the backend blipped, a unit of work would not open — keeps +// the last render on screen and is retried on the next tick without printing: +// the watch is a live view, and a line (or, under --json, an error object) on +// every tick for a condition the user can already see would bury it. It +// returns when stop fires, or immediately, reporting false, when the initial +// render found nothing to watch. +func watchIssueLoop(ctx context.Context, src issueWatchSource, tick <-chan time.Time, stop <-chan os.Signal) bool { + issue := src.render(ctx) if issue == nil { - return + return false } lastSnapshot := singleIssueSnapshot(issue) fmt.Fprintf(os.Stderr, "\nWatching for changes... (Press Ctrl+C to exit)\n") - // Handle Ctrl+C — deferred Stop prevents signal handler leak - sigChan := make(chan os.Signal, 1) - signal.Notify(sigChan, os.Interrupt, syscall.SIGTERM) - defer signal.Stop(sigChan) - - pollInterval := 2 * time.Second - ticker := time.NewTicker(pollInterval) - defer ticker.Stop() - for { select { - case <-sigChan: + case <-stop: fmt.Fprintf(os.Stderr, "\nStopped watching.\n") - return - case <-ticker.C: - issue := fetchIssue(ctx, issueID) - if issue == nil { + return true + case <-tick: + issue, err := src.fetch(ctx) + if err != nil || issue == nil { continue } snap := singleIssueSnapshot(issue) if snap != lastSnapshot { lastSnapshot = snap - displayShowIssue(ctx, issueID) + src.render(ctx) fmt.Fprintf(os.Stderr, "\nWatching for changes... (Press Ctrl+C to exit)\n") } } } } -// fetchIssue retrieves a single issue by ID, returning nil on error. -func fetchIssue(ctx context.Context, issueID string) *types.Issue { +// fetchIssue retrieves a single issue by ID without printing anything. +func fetchIssue(ctx context.Context, issueID string) (*types.Issue, error) { result, err := resolveAndGetIssueWithRouting(ctx, store, issueID) if result != nil { defer result.Close() } - if err != nil || result == nil || result.Issue == nil { - return nil + if err != nil { + return nil, err } - return result.Issue + if result == nil || result.Issue == nil { + return nil, fmt.Errorf("issue %s not found", issueID) + } + return result.Issue, nil } // displayShowIssueReturn displays a single issue and returns it for snapshot use. +// Matches the full bd show output: header, metadata, content, labels, deps, comments. func displayShowIssueReturn(ctx context.Context, issueID string) *types.Issue { result, err := resolveAndGetIssueWithRouting(ctx, store, issueID) if result != nil { diff --git a/cmd/bd/show_embedded_test.go b/cmd/bd/show_embedded_test.go index 87df8f99d..8780a090c 100644 --- a/cmd/bd/show_embedded_test.go +++ b/cmd/bd/show_embedded_test.go @@ -3,13 +3,16 @@ package main import ( + "context" "encoding/json" + "errors" "fmt" "os" "os/exec" "strings" "sync" "testing" + "time" "github.com/steveyegge/beads/internal/types" ) @@ -148,6 +151,28 @@ func TestEmbeddedShow(t *testing.T) { bdShowFail2(t, bd, dir, "ts-nonexistent999") }) + // A watch on an id that does not exist used to print "Issue not found" + // and exit 0 on this route, watching nothing. It now exits 1 like plain + // `bd show `, and like the proxied route. + t.Run("show_watch_nonexistent_id_exits_1", func(t *testing.T) { + ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, bd, "show", "ts-nonexistent999", "--watch") + cmd.Dir = dir + cmd.Env = bdEnv(dir) + out, err := cmd.CombinedOutput() + if ctx.Err() != nil { + t.Fatalf("bd show --watch on a missing id kept watching:\n%s", out) + } + var exitErr *exec.ExitError + if !errors.As(err, &exitErr) || exitErr.ExitCode() != 1 { + t.Fatalf("bd show --watch on a missing id: err=%v, want exit 1\n%s", err, out) + } + if !strings.Contains(string(out), "ts-nonexistent999") { + t.Errorf("output does not name the missing id:\n%s", out) + } + }) + t.Run("show_no_args", func(t *testing.T) { bdShowFail2(t, bd, dir) }) diff --git a/cmd/bd/show_proxied_integration_test.go b/cmd/bd/show_proxied_integration_test.go index b31bc6dad..0dc2c961d 100644 --- a/cmd/bd/show_proxied_integration_test.go +++ b/cmd/bd/show_proxied_integration_test.go @@ -908,17 +908,6 @@ func TestProxiedServerShow3(t *testing.T) { } }) - t.Run("show_watch_rejected_in_proxied_mode", func(t *testing.T) { - t.Parallel() - p := newSharedProxiedProject(t, bd, "swt") - issue := bdProxiedCreate(t, bd, p.dir, "Watch test", "--type", "task") - stdout, stderr := bdProxiedShowFail(t, bd, p.dir, issue.ID, "--watch") - combined := stdout + stderr - if !strings.Contains(combined, "watch mode not supported in proxied-server mode") { - t.Errorf("expected proxied watch-rejection error, got: %s", combined) - } - }) - t.Run("show_wisp_comments_default_count_only", func(t *testing.T) { t.Parallel() p := newSharedProxiedProject(t, bd, "swcc") diff --git a/cmd/bd/show_proxied_server.go b/cmd/bd/show_proxied_server.go index 26a9d3826..fee04107b 100644 --- a/cmd/bd/show_proxied_server.go +++ b/cmd/bd/show_proxied_server.go @@ -94,16 +94,47 @@ func proxiedIssueReader() (issueops.Reader, error) { func runShowProxiedServer(cmd *cobra.Command, ctx context.Context, args []string) error { in := gatherShowProxiedInput(cmd, args) - if in.watchMode { - return HandleProxyCapabilityError(AssertProxyCommandCapability("show", ProxyModeProxied, ProxyCapWatch)) - } - uw, err := proxiedOpenReadUOW(ctx) if err != nil { return err } + + if err := resolveShowProxiedIDs(ctx, uw, in); err != nil { + uw.Close(ctx) + return err + } + + // Same precedence as the direct route: --as-of answers a point-in-time + // read, which has nothing to watch. + if in.watchMode && in.asOfRef == "" { + // The watch opens a fresh unit of work per poll; holding this one + // would pin a transaction for as long as the user watches. + uw.Close(ctx) + if len(in.ids) != 1 { + return HandleErrorRespectJSON("watch mode requires exactly one issue ID") + } + return runIssueWatch(ctx, proxiedIssueWatchSource(in)) + } defer uw.Close(ctx) + switch { + case in.asOfRef != "": + runShowProxiedAsOf(ctx, uw, in) + case in.thread: + return runShowProxiedThread(ctx, uw, in) + case in.refs: + runShowProxiedRefs(ctx, uw, in) + case in.children: + runShowProxiedChildren(ctx, uw, in) + default: + return runShowProxiedDefault(ctx, uw, in) + } + return nil +} + +// resolveShowProxiedIDs turns --current into an explicit id and rejects an +// empty id list, the same checks the direct route makes before dispatching. +func resolveShowProxiedIDs(ctx context.Context, uw uow.UnitOfWork, in *showProxiedInput) error { if in.currentMode { if len(in.ids) > 0 { return HandleErrorRespectJSON("--current cannot be combined with explicit issue IDs") @@ -118,19 +149,6 @@ func runShowProxiedServer(cmd *cobra.Command, ctx context.Context, args []string if len(in.ids) == 0 { return HandleErrorRespectJSON("at least one issue ID is required (use positional args, --id flag, or --current)") } - - switch { - case in.asOfRef != "": - runShowProxiedAsOf(ctx, uw, in) - case in.thread: - return runShowProxiedThread(ctx, uw, in) - case in.refs: - runShowProxiedRefs(ctx, uw, in) - case in.children: - runShowProxiedChildren(ctx, uw, in) - default: - return runShowProxiedDefault(ctx, uw, in) - } return nil } @@ -504,9 +522,10 @@ func proxiedRenderIssue(ctx context.Context, uw uow.UnitOfWork, issue *types.Iss // A READ on an ALTERNATE view. `bd show`'s detail view is on // issueops.Reader on both routes and gets its labels hydrated there; this - // renderer serves --refs, --children, --thread and --as-of, which answer - // with shapes the Reader contract does not describe, from a unit of work - // the caller already holds and has already read the issue from. Asking the + // renderer serves the text views the Reader contract does not describe — + // --refs, --children, --thread, --as-of, and every render of --watch + // (show_proxied_watch.go) — from a unit of work the caller already holds + // and has already read the issue from. Asking the // role here would open a second transaction to re-fetch a row this function // was handed. Alternate views reaching roles of their own is the follow-up // (ga-2ltro.12). diff --git a/cmd/bd/show_proxied_watch.go b/cmd/bd/show_proxied_watch.go new file mode 100644 index 000000000..16e97f663 --- /dev/null +++ b/cmd/bd/show_proxied_watch.go @@ -0,0 +1,57 @@ +package main + +import ( + "context" + "errors" + "time" + + "github.com/steveyegge/beads/internal/types" + "github.com/steveyegge/beads/internal/workapi" +) + +// proxiedIssueWatchSource feeds `bd show --watch` from the proxied-server +// provider. Every render and every poll opens its own short unit of work, so a +// long watch never pins a transaction and each read sees the latest commit. +// After the first render the resolved id is used, so a partial id is resolved +// once, as the direct route's first render does. +func proxiedIssueWatchSource(in *showProxiedInput) issueWatchSource { + id := in.ids[0] + formatTime := func(t time.Time) string { + if in.localTime { + t = t.Local() + } + return t.Format("2006-01-02 15:04") + } + return issueWatchSource{ + render: func(ctx context.Context) *types.Issue { + uw, err := proxiedOpenReadUOW(ctx) + if err != nil { + return nil + } + defer uw.Close(ctx) + issue, isWisp, err := workapi.GetIssueOrWisp(ctx, workapi.NewUOWDetailSource(uw), id) + if err != nil { + reportIssueLookupFailure("fetching", id, err) + return nil + } + id = issue.ID + proxiedRenderIssue(ctx, uw, issue, isWisp, in, 0, formatTime) + return issue + }, + // Opens its unit of work directly rather than through + // proxiedOpenReadUOW, which reports its own failure: a poll must stay + // silent and leave the verdict to watchIssueLoop. + fetch: func(ctx context.Context) (*types.Issue, error) { + if uowProvider == nil { + return nil, errors.New("proxied-server UOW provider not initialized") + } + uw, err := uowProvider.NewUOW(ctx) + if err != nil { + return nil, err + } + defer uw.Close(ctx) + issue, _, err := workapi.GetIssueOrWisp(ctx, workapi.NewUOWDetailSource(uw), id) + return issue, err + }, + } +} diff --git a/cmd/bd/show_proxied_watch_integration_test.go b/cmd/bd/show_proxied_watch_integration_test.go new file mode 100644 index 000000000..76cd72f23 --- /dev/null +++ b/cmd/bd/show_proxied_watch_integration_test.go @@ -0,0 +1,216 @@ +//go:build cgo + +package main + +import ( + "bytes" + "os/exec" + "strings" + "sync" + "syscall" + "testing" + "time" +) + +// watchBannerWriter buffers a stream and signals every time the watch banner +// is printed, so a test can wait for the Nth render instead of sleeping. +type watchBannerWriter struct { + mu sync.Mutex + buf bytes.Buffer + count int + renders chan int +} + +const showWatchBanner = "Watching for changes..." + +func (w *watchBannerWriter) Write(p []byte) (int, error) { + w.mu.Lock() + defer w.mu.Unlock() + n, err := w.buf.Write(p) + if now := strings.Count(w.buf.String(), showWatchBanner); now > w.count { + w.count = now + select { + case w.renders <- now: + default: + } + } + return n, err +} + +func (w *watchBannerWriter) String() string { + w.mu.Lock() + defer w.mu.Unlock() + return w.buf.String() +} + +// waitForRender blocks until the banner has been printed at least want times, +// giving up when the process exits or the timeout elapses. +func (w *watchBannerWriter) waitForRender(want int, timeout time.Duration, exited <-chan struct{}) bool { + deadline := time.After(timeout) + for { + w.mu.Lock() + got := w.count + w.mu.Unlock() + if got >= want { + return true + } + select { + case <-w.renders: + case <-exited: + w.mu.Lock() + defer w.mu.Unlock() + return w.count >= want + case <-deadline: + return false + } + } +} + +// TestProxiedServerShowWatch pins `bd show --watch` under --proxied-server to +// the direct route's contract: render once, redraw when the issue's +// status/updated_at snapshot changes, and stop cleanly on SIGINT. Proxied mode +// used to refuse it outright (proxy.watch.unsupported), so every consumer on +// the default proxied transport had to reimplement the poll itself. +func TestProxiedServerShowWatch(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + + t.Run("redraws_on_change_and_stops_on_interrupt", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "swr") + issue := bdProxiedCreate(t, bd, p.dir, "Watch me", "--type", "task") + + var stdout bytes.Buffer + stderr := &watchBannerWriter{renders: make(chan int, 1)} + cmd := exec.Command(bd, "show", issue.ID, "--watch") + cmd.Dir = p.dir + cmd.Env = bdProxiedEnv(p.dir) + cmd.Stdout = &stdout + cmd.Stderr = stderr + if err := cmd.Start(); err != nil { + t.Fatalf("start bd show --watch: %v", err) + } + var waitErr error + exited := make(chan struct{}) + go func() { + waitErr = cmd.Wait() + close(exited) + }() + t.Cleanup(func() { + _ = cmd.Process.Kill() + <-exited + }) + + if !stderr.waitForRender(1, 60*time.Second, exited) { + t.Fatalf("bd show --watch never started watching\nstdout:\n%s\nstderr:\n%s", stdout.String(), stderr.String()) + } + if strings.Contains(stderr.String(), "not supported") { + t.Fatalf("bd show --watch refused under --proxied-server:\n%s", stderr.String()) + } + + bdProxiedUpdateOne(t, bd, p.dir, issue.ID, "--status", "in_progress") + + if !stderr.waitForRender(2, 60*time.Second, exited) { + t.Fatalf("bd show --watch did not redraw after the status change\nstdout:\n%s\nstderr:\n%s", stdout.String(), stderr.String()) + } + + if err := cmd.Process.Signal(syscall.SIGINT); err != nil { + t.Fatalf("signal bd show --watch: %v", err) + } + select { + case <-exited: + if waitErr != nil { + t.Fatalf("bd show --watch exited with %v after SIGINT, want 0\nstderr:\n%s", waitErr, stderr.String()) + } + case <-time.After(30 * time.Second): + t.Fatalf("bd show --watch ignored SIGINT\nstderr:\n%s", stderr.String()) + } + if !strings.Contains(stderr.String(), "Stopped watching.") { + t.Errorf("stderr lacks the stop line:\n%s", stderr.String()) + } + + out := stdout.String() + if got := strings.Count(out, "Watch me"); got < 2 { + t.Errorf("stdout rendered the issue %d time(s), want an initial render and a redraw:\n%s", got, out) + } + if !strings.Contains(strings.ToUpper(out), "IN_PROGRESS") { + t.Errorf("redraw does not show the new status:\n%s", out) + } + }) + + // A poll that fails keeps the last render and prints nothing: the route + // used to print "Error refreshing " every 2s once the issue was + // deleted, and under --json an error object on stdout each time. + t.Run("deleted_issue_polls_stay_quiet", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "swd") + issue := bdProxiedCreate(t, bd, p.dir, "Watch then delete", "--type", "task") + + // Locked: stdout is read while the child is still writing to it. + stdout := &watchBannerWriter{renders: make(chan int, 1)} + stderr := &watchBannerWriter{renders: make(chan int, 1)} + cmd := exec.Command(bd, "--json", "show", issue.ID, "--watch") + cmd.Dir = p.dir + cmd.Env = bdProxiedEnv(p.dir) + cmd.Stdout = stdout + cmd.Stderr = stderr + if err := cmd.Start(); err != nil { + t.Fatalf("start bd show --watch: %v", err) + } + exited := make(chan struct{}) + go func() { + _ = cmd.Wait() + close(exited) + }() + t.Cleanup(func() { + _ = cmd.Process.Kill() + <-exited + }) + + if !stderr.waitForRender(1, 60*time.Second, exited) { + t.Fatalf("bd show --watch never started watching\nstdout:\n%s\nstderr:\n%s", stdout.String(), stderr.String()) + } + before := len(stdout.String()) + bdProxiedDelete(t, bd, p.dir, issue.ID, "--force") + + // Several poll intervals, so a per-tick report would have fired. + select { + case <-exited: + t.Fatalf("bd show --watch exited after the issue was deleted\nstderr:\n%s", stderr.String()) + case <-time.After(3 * showWatchPollInterval): + } + if after := stdout.String()[before:]; after != "" { + t.Errorf("stdout grew after the delete:\n%s", after) + } + if strings.Contains(stderr.String(), "Error") || strings.Contains(stderr.String(), "not found") { + t.Errorf("stderr reported the failed polls:\n%s", stderr.String()) + } + }) + + t.Run("missing_id_fails_instead_of_watching", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "swm") + stdout, stderr, err, timedOut := bdProxiedRunDeadline(t, bd, p.dir, 60*time.Second, "show", "swm-nonexistent999", "--watch") + if timedOut { + t.Fatalf("bd show --watch on a missing id kept watching\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + if err == nil { + t.Fatalf("bd show --watch on a missing id exited 0\nstdout:\n%s\nstderr:\n%s", stdout, stderr) + } + if want := "Issue swm-nonexistent999 not found"; !strings.Contains(stderr, want) { + t.Errorf("stderr = %q, want it to contain %q", stderr, want) + } + }) + + t.Run("requires_exactly_one_id", func(t *testing.T) { + t.Parallel() + p := newSharedProxiedProject(t, bd, "swo") + a := bdProxiedCreate(t, bd, p.dir, "Watch A", "--type", "task") + b := bdProxiedCreate(t, bd, p.dir, "Watch B", "--type", "task") + stdout, stderr := bdProxiedShowFail(t, bd, p.dir, a.ID, b.ID, "--watch") + if combined := stdout + stderr; !strings.Contains(combined, "watch mode requires exactly one issue ID") { + t.Errorf("want the direct route's one-id error, got:\n%s", combined) + } + }) +} diff --git a/cmd/bd/show_watch_loop_test.go b/cmd/bd/show_watch_loop_test.go new file mode 100644 index 000000000..d45b75c08 --- /dev/null +++ b/cmd/bd/show_watch_loop_test.go @@ -0,0 +1,152 @@ +package main + +import ( + "context" + "errors" + "os" + "slices" + "strings" + "testing" + "time" + + "github.com/steveyegge/beads/internal/types" +) + +// watchPoll is one scripted fetch result. +type watchPoll struct { + issue *types.Issue + err error +} + +// scriptedWatchSource replays a fixed sequence of fetch results and counts +// renders, so the loop's redraw rule can be checked without a store. +type scriptedWatchSource struct { + initial *types.Issue + polls []watchPoll + renders int + // rendersAtFetch records the render count as each poll starts, i.e. the + // result of the poll before it. Written only on the loop's goroutine and + // read after the loop returns. + rendersAtFetch []int +} + +func (s *scriptedWatchSource) source() issueWatchSource { + return issueWatchSource{ + render: func(context.Context) *types.Issue { + s.renders++ + return s.initial + }, + fetch: func(context.Context) (*types.Issue, error) { + s.rendersAtFetch = append(s.rendersAtFetch, s.renders) + next := s.polls[0] + s.polls = s.polls[1:] + return next.issue, next.err + }, + } +} + +// driveWatchLoop runs the loop over src for every scripted poll, stops it, and +// returns whether it reported watching plus everything it wrote to stderr. +func driveWatchLoop(t *testing.T, src *scriptedWatchSource) (bool, string) { + t.Helper() + polls := len(src.polls) + var watched bool + stderr := captureStderr(t, func() { + tick := make(chan time.Time) + stop := make(chan os.Signal, 1) + done := make(chan bool) + go func() { done <- watchIssueLoop(context.Background(), src.source(), tick, stop) }() + // tick is unbuffered, so each send lands only once the loop is back in + // its select, i.e. after the previous poll (and any redraw) finished. + for range polls { + tick <- time.Time{} + } + stop <- os.Interrupt + watched = <-done + }) + return watched, stderr +} + +// TestWatchIssueLoopRedrawsOnlyOnSnapshotChange pins the redraw rule both +// `bd show --watch` routes share: an unchanged snapshot or a failed read does +// not redraw, a status or updated_at change does, and stop ends the loop. +func TestWatchIssueLoopRedrawsOnlyOnSnapshotChange(t *testing.T) { + base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) + open := &types.Issue{ID: "w-1", Status: types.StatusOpen, UpdatedAt: base} + sameAgain := &types.Issue{ID: "w-1", Status: types.StatusOpen, UpdatedAt: base} + inProgress := &types.Issue{ID: "w-1", Status: types.StatusInProgress, UpdatedAt: base.Add(time.Second)} + touched := &types.Issue{ID: "w-1", Status: types.StatusInProgress, UpdatedAt: base.Add(2 * time.Second)} + + src := &scriptedWatchSource{ + initial: open, + polls: []watchPoll{ + {issue: sameAgain}, + {err: errors.New("backend blip")}, + {issue: inProgress}, + {issue: inProgress}, + {issue: touched}, + }, + } + watched, _ := driveWatchLoop(t, src) + if !watched { + t.Fatal("watchIssueLoop reported nothing watched after a successful render") + } + // Poll order: unchanged, failed read, status change, unchanged, + // updated_at change. + if want := []int{1, 1, 1, 2, 2}; !slices.Equal(src.rendersAtFetch, want) { + t.Fatalf("renders before each poll = %v, want %v", src.rendersAtFetch, want) + } + if src.renders != 3 { + t.Fatalf("renders = %d, want 3 (initial, status change, updated_at change)", src.renders) + } +} + +// TestWatchIssueLoopFailedPollsStayQuiet pins the poll-failure rule both routes +// share: a poll that fails (a deleted issue, a unit of work that will not open) +// keeps the last render and prints nothing, however many ticks it lasts, and a +// later successful poll with a change redraws as usual. The proxied route used +// to print an error line — under --json, an error object — on every tick. +func TestWatchIssueLoopFailedPollsStayQuiet(t *testing.T) { + base := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC) + open := &types.Issue{ID: "w-2", Status: types.StatusOpen, UpdatedAt: base} + closed := &types.Issue{ID: "w-2", Status: types.StatusClosed, UpdatedAt: base.Add(time.Second)} + + src := &scriptedWatchSource{ + initial: open, + polls: []watchPoll{ + {err: errors.New("issue w-2 not found")}, + {err: errors.New("open unit of work: connection refused")}, + {err: errors.New("issue w-2 not found")}, + {issue: closed}, + }, + } + watched, stderr := driveWatchLoop(t, src) + if !watched { + t.Fatal("watchIssueLoop reported nothing watched after a successful render") + } + if strings.Contains(stderr, "not found") || strings.Contains(stderr, "connection refused") || strings.Contains(stderr, "Error") { + t.Fatalf("failed polls printed to stderr:\n%s", stderr) + } + // Initial banner, the redraw's banner after the recovery, and the stop line. + if got := strings.Count(stderr, "Watching for changes..."); got != 2 { + t.Fatalf("banner printed %d times, want 2 (initial + one redraw):\n%s", got, stderr) + } + if src.renders != 2 { + t.Fatalf("renders = %d, want 2 (initial, redraw after recovery)", src.renders) + } +} + +// TestWatchIssueLoopStopsWhenNothingToWatch pins that a failed initial render +// returns at once rather than polling an issue that was never shown. +func TestWatchIssueLoopStopsWhenNothingToWatch(t *testing.T) { + src := issueWatchSource{ + render: func(context.Context) *types.Issue { return nil }, + fetch: func(context.Context) (*types.Issue, error) { + t.Fatal("fetch called after a failed initial render") + return nil, nil + }, + } + if watchIssueLoop(context.Background(), src, make(chan time.Time), make(chan os.Signal)) { + t.Fatal("watchIssueLoop reported watching after a failed initial render") + } +} From b2faf48e84e259af6234fd8e4dafff376e80b498 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 29 Sep 2026 01:48:15 -0700 Subject: [PATCH 31/45] fix(dbproxy): retire the proxy when its Dolt backend exits cleanly (#6937) * fix(dbproxy): retire the proxy when its Dolt backend exits cleanly DoltServer supervised `dolt sql-server` with cmd.Wait inside an errgroup, and Running read the errgroup context. errgroup cancels that context only for a non-nil result, so a backend that shut down gracefully (dolt exits 0 on SIGTERM) was reported as running forever. The proxy's backend health watcher never fired: the proxy stayed up and adoptable in front of a dead store. With --proxied-server-idle-timeout 0 (the setting Gas City uses) it never went away; `bd dolt status` said "running" and every command, `bd ping` included, failed until `bd dolt stop`. The supervising goroutine now reports a clean exit as errBackendExited, so Running flips on any backend exit and Stop treats it as benign. This follows the existing proxied lifecycle contract (child death retires the owned proxy; the next bd invocation starts a fresh proxy/backend pair) rather than adding in-proxy restarts: spawn, adoption, stop epochs and identity already live in the on-demand start path, and a second restart loop inside the proxy would duplicate that and could mask a crash-looping backend. `bd dolt status` no longer headlines a proxy whose managed backend is gone as "running"; it says the proxy is up but not serving and points at `bd dolt stop` (for proxies from an older bd). The JSON payload is unchanged. `bd ping` already verifies by running SELECT 1 through the proxy, so after this fix a successful ping reflects a live pair. Tests: DoltServer Running goes false after SIGTERM (clean exit) and SIGKILL, and Stop succeeds; managed-local lane test that SIGTERMs the backend with idle timeout 0 and requires the proxy to retire and `bd ping --json` to succeed on a fresh pair; status render case. Co-Authored-By: Claude Opus 5.5 * fix(dolt status): report running=false for a proxy over a dead managed backend Review follow-up: the JSON running field now means "serving" (a live proxy and, on a managed topology, a live dolt behind it), so scripts that key off running alone are not fooled by a stranded proxy left by an older bd. proxy_pid/proxy_port and backend_* are unchanged. Adds the PR link to the CHANGELOG entry. Co-Authored-By: Claude Opus 5.5 * docs(changelog): resolve merge markers left by the origin/main merge Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Gas City Adopt-PR Co-authored-by: Claude Opus 5.5 (cherry picked from commit d803ec9126028a76124c81ee90ca8068a1c744df) hand-resolved: .github/workflows/proxied-local-smoke.yml: the hotfix's lifecycle-test list differs in order and content, so TestManagedLocalProxiedBackendCleanExitRetiresProxy was appended to the hotfix list. CHANGELOG entry placed under the hotfix's [Unreleased] ### Fixed; cmd/bd/BUILD.bazel and internal/storage/dbproxy/server/BUILD.bazel dropped (no Bazel on hotfix/1.3.1). --- .github/workflows/proxied-local-smoke.yml | 3 +- CHANGELOG.md | 13 ++++ cmd/bd/dolt_proxied_lifecycle.go | 35 +++++++-- cmd/bd/dolt_proxied_lifecycle_test.go | 40 +++++++++++ cmd/bd/proxied_local_backend_exit_test.go | 71 +++++++++++++++++++ internal/storage/dbproxy/server/doltserver.go | 14 +++- .../server/doltserver_exit_unix_test.go | 64 +++++++++++++++++ 7 files changed, 231 insertions(+), 9 deletions(-) create mode 100644 cmd/bd/proxied_local_backend_exit_test.go create mode 100644 internal/storage/dbproxy/server/doltserver_exit_unix_test.go diff --git a/.github/workflows/proxied-local-smoke.yml b/.github/workflows/proxied-local-smoke.yml index 7033f3faa..047f1aa6e 100644 --- a/.github/workflows/proxied-local-smoke.yml +++ b/.github/workflows/proxied-local-smoke.yml @@ -119,7 +119,8 @@ jobs: TestManagedLocalProxiedCapabilityMatrix \ TestManagedLocalProxiedBackupRoundTrip \ TestManagedLocalProxiedAutoBackup \ - TestManagedLocalProxiedBackupRestoreRefusedWhileAttached + TestManagedLocalProxiedBackupRestoreRefusedWhileAttached \ + TestManagedLocalProxiedBackendCleanExitRetiresProxy do if ! grep -qx "$test_name" <<<"$listed"; then echo "managed-local lifecycle test $test_name is missing; got: '$listed'" >&2 diff --git a/CHANGELOG.md b/CHANGELOG.md index 827c55fe9..f046b0281 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,6 +126,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **A proxied workspace's proxy retires when its Dolt backend exits + cleanly.** The proxy noticed its `dolt sql-server` child exiting only when + the exit status was non-zero. A backend that shut down gracefully (for + example on SIGTERM) left the proxy up and adoptable in front of a dead store: + with `--proxied-server-idle-timeout 0` it never went away, `bd dolt status` + called it running, and every command failed until `bd dolt stop`. The proxy + now retires on any backend exit, as it already did on a crash, so the next bd + command (including `bd ping`) starts a fresh proxy and backend. `bd dolt + status` no longer reports a proxy whose managed backend is gone as running: + the text says "not serving" and the JSON reports `running: false` (with + `proxy_pid` still set), including for a stranded proxy started by an older + bd. ([#6937](https://github.com/gastownhall/beads/pull/6937)) + - **`bd show --watch` works under `--proxied-server`.** It was refused with `proxy.watch.unsupported` although the provider can answer the poll the way `bd list --watch` already does, so every proxied workspace — the default diff --git a/cmd/bd/dolt_proxied_lifecycle.go b/cmd/bd/dolt_proxied_lifecycle.go index 31b7db6cb..28d06357f 100644 --- a/cmd/bd/dolt_proxied_lifecycle.go +++ b/cmd/bd/dolt_proxied_lifecycle.go @@ -43,9 +43,12 @@ func proxiedDoltStartRefusal() *ProxyCapabilityError { // proxiedDoltStatus is the `bd dolt status --json` payload for a proxied // workspace. It reports the two processes separately because they fail -// separately and an operator's next move differs: Running describes the proxy, -// which is the endpoint every bd command connects through, while the Backend* -// fields describe the dolt server behind it. +// separately and an operator's next move differs: Running says whether the +// workspace is serving (a live proxy, and on a managed topology a live dolt +// behind it), Proxy* describe the proxy every bd command connects through, and +// the Backend* fields describe the dolt server behind it. A proxy left in front +// of a dead managed backend (possible with proxies started by an older bd) +// therefore reports running=false with proxy_pid still set. type proxiedDoltStatus struct { Mode string `json:"mode"` Root string `json:"root"` @@ -98,9 +101,17 @@ func collectProxiedDoltStatus(beadsDir string) (proxiedDoltStatus, error) { status.BackendEndpoint = externalDoltEndpoint(*info.External) } } + status.Running = proxiedStatusServing(live.ProxyRunning, status.BackendManaged, status.BackendRunning) return status, nil } +// proxiedStatusServing reports whether a proxied workspace is serving: its +// proxy is live and, when bd manages the backend, that dolt server is live +// too. A proxy in front of a dead managed backend serves nothing. +func proxiedStatusServing(proxyRunning, backendManaged, backendRunning bool) bool { + return proxyRunning && (!backendManaged || backendRunning) +} + func externalDoltEndpoint(cfg configfile.ExternalDoltConfig) string { if cfg.Socket != "" { return "unix:" + cfg.Socket @@ -116,11 +127,20 @@ func renderProxiedDoltStatus(status proxiedDoltStatus) { return } - if status.Running { + // A live proxy in front of a managed backend that is gone serves nothing, + // so do not call it running. The proxy retires itself once it notices + // the backend exit; a proxy from an older bd may not. + backendDown := !status.Running && status.ProxyPID != 0 && status.BackendManaged && !status.BackendRunning + switch { + case backendDown: + fmt.Printf("Dolt server: not serving (%s): proxy is up but its dolt backend is not running\n", status.Mode) + fmt.Printf(" Proxy PID: %d\n", status.ProxyPID) + fmt.Printf(" Proxy port: %d\n", status.ProxyPort) + case status.Running: fmt.Printf("Dolt server: running (%s)\n", status.Mode) fmt.Printf(" Proxy PID: %d\n", status.ProxyPID) fmt.Printf(" Proxy port: %d\n", status.ProxyPort) - } else { + default: fmt.Printf("Dolt server: not running (%s)\n", status.Mode) } fmt.Printf(" Root: %s\n", status.Root) @@ -135,9 +155,12 @@ func renderProxiedDoltStatus(status proxiedDoltStatus) { if status.IdleTimeout != "" { fmt.Printf(" Idle timeout: %s\n", status.IdleTimeout) } - if !status.Running { + if !status.Running && !backendDown { fmt.Println(" The proxy starts on demand; the next bd command launches it.") } + if backendDown { + fmt.Println(" Run 'bd dolt stop' to retire the proxy; the next bd command starts a fresh proxy and backend.") + } if isDoltLocalOnly() { fmt.Println(" Remote sync: disabled (dolt.local-only=true)") } diff --git a/cmd/bd/dolt_proxied_lifecycle_test.go b/cmd/bd/dolt_proxied_lifecycle_test.go index d12f87d0b..aa31c22e6 100644 --- a/cmd/bd/dolt_proxied_lifecycle_test.go +++ b/cmd/bd/dolt_proxied_lifecycle_test.go @@ -151,6 +151,28 @@ func TestRenderProxiedDoltStatus(t *testing.T) { } }) + t.Run("text does not call a proxy over a dead managed backend running", func(t *testing.T) { + defer restoreJSONOutput(t)() + jsonOutput = false + stranded := running + stranded.Running = false + stranded.BackendRunning, stranded.BackendPID, stranded.BackendPort = false, 0, 0 + out := captureStdout(t, func() error { renderProxiedDoltStatus(stranded); return nil }) + if strings.Contains(out, "Dolt server: running") { + t.Errorf("proxy with a dead backend reported as running:\n%s", out) + } + for _, want := range []string{ + "Dolt server: not serving (proxied-server)", + "Proxy PID: 111", + "Backend: not running", + "bd dolt stop", + } { + if !strings.Contains(out, want) { + t.Errorf("missing %q in:\n%s", want, out) + } + } + }) + t.Run("text marks an external backend as somebody else's", func(t *testing.T) { defer restoreJSONOutput(t)() jsonOutput = false @@ -180,6 +202,24 @@ func TestRenderProxiedDoltStatus(t *testing.T) { }) } +func TestProxiedStatusServing(t *testing.T) { + for _, tc := range []struct { + name string + proxyRunning, managed, backendRunning bool + want bool + }{ + {"managed pair live", true, true, true, true}, + {"proxy over dead managed backend", true, true, false, false}, + {"external backend", true, false, false, true}, + {"proxy down", false, true, true, false}, + {"nothing running", false, true, false, false}, + } { + if got := proxiedStatusServing(tc.proxyRunning, tc.managed, tc.backendRunning); got != tc.want { + t.Errorf("%s: proxiedStatusServing(%v, %v, %v) = %v, want %v", tc.name, tc.proxyRunning, tc.managed, tc.backendRunning, got, tc.want) + } + } +} + func restoreJSONOutput(t *testing.T) func() { t.Helper() orig := jsonOutput diff --git a/cmd/bd/proxied_local_backend_exit_test.go b/cmd/bd/proxied_local_backend_exit_test.go new file mode 100644 index 000000000..f4b5e336d --- /dev/null +++ b/cmd/bd/proxied_local_backend_exit_test.go @@ -0,0 +1,71 @@ +//go:build cgo && unix + +package main + +import ( + "encoding/json" + "strings" + "syscall" + "testing" + "time" +) + +// backendCleanExitRetireTimeout bounds how long the proxy may outlive a dolt +// backend that shut down gracefully. dolt's own SIGTERM shutdown dominates it; +// the proxy's health watcher polls every 100ms. +const backendCleanExitRetireTimeout = 30 * time.Second + +// TestManagedLocalProxiedBackendCleanExitRetiresProxy extends the child-death +// contract (see TestManagedLocalProxiedOutageReconnectContract, which kills +// the backend) to a backend that exits CLEANLY: SIGTERM makes dolt shut down +// gracefully with status 0. The proxy must retire just as it does for a +// crash, so the next bd command starts a fresh proxy/backend pair and `bd +// ping` answers from a live store. +// +// Before the fix the proxy only noticed non-zero exits. With idle timeout 0 +// (never idle out) it stayed up and adoptable in front of a dead backend +// indefinitely: `bd dolt status` said "running", and every command, `bd ping` +// included, failed until an operator ran `bd dolt stop`. +func TestManagedLocalProxiedBackendCleanExitRetiresProxy(t *testing.T) { + requireManagedLocalProxiedEnv(t) + bd := buildEmbeddedBD(t) + p := bdProxiedInit(t, bd, "clean_exit", "--proxied-server-idle-timeout", "0") + sentinel := bdProxiedCreate(t, bd, p.dir, "clean exit sentinel") + + proxyBefore := readManagedProxyPidFile(t, p) + if proxyBefore == nil || !processAlive(proxyBefore.Pid) { + t.Fatalf("managed-local proxy is not running: %+v", proxyBefore) + } + backendBefore := readManagedBackendPidFile(t, p) + if backendBefore == nil || !processAlive(backendBefore.Pid) { + t.Fatalf("managed-local backend is not running: %+v", backendBefore) + } + + if err := syscall.Kill(backendBefore.Pid, syscall.SIGTERM); err != nil { + t.Fatalf("SIGTERM dolt backend pid %d: %v", backendBefore.Pid, err) + } + waitForManagedProxiedShutdown(t, p, proxyBefore.Pid, backendBefore.Pid, backendCleanExitRetireTimeout) + + out, err := bdProxiedRun(t, bd, p.dir, "ping", "--json") + if err != nil { + t.Fatalf("bd ping after the backend exited: %v\n%s", err, out) + } + var ping struct { + Status string `json:"status"` + } + if err := json.Unmarshal([]byte(strings.TrimSpace(string(out))), &ping); err != nil || ping.Status != "ok" { + t.Fatalf("bd ping --json: status=%q err=%v\n%s", ping.Status, err, out) + } + if got := bdProxiedShow(t, bd, p.dir, sentinel.ID); got.Title != "clean exit sentinel" { + t.Fatalf("sentinel after restart: %+v", got) + } + + proxyAfter := readManagedProxyPidFile(t, p) + backendAfter := readManagedBackendPidFile(t, p) + if proxyAfter == nil || proxyAfter.Pid == proxyBefore.Pid || !processAlive(proxyAfter.Pid) { + t.Fatalf("expected a fresh live proxy: before=%+v after=%+v", proxyBefore, proxyAfter) + } + if backendAfter == nil || backendAfter.Pid == backendBefore.Pid || !processAlive(backendAfter.Pid) { + t.Fatalf("expected a fresh live backend: before=%+v after=%+v", backendBefore, backendAfter) + } +} diff --git a/internal/storage/dbproxy/server/doltserver.go b/internal/storage/dbproxy/server/doltserver.go index 52ea71a23..0cd1a8a4b 100644 --- a/internal/storage/dbproxy/server/doltserver.go +++ b/internal/storage/dbproxy/server/doltserver.go @@ -36,6 +36,13 @@ const ( LockFileName = "proxy-child.lock" ) +// errBackendExited is the result of the supervising goroutine when the dolt +// sql-server exits on its own with status 0 (for example dolt's graceful +// shutdown on SIGTERM). errgroup cancels egCtx only for a non-nil result, and +// Running reads egCtx, so without it a cleanly exited backend would be +// reported as running forever. +var errBackendExited = errors.New("dolt sql-server exited") + const ( startReadyTimeout = 30 * time.Second startReadyPollInterval = 50 * time.Millisecond @@ -303,7 +310,10 @@ func (s *DoltServer) Start(ctx context.Context) error { eg.Go(func() error { defer lock.Unlock() - return cmd.Wait() + if err := cmd.Wait(); err != nil { + return err + } + return errBackendExited }) if err := s.waitReady(ctx); err != nil { @@ -358,7 +368,7 @@ func (s *DoltServer) Stop(ctx context.Context) error { if s.eg != nil { waitErr = s.eg.Wait() var exitErr *exec.ExitError - if errors.As(waitErr, &exitErr) || errors.Is(waitErr, context.Canceled) { + if errors.As(waitErr, &exitErr) || errors.Is(waitErr, context.Canceled) || errors.Is(waitErr, errBackendExited) { waitErr = nil } } diff --git a/internal/storage/dbproxy/server/doltserver_exit_unix_test.go b/internal/storage/dbproxy/server/doltserver_exit_unix_test.go new file mode 100644 index 000000000..0dcc84fc9 --- /dev/null +++ b/internal/storage/dbproxy/server/doltserver_exit_unix_test.go @@ -0,0 +1,64 @@ +//go:build unix + +package server_test + +import ( + "context" + "syscall" + "testing" + "time" + + "github.com/steveyegge/beads/internal/storage/dbproxy/pidfile" + "github.com/steveyegge/beads/internal/storage/dbproxy/server" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// backendExitNoticeTimeout bounds how long Running may keep reporting a +// backend that has left the process table. dolt's own graceful shutdown on +// SIGTERM dominates this; the detection itself is immediate. +const ( + backendExitNoticeTimeout = 30 * time.Second + backendExitNoticePoll = 50 * time.Millisecond +) + +// TestDoltServer_RunningFalseAfterBackendExits verifies Running observes the +// dolt sql-server leaving on its own, whatever its exit status. A SIGTERM'd +// dolt shuts down gracefully and exits 0; before the fix only a non-zero exit +// flipped Running, so the proxy's backend health watcher never fired and the +// proxy stayed up (and adoptable) in front of a dead backend. +func TestDoltServer_RunningFalseAfterBackendExits(t *testing.T) { + for _, tc := range []struct { + name string + sig syscall.Signal + }{ + {name: "clean exit on SIGTERM", sig: syscall.SIGTERM}, + {name: "killed", sig: syscall.SIGKILL}, + } { + t.Run(tc.name, func(t *testing.T) { + s, rootDir := newDoltServer(t) + ctx := context.Background() + require.NoError(t, s.Start(ctx)) + require.True(t, s.Running(ctx)) + + pf, err := pidfile.Read(rootDir, server.PIDFileName) + require.NoError(t, err) + require.NotNil(t, pf) + require.NoError(t, syscall.Kill(pf.Pid, tc.sig)) + + deadline := time.Now().Add(backendExitNoticeTimeout) + for s.Running(ctx) && time.Now().Before(deadline) { + time.Sleep(backendExitNoticePoll) + } + require.False(t, s.Running(ctx), "Running still true %s after the dolt sql-server (pid %d) got %s", backendExitNoticeTimeout, pf.Pid, tc.sig) + + // A backend that exited on its own is not a Stop failure. + stopCtx, cancel := context.WithTimeout(ctx, stopTimeout) + defer cancel() + require.NoError(t, s.Stop(stopCtx)) + left, err := pidfile.Read(rootDir, server.PIDFileName) + require.NoError(t, err) + assert.Nil(t, left, "Stop must remove the backend pid record") + }) + } +} From b4d81fdf5bbf7d2f9dc6e888f784496b57f6168e Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 29 Sep 2026 02:26:09 -0700 Subject: [PATCH 32/45] fix(beads): make an explicit BEADS_DIR authoritative during discovery (#6938) * fix(beads): make an explicit BEADS_DIR authoritative during discovery FindBeadsDir ignored a BEADS_DIR that did not hold project files yet (missing, empty, or uninitialized) and walked up from the current directory instead. PersistentPreRun then rebound BEADS_DIR to whatever ancestor workspace the walk found, so: - `bd init` refused with "already initialized" because of the parent workspace instead of initializing the named directory; - data commands (`bd list`, `bd create`, ...) read and wrote the parent's store. FindBeadsDir now returns "" for such a BEADS_DIR, matching FindDatabasePath, which already refused to fall back. Commands that create a workspace (import/setup default path, bootstrap's fresh-clone synthesis) target the named directory via the new beads.ExplicitBeadsDir helper instead of CWD. Behaviour with BEADS_DIR unset is unchanged. Tests: FindBeadsDir unit cases (missing/empty/non-project BEADS_DIR, initialized BEADS_DIR, unset BEADS_DIR still walks up), and a bd subprocess regression in embedded and proxied-server modes (init under an initialized parent, list refuses, create lands in the new store). Co-Authored-By: Claude Opus 5.5 * docs(changelog): note that BEADS_DIR at a project root now errors Review follow-up for the explicit-BEADS_DIR fix: call out the behaviour change for BEADS_DIR pointing at a project root (or a not-yet-created .beads), which previously worked only because discovery walked up. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Gas City Adopt-PR Co-authored-by: Claude Opus 5.5 (cherry picked from commit 51a7c8e324217a25a97301d303862d1f6865f869) hand-resolved: CHANGELOG entry placed under the hotfix's [Unreleased] ### Fixed; cmd/bd/BUILD.bazel and internal/beads/BUILD.bazel dropped (no Bazel on hotfix/1.3.1). Go code applied unchanged. --- CHANGELOG.md | 18 +++ cmd/bd/bootstrap.go | 4 +- .../init_explicit_beads_dir_embedded_test.go | 146 ++++++++++++++++++ cmd/bd/main.go | 5 + internal/beads/beads.go | 31 +++- internal/beads/explicit_beads_dir_test.go | 116 ++++++++++++++ 6 files changed, 312 insertions(+), 8 deletions(-) create mode 100644 cmd/bd/init_explicit_beads_dir_embedded_test.go create mode 100644 internal/beads/explicit_beads_dir_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index f046b0281..cdf74d896 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -126,6 +126,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- **An explicit `BEADS_DIR` is authoritative during workspace discovery.** + When `BEADS_DIR` named a directory without project files yet (missing, + empty, or not initialized), discovery ignored it and walked up from the + current directory instead, then rebound `BEADS_DIR` to whatever ancestor + workspace it found. `bd init` refused with "already initialized" because of + the parent workspace, and data commands such as `bd create` and `bd list` + read and wrote the parent's store. Discovery now reports no workspace for + such a `BEADS_DIR`, so `bd init` initializes the named directory and other + commands fail with "no beads database found". `bd import`/`bd setup` and + `bd bootstrap` target the named directory when they create a workspace. + Behaviour with `BEADS_DIR` unset is unchanged. **Behaviour change:** a + `BEADS_DIR` that points at a project root rather than its `.beads` + directory (for example `BEADS_DIR=$repo` instead of `BEADS_DIR=$repo/.beads`), + or at a `.beads` that does not exist yet, used to work by accident because + discovery walked up to the nearest workspace; it now fails with "no beads + database found". Point `BEADS_DIR` at the `.beads` directory itself. + ([#6938](https://github.com/gastownhall/beads/pull/6938)) + - **A proxied workspace's proxy retires when its Dolt backend exits cleanly.** The proxy noticed its `dolt sql-server` child exiting only when the exit status was non-zero. A backend that shut down gracefully (for diff --git a/cmd/bd/bootstrap.go b/cmd/bd/bootstrap.go index 95ba55b57..61900646e 100644 --- a/cmd/bd/bootstrap.go +++ b/cmd/bd/bootstrap.go @@ -186,7 +186,9 @@ Examples: if isGitRepo() && !isBareGitRepo() { if originURL, err := gitOriginGetURL(); err == nil && originURL != "" { if gitOriginHasDoltDataRef() { - if fallbackDir := beads.GetWorktreeFallbackBeadsDir(); fallbackDir != "" { + if explicitDir := beads.ExplicitBeadsDir(); explicitDir != "" { + beadsDir = explicitDir + } else if fallbackDir := beads.GetWorktreeFallbackBeadsDir(); fallbackDir != "" { beadsDir = fallbackDir } else { cwd, err := os.Getwd() diff --git a/cmd/bd/init_explicit_beads_dir_embedded_test.go b/cmd/bd/init_explicit_beads_dir_embedded_test.go new file mode 100644 index 000000000..c6bae6f68 --- /dev/null +++ b/cmd/bd/init_explicit_beads_dir_embedded_test.go @@ -0,0 +1,146 @@ +//go:build cgo + +package main + +import ( + "bytes" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + + "github.com/steveyegge/beads/internal/configfile" + "github.com/steveyegge/beads/internal/storage/dbproxy/proxy" +) + +// explicitBeadsDirMode describes how a storage mode's bd subprocesses are +// initialized and run for the explicit-BEADS_DIR regression tests. +type explicitBeadsDirMode struct { + initArgs []string + env func(home string) []string + proxied bool +} + +// runExplicitBeadsDirBD runs bd in dir with the mode's isolated environment. +// beadsDir, when non-empty, is passed as an explicit BEADS_DIR. +func runExplicitBeadsDirBD(t *testing.T, bd string, mode explicitBeadsDirMode, home, dir, beadsDir string, args ...string) (string, string, error) { + t.Helper() + cmd := exec.Command(bd, args...) + cmd.Dir = dir + env := append(mode.env(home), "XDG_CONFIG_HOME="+filepath.Join(home, ".config")) + if beadsDir != "" { + env = append(env, "BEADS_DIR="+beadsDir) + } + cmd.Env = env + var stdout, stderr bytes.Buffer + cmd.Stdout = &stdout + cmd.Stderr = &stderr + err := cmd.Run() + return stdout.String(), stderr.String(), err +} + +// testExplicitBeadsDirIsAuthoritative covers the regression where an explicit +// BEADS_DIR without project files was ignored during discovery: bd walked up +// from CWD, bound the parent workspace, and `bd init` then refused ("already +// initialized") while data commands read and wrote the parent's store. +func testExplicitBeadsDirIsAuthoritative(t *testing.T, bd string, mode explicitBeadsDirMode) { + t.Helper() + home := t.TempDir() + parent := t.TempDir() + initGitRepoAt(t, parent) + parentBeadsDir := filepath.Join(parent, ".beads") + child := filepath.Join(parent, "child") + explicit := filepath.Join(child, ".beads") + if err := os.MkdirAll(child, 0o755); err != nil { + t.Fatal(err) + } + if mode.proxied { + for _, root := range []string{filepath.Join(parentBeadsDir, "dolt"), filepath.Join(explicit, "dolt")} { + root := root + t.Cleanup(func() { + if _, err := os.Stat(root); err != nil { + return + } + if err := proxy.Shutdown(root); err != nil { + t.Logf("proxy.Shutdown(%s): %v", root, err) + } + }) + shutdownProxyOnInterrupt(t, root) + } + } + + initArgs := append([]string{"init", "--quiet", "--non-interactive", "--skip-agents", "--skip-hooks"}, mode.initArgs...) + if stdout, stderr, err := runExplicitBeadsDirBD(t, bd, mode, home, parent, "", append(initArgs, "--prefix", "par")...); err != nil { + t.Fatalf("parent bd init: %v\nstdout:\n%s\nstderr:\n%s", err, stdout, stderr) + } + parentID, stderr, err := runExplicitBeadsDirBD(t, bd, mode, home, parent, "", "create", "--silent", "parent issue") + if err != nil { + t.Fatalf("parent bd create: %v\nstderr:\n%s", err, stderr) + } + parentID = strings.TrimSpace(parentID) + + // A data command against the uninitialized explicit dir must not fall + // back to the parent workspace. + stdout, stderr, err := runExplicitBeadsDirBD(t, bd, mode, home, child, explicit, "list", "--json") + if err == nil || strings.Contains(stdout, parentID) { + t.Errorf("bd list with BEADS_DIR=%s (no workspace yet) should fail without reading the parent store; err=%v\nstdout:\n%s\nstderr:\n%s", explicit, err, stdout, stderr) + } else if !strings.Contains(stderr, "no beads database found") { + t.Errorf("bd list with an uninitialized BEADS_DIR: want \"no beads database found\", got stderr:\n%s", stderr) + } + + // bd init must initialize the explicit dir (existing but empty, as a + // caller that pre-creates it would leave it), not refuse because of the + // parent workspace. + if err := os.MkdirAll(explicit, 0o755); err != nil { + t.Fatal(err) + } + if stdout, stderr, err := runExplicitBeadsDirBD(t, bd, mode, home, child, explicit, append(initArgs, "--prefix", "kid")...); err != nil { + t.Fatalf("bd init with BEADS_DIR=%s: %v\nstdout:\n%s\nstderr:\n%s", explicit, err, stdout, stderr) + } + childCfg, err := configfile.Load(explicit) + if err != nil || childCfg == nil { + t.Fatalf("load %s metadata: cfg=%v err=%v", explicit, childCfg, err) + } + if childCfg.DoltDatabase != "kid" { + t.Fatalf("explicit BEADS_DIR dolt_database = %q, want %q", childCfg.DoltDatabase, "kid") + } + parentCfg, err := configfile.Load(parentBeadsDir) + if err != nil || parentCfg == nil || parentCfg.DoltDatabase != "par" { + t.Fatalf("parent metadata changed: cfg=%+v err=%v", parentCfg, err) + } + + // Writes through the explicit dir land in the new workspace only. + childID, stderr, err := runExplicitBeadsDirBD(t, bd, mode, home, child, explicit, "create", "--silent", "child issue") + if err != nil { + t.Fatalf("bd create with BEADS_DIR=%s: %v\nstderr:\n%s", explicit, err, stderr) + } + childID = strings.TrimSpace(childID) + if !strings.HasPrefix(childID, "kid-") { + t.Fatalf("bd create with BEADS_DIR=%s minted %q, want a kid- ID", explicit, childID) + } + parentList, stderr, err := runExplicitBeadsDirBD(t, bd, mode, home, parent, "", "list", "--json") + if err != nil { + t.Fatalf("parent bd list: %v\nstderr:\n%s", err, stderr) + } + if strings.Contains(parentList, childID) || !strings.Contains(parentList, parentID) { + t.Fatalf("parent store should hold only %s, got:\n%s", parentID, parentList) + } +} + +func TestEmbeddedInitExplicitBeadsDirIsAuthoritative(t *testing.T) { + if os.Getenv("BEADS_TEST_EMBEDDED_DOLT") != "1" { + t.Skip("set BEADS_TEST_EMBEDDED_DOLT=1 to run embedded dolt init tests") + } + t.Parallel() + testExplicitBeadsDirIsAuthoritative(t, buildEmbeddedBD(t), explicitBeadsDirMode{env: bdEnv}) +} + +func TestProxiedServerInitExplicitBeadsDirIsAuthoritative(t *testing.T) { + requireProxiedServerEnv(t) + testExplicitBeadsDirIsAuthoritative(t, buildEmbeddedBD(t), explicitBeadsDirMode{ + initArgs: []string{"--proxied-server"}, + env: bdProxiedEnv, + proxied: true, + }) +} diff --git a/cmd/bd/main.go b/cmd/bd/main.go index 0943da305..93ec76c45 100644 --- a/cmd/bd/main.go +++ b/cmd/bd/main.go @@ -1443,6 +1443,11 @@ var rootCmd = &cobra.Command{ // would create a local database instead of using the redirect target. // (GH#bd-0qel) targetBeadsDir := beads.FindBeadsDir() + if targetBeadsDir == "" { + // An explicit BEADS_DIR is authoritative even + // before it holds project files. + targetBeadsDir = beads.ExplicitBeadsDir() + } if targetBeadsDir == "" { targetBeadsDir = ".beads" } diff --git a/internal/beads/beads.go b/internal/beads/beads.go index 5087ae81e..d23141623 100644 --- a/internal/beads/beads.go +++ b/internal/beads/beads.go @@ -737,11 +737,26 @@ func hasBeadsDatabase(beadsDir string) bool { return false } +// ExplicitBeadsDir returns the .beads directory named by the BEADS_DIR +// environment variable, canonicalized and with any redirect followed, or "" +// when BEADS_DIR is unset. Unlike FindBeadsDir it does not require the +// directory to exist or to hold project files, so commands that create a +// workspace can target the caller's explicit choice instead of CWD. +func ExplicitBeadsDir() string { + beadsDir := os.Getenv("BEADS_DIR") + if beadsDir == "" { + return "" + } + return FollowRedirect(canonicalizeBeadsDirPath(beadsDir)) +} + // FindBeadsDir finds the .beads/ directory in the current directory tree. // Returns empty string if not found. // // Resolution order: -// 1. BEADS_DIR environment variable (highest priority) +// 1. BEADS_DIR environment variable (highest priority). An explicit +// BEADS_DIR is authoritative: when it does not (yet) hold project files, +// FindBeadsDir returns "" instead of discovering some other workspace. // 2. Walk up from CWD toward repo root boundary, checking each directory // for .beads/ with valid project files. For worktrees, stops at the // worktree root; for non-worktrees, stops at the git root. @@ -756,18 +771,20 @@ func hasBeadsDatabase(beadsDir string) bool { // contents are used as the actual .beads directory path. func FindBeadsDir() string { // 1. Check BEADS_DIR environment variable (preferred) - if beadsDir := os.Getenv("BEADS_DIR"); beadsDir != "" { - absBeadsDir := canonicalizeBeadsDirPath(beadsDir) - - // Follow redirect if present - absBeadsDir = FollowRedirect(absBeadsDir) - + if absBeadsDir := ExplicitBeadsDir(); absBeadsDir != "" { if info, err := os.Stat(absBeadsDir); err == nil && info.IsDir() { // Validate directory contains actual project files if hasBeadsProjectFiles(absBeadsDir) { return absBeadsDir } } + + // The caller named this directory explicitly. Walking up from CWD + // here would bind an unrelated ancestor workspace (and PersistentPreRun + // would then rewrite BEADS_DIR to it), so `bd init` and every other + // command would act on the wrong store. Report "not found" instead, + // matching FindDatabasePath. + return "" } // 2. Walk up from CWD toward the repo root, checking each directory for .beads/. diff --git a/internal/beads/explicit_beads_dir_test.go b/internal/beads/explicit_beads_dir_test.go new file mode 100644 index 000000000..527be25c9 --- /dev/null +++ b/internal/beads/explicit_beads_dir_test.go @@ -0,0 +1,116 @@ +package beads + +import ( + "os" + "os/exec" + "path/filepath" + "testing" +) + +// explicitBeadsDirFixture builds a git repo whose root holds an initialized +// workspace (parent/.beads with metadata.json) and a child directory that has +// not been initialized yet. It chdirs into the child, so an unconstrained walk +// up from CWD would bind the parent workspace. +func explicitBeadsDirFixture(t *testing.T) (parentBeadsDir, childDir string) { + t.Helper() + root := t.TempDir() + cmd := exec.Command("git", "init", "--quiet", root) + cmd.Env = append(os.Environ(), "GIT_CONFIG_NOSYSTEM=1", "HOME="+root, "XDG_CONFIG_HOME="+root) + if out, err := cmd.CombinedOutput(); err != nil { + t.Skipf("git not available: %v: %s", err, out) + } + parentBeadsDir = filepath.Join(root, ".beads") + if err := os.MkdirAll(parentBeadsDir, 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(parentBeadsDir, "metadata.json"), []byte(`{"backend":"dolt","dolt_database":"parent"}`), 0o600); err != nil { + t.Fatal(err) + } + childDir = filepath.Join(root, "child") + if err := os.MkdirAll(childDir, 0o700); err != nil { + t.Fatal(err) + } + t.Chdir(childDir) + return parentBeadsDir, childDir +} + +// TestFindBeadsDir_ExplicitBEADS_DIRIsAuthoritative verifies that an explicit +// BEADS_DIR which does not (yet) hold project files is never replaced by a +// workspace discovered by walking up from CWD. Before the fix, FindBeadsDir +// silently fell back to the ancestor walk, so every command — including +// `bd init` after PersistentPreRun rebinds BEADS_DIR — targeted the parent +// workspace instead of the directory the caller named. +func TestFindBeadsDir_ExplicitBEADS_DIRIsAuthoritative(t *testing.T) { + cases := []struct { + name string + setup func(t *testing.T, explicit string) + }{ + {name: "missing", setup: func(*testing.T, string) {}}, + {name: "empty", setup: func(t *testing.T, explicit string) { + if err := os.MkdirAll(explicit, 0o700); err != nil { + t.Fatal(err) + } + }}, + {name: "non-project files only", setup: func(t *testing.T, explicit string) { + if err := os.MkdirAll(explicit, 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(explicit, "registry.json"), []byte("[]"), 0o600); err != nil { + t.Fatal(err) + } + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + parentBeadsDir, childDir := explicitBeadsDirFixture(t) + explicit := filepath.Join(childDir, ".beads") + tc.setup(t, explicit) + t.Setenv("BEADS_DIR", explicit) + + if got := FindBeadsDir(); got != "" { + t.Fatalf("FindBeadsDir() = %q with BEADS_DIR=%q; want \"\" (explicit BEADS_DIR must not fall back to the ancestor workspace %q)", got, explicit, parentBeadsDir) + } + if got := FindDatabasePath(); got != "" { + t.Fatalf("FindDatabasePath() = %q with BEADS_DIR=%q; want \"\"", got, explicit) + } + }) + } +} + +// TestFindBeadsDir_ExplicitBEADS_DIRSelectedOnceInitialized verifies the +// explicit directory is returned as soon as it holds project files, even with +// an initialized ancestor workspace. +func TestFindBeadsDir_ExplicitBEADS_DIRSelectedOnceInitialized(t *testing.T) { + _, childDir := explicitBeadsDirFixture(t) + explicit := filepath.Join(childDir, ".beads") + if err := os.MkdirAll(explicit, 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(explicit, "metadata.json"), []byte(`{"backend":"dolt","dolt_database":"child"}`), 0o600); err != nil { + t.Fatal(err) + } + t.Setenv("BEADS_DIR", explicit) + + got, _ := filepath.EvalSymlinks(FindBeadsDir()) + want, _ := filepath.EvalSymlinks(explicit) + if got != want { + t.Fatalf("FindBeadsDir() = %q; want explicit BEADS_DIR %q", got, want) + } +} + +// TestFindBeadsDir_UnsetBEADS_DIRStillWalksUp pins the unchanged behaviour +// when BEADS_DIR is not set: discovery walks up from CWD to the ancestor +// workspace. +func TestFindBeadsDir_UnsetBEADS_DIRStillWalksUp(t *testing.T) { + parentBeadsDir, _ := explicitBeadsDirFixture(t) + t.Setenv("BEADS_DIR", "") + if err := os.Unsetenv("BEADS_DIR"); err != nil { + t.Fatal(err) + } + + got, _ := filepath.EvalSymlinks(FindBeadsDir()) + want, _ := filepath.EvalSymlinks(parentBeadsDir) + if got != want { + t.Fatalf("FindBeadsDir() = %q; want ancestor workspace %q", got, want) + } +} From 4b49bf2e8022144436911c1fd76f97fb86d518f3 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Tue, 29 Sep 2026 03:07:09 -0700 Subject: [PATCH 33/45] fix(storage): recheck blocked state after commit on the proxied (uow) path and the remaining store routes (#6942) Extends #6719's post-commit blocked-state recheck (gastownhall/beads#6716) to every write runner #6719 left uncovered: - the proxied (uow) write path: every proxied CLI write, bd batch when proxied, and bd serve; - DoltStore.RunInTransaction. It also settles when the regular SQL commit lands and only the Dolt commit after it fails (ErrCommitIndeterminate); - the raw-transaction wisp update, close and delete writers; - both arms of RemoveDependencyWithOptions; - the domain/db recomputes that bypassed issueops. The recording shapes are shared issueops Note* helpers. A failed recheck is reported through issueops.ReportBlockedRecheckFailure (the bd.db.blocked_recheck_dropped counter plus a warning) and never returned. Regression tests fail on main and pass here: a deterministic proxied uow race in six shapes, RunInTransaction and concurrent wisp-close races, sqlmock phase tests for the indeterminate commit, and a proxied CLI fan-in test that runs concurrent `bd close` processes and then checks `bd ready`. Co-Authored-By: Claude Opus 5.5 (cherry picked from commit cda2407115fa13e9f0d781692863b3ba59b6be17) Hand-ported to hotfix/1.3.1, per the #6942 review (section 5). The code hunks are unchanged; every conflict came from main-only context: - Versioned-history scope. It does not exist on the release line. Main's context installs ScopeVersionedHistoryTransaction/clearVersionScope/ releaseVersionScope in internal/storage/dolt/dependencies.go (2 places), dolt/transaction.go (runDoltTransactionRecording), uow/dolt_sql_provider.go (BeginTx) and uow/doltserver_tx.go (struct, releaseConn, poisonConn, release helpers). Resolved by keeping the release code without those lines and adding only the blocked-recheck-scope lines. In dolt_sql_provider.go the release line keeps its journal-only comment wording, extended with the recheck-scope sentence. - Close label. On the release line, issueops/close.go records with noteBlockedRecheck(tx, "close of "+id, ...), the label from the #6719 hand port. It now calls NoteStatusChangeBlockedRecheck, which produces the same "close of " label through statusChangeRecheckLabel. - BUILD.bazel (cmd/bd, dolt, issueops, uow) and dolt/version_scope_completeness_test.go do not exist on the release line and are dropped. - CHANGELOG.md: the release CHANGELOG is kept. This entry is added verbatim before the #6716 entry from the #6719 port, and that entry's list of uncovered routes now points at this entry, as on main. No schema change. go build ./... and go vet (internal/storage, cmd/bd) are clean. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 25 ++- .../close_fanin_proxied_integration_test.go | 84 +++++++++ .../dolt/blocked_recheck_phase_test.go | 70 +++++++ internal/storage/dolt/blocked_recheck_raw.go | 30 +++ .../storage/dolt/blocked_recheck_raw_test.go | 143 ++++++++++++++ internal/storage/dolt/dependencies.go | 10 +- .../dolt/journal_scope_completeness_test.go | 4 +- internal/storage/dolt/store.go | 34 ++-- internal/storage/dolt/transaction.go | 60 +++++- internal/storage/dolt/wisps.go | 26 ++- .../storage/domain/db/delete_repo_test.go | 6 +- internal/storage/domain/db/dependency.go | 1 + internal/storage/domain/db/issue.go | 11 +- internal/storage/domain/issue.go | 5 +- internal/storage/domain/issue_delete.go | 2 +- internal/storage/issueops/blocked_recheck.go | 83 ++++++--- .../issueops/blocked_recheck_report.go | 39 ++++ internal/storage/issueops/bulk_ops.go | 2 +- internal/storage/issueops/close.go | 2 +- internal/storage/issueops/delete.go | 4 +- internal/storage/issueops/dependencies.go | 4 +- internal/storage/issueops/update.go | 2 +- internal/storage/uow/blocked_recheck.go | 80 ++++++++ .../storage/uow/blocked_recheck_race_test.go | 174 ++++++++++++++++++ internal/storage/uow/dolt_sql_provider.go | 5 +- internal/storage/uow/doltserver_tx.go | 94 +++++++--- 26 files changed, 885 insertions(+), 115 deletions(-) create mode 100644 cmd/bd/close_fanin_proxied_integration_test.go create mode 100644 internal/storage/dolt/blocked_recheck_phase_test.go create mode 100644 internal/storage/dolt/blocked_recheck_raw.go create mode 100644 internal/storage/dolt/blocked_recheck_raw_test.go create mode 100644 internal/storage/issueops/blocked_recheck_report.go create mode 100644 internal/storage/uow/blocked_recheck.go create mode 100644 internal/storage/uow/blocked_recheck_race_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index cdf74d896..bcffc2e42 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -217,6 +217,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 statements it cannot parse (for example `PRAGMA`) instead of treating them as reads. +- **The #6716 fan-in stall is fixed on the proxied-server route and the + remaining store routes.** Two blockers of one dependent taken away at the + same time (parallel workers closing both, or a close racing a + `bd dep remove` or a delete of the other) could leave the dependent + `is_blocked=1` and missing from `bd ready` until `bd recompute-blocked` + ([#6716](https://github.com/gastownhall/beads/issues/6716)). The post-commit + recheck the Dolt store write transactions gained for that issue now also + runs on every write the proxied-server (uow/domain-db) route serves, which + under `--proxied-server` — the default topology — is `bd close` (single + and batch), `bd update --status`, `bd dep remove`, `bd delete`, `bd batch` + and `bd serve`; on `RunInTransaction` (`bd batch` direct, `bd cook`, + `bd mol squash`/`burn`, and SDK callers that close or update inside a + transaction); and on the wisp close, update and delete writers and the + legacy dependency removal behind `bd duplicates --merge`. Demote-to-wisp + moves a row between planes without taking a blocker away and is unchanged. + - Unblocking two blockers of one dependent at the same time — closing both, or a close racing a `bd dep remove` or a delete of the other — no longer leaves the dependent stuck as blocked and hidden from `bd ready` until @@ -224,12 +240,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 update to an inactive status, a dependency removal, a delete) and runs through a Dolt store write transaction now rechecks the dependents it recomputed once that transaction has committed - ([#6716](https://github.com/gastownhall/beads/issues/6716)). Writes that - reach the database another way still need the `bd doctor` / - `bd recompute-blocked` repair they needed before: `bd batch` (on both its - plain and its proxied transaction), `bd cook`, `bd mol squash`, - `bd mol burn`, `bd duplicates --merge`, and the wisp writes — closes, - updates, deletes and demote-to-wisp. + ([#6716](https://github.com/gastownhall/beads/issues/6716)). The + proxied-server route and the remaining store routes are covered by the + entry above. - **`bd close` now exits non-zero when any issue in a batch fails to close** ([#6648](https://github.com/gastownhall/beads/issues/6648)). A batch with one diff --git a/cmd/bd/close_fanin_proxied_integration_test.go b/cmd/bd/close_fanin_proxied_integration_test.go new file mode 100644 index 000000000..ad15ff1a7 --- /dev/null +++ b/cmd/bd/close_fanin_proxied_integration_test.go @@ -0,0 +1,84 @@ +//go:build cgo + +package main + +import ( + "bytes" + "encoding/json" + "fmt" + "os/exec" + "strings" + "sync" + "testing" + + "github.com/steveyegge/beads/internal/types" +) + +// TestProxiedServerCloseSiblingBlockersFanIn is gastownhall/beads#6716 end to +// end on the proxied CLI: the fan-in of a formula whose parallel workers +// close the two blockers of one step at the same time. Each round closes the +// blockers with two concurrent `bd close` processes and then requires the +// dependent in `bd ready`. Without the post-commit recheck on the uow path a +// round whose closes overlap leaves the dependent is_blocked=1 and missing +// from ready work. The interleaving is not pinned here (see +// TestProxiedBlockedRecheckAfterRacingUnblocks in internal/storage/uow for +// the deterministic race); this proves the settled CLI outcome. +func TestProxiedServerCloseSiblingBlockersFanIn(t *testing.T) { + requireSharedProxiedServer(t) + t.Parallel() + bd := buildEmbeddedBD(t) + p := newSharedProxiedProject(t, bd, "fan") + + const rounds = 6 + for round := 0; round < rounds; round++ { + a := bdProxiedCreate(t, bd, p.dir, fmt.Sprintf("round %d blocker a", round)).ID + b := bdProxiedCreate(t, bd, p.dir, fmt.Sprintf("round %d blocker b", round)).ID + c := bdProxiedCreate(t, bd, p.dir, fmt.Sprintf("round %d fan-in", round)).ID + for _, blocker := range []string{a, b} { + if out, err := bdProxiedRun(t, bd, p.dir, "dep", "add", c, blocker); err != nil { + t.Fatalf("round %d: dep add %s %s: %v\n%s", round, c, blocker, err, out) + } + } + + var wg sync.WaitGroup + errs := make([]string, 2) + for i, id := range []string{a, b} { + wg.Add(1) + go func() { + defer wg.Done() + cmd := exec.Command(bd, "close", id) + cmd.Dir = p.dir + cmd.Env = bdProxiedEnv(p.dir) + var stderr bytes.Buffer + cmd.Stderr = &stderr + if err := cmd.Run(); err != nil { + errs[i] = fmt.Sprintf("close %s: %v\n%s", id, err, stderr.String()) + } + }() + } + wg.Wait() + if joined := strings.TrimSpace(strings.Join(errs, "")); joined != "" { + t.Fatalf("round %d: %s", round, joined) + } + + out, err := bdProxiedRun(t, bd, p.dir, "ready", "--json", "--limit", "0") + if err != nil { + t.Fatalf("round %d: bd ready: %v\n%s", round, err, out) + } + start := bytes.IndexByte(out, '[') + if start < 0 { + t.Fatalf("round %d: bd ready printed no JSON array: %s", round, out) + } + var ready []types.Issue + if err := json.Unmarshal(out[start:], &ready); err != nil { + t.Fatalf("round %d: parse bd ready: %v\n%s", round, err, out) + } + found := false + for _, issue := range ready { + found = found || issue.ID == c + } + if !found { + t.Fatalf("round %d: %s has both blockers closed but is missing from bd ready", round, c) + } + } +} diff --git a/internal/storage/dolt/blocked_recheck_phase_test.go b/internal/storage/dolt/blocked_recheck_phase_test.go new file mode 100644 index 000000000..337073804 --- /dev/null +++ b/internal/storage/dolt/blocked_recheck_phase_test.go @@ -0,0 +1,70 @@ +package dolt + +import ( + "context" + "errors" + "slices" + "testing" + + sqlmock "github.com/DATA-DOG/go-sqlmock" + mysql "github.com/go-sql-driver/mysql" + + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// scopeAndRecordDependent scopes both of tx's SQL transactions for the +// blocked recheck, as runDoltTransactionRecording does, and records one +// dependent on each, as a close on each plane would. +func scopeAndRecordDependent(t *testing.T, tx *doltTransaction) { + t.Helper() + t.Cleanup(issueops.ScopeBlockedRecheckTransaction(tx.regularTx)) + t.Cleanup(issueops.ScopeBlockedRecheckTransaction(tx.ignoredTx)) + issueops.NoteStatusChangeBlockedRecheck(tx.regularTx, "rp-a", string(types.StatusClosed), []string{"rp-c"}, nil) + issueops.NoteStatusChangeBlockedRecheck(tx.ignoredTx, "rp-wa", string(types.StatusClosed), nil, []string{"rp-wc"}) +} + +// TestFinishDoltTransactionRecordingKeepsRecheckWhenDoltCommitFails pins the +// RunInTransaction half of #6716 where the regular SQL commit lands and the +// Dolt commit after it fails. The rows are committed, the error is +// ErrCommitIndeterminate and the write is never replayed, so the recorded +// dependents must still reach the post-commit recheck. +func TestFinishDoltTransactionRecordingKeepsRecheckWhenDoltCommitFails(t *testing.T) { + store, conn, tx, regularMock, ignoredMock := newTransactionPhaseFixture(t) + scopeAndRecordDependent(t, tx) + tx.dirty.MarkDirty("issues") + regularMock.ExpectCommit() + regularMock.ExpectQuery("SELECT COUNT\\(\\*\\) FROM dolt_status s"). + WillReturnRows(sqlmock.NewRows([]string{"count"}).AddRow(1)) + regularMock.ExpectExec("CALL DOLT_ADD\\(\\?\\)").WithArgs("issues"). + WillReturnError(errors.New("invalid connection")) + ignoredMock.ExpectRollback() + + pending, err := store.finishDoltTransactionRecording(context.Background(), conn, tx, "test: dolt commit fails") + if !errors.Is(err, ErrCommitIndeterminate) { + t.Fatalf("error = %v, want ErrCommitIndeterminate", err) + } + if !slices.Equal(pending.IssueIDs, []string{"rp-c"}) || !slices.Equal(pending.WispIDs, []string{"rp-wc"}) { + t.Fatalf("pending = %+v, want the dependents recorded on both transactions", pending) + } + requireTransactionPhaseMocks(t, regularMock, ignoredMock) +} + +// TestFinishDoltTransactionRecordingDropsRecheckWhenSQLCommitRefused is the +// other side: a regular SQL commit the server refused rolled back, so there +// is nothing committed to recheck. +func TestFinishDoltTransactionRecordingDropsRecheckWhenSQLCommitRefused(t *testing.T) { + store, conn, tx, regularMock, ignoredMock := newTransactionPhaseFixture(t) + scopeAndRecordDependent(t, tx) + regularMock.ExpectCommit().WillReturnError(&mysql.MySQLError{Number: 1213, Message: "serialization failure"}) + ignoredMock.ExpectRollback() + + pending, err := store.finishDoltTransactionRecording(context.Background(), conn, tx, "test: sql commit refused") + if err == nil || errors.Is(err, ErrCommitIndeterminate) { + t.Fatalf("error = %v, want a determinate commit failure", err) + } + if !pending.Empty() { + t.Fatalf("pending = %+v, want nothing after a refused SQL commit", pending) + } + requireTransactionPhaseMocks(t, regularMock, ignoredMock) +} diff --git a/internal/storage/dolt/blocked_recheck_raw.go b/internal/storage/dolt/blocked_recheck_raw.go new file mode 100644 index 000000000..cd9130ba9 --- /dev/null +++ b/internal/storage/dolt/blocked_recheck_raw.go @@ -0,0 +1,30 @@ +package dolt + +import ( + "context" + "database/sql" + + "github.com/steveyegge/beads/internal/storage/issueops" +) + +// settleBlockedRecheck runs the post-commit blocked-state recheck of what a +// committed write recorded (gastownhall/beads#6716) and logs a failure +// instead of returning it: the write is durable, and an error would read as +// "the write did not land" to every caller. +func (s *DoltStore) settleBlockedRecheck(ctx context.Context, pending issueops.BlockedRecheck) { + logBlockedRecheckFailure(ctx, pending, s.recheckBlockedAfterCommit(ctx, pending)) +} + +// commitSQLTxAndRecheck is commitSQLTx for a raw write transaction that was +// scoped with issueops.ScopeBlockedRecheckTransaction: once tx has committed +// it settles the dependents tx's unblocking writes recorded. The wisp writers +// and the legacy dependency removal run on raw transactions outside +// withWriteTx, so without this their writes would record into nothing and +// keep the #6716 race. +func (s *DoltStore) commitSQLTxAndRecheck(ctx context.Context, op string, tx *sql.Tx) error { + if err := s.commitSQLTx(ctx, op, tx); err != nil { + return err + } + s.settleBlockedRecheck(ctx, issueops.TakeBlockedRecheck(tx)) + return nil +} diff --git a/internal/storage/dolt/blocked_recheck_raw_test.go b/internal/storage/dolt/blocked_recheck_raw_test.go new file mode 100644 index 000000000..a7609d68f --- /dev/null +++ b/internal/storage/dolt/blocked_recheck_raw_test.go @@ -0,0 +1,143 @@ +package dolt + +import ( + "context" + "database/sql" + "fmt" + "slices" + "sync" + "testing" + + "github.com/steveyegge/beads/internal/storage" + "github.com/steveyegge/beads/internal/storage/issueops" + "github.com/steveyegge/beads/internal/types" +) + +// seedBlockedDependent creates blockers a and b (wisps when wisp is set) and +// the permanent dependent c that both block. +func seedBlockedDependent(t *testing.T, ctx context.Context, store *DoltStore, a, b, c string, wisp bool) { + t.Helper() + for _, id := range []string{a, b} { + if wisp { + createWisp(t, ctx, store, id) + } else { + createPerm(t, ctx, store, id) + } + } + createPerm(t, ctx, store, c) + addDependency(t, ctx, store, c, a, types.DepBlocks) + addDependency(t, ctx, store, c, b, types.DepBlocks) + assertIsBlocked(t, ctx, store, "issues", c, true) +} + +// closeOnSeparatePool closes id and commits it on a pool of its own, with the +// ordinary in-transaction recompute. The store's own transaction may hold +// every connection its pool has, so the sibling writer cannot borrow one. +func closeOnSeparatePool(t *testing.T, ctx context.Context, store *DoltStore, id string) { + t.Helper() + db, err := sql.Open("mysql", store.connStr) + if err != nil { + t.Fatalf("open sibling pool: %v", err) + } + defer func() { _ = db.Close() }() + tx, err := db.BeginTx(ctx, nil) + if err != nil { + t.Fatalf("begin sibling tx: %v", err) + } + if _, err := issueops.CloseIssueInTx(ctx, tx, id, "done", "sibling", ""); err != nil { + _ = tx.Rollback() + t.Fatalf("sibling close of %s: %v", id, err) + } + if err := tx.Commit(); err != nil { + t.Fatalf("commit sibling close of %s: %v", id, err) + } +} + +func assertReady(t *testing.T, ctx context.Context, store *DoltStore, id string) { + t.Helper() + assertIsBlocked(t, ctx, store, "issues", id, false) + ready, err := store.GetReadyWork(ctx, types.WorkFilter{}) + if err != nil { + t.Fatalf("GetReadyWork: %v", err) + } + if !slices.ContainsFunc(ready, func(i *types.Issue) bool { return i.ID == id }) { + t.Fatalf("%s has no open blockers but is missing from ready work", id) + } +} + +// TestBlockedRecheck_RunInTransactionCommittingLast is gastownhall/beads#6716 +// on DoltStore.RunInTransaction, the storage.Transaction runner `bd batch`, +// `bd cook`, `bd mol squash`/`burn` and SDK callers (Gas City's native store +// closes and status updates) write through. The transaction closes one +// blocker while both are open, a sibling close of the other commits, and the +// transaction commits last: without a post-commit recheck the dependent stays +// blocked. +func TestBlockedRecheck_RunInTransactionCommittingLast(t *testing.T) { + cases := []struct { + name string + wisp bool + write func(ctx context.Context, tx storage.Transaction, id string) error + }{ + {"close", false, func(ctx context.Context, tx storage.Transaction, id string) error { + return tx.CloseIssue(ctx, id, "done", "tester", "") + }}, + {"update to closed", false, func(ctx context.Context, tx storage.Transaction, id string) error { + return tx.UpdateIssue(ctx, id, map[string]interface{}{"status": types.StatusClosed}, "tester") + }}, + {"wisp close", true, func(ctx context.Context, tx storage.Transaction, id string) error { + return tx.CloseIssue(ctx, id, "done", "tester", "") + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + store, cleanup := setupConcurrentTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + seedBlockedDependent(t, ctx, store, "rt-a", "rt-b", "rt-c", tc.wisp) + + if err := store.RunInTransaction(ctx, "close rt-b", func(tx storage.Transaction) error { + if err := tc.write(ctx, tx, "rt-b"); err != nil { + return err + } + closeOnSeparatePool(t, ctx, store, "rt-a") + return nil + }); err != nil { + t.Fatalf("RunInTransaction: %v", err) + } + + assertReady(t, ctx, store, "rt-c") + }) + } +} + +// TestBlockedRecheck_ConcurrentWispCloses races the raw-transaction wisp +// close path (closeWisp) against itself. It cannot pin the interleaving the +// way the tests above do, so it only proves the settled outcome: whichever +// close commits last, the dependent ends up ready. +func TestBlockedRecheck_ConcurrentWispCloses(t *testing.T) { + store, cleanup := setupConcurrentTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + for round := 0; round < 5; round++ { + a, b, c := fmt.Sprintf("rw-%da", round), fmt.Sprintf("rw-%db", round), fmt.Sprintf("rw-%dc", round) + seedBlockedDependent(t, ctx, store, a, b, c, true) + var wg sync.WaitGroup + errs := make([]error, 2) + for i, id := range []string{a, b} { + wg.Add(1) + go func() { + defer wg.Done() + errs[i] = store.CloseIssue(ctx, id, "done", "tester", "") + }() + } + wg.Wait() + for _, err := range errs { + if err != nil { + t.Fatalf("round %d close: %v", round, err) + } + } + assertReady(t, ctx, store, c) + } +} diff --git a/internal/storage/dolt/dependencies.go b/internal/storage/dolt/dependencies.go index b909889ea..24e33bb76 100644 --- a/internal/storage/dolt/dependencies.go +++ b/internal/storage/dolt/dependencies.go @@ -104,10 +104,12 @@ func (s *DoltStore) RemoveDependencyWithOptions(ctx context.Context, issueID, de defer func() { _ = tx.Rollback() }() clearJournalScope := s.scopeEventsJournalTransaction(tx) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() if _, err := issueops.RemoveDependencyInTx(ctx, tx, issueID, dependsOnID, actor, rmOpts.EmitEvent); err != nil { return err } - if err := s.commitSQLTx(ctx, "commit remove wisp dependency", tx); err != nil { + if err := s.commitSQLTxAndRecheck(ctx, "commit remove wisp dependency", tx); err != nil { return err } return nil @@ -121,6 +123,8 @@ func (s *DoltStore) RemoveDependencyWithOptions(ctx context.Context, issueID, de clearJournalScope := s.scopeEventsJournalTransaction(tx) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() eventWritten, err := issueops.RemoveDependencyInTx(ctx, tx, issueID, dependsOnID, actor, rmOpts.EmitEvent) if err != nil { @@ -130,6 +134,10 @@ func (s *DoltStore) RemoveDependencyWithOptions(ctx context.Context, issueID, de if err := s.commitSQLTx(ctx, "sql commit", tx); err != nil { return err } + // The removal is durable; its dependents are settled after this + // removal's own Dolt commit so the recheck's commit follows it. + pending := issueops.TakeBlockedRecheck(tx) + defer s.settleBlockedRecheck(ctx, pending) // GH#2455: Use explicit DOLT_ADD to avoid sweeping up stale config changes. // Stage events only when RemoveDependencyInTx actually recorded a // dependency_removed event (explicit verb + genuine edge removal). A diff --git a/internal/storage/dolt/journal_scope_completeness_test.go b/internal/storage/dolt/journal_scope_completeness_test.go index 491eae888..0e709acc5 100644 --- a/internal/storage/dolt/journal_scope_completeness_test.go +++ b/internal/storage/dolt/journal_scope_completeness_test.go @@ -8,7 +8,7 @@ import ( // scopeCalls are the ways a function binds journal activation to the concrete // transaction it just began: the store helper, or the issueops primitive it -// wraps (runDoltTransaction calls the latter directly). +// wraps (runDoltTransactionRecording calls the latter directly). var scopeCalls = map[string]bool{ "scopeEventsJournalTransaction": true, "ScopeEventsJournalTransaction": true, @@ -88,7 +88,7 @@ func TestEveryRawTxJournalScopeIsScopedOrExempt(t *testing.T) { for key, f := range doltFns { // In scope: a function that mints its own transaction AND hands it to a // journaling mutator IN ITS OWN BODY. A function given a tx by - // withWriteTx/runDoltTransaction inherits their scoping, and those two + // withWriteTx/runDoltTransactionRecording inherits their scoping, and those two // are checked here in their own right. // // Reachability is deliberately DIRECT rather than a call-graph fixpoint. diff --git a/internal/storage/dolt/store.go b/internal/storage/dolt/store.go index 1c5a922ff..58c182966 100644 --- a/internal/storage/dolt/store.go +++ b/internal/storage/dolt/store.go @@ -873,20 +873,19 @@ var doltTracer = otel.Tracer("github.com/steveyegge/beads/storage/dolt") // Instruments are registered against the global delegating provider at init time, // so they automatically forward to the real provider once telemetry.Init() runs. var doltMetrics struct { - retryCount metric.Int64Counter - lockWaitMs metric.Float64Histogram - circuitTrips metric.Int64Counter - circuitRejected metric.Int64Counter - serializationErrors metric.Int64Counter - writeRetries metric.Int64Counter - postTxCommitDropped metric.Int64Counter - blockedRecheckDropped metric.Int64Counter - connAcquireMs metric.Float64Histogram - poolWaitCount metric.Int64Counter - poolWaitMs metric.Float64Histogram - claimVerifyLost metric.Int64Counter - claimVerifyRecovered metric.Int64Counter - ignoredTxFreshPool metric.Int64Counter + retryCount metric.Int64Counter + lockWaitMs metric.Float64Histogram + circuitTrips metric.Int64Counter + circuitRejected metric.Int64Counter + serializationErrors metric.Int64Counter + writeRetries metric.Int64Counter + postTxCommitDropped metric.Int64Counter + connAcquireMs metric.Float64Histogram + poolWaitCount metric.Int64Counter + poolWaitMs metric.Float64Histogram + claimVerifyLost metric.Int64Counter + claimVerifyRecovered metric.Int64Counter + ignoredTxFreshPool metric.Int64Counter } func init() { @@ -919,10 +918,6 @@ func init() { metric.WithDescription("Post-tx dolt commits abandoned after retries; the data landed but no dolt commit was minted (change rides the next commit on the branch)"), metric.WithUnit("{commit}"), ) - doltMetrics.blockedRecheckDropped, _ = m.Int64Counter("bd.db.blocked_recheck_dropped", - metric.WithDescription("Post-commit blocked-state rechecks abandoned; the write landed but dependents may carry a stale is_blocked flag until `bd recompute-blocked`"), - metric.WithUnit("{recheck}"), - ) doltMetrics.connAcquireMs, _ = m.Float64Histogram("bd.db.conn_acquire_ms", metric.WithDescription("Time to acquire a pooled connection for a Dolt transaction"), metric.WithUnit("ms"), @@ -1300,8 +1295,7 @@ func logBlockedRecheckFailure(ctx context.Context, pending issueops.BlockedReche if err == nil { return } - doltMetrics.blockedRecheckDropped.Add(ctx, 1) - log.Printf("warning: %s", issueops.BlockedRecheckFailureMessage(pending, err)) + issueops.ReportBlockedRecheckFailure(ctx, pending, err) } // SetEventsJournalEnabled activates the journal for this store instance only. diff --git a/internal/storage/dolt/transaction.go b/internal/storage/dolt/transaction.go index cd8af9c7b..e5aab74db 100644 --- a/internal/storage/dolt/transaction.go +++ b/internal/storage/dolt/transaction.go @@ -156,6 +156,21 @@ func (t *doltTransaction) dirtyTableNames() []string { } func (s *DoltStore) runDoltTransaction(ctx context.Context, commitMsg string, fn func(tx storage.Transaction) error) error { + pending, err := s.runDoltTransactionRecording(ctx, commitMsg, fn) + // Settled here, after the pinned connection and any borrowed ignored-table + // connection have gone back to the pool: the recheck needs one of its own. + // pending is non-empty only when the regular SQL commit landed or may have + // landed, so this also runs when the Dolt commit after it failed + // (ErrCommitIndeterminate): those rows are committed all the same. + s.settleBlockedRecheck(ctx, pending) + return err +} + +// runDoltTransactionRecording is runDoltTransaction's transaction. Once it has +// committed it hands back the dependents the callback's unblocking writes +// recorded on either of its SQL transactions, for the post-commit +// blocked-state recheck (gastownhall/beads#6716). +func (s *DoltStore) runDoltTransactionRecording(ctx context.Context, commitMsg string, fn func(tx storage.Transaction) error) (issueops.BlockedRecheck, error) { // Pin a single connection for the entire operation: SQL transaction, // config protection, and DOLT_COMMIT must all run on the same Dolt // session. Each pool connection has an independent working set in Dolt @@ -182,18 +197,18 @@ func (s *DoltStore) runDoltTransaction(ctx context.Context, commitMsg string, fn } if err != nil { - return fmt.Errorf("failed to acquire connection: %w", err) + return issueops.BlockedRecheck{}, fmt.Errorf("failed to acquire connection: %w", err) } defer conn.Close() var currentBranch string if err := conn.QueryRowContext(ctx, "SELECT active_branch()").Scan(¤tBranch); err != nil { - return fmt.Errorf("failed to read active branch: %w", err) + return issueops.BlockedRecheck{}, fmt.Errorf("failed to read active branch: %w", err) } regularTx, err := conn.BeginTx(ctx, nil) if err != nil { - return fmt.Errorf("failed to begin regular tx: %w", err) + return issueops.BlockedRecheck{}, fmt.Errorf("failed to begin regular tx: %w", err) } // The journal counter and rows must commit in the SAME SQL transaction as @@ -220,12 +235,21 @@ func (s *DoltStore) runDoltTransaction(ctx context.Context, commitMsg string, fn ignoredCleanup, ignoredTx, err = s.beginIgnoredTxOnBranch(ctx, currentBranch) if err != nil { _ = regularTx.Rollback() - return err + return issueops.BlockedRecheck{}, err } defer ignoredCleanup() } clearJournalScope := issueops.ScopeEventsJournalTransaction(regularTx, journalEnabled) defer clearJournalScope() + // Unblocking writes record their dependents on whichever transaction they + // ran in: the regular one, or the ignored one wisps use when the journal + // does not pin both planes together. + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(regularTx) + defer clearRecheckScope() + if !journalEnabled { + clearIgnoredRecheckScope := issueops.ScopeBlockedRecheckTransaction(ignoredTx) + defer clearIgnoredRecheckScope() + } tx := &doltTransaction{regularTx: regularTx, ignoredTx: ignoredTx, store: s, journalPinned: journalEnabled} @@ -244,10 +268,34 @@ func (s *DoltStore) runDoltTransaction(ctx context.Context, commitMsg string, fn if !journalEnabled { _ = ignoredTx.Rollback() } - return err + return issueops.BlockedRecheck{}, err } - return s.finishDoltTransaction(ctx, conn, tx, commitMsg) + return s.finishDoltTransactionRecording(ctx, conn, tx, commitMsg) +} + +// finishDoltTransactionRecording is finishDoltTransaction that also hands back +// the dependents the transaction's unblocking writes recorded, on either of its +// SQL transactions, for the post-commit blocked-state recheck. +// +// They are handed back with the error when the commit is indeterminate +// (ErrCommitIndeterminate): the regular SQL commit may have landed, or did land +// and only the Dolt commit or the ignored-table commit after it failed. The +// rows are then committed without a caller being able to replay the write, so +// the recheck is the only thing that can settle their dependents; being a full +// recompute of committed state, it is harmless if the commit did not land. +// Any other failure means the regular SQL commit was refused and rolled back, +// and nothing is handed back. +func (s *DoltStore) finishDoltTransactionRecording(ctx context.Context, conn *sql.Conn, tx *doltTransaction, commitMsg string) (issueops.BlockedRecheck, error) { + err := s.finishDoltTransaction(ctx, conn, tx, commitMsg) + if err != nil && !errors.Is(err, ErrCommitIndeterminate) { + return issueops.BlockedRecheck{}, err + } + pending := issueops.TakeBlockedRecheck(tx.regularTx) + if !tx.journalPinned { + pending = pending.Merge(issueops.TakeBlockedRecheck(tx.ignoredTx)) + } + return pending, err } // finishDoltTransaction commits the regular SQL transaction, its associated diff --git a/internal/storage/dolt/wisps.go b/internal/storage/dolt/wisps.go index 36b6bee66..e8452a674 100644 --- a/internal/storage/dolt/wisps.go +++ b/internal/storage/dolt/wisps.go @@ -194,12 +194,14 @@ func (s *DoltStore) updateWisp(ctx context.Context, id string, updates map[strin clearJournalScope := s.scopeEventsJournalTransaction(tx) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() if _, err := issueops.UpdateIssueInTx(ctx, tx, id, updates, actor); err != nil { return err } - return s.commitSQLTx(ctx, "commit update wisp", tx) + return s.commitSQLTxAndRecheck(ctx, "commit update wisp", tx) } // updateWispChecked updates a wisp with the optional atomic preconditions of @@ -221,6 +223,8 @@ func (s *DoltStore) updateWispChecked(ctx context.Context, id string, updates ma clearJournalScope := s.scopeEventsJournalTransaction(tx) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() if opts.ExpectedVersion != nil { if err := issueops.CheckVersionInTx(ctx, tx, id, *opts.ExpectedVersion); err != nil { @@ -234,7 +238,7 @@ func (s *DoltStore) updateWispChecked(ctx context.Context, id string, updates ma return err } - return s.commitSQLTx(ctx, "commit update wisp", tx) + return s.commitSQLTxAndRecheck(ctx, "commit update wisp", tx) } // closeWisp closes a wisp in the wisps table. @@ -249,12 +253,14 @@ func (s *DoltStore) closeWisp(ctx context.Context, id string, reason string, act clearJournalScope := s.scopeEventsJournalTransaction(tx) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() if _, err := issueops.CloseIssueInTx(ctx, tx, id, reason, actor, session); err != nil { return err } - return s.commitSQLTx(ctx, "commit close wisp", tx) + return s.commitSQLTxAndRecheck(ctx, "commit close wisp", tx) } // closeWispChecked closes a wisp with the is_blocked guard, mirroring closeWisp @@ -282,13 +288,15 @@ func (s *DoltStore) closeWispChecked(ctx context.Context, id string, actor strin clearJournalScope := s.scopeEventsJournalTransaction(tx) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() res, err := issueops.CloseIssueCheckedInTx(ctx, tx, id, opts.Reason, actor, opts.Session, opts.Force, opts.ExpectedVersion) if err != nil { return storage.CloseIssueResult{}, err } - if err := s.commitSQLTx(ctx, "commit close wisp", tx); err != nil { + if err := s.commitSQLTxAndRecheck(ctx, "commit close wisp", tx); err != nil { return storage.CloseIssueResult{}, err } return storage.CloseIssueResult{Unchanged: res.AlreadyClosed, OpenChildren: res.OpenChildren}, nil @@ -304,6 +312,8 @@ func (s *DoltStore) deleteWisp(ctx context.Context, id string) error { clearJournalScope := s.scopeEventsJournalTransaction(tx) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() affectedIssues, affectedWisps, aerr := issueops.AffectedByDeletionInTx(ctx, tx, nil, []string{id}) if aerr != nil { @@ -347,8 +357,9 @@ func (s *DoltStore) deleteWisp(ctx context.Context, id string) error { if err := issueops.RecomputeIsBlockedInTx(ctx, tx, affectedIssues, affectedWisps); err != nil { return fmt.Errorf("recompute is_blocked after wisp delete for %s: %w", id, err) } + issueops.NoteDeleteBlockedRecheck(tx, []string{id}, "", affectedIssues, affectedWisps) - return s.commitSQLTx(ctx, "commit delete wisp", tx) + return s.commitSQLTxAndRecheck(ctx, "commit delete wisp", tx) } // deleteWispBatch permanently removes multiple wisps using one transaction per @@ -397,6 +408,8 @@ func (s *DoltStore) deleteWispBatchTx(ctx context.Context, ids []string) (int, e clearJournalScope := s.scopeEventsJournalTransaction(tx) defer clearJournalScope() + clearRecheckScope := issueops.ScopeBlockedRecheckTransaction(tx) + defer clearRecheckScope() affectedIssues, affectedWisps, aerr := issueops.AffectedByDeletionInTx(ctx, tx, nil, ids) if aerr != nil { @@ -448,8 +461,9 @@ func (s *DoltStore) deleteWispBatchTx(ctx context.Context, ids []string) (int, e if err := issueops.RecomputeIsBlockedInTx(ctx, tx, affectedIssues, affectedWisps); err != nil { return 0, fmt.Errorf("recompute is_blocked after batched wisp delete: %w", err) } + issueops.NoteDeleteBlockedRecheck(tx, ids, "", affectedIssues, affectedWisps) - if err := s.commitSQLTx(ctx, "commit batch wisp delete", tx); err != nil { + if err := s.commitSQLTxAndRecheck(ctx, "commit batch wisp delete", tx); err != nil { return 0, err } diff --git a/internal/storage/domain/db/delete_repo_test.go b/internal/storage/domain/db/delete_repo_test.go index d81095731..d6857d38f 100644 --- a/internal/storage/domain/db/delete_repo_test.go +++ b/internal/storage/domain/db/delete_repo_test.go @@ -237,7 +237,7 @@ func (s *testSuite) issueAffectedByDeletionDependers() { } func (s *testSuite) issueRecomputeIsBlockedEmpty() { - s.Require().NoError(s.issueRepo().RecomputeIsBlocked(s.Ctx(), nil, nil)) + s.Require().NoError(s.issueRepo().RecomputeIsBlockedAfterDelete(s.Ctx(), nil, nil, nil)) } func (s *testSuite) issueRecomputeIsBlockedFlips() { @@ -247,7 +247,7 @@ func (s *testSuite) issueRecomputeIsBlockedFlips() { newDep("bd-del-rib-depender", "bd-del-rib-blocker", types.DepBlocks), "tester", domain.DepInsertOpts{})) - s.Require().NoError(s.issueRepo().RecomputeIsBlocked(s.Ctx(), + s.Require().NoError(s.issueRepo().RecomputeIsBlockedAfterDelete(s.Ctx(), nil, []string{"bd-del-rib-depender"}, nil)) var blocked int s.Require().NoError(s.Runner().QueryRowContext(s.Ctx(), @@ -257,7 +257,7 @@ func (s *testSuite) issueRecomputeIsBlockedFlips() { _, err := s.Runner().ExecContext(s.Ctx(), "UPDATE issues SET status = 'closed' WHERE id = ?", "bd-del-rib-blocker") s.Require().NoError(err) - s.Require().NoError(s.issueRepo().RecomputeIsBlocked(s.Ctx(), + s.Require().NoError(s.issueRepo().RecomputeIsBlockedAfterDelete(s.Ctx(), nil, []string{"bd-del-rib-depender"}, nil)) s.Require().NoError(s.Runner().QueryRowContext(s.Ctx(), "SELECT is_blocked FROM issues WHERE id = ?", "bd-del-rib-depender").Scan(&blocked)) diff --git a/internal/storage/domain/db/dependency.go b/internal/storage/domain/db/dependency.go index 61819c02d..6b45e672f 100644 --- a/internal/storage/domain/db/dependency.go +++ b/internal/storage/domain/db/dependency.go @@ -383,6 +383,7 @@ func (r *dependencySQLRepositoryImpl) Delete(ctx context.Context, issueID, depen if err := issueops.RecomputeIsBlockedInTx(ctx, r.runner, affectedIssues, affectedWisps); err != nil { return domain.DepDeleteResult{}, fmt.Errorf("db: DependencySQLRepository.Delete: recompute is_blocked: %w", err) } + issueops.NoteDependencyRemovalBlockedRecheck(r.runner, issueID, dependsOnID, affectedIssues, affectedWisps) // Snapshot only after all derived blocked-state maintenance has completed. // Never gated on opts.EmitEvent — a structural removal is as real to a diff --git a/internal/storage/domain/db/issue.go b/internal/storage/domain/db/issue.go index b44cdc2a1..eb19947af 100644 --- a/internal/storage/domain/db/issue.go +++ b/internal/storage/domain/db/issue.go @@ -323,6 +323,9 @@ func (r *issueSQLRepositoryImpl) Update(ctx context.Context, id string, updates if err := issueops.RecomputeIsBlockedInTx(ctx, r.runner, affectedIssues, affectedWisps); err != nil { return fmt.Errorf("db: Update %s: recompute is_blocked: %w", id, err) } + if !newActive { + issueops.NoteStatusChangeBlockedRecheck(r.runner, id, string(newStatus), affectedIssues, affectedWisps) + } } } // Snapshot only after all derived blocked-state maintenance has completed. @@ -1070,8 +1073,12 @@ func (r *issueSQLRepositoryImpl) AffectedByDeletion(ctx context.Context, issueID return issueops.AffectedByDeletionInTx(ctx, r.runner, issueIDs, wispIDs) } -func (r *issueSQLRepositoryImpl) RecomputeIsBlocked(ctx context.Context, issueIDs, wispIDs []string) error { - return issueops.RecomputeIsBlockedInTx(ctx, r.runner, issueIDs, wispIDs) +func (r *issueSQLRepositoryImpl) RecomputeIsBlockedAfterDelete(ctx context.Context, deletedIDs, issueIDs, wispIDs []string) error { + if err := issueops.RecomputeIsBlockedInTx(ctx, r.runner, issueIDs, wispIDs); err != nil { + return err + } + issueops.NoteDeleteBlockedRecheck(r.runner, deletedIDs, "", issueIDs, wispIDs) + return nil } func (r *issueSQLRepositoryImpl) AsOf(ctx context.Context, id, ref string) (*types.Issue, error) { diff --git a/internal/storage/domain/issue.go b/internal/storage/domain/issue.go index a8137338a..c73f2c4d2 100644 --- a/internal/storage/domain/issue.go +++ b/internal/storage/domain/issue.go @@ -89,7 +89,10 @@ type IssueSQLRepository interface { FindAllDependents(ctx context.Context, ids []string) ([]string, error) FindWispDependentsRecursive(ctx context.Context, ids []string) (map[string]bool, error) AffectedByDeletion(ctx context.Context, issueIDs, wispIDs []string) (affectedIssues, affectedWisps []string, err error) - RecomputeIsBlocked(ctx context.Context, issueIDs, wispIDs []string) error + // RecomputeIsBlockedAfterDelete recomputes the blocked state of the + // dependents a delete of deletedIDs affected, and records them for the + // post-commit recheck the unit of work runs (gastownhall/beads#6716). + RecomputeIsBlockedAfterDelete(ctx context.Context, deletedIDs, issueIDs, wispIDs []string) error Close(ctx context.Context, id string, params CloseRowParams, actor string, opts IssueTableOpts) (CloseRowResult, error) CloseChecked(ctx context.Context, id string, params CloseRowParams, actor string, force bool) (CloseRowResult, error) Reopen(ctx context.Context, id string, params ReopenRowParams, actor string, opts IssueTableOpts) (ReopenRowResult, error) diff --git a/internal/storage/domain/issue_delete.go b/internal/storage/domain/issue_delete.go index a07a7fb5d..881db741b 100644 --- a/internal/storage/domain/issue_delete.go +++ b/internal/storage/domain/issue_delete.go @@ -211,7 +211,7 @@ func (u *issueUseCaseImpl) deleteMany(ctx context.Context, params DeleteIssuesPa result.ReferencesUpdated = refs } - if err := u.issueRepo.RecomputeIsBlocked(ctx, affectedIssues, affectedWisps); err != nil { + if err := u.issueRepo.RecomputeIsBlockedAfterDelete(ctx, allIDs, affectedIssues, affectedWisps); err != nil { return result, fmt.Errorf("delete: recompute is_blocked: %w", err) } diff --git a/internal/storage/issueops/blocked_recheck.go b/internal/storage/issueops/blocked_recheck.go index 7204704bd..d31b67c66 100644 --- a/internal/storage/issueops/blocked_recheck.go +++ b/internal/storage/issueops/blocked_recheck.go @@ -33,36 +33,26 @@ import ( // // # What is scoped, and what is not // -// Recording only reaches a recheck through a transaction a store scoped (see -// ScopeBlockedRecheckTransaction), and exactly two write runners scope one: -// DoltStore's commitWriteTx and EmbeddedDoltStore's commitConn. A write that -// reaches this package on any other transaction records nothing and behaves -// exactly as it did before the recheck existed — the skew above survives on -// it. Those routes are known and deliberately out of scope here: +// Recording only reaches a recheck through a transaction a write runner +// scoped (see ScopeBlockedRecheckTransaction). The runners that scope one, +// and settle what it recorded once it has committed, are: // -// - the uow/domain-db routing system (internal/storage/uow, with the -// in-transaction recomputes at internal/storage/domain/db/issue.go and -// dependency.go), which serves `bd batch` when proxied and every write a -// uow provider serves. Scoping it needs more than these two lines: -// doltServerTx.Runner() hands out the pinned *sql.Conn, not the *sql.Tx -// this package keys its scope on, so the key has to be threaded through -// the wrapper — the caveat below, already real rather than hypothetical. -// - storage.Transaction (internal/storage/dolt/transaction.go, reached -// through DoltStore.RunInTransaction on a pinned connection), which is -// how `bd batch` closes, status updates and dep removals land when not -// proxied, and how `bd cook`'s proto-subgraph delete, `bd mol squash` and -// `bd mol burn` delete and close. -// - the legacy DoltStore.RemoveDependencyWithOptions (raw BeginTx in -// internal/storage/dolt/dependencies.go), which `bd duplicates --merge` -// uses to remove parent-child edges. -// - the wisp writers (internal/storage/dolt/wisps.go closeWisp/updateWisp/ -// deleteWisp and ephemeral_routing.go's demote-to-wisp), which run raw -// unscoped transactions. The recompute joins depends_on_wisp_id, so a -// permanent issue blocked by two wisps whose removals race keeps the same -// stale flag. +// - DoltStore's commitWriteTx (withWriteTx/withRetryTx), its storage.Transaction +// runner (RunInTransaction, which scopes both its regular and its ignored +// transaction), and the raw transactions of its wisp writers and of the +// legacy RemoveDependencyWithOptions; +// - EmbeddedDoltStore's commitConn, which every embedded write, including +// RunInTransaction, goes through; +// - the uow Tx (internal/storage/uow), which serves every write a uow +// provider serves: the proxied CLI, `bd batch` when proxied, and bd serve. +// It scopes the pinned *sql.Conn its repositories run on, and the +// domain/db repositories record through the Note* helpers below exactly +// where the issueops write paths do. // -// Every one of these leaves the pre-existing, operator-repairable state -// (`bd doctor`, `bd recompute-blocked`); none is made worse by the recheck. +// A write that reaches this package on any other transaction records nothing +// and behaves exactly as it did before the recheck existed; that leaves the +// operator-repairable state (`bd doctor`, `bd recompute-blocked`) the recheck +// was added to prevent. type BlockedRecheck struct { IssueIDs []string WispIDs []string @@ -222,6 +212,43 @@ func TakeBlockedRecheck(tx DBTX) BlockedRecheck { return taken } +// Merge returns the union of r and other: every id either recorded, and the +// sources and counts of both. A runner that scoped two transactions for one +// write (DoltStore's split regular/ignored transactions) settles them with one +// recheck. +func (r BlockedRecheck) Merge(other BlockedRecheck) BlockedRecheck { + merged := BlockedRecheck{SourceCount: r.SourceCount + other.SourceCount} + merged.IssueIDs, _ = appendRecheckIDs(append([]string(nil), r.IssueIDs...), other.IssueIDs, nil) + merged.WispIDs, _ = appendRecheckIDs(append([]string(nil), r.WispIDs...), other.WispIDs, nil) + merged.Sources, _ = appendRecheckIDs(append([]string(nil), r.Sources...), other.Sources, nil) + if len(merged.Sources) > recheckSourceLimit { + merged.Sources = merged.Sources[:recheckSourceLimit] + } + return merged +} + +// NoteStatusChangeBlockedRecheck records, for the post-commit recheck, the +// dependents a move of id to the inactive newStatus (a close included) +// recomputed in tx. The issue itself is excluded: a concurrent writer +// conflicts on its row rather than racing past it. +func NoteStatusChangeBlockedRecheck(tx DBTX, id, newStatus string, affectedIssues, affectedWisps []string) { + noteBlockedRecheck(tx, statusChangeRecheckLabel(id, newStatus), []string{id}, affectedIssues, affectedWisps) +} + +// NoteDependencyRemovalBlockedRecheck records the dependents a removal of the +// issueID -> dependsOnID edge recomputed in tx. Nothing is excluded: the +// dependent is exactly the row a racing unblocking write can leave stale. +func NoteDependencyRemovalBlockedRecheck(tx DBTX, issueID, dependsOnID string, affectedIssues, affectedWisps []string) { + noteBlockedRecheck(tx, fmt.Sprintf("dependency removal %s -> %s", issueID, dependsOnID), nil, affectedIssues, affectedWisps) +} + +// NoteDeleteBlockedRecheck records the dependents a delete of deletedIDs +// recomputed in tx, leaving out the deleted rows themselves. scope, when set, +// says where the ids came from ("from "). +func NoteDeleteBlockedRecheck(tx DBTX, deletedIDs []string, scope string, affectedIssues, affectedWisps []string) { + noteBlockedRecheck(tx, deleteRecheckLabel(deletedIDs, scope), deletedIDs, affectedIssues, affectedWisps) +} + // noteBlockedRecheck records the dependents a write recomputed in tx. source // labels the write for the recheck's commit message. exclude names ids this // call leaves out: for a close or status update, the issue whose row this diff --git a/internal/storage/issueops/blocked_recheck_report.go b/internal/storage/issueops/blocked_recheck_report.go new file mode 100644 index 000000000..eb09b5ba5 --- /dev/null +++ b/internal/storage/issueops/blocked_recheck_report.go @@ -0,0 +1,39 @@ +package issueops + +import ( + "context" + "log" + + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/metric" +) + +// blockedRecheckDropped counts post-commit rechecks that failed. It is +// registered on the storage meter every write runner reports through, against +// the global delegating provider, so it forwards once telemetry.Init() runs. +var blockedRecheckDropped metric.Int64Counter + +func init() { + blockedRecheckDropped, _ = otel.Meter("github.com/steveyegge/beads/storage/dolt").Int64Counter("bd.db.blocked_recheck_dropped", + metric.WithDescription("Post-commit blocked-state rechecks abandoned; the write landed but dependents may carry a stale is_blocked flag until `bd recompute-blocked`"), + metric.WithUnit("{recheck}"), + ) +} + +// ReportBlockedRecheckFailure is the one channel a write runner reports a +// failed post-commit recheck through, instead of returning it: the write it +// followed is committed and durable, and every caller of a store write reads +// an error as "the mutation did not land", so surfacing this one would make +// automated callers retry and double-apply. What is left behind is the stale +// is_blocked flag `bd doctor` and `bd recompute-blocked` repair. +// +// Because the failure stops here, the counter and the line are its only +// trace. The counter is what a fleet alerts on; the line names the rows to +// repair. +func ReportBlockedRecheckFailure(ctx context.Context, pending BlockedRecheck, err error) { + if err == nil { + return + } + blockedRecheckDropped.Add(ctx, 1) + log.Printf("warning: %s", BlockedRecheckFailureMessage(pending, err)) +} diff --git a/internal/storage/issueops/bulk_ops.go b/internal/storage/issueops/bulk_ops.go index e96d5b513..636ae1819 100644 --- a/internal/storage/issueops/bulk_ops.go +++ b/internal/storage/issueops/bulk_ops.go @@ -234,7 +234,7 @@ func DeleteIssuesBySourceRepoInTx(ctx context.Context, tx *sql.Tx, sourceRepo st if err := RecomputeIsBlockedInTx(ctx, tx, affectedIssues, affectedWisps); err != nil { return int(rowsAffected), fmt.Errorf("recompute is_blocked after source-repo delete: %w", err) } - noteBlockedRecheck(tx, deleteRecheckLabel(issueIDs, "from "+sourceRepo), issueIDs, affectedIssues, affectedWisps) + NoteDeleteBlockedRecheck(tx, issueIDs, "from "+sourceRepo, affectedIssues, affectedWisps) return int(rowsAffected), nil } diff --git a/internal/storage/issueops/close.go b/internal/storage/issueops/close.go index 6b8242996..170a0218b 100644 --- a/internal/storage/issueops/close.go +++ b/internal/storage/issueops/close.go @@ -381,7 +381,7 @@ func closeIssueInTx(ctx context.Context, tx DBTX, id string, reason, actor, sess if err != nil { return nil, fmt.Errorf("recompute is_blocked after close for %s: %w", id, err) } - noteBlockedRecheck(tx, "close of "+id, []string{id}, affectedIssues, affectedWisps) + NoteStatusChangeBlockedRecheck(tx, id, string(types.StatusClosed), affectedIssues, affectedWisps) // Snapshot only after all derived blocked-state maintenance has completed. // recordEvent gates the human audit event, never the journal. diff --git a/internal/storage/issueops/delete.go b/internal/storage/issueops/delete.go index e0406c8f2..f94f48ec2 100644 --- a/internal/storage/issueops/delete.go +++ b/internal/storage/issueops/delete.go @@ -51,7 +51,7 @@ func DeleteIssueInTx(ctx context.Context, tx *sql.Tx, id string, actor string) e if err := RecomputeIsBlockedInTx(ctx, tx, affectedIssues, affectedWisps); err != nil { return fmt.Errorf("recompute is_blocked after delete for %s: %w", id, err) } - noteBlockedRecheck(tx, deleteRecheckLabel([]string{id}, ""), []string{id}, affectedIssues, affectedWisps) + NoteDeleteBlockedRecheck(tx, []string{id}, "", affectedIssues, affectedWisps) return nil } @@ -392,7 +392,7 @@ func DeleteResolvedSetInTx(ctx context.Context, tx *sql.Tx, set DeletionSet, dry if err := RecomputeIsBlockedInTx(ctx, tx, affectedIssues, affectedWisps); err != nil { return nil, fmt.Errorf("recompute is_blocked after batch delete: %w", err) } - noteBlockedRecheck(tx, deleteRecheckLabel(set.All, ""), set.All, affectedIssues, affectedWisps) + NoteDeleteBlockedRecheck(tx, set.All, "", affectedIssues, affectedWisps) return result, nil } diff --git a/internal/storage/issueops/dependencies.go b/internal/storage/issueops/dependencies.go index 755ec8da8..01efd856c 100644 --- a/internal/storage/issueops/dependencies.go +++ b/internal/storage/issueops/dependencies.go @@ -1062,9 +1062,7 @@ func removeDependencyInTx(ctx context.Context, tx *sql.Tx, issueID, dependsOnID, return false, fmt.Errorf("recompute is_blocked after remove dependency %s -> %s: %w", issueID, dependsOnID, err) } mergeRecomputeIsBlockedResult(recomputeResult, recomputed) - // The dependent is exactly the row a racing unblocking write can leave - // stale, so nothing is excluded from the post-commit recheck. - noteBlockedRecheck(tx, fmt.Sprintf("dependency removal %s -> %s", issueID, dependsOnID), nil, affectedIssues, affectedWisps) + NoteDependencyRemovalBlockedRecheck(tx, issueID, dependsOnID, affectedIssues, affectedWisps) // Snapshot only after all derived blocked-state maintenance has completed. // Never gated on emitEvent — a structural removal is as real to a replaying // consumer as one from an explicit dep verb. diff --git a/internal/storage/issueops/update.go b/internal/storage/issueops/update.go index 10fda9f09..c0958ed6a 100644 --- a/internal/storage/issueops/update.go +++ b/internal/storage/issueops/update.go @@ -535,7 +535,7 @@ func updateIssueInTx(ctx context.Context, tx DBTX, id string, updates map[string updateResult.IssueRowsChanged = !isWisp || recompute.IssueRowsChanged updateResult.WispRowsChanged = isWisp || recompute.WispRowsChanged if !newActive { - noteBlockedRecheck(tx, statusChangeRecheckLabel(id, newStatus), []string{id}, affectedIssues, affectedWisps) + NoteStatusChangeBlockedRecheck(tx, id, newStatus, affectedIssues, affectedWisps) } } } diff --git a/internal/storage/uow/blocked_recheck.go b/internal/storage/uow/blocked_recheck.go new file mode 100644 index 000000000..4fc2ca8eb --- /dev/null +++ b/internal/storage/uow/blocked_recheck.go @@ -0,0 +1,80 @@ +package uow + +import ( + "context" + "fmt" + + "github.com/cenkalti/backoff/v4" + + "github.com/steveyegge/beads/internal/storage/issueops" +) + +// recheckBlockedAfterCommit recomputes, on the pinned session and a fresh +// snapshot, the blocked state of the dependents a committed unit of work's +// unblocking writes recorded (gastownhall/beads#6716). The in-transaction +// recompute read that unit of work's start snapshot, so two proxied writes +// racing on the blockers of one dependent each saw the other blocker still in +// place and both committed leaving it blocked and hidden from `bd ready`. A +// recompute over the same ids after commit sees both writes. +// +// It runs no SQL when nothing was recorded. It commits the way the unit of +// work did (commitStatement): a Dolt commit named for the writes when it +// changed an issues row and version commits are not deferred, a plain COMMIT +// otherwise. A serialization loss is retried on a new transaction. +// +// The write it follows is durable, so a failure is reported through +// issueops.ReportBlockedRecheckFailure (the counter and warning the store +// runners use), never returned: an error from Commit tells every caller the +// write did not land. What a failure +// leaves is the stale flag `bd doctor` and `bd recompute-blocked` repair. It +// reports whether the session is clean enough to go back to the pool; false +// means a transaction may still be open on it and it must be poisoned. +func (t *doltServerTx) recheckBlockedAfterCommit(ctx context.Context, pending issueops.BlockedRecheck) bool { + if pending.Empty() || issueops.InBlockedRecheck(ctx) { + return true + } + ctx, cancel := issueops.BlockedRecheckContext(ctx) + defer cancel() + + bo := backoff.NewExponentialBackOff() + bo.InitialInterval = txRetryInitialInterval + bo.MaxElapsedTime = DefaultTxRetryMaxElapsed + clean := true + err := backoff.Retry(func() error { + err := t.recheckBlockedOnce(ctx, pending) + if err == nil || isSerializationError(err) { + // A serialization failure means the server already rolled the + // transaction back, so the session is idle for the next attempt. + return err + } + if _, rbErr := t.conn.ExecContext(ctx, "ROLLBACK;"); rbErr != nil { + clean = false + } + return backoff.Permanent(err) + }, backoff.WithContext(bo, ctx)) + if err != nil { + issueops.ReportBlockedRecheckFailure(ctx, pending, issueops.BlockedRecheckFailed(err)) + } + return clean +} + +// recheckBlockedOnce is one recheck transaction on the pinned session. +func (t *doltServerTx) recheckBlockedOnce(ctx context.Context, pending issueops.BlockedRecheck) error { + if _, err := t.conn.ExecContext(ctx, "START TRANSACTION;"); err != nil { + return fmt.Errorf("begin recheck: %w", err) + } + result, err := issueops.RecomputeIsBlockedInTxWithResult(ctx, t.conn, pending.IssueIDs, pending.WispIDs) + if err != nil { + return err + } + message := "" + if result.IssueRowsChanged { + message = pending.CommitMessage() + } + stmt, args, err := t.commitStatement(ctx, message) + if err != nil { + return err + } + _, err = t.conn.ExecContext(ctx, stmt, args...) + return err +} diff --git a/internal/storage/uow/blocked_recheck_race_test.go b/internal/storage/uow/blocked_recheck_race_test.go new file mode 100644 index 000000000..c03cfb386 --- /dev/null +++ b/internal/storage/uow/blocked_recheck_race_test.go @@ -0,0 +1,174 @@ +package uow + +import ( + "context" + "slices" + "testing" + "time" + + "github.com/steveyegge/beads/internal/storage/domain" + "github.com/steveyegge/beads/internal/types" +) + +// TestProxiedBlockedRecheckAfterRacingUnblocks is gastownhall/beads#6716 on +// the proxied (uow) write path, the one every proxied `bd close`, +// `bd update --status closed`, `bd dep remove`, `bd delete`, proxied batch +// close and bd serve reach. +// +// Two blockers A and B of one dependent C are taken away by two units of +// work whose transactions overlap: the first begins and writes while both +// blockers are open, the second then writes and commits, and the first +// commits last. Each in-transaction recompute saw the other blocker still in +// place, so neither wrote C, and without a post-commit recheck C stays +// is_blocked=1 and hidden from ready work on every committed snapshot. +func TestProxiedBlockedRecheckAfterRacingUnblocks(t *testing.T) { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + const prefix = "rbr" + provider := newUOWRoleFixtureProvider(t, ctx, prefix) + kit := newUOWRoleFixtureKit(provider, prefix) + + closeIssue := func(id string) func(context.Context, UnitOfWork) error { + return func(ctx context.Context, uw UnitOfWork) error { + _, err := uw.IssueUseCase().CloseIssue(ctx, id, domain.CloseIssueParams{Reason: "done"}, "racer") + return err + } + } + closeChecked := func(id string) func(context.Context, UnitOfWork) error { + return func(ctx context.Context, uw UnitOfWork) error { + _, err := uw.IssueUseCase().CloseIssueChecked(ctx, id, domain.CloseIssueParams{Reason: "done"}, "racer", false) + return err + } + } + updateClosed := func(id string) func(context.Context, UnitOfWork) error { + return func(ctx context.Context, uw UnitOfWork) error { + return uw.IssueUseCase().UpdateIssue(ctx, id, map[string]any{"status": string(types.StatusClosed)}, "racer") + } + } + closeWisp := func(id string) func(context.Context, UnitOfWork) error { + return func(ctx context.Context, uw UnitOfWork) error { + _, err := uw.IssueUseCase().CloseWisp(ctx, id, domain.CloseIssueParams{Reason: "done"}, "racer") + return err + } + } + + cases := []struct { + name string + // wispBlockers seeds A and B as wisps instead of issues. + wispBlockers bool + // first and second take blocker A and blocker B away; first commits last. + first, second func(a, b, c string) func(context.Context, UnitOfWork) error + }{ + { + name: "close vs close", + first: func(a, _, _ string) func(context.Context, UnitOfWork) error { return closeIssue(a) }, + second: func(_, b, _ string) func(context.Context, UnitOfWork) error { return closeIssue(b) }, + }, + { + // The batch closer's per-item close (bd close a b, proxied). + name: "checked close vs checked close", + first: func(a, _, _ string) func(context.Context, UnitOfWork) error { return closeChecked(a) }, + second: func(_, b, _ string) func(context.Context, UnitOfWork) error { return closeChecked(b) }, + }, + { + // bd update --status closed, proxied. + name: "update to closed vs close", + first: func(a, _, _ string) func(context.Context, UnitOfWork) error { return updateClosed(a) }, + second: func(_, b, _ string) func(context.Context, UnitOfWork) error { return closeIssue(b) }, + }, + { + name: "dependency removal vs close", + first: func(a, _, c string) func(context.Context, UnitOfWork) error { + return func(ctx context.Context, uw UnitOfWork) error { + return uw.DependencyUseCase().RemoveDependency(ctx, c, a, "racer") + } + }, + second: func(_, b, _ string) func(context.Context, UnitOfWork) error { return closeIssue(b) }, + }, + { + name: "delete vs close", + first: func(a, _, _ string) func(context.Context, UnitOfWork) error { + return func(ctx context.Context, uw UnitOfWork) error { + _, err := uw.IssueUseCase().DeleteIssues(ctx, domain.DeleteIssuesParams{IDs: []string{a}, EnforceCascadePolicy: true, Force: true}, "racer") + return err + } + }, + second: func(_, b, _ string) func(context.Context, UnitOfWork) error { return closeIssue(b) }, + }, + { + name: "wisp close vs wisp close", + wispBlockers: true, + first: func(a, _, _ string) func(context.Context, UnitOfWork) error { return closeWisp(a) }, + second: func(_, b, _ string) func(context.Context, UnitOfWork) error { return closeWisp(b) }, + }, + } + + for i, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + n := string(rune('a' + i)) + a, b, c := prefix+"-"+n+"blka", prefix+"-"+n+"blkb", prefix+"-"+n+"dep" + seed := kit.CreateIssue + if tc.wispBlockers { + seed = kit.CreateWisp + } + for _, id := range []string{a, b} { + if err := seed(ctx, &types.Issue{ID: id, Title: id, IssueType: types.TypeTask, Status: types.StatusOpen, Priority: 2}, "seeder"); err != nil { + t.Fatalf("seed %s: %v", id, err) + } + } + if err := kit.CreateIssue(ctx, &types.Issue{ID: c, Title: c, IssueType: types.TypeTask, Status: types.StatusOpen, Priority: 2}, "seeder"); err != nil { + t.Fatalf("seed %s: %v", c, err) + } + for _, blocker := range []string{a, b} { + if err := kit.AddDependency(ctx, &types.Dependency{IssueID: c, DependsOnID: blocker, Type: types.DepBlocks}, "seeder"); err != nil { + t.Fatalf("add %s -> %s: %v", c, blocker, err) + } + } + requireBlocked(t, ctx, kit, c, true) + + first, err := provider.NewUOW(ctx) + if err != nil { + t.Fatalf("open first unit of work: %v", err) + } + defer first.Close(ctx) + if err := tc.first(a, b, c)(ctx, first); err != nil { + t.Fatalf("first write: %v", err) + } + + if err := RunTx(ctx, provider, func(ctx context.Context, uw UnitOfWork) (string, error) { + return "second racing write", tc.second(a, b, c)(ctx, uw) + }); err != nil { + t.Fatalf("second write: %v", err) + } + // The second write's own recheck ran while the first was still + // uncommitted, so the dependent is legitimately still blocked. + requireBlocked(t, ctx, kit, c, true) + + if err := first.Commit(ctx, "first racing write"); err != nil { + t.Fatalf("commit first write: %v", err) + } + + requireBlocked(t, ctx, kit, c, false) + ready, err := RunTxRead(ctx, provider, func(ctx context.Context, uw UnitOfWork) (domain.SearchPage, error) { + return uw.IssueUseCase().GetReadyWork(ctx, types.WorkFilter{}) + }) + if err != nil { + t.Fatalf("ready work: %v", err) + } + if !slices.ContainsFunc(ready.Items, func(i *types.Issue) bool { return i.ID == c }) { + t.Fatalf("%s has no open blockers but is missing from ready work", c) + } + }) + } +} + +func requireBlocked(t *testing.T, ctx context.Context, kit roleFixtureKit, id string, want bool) { + t.Helper() + var blocked bool + if err := kit.QueryScalar(ctx, "SELECT is_blocked FROM issues WHERE id = ?", []any{id}, &blocked); err != nil { + t.Fatalf("read is_blocked of %s: %v", id, err) + } + if blocked != want { + t.Fatalf("%s is_blocked = %v, want %v", id, blocked, want) + } +} diff --git a/internal/storage/uow/dolt_sql_provider.go b/internal/storage/uow/dolt_sql_provider.go index 371dc53ff..b32def7ea 100644 --- a/internal/storage/uow/dolt_sql_provider.go +++ b/internal/storage/uow/dolt_sql_provider.go @@ -202,10 +202,13 @@ func (p *doltSQLProvider) BeginTx(ctx context.Context) (Tx, error) { // AFTER START TRANSACTION so the seq allocation's UPDATE and the SELECT that // must observe it are inside one transaction on one session. The scope is // released when the connection is (doltServerTx.releaseConn / poisonConn), - // so an entry cannot outlive its transaction. + // so an entry cannot outlive its transaction. The blocked-recheck scope is + // bound and released the same way; Commit takes what it recorded once the + // transaction has committed. return &doltServerTx{ conn: conn, clearJournalScope: issueops.ScopeEventsJournalTransaction(conn, p.eventsJournalEnabled.Load()), + clearRecheckScope: issueops.ScopeBlockedRecheckTransaction(conn), }, nil } diff --git a/internal/storage/uow/doltserver_tx.go b/internal/storage/uow/doltserver_tx.go index ee22e35d6..3a98f27ac 100644 --- a/internal/storage/uow/doltserver_tx.go +++ b/internal/storage/uow/doltserver_tx.go @@ -18,6 +18,10 @@ type doltServerTx struct { // releaseConn and poisonConn — so the activation entry cannot outlive the // transaction it describes, whichever way the transaction ends. clearJournalScope func() + // clearRecheckScope is the same binding for the dependents this + // transaction's unblocking writes record for the post-commit + // blocked-state recheck (gastownhall/beads#6716). + clearRecheckScope func() } var _ Tx = (*doltServerTx)(nil) @@ -30,6 +34,51 @@ func (t *doltServerTx) Commit(ctx context.Context, message string) error { if t.done { return errors.New("uow: commit: already done") } + stmt, args, err := t.commitStatement(ctx, message) + if err != nil { + if isSerializationError(err) { + // As below: the server already rolled the transaction back and + // the caller retries the whole unit of work, so leave the + // pinned session in place for the retry. + return err + } + // A failed status check leaves the transaction open on the pinned + // session, exactly like a failed DOLT_COMMIT below: roll back + // before release, or poison the connection. + return t.closeOpenTxAfterFailure(ctx, err) + } + _, err = t.conn.ExecContext(ctx, stmt, args...) + if err == nil { + t.done = true + // The write is durable. Settle the dependents its unblocking writes + // recorded on a fresh snapshot before the pinned session goes back + // to the pool (gastownhall/beads#6716). A recheck failure is logged, + // never returned: the caller must not read a committed write as + // failed and replay it. + if t.recheckBlockedAfterCommit(ctx, issueops.TakeBlockedRecheck(t.conn)) { + t.releaseConn() + } else { + t.poisonConn() + } + return nil + } + if isSerializationError(err) { + // Serialization failures guarantee the transaction was already rolled + // back and the caller retries them, so leave the pinned session in place + // for the retry rather than tearing it down here. + return err + } + // A non-serialization DOLT_COMMIT failure leaves the transaction open on the + // pinned session. Roll it back before releasing the connection so the next + // borrower cannot inherit and implicitly commit the orphaned writes. If the + // rollback also fails the session state is unknown, so poison the connection + // and let the pool discard it instead of handing it out again. + return t.closeOpenTxAfterFailure(ctx, err) +} + +// commitStatement picks the statement that ends the open transaction on the +// pinned session for a commit named message. +func (t *doltServerTx) commitStatement(ctx context.Context, message string) (string, []interface{}, error) { // dolt.auto-commit=batch/off (GH#4995): defer the Dolt version commit to a // later explicit commit point (bd dolt commit), the same as the embedded // and direct SQL-server backends already do at their own commit sites @@ -86,41 +135,15 @@ func (t *doltServerTx) Commit(ctx context.Context, message string) error { // uow layer lets it surface as a lost-update signal // (lostupdate_dolt_test.go) — swallowing it a layer down could mask a // silently lost write. - pending, perr := issueops.HasPendingChanges(ctx, t.conn) - if perr != nil { - if isSerializationError(perr) { - // As below: the server already rolled the transaction back and - // the caller retries the whole unit of work, so leave the - // pinned session in place for the retry. - return perr - } - // A failed status check leaves the transaction open on the pinned - // session, exactly like a failed DOLT_COMMIT below: roll back - // before release, or poison the connection. - return t.closeOpenTxAfterFailure(ctx, perr) + pending, err := issueops.HasPendingChanges(ctx, t.conn) + if err != nil { + return "", nil, err } if !pending { stmt, args = "COMMIT;", nil } } - _, err := t.conn.ExecContext(ctx, stmt, args...) - if err == nil { - t.done = true - t.releaseConn() - return nil - } - if isSerializationError(err) { - // Serialization failures guarantee the transaction was already rolled - // back and the caller retries them, so leave the pinned session in place - // for the retry rather than tearing it down here. - return err - } - // A non-serialization DOLT_COMMIT failure leaves the transaction open on the - // pinned session. Roll it back before releasing the connection so the next - // borrower cannot inherit and implicitly commit the orphaned writes. If the - // rollback also fails the session state is unknown, so poison the connection - // and let the pool discard it instead of handing it out again. - return t.closeOpenTxAfterFailure(ctx, err) + return stmt, args, nil } // closeOpenTxAfterFailure finishes a Commit attempt that failed with the @@ -170,6 +193,7 @@ func (t *doltServerTx) rollbackConn(ctx context.Context) error { func (t *doltServerTx) releaseConn() { t.releaseJournalScope() + t.releaseRecheckScope() if t.conn != nil { _ = t.conn.Close() t.conn = nil @@ -186,6 +210,15 @@ func (t *doltServerTx) releaseJournalScope() { } } +// releaseRecheckScope is releaseJournalScope's counterpart for the +// blocked-recheck scope. Idempotent for the same reason. +func (t *doltServerTx) releaseRecheckScope() { + if t.clearRecheckScope != nil { + t.clearRecheckScope() + t.clearRecheckScope = nil + } +} + // poisonConn discards the pinned session instead of returning it to the pool. // A session whose transaction may still be open must never be reused: because // go-sql-driver's ResetSession only performs a liveness check (no @@ -194,6 +227,7 @@ func (t *doltServerTx) releaseJournalScope() { // database/sql close the connection and drop it from the pool. func (t *doltServerTx) poisonConn() { t.releaseJournalScope() + t.releaseRecheckScope() if t.conn == nil { return } From f83654b1186dd12462bc149f327fb7aa04058b28 Mon Sep 17 00:00:00 2001 From: Gas City Adopt-PR Date: Tue, 29 Sep 2026 16:08:20 +0000 Subject: [PATCH 34/45] release: prep v1.3.1-rc.2 RC bump per RELEASING.md "Cut an RC", on the #6593 model, cut on hotfix/1.3.1 (the owner's release branch for 1.3.1, cut from release/1.3.0). ./scripts/update-versions.sh 1.3.1-rc.2 stamps version.go, the plugin manifests, marketplace.json, npm package.json, the MCP pyproject/__init__/uv.lock (1.3.1rc2), default.nix, the .githooks markers and the winres strings; the PE numeric fields stay at the 1.3.1 base. ./scripts/check-versions.sh is green at 1.3.1-rc.2. CHANGELOG: a dated [1.3.1-rc.2] section covers everything since v1.3.1-rc.1: the five release/1.3.0 merges after rc.1 (#6653, #6659, #6695, #6547, #6652) and the hotfix batches (#6582, #6584, #6694, #6883, #6719, #6740+#6771, #6563+#6627, #6499, #6744, #6879, #6932-#6935, #6937, #6938, #6942). Entries parked under [Unreleased] and the post-rc.1 additions inside [1.3.1] move into it, with an upgrade-notes list of the behaviour changes. The two #6716 bullets are merged into one, the #6584 paragraph that said auto-backup was not wired now defers to the #6879 entry, and #6499, #6563/#6627 and #6547 get entries. The section rc.1 shipped as [1.3.1] is renamed [1.3.1-rc.1] with its body byte-identical to the v1.3.1-rc.1 tag, following the [1.1.0-rc.N] precedent; the final 1.3.1 prep creates [1.3.1]. [Unreleased] stays empty. cmd/bd/info.go: a 1.3.1-rc.2 digest at index 0 of versionChanges. Co-Authored-By: Claude Opus 5.5 --- .claude-plugin/marketplace.json | 2 +- .githooks/post-checkout | 4 +- .githooks/post-merge | 4 +- .githooks/pre-commit | 4 +- .githooks/pre-push | 4 +- .githooks/prepare-commit-msg | 4 +- CHANGELOG.md | 569 ++++++++++-------- cmd/bd/info.go | 25 + cmd/bd/version.go | 2 +- cmd/bd/winres/winres.json | 4 +- default.nix | 2 +- integrations/beads-mcp/pyproject.toml | 2 +- .../beads-mcp/src/beads_mcp/__init__.py | 2 +- integrations/beads-mcp/uv.lock | 2 +- npm-package/package.json | 2 +- plugins/beads/.claude-plugin/plugin.json | 2 +- plugins/beads/.codex-plugin/plugin.json | 2 +- plugins/beads/.copilot-plugin/plugin.json | 2 +- 18 files changed, 370 insertions(+), 268 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 575163c11..f8f85baf2 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ "name": "beads", "source": "./plugins/beads", "description": "AI-supervised issue tracker for coding workflows", - "version": "1.3.1-rc.1" + "version": "1.3.1-rc.2" } ] } diff --git a/.githooks/post-checkout b/.githooks/post-checkout index 3e06c7b41..af46d18a8 100755 --- a/.githooks/post-checkout +++ b/.githooks/post-checkout @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.1 --- +# --- END BEADS INTEGRATION v1.3.1-rc.2 --- diff --git a/.githooks/post-merge b/.githooks/post-merge index 039402e0b..2c1d4a142 100755 --- a/.githooks/post-merge +++ b/.githooks/post-merge @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.1 --- +# --- END BEADS INTEGRATION v1.3.1-rc.2 --- diff --git a/.githooks/pre-commit b/.githooks/pre-commit index a667ef2b0..4bb400076 100755 --- a/.githooks/pre-commit +++ b/.githooks/pre-commit @@ -40,7 +40,7 @@ if [ -n "$staged_go_files" ]; then echo "$staged_go_files" | xargs git add fi -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -97,4 +97,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.1 --- +# --- END BEADS INTEGRATION v1.3.1-rc.2 --- diff --git a/.githooks/pre-push b/.githooks/pre-push index 096439496..e732dc462 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -39,7 +39,7 @@ fi exec < <(printf '%s\n' "$push_refs") -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -96,4 +96,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.1 --- +# --- END BEADS INTEGRATION v1.3.1-rc.2 --- diff --git a/.githooks/prepare-commit-msg b/.githooks/prepare-commit-msg index 1c24adc7c..3bed11aa6 100755 --- a/.githooks/prepare-commit-msg +++ b/.githooks/prepare-commit-msg @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.1 --- +# --- END BEADS INTEGRATION v1.3.1-rc.2 --- diff --git a/CHANGELOG.md b/CHANGELOG.md index bcffc2e42..314de4705 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,10 +7,57 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.3.1-rc.2] - 2026-09-29 + +Second release candidate for 1.3.1, still a patch line on top of 1.3.0 with +**no schema migration**: upgrading from 1.3.0 or 1.3.1-rc.1 is a binary swap. +It adds the proxied-server `bd backup` family and opt-in auto-backup on +managed-local workspaces, a `bd purge` that can run an orchestrator's wisp +retention sweep, and fixes for the #6716 fan-in stall, partial batch closes, +`bd sql` result sets, `bd show --watch`, `bd types`, `BEADS_DIR` discovery and +proxy liveness, plus eleven fixes from `main` ([#6653](https://github.com/gastownhall/beads/pull/6653)) and the smart-gate +data-behind stop ([#6659](https://github.com/gastownhall/beads/pull/6659), [#6695](https://github.com/gastownhall/beads/pull/6695)). Engineering notes for embedders +reading proxied-server surfaces are in `engdocs/` ([#6652](https://github.com/gastownhall/beads/pull/6652)). + +### Upgrade notes + +These behaviour changes can affect scripts; each is detailed below. + +- `bd show --watch ` exits 1 when the id cannot be found (it used to exit + 0), on both the direct and proxied routes. +- `bd sql` in direct server mode prints `OK` / `{"status":"ok"}` for a + multi-statement write or a write the parser cannot classify, instead of + `OK, N rows affected` / `{"rows_affected":N}`. `bd sql --readonly` refuses + statements it cannot parse (for example `PRAGMA`) instead of running them + as reads. +- A failure to read the custom issue types now fails `bd types`, + `bd create --graph` and `bd config set storage-class.*` instead of being + treated as "no custom types". +- `BEADS_DIR` must point at the `.beads` directory itself. A `BEADS_DIR` that + names a project root, or a `.beads` that does not exist yet, now fails with + "no beads database found" instead of silently resolving to the nearest + workspace above the current directory. +- `bd purge` always keeps closed beads a live bead depends on + (`live_dependent_skipped`); there is no flag to turn this off. +- `--older-than` on `bd purge` and `bd prune` is exact at hour precision: + `12h` no longer rounds up to a day, so `bd prune --older-than 12h` deletes + rows it used to keep. +- `bd close` with several ids exits non-zero when any of them fails to close. +- Unflagged piped `bd query` is no longer capped at 50 rows. +- On a proxied workspace with `dolt.auto-commit` set to `batch` or `off`, + writes are no longer committed to Dolt history one by one; run + `bd dolt commit` to flush them. +- `bd backup sync` fails after waiting 5 seconds if another backup of the same + workspace is running. +- `bd ready` / `bd list --ready` include issues in custom `active`-category + statuses, and a store open against an unreachable Dolt server on a + proxied-server workspace or under `bd serve` retries for up to ~40s before + failing. Both come from [#6653](https://github.com/gastownhall/beads/pull/6653); see **Changed**. + ### Added - **`bd backup` works on a proxied-server workspace bd runs the Dolt server - for.** `bd backup init`, `sync`, `remove`, `status` and `restore` are routed + for** ([#6584](https://github.com/gastownhall/beads/pull/6584)). `bd backup init`, `sync`, `remove`, `status` and `restore` are routed over the proxied provider; before this, a proxied workspace — the default topology since 1.3.0 — had no backup path at all. `bd backup restore` stops the proxy and its Dolt child before replacing the database and leaves the @@ -32,14 +79,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 network from its own environment, which is the same credential question the next slice has to answer for `dolt push`. - Backups on this topology are **explicit only**. Auto-backup has no - post-command hook on the proxied path, so `backup.enabled=true` does nothing - there; `bd backup status` and `bd backup` help now say so instead of - attributing the OFF to a missing git remote, which was never the reason on a - server topology. Wiring an auto-backup hook through the new routes is left to - a later slice. + `bd backup status` and `bd backup` help no longer attribute auto-backup being + OFF on a server topology to a missing git remote, which was never the reason + there ([#6694](https://github.com/gastownhall/beads/pull/6694)). Auto-backup + itself is opt-in on this topology; see the next entry. -- **Auto-backup runs on a managed-local proxied-server workspace.** The +- **Auto-backup runs on a managed-local proxied-server workspace** + ([#6879](https://github.com/gastownhall/beads/pull/6879)). The proxied arm of the post-command hook now calls auto-backup, so an explicit `backup.enabled=true` (or `BD_BACKUP_ENABLED=1`) takes the same throttled, change-detected Dolt-native backup into `.beads/backup` it takes in direct @@ -60,7 +106,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 up to 5 seconds (the bound restore uses for the workspace gate) and then fails with "another backup is running for this workspace". -- **`bd purge` covers an orchestrator's wisp retention sweep.** Three changes +- **`bd purge` covers an orchestrator's wisp retention sweep** + ([#6883](https://github.com/gastownhall/beads/pull/6883)). Three changes that together let `bd purge --wisps-plane --older-than 168h --force` replace a raw `DELETE FROM wisps` retention step: - **Live-dependent protection, always on.** `bd purge` no longer deletes a @@ -94,7 +141,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- **Proxied-server refusals now say *why* they refuse.** The JSON a refused +- **Proxied-server refusals now say *why* they refuse** ([#6582](https://github.com/gastownhall/beads/pull/6582)). The JSON a refused command prints gains a `reason` field next to the existing `code`, `error` and `mutates`: `design` for a refusal that is expected to stay (shared history, multi-repo routing, destructive admin, strict `--readonly`) and @@ -118,209 +165,34 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 instead of the message followed by `Error: exit code 1` and the full usage text. Exit statuses are unchanged; this applies on every topology. -- **`bd purge` keeps closed beads a live bead depends on.** A closed bead - that a not-done bead depends on through `parent-child`, `tracks` or - `blocks` is no longer purged (reported as `live_dependent_skipped`), so a - purge that used to delete a closed molecule root under a live step now - leaves it. See the `--wisps-plane` entry under Added. +- **`bd purge` keeps closed beads a live bead depends on, always** + ([#6883](https://github.com/gastownhall/beads/pull/6883)). A closed bead that a not-done bead depends on through + `parent-child`, `tracks` or `blocks` is no longer purged (reported as + `live_dependent_skipped`), so a purge that used to delete a closed molecule + root under a live step now leaves it. There is no flag to turn this off. + See the `--wisps-plane` entry under Added. -### Fixed +- **`--older-than` on `bd purge` and `bd prune` is taken at hour precision** + ([#6883](https://github.com/gastownhall/beads/pull/6883)). An `Nh` value is no longer rounded to whole days: `12h` used to + round UP to one day and now means 12 hours, so `bd prune --older-than 12h` + deletes rows closed 12–24 hours ago that it used to keep; `36h` used to floor + to one day and now means 36 hours. Day values (`7`, `7d`, `2w`) are + unchanged. See the `bd purge` entry under Added. -- **An explicit `BEADS_DIR` is authoritative during workspace discovery.** - When `BEADS_DIR` named a directory without project files yet (missing, - empty, or not initialized), discovery ignored it and walked up from the - current directory instead, then rebound `BEADS_DIR` to whatever ancestor - workspace it found. `bd init` refused with "already initialized" because of - the parent workspace, and data commands such as `bd create` and `bd list` - read and wrote the parent's store. Discovery now reports no workspace for - such a `BEADS_DIR`, so `bd init` initializes the named directory and other - commands fail with "no beads database found". `bd import`/`bd setup` and - `bd bootstrap` target the named directory when they create a workspace. - Behaviour with `BEADS_DIR` unset is unchanged. **Behaviour change:** a - `BEADS_DIR` that points at a project root rather than its `.beads` - directory (for example `BEADS_DIR=$repo` instead of `BEADS_DIR=$repo/.beads`), - or at a `.beads` that does not exist yet, used to work by accident because - discovery walked up to the nearest workspace; it now fails with "no beads - database found". Point `BEADS_DIR` at the `.beads` directory itself. - ([#6938](https://github.com/gastownhall/beads/pull/6938)) - -- **A proxied workspace's proxy retires when its Dolt backend exits - cleanly.** The proxy noticed its `dolt sql-server` child exiting only when - the exit status was non-zero. A backend that shut down gracefully (for - example on SIGTERM) left the proxy up and adoptable in front of a dead store: - with `--proxied-server-idle-timeout 0` it never went away, `bd dolt status` - called it running, and every command failed until `bd dolt stop`. The proxy - now retires on any backend exit, as it already did on a crash, so the next bd - command (including `bd ping`) starts a fresh proxy and backend. `bd dolt - status` no longer reports a proxy whose managed backend is gone as running: - the text says "not serving" and the JSON reports `running: false` (with - `proxy_pid` still set), including for a stranded proxy started by an older - bd. ([#6937](https://github.com/gastownhall/beads/pull/6937)) - -- **`bd show --watch` works under `--proxied-server`.** It was refused with - `proxy.watch.unsupported` although the provider can answer the poll the way - `bd list --watch` already does, so every proxied workspace — the default - transport for managed-local setups — had no way to watch a bead. The proxied - route now shares the direct route's loop: render once, re-read every 2s, - redraw only when the issue's status or `updated_at` changes, stop cleanly on - Ctrl+C/SIGTERM, and require exactly one id. Each poll opens its own short - unit of work, so a long watch never pins a transaction, and a poll that - fails (the issue was deleted, the backend blipped) keeps the last render - without printing, as on the direct route. On both routes a watch whose id - cannot be found now exits non-zero instead of 0. - -- **A server-mode workspace with an empty `.beads/dolt` is no longer refused - as legacy** - ([#5682](https://github.com/gastownhall/beads/issues/5682), - [#6935](https://github.com/gastownhall/beads/pull/6935)). bd creates `.beads/dolt` on use even when the data lives on an - external Dolt server, and leaves it empty. When the gitignored - `.local_version` witness was missing — after a fresh checkout, or when a - provisioner created the empty root before `bd init --server --external` — - the legacy-upgrade guard refused every command, `bd init` included, as a - "legacy Dolt server workspace", and no bd command could repair it. An empty - root holds nothing a legacy release could have left, so the guard now admits - it: `bd init` writes the witness, and an existing workspace in that state - re-seeds it on the next command. A pre-1.0 witness, or a `.beads/dolt` that - holds anything, is still refused. - -- **`bd types` now lists exactly the types `bd create --type` accepts** - ([#6934](https://github.com/gastownhall/beads/pull/6934)). The two resolved - the type set through different code and could disagree: a server-mode - workspace's `bd types` dropped `.beads/config.yaml` `types.custom` entries - once the database had any custom type, although create accepts them; in - proxied mode `bd config set storage-class.` checked `config.yaml` alone - and refused custom types registered in the database; and the built-in system - types `message`, `molecule`, `gate` and `event` were never listed. Every - mode now composes custom types by one rule (the `custom_types` table, else - the `types.custom` config row, always unioned with `config.yaml`). `bd types` - gains a "System types" section, and `bd types --json` an additive - `system_types` field; `core_types` and `custom_types` are unchanged. A - failure to read the custom types now fails `bd types`, `bd create --graph` - and `bd config set storage-class.*` instead of being silently treated as "no - custom types". - -- **`bd sql` no longer drops the rows of CTE queries or CALL result sets** - ([#6932](https://github.com/gastownhall/beads/pull/6932)). In proxied-server - mode, `WITH name(cols) AS (...) SELECT ...` and `WITH RECURSIVE ...` queries - were misread as writes and printed `OK, 0 rows affected` with no error; they - now return their rows. Statements are now classified with the Dolt SQL - parser (shared by the proxied and direct paths): reads return rows, plain - writes (including CTE-prefixed `UPDATE`/`DELETE`) commit and report - `rows_affected`, and anything that may both write and return rows (`CALL`, - `EXPLAIN ANALYZE` of a write, `RETURNING`) or that the parser cannot - classify is committed and prints whatever rows it returns, so - `CALL DOLT_BRANCH(...)`-style result sets are rendered instead of discarded. - Visible changes for scripts: in direct server mode a multi-statement write, - or a single write statement the parser cannot classify, now prints `OK` / - `{"status":"ok"}` (as proxied mode already did) instead of - `OK, N rows affected` / `{"rows_affected":N}`; and `--readonly` now refuses - statements it cannot parse (for example `PRAGMA`) instead of treating them - as reads. - -- **The #6716 fan-in stall is fixed on the proxied-server route and the - remaining store routes.** Two blockers of one dependent taken away at the - same time (parallel workers closing both, or a close racing a - `bd dep remove` or a delete of the other) could leave the dependent - `is_blocked=1` and missing from `bd ready` until `bd recompute-blocked` - ([#6716](https://github.com/gastownhall/beads/issues/6716)). The post-commit - recheck the Dolt store write transactions gained for that issue now also - runs on every write the proxied-server (uow/domain-db) route serves, which - under `--proxied-server` — the default topology — is `bd close` (single - and batch), `bd update --status`, `bd dep remove`, `bd delete`, `bd batch` - and `bd serve`; on `RunInTransaction` (`bd batch` direct, `bd cook`, - `bd mol squash`/`burn`, and SDK callers that close or update inside a - transaction); and on the wisp close, update and delete writers and the - legacy dependency removal behind `bd duplicates --merge`. Demote-to-wisp - moves a row between planes without taking a blocker away and is unchanged. - -- Unblocking two blockers of one dependent at the same time — closing both, - or a close racing a `bd dep remove` or a delete of the other — no longer - leaves the dependent stuck as blocked and hidden from `bd ready` until - `bd recompute-blocked`: a write that takes a blocker away (a close, an - update to an inactive status, a dependency removal, a delete) and runs - through a Dolt store write transaction now rechecks the dependents it - recomputed once that transaction has committed - ([#6716](https://github.com/gastownhall/beads/issues/6716)). The - proxied-server route and the remaining store routes are covered by the - entry above. - -- **`bd close` now exits non-zero when any issue in a batch fails to close** - ([#6648](https://github.com/gastownhall/beads/issues/6648)). A batch with one - refused id used to exit 0 as long as another id closed, so scripts could not - tell that part of the batch was left open. The closable ids still close, the - refusals are still printed, and a final `Error: N of M issues failed to close` - line is added; ids that were already closed still count as success. Both the - direct and proxied-server routes behave this way for ids the close policy or - the engine refuses. (An id that cannot be *resolved* at all is unchanged and - still differs by route: the direct route aborts the whole command before any - close runs, while the proxied route refuses that argument and closes the - rest.) In `--json` mode the summary is instead a compact JSON line on stderr - naming the failed ids, matching `bd update`'s partial-failure report, while - stdout keeps the usual closed-issues array. Each `failed[]` entry carries the - refusal as the engine worded it, identically on both routes; the `--force` - hint and the route's own framing stay on the human-readable stderr line, which - is unchanged. `--claim-next` still claims when - part of the batch closed — the claim commits inside the batch's own - transaction and a sibling's refusal does not roll it back — so the summary - names the claimed id rather than leaving it silently assigned. - -- **`bd query` no longer silently stops at 50 rows when its output is piped** - ([#6229](https://github.com/gastownhall/beads/issues/6229)). `bd list` has - treated piped stdout as unlimited since GH#4094, but `bd query` read its - `--limit` default of 50 straight through, so `bd query '...' | consumer` - dropped every row past 50 with no hint. An unflagged `bd query` now resolves - through the same policy as `bd list`: an explicit `--limit` wins, piped - stdout is unlimited, agent mode on a terminal gets 20, and a terminal gets 50. - -## [1.3.1] - 2026-09-16 - -### Changed - -- **`bd dolt status --json` changes shape on a proxied workspace** - ([#6580](https://github.com/gastownhall/beads/pull/6580)). It now emits - `{"mode": "proxied-server", "root": ..., "running": , "proxy_pid": - ..., "proxy_port": ..., "backend_managed": ..., "backend_running": ..., - "backend_pid": ..., "backend_port": ..., "backend_endpoint": ..., - "idle_timeout": ...}` instead of the always-false `{"running": false, "pid": - 0, "port": 0}` — `pid` and `port` are gone, replaced by the `proxy_*` and - `backend_*` pairs. `running` no longer reports a bd-managed dolt PID file; it - describes the proxy, the endpoint every bd command connects through, so it is - now usually true where it used to be always false. `backend_managed` is false - on external proxied topologies, where the dolt server is not bd's process to - report on and `backend_endpoint` names it instead. Anything parsing the old - payload changes behaviour on upgrade. Direct-server, shared, embedded and - externally-managed workspaces keep the output they had. - -- **`bd config set ` deletes a flat top-level key of the same - literal name** ([#6578](https://github.com/gastownhall/beads/pull/6578)). - Dotted keys are now written as a nested mapping (see **Fixed** below for why), - and where the file already carried the flat spelling — a top-level key whose - *name* contains the dot, such as `dolt.host: "10.0.0.1"` — that entry is - removed and the value re-emitted under `dolt:`. Only the key being written is - touched; other dotted keys, including ones bd did not write, are left exactly - as they are. - - `.beads/config.yaml` is not exclusively bd's, so this can be visible outside - bd: an external tool that both writes *and* reads flat dotted keys there will - stop seeing any value bd migrates this way. Any dotted key bd stores in - `config.yaml` rather than the database can collide; in practice that means - `dolt.*`, `export.auto`, and `backup.enabled`. If the other writer then - re-adds its flat key, the file carries both spellings and bd's own readers - disagree — Viper-backed reads (`bd config get`, and the `dolt.auto-start` - lookup on the store-open path) return the **flat** value, while - `GetStringFromDir` returns the **nested** one, so a `bd config set` can be - shadowed by a stale flat entry. Tracked as - [#6594](https://github.com/gastownhall/beads/issues/6594). Until that lands, - in a workspace whose config another tool maintains, let that tool own its - keys, or check `.beads/config.yaml` for a duplicated key after running - `bd config set`. - -- **Two `bd config set` writes that used to exit 0 now exit 1** - ([#6578](https://github.com/gastownhall/beads/pull/6578)), because both wrote - a value no reader could see: setting a dotted key under a parent that already - holds a scalar (`sync: enabled`, then `bd config set sync.remote`), and - setting one in a file whose top level is not a mapping. A script that - tolerated the old exit 0 now has to handle the failure. +- **`dolt.auto-commit=batch` and `off` now take effect on a proxied-server + workspace, and `bd dolt commit` works there** ([#6499](https://github.com/gastownhall/beads/pull/6499), + [#4995](https://github.com/gastownhall/beads/issues/4995)). Every proxied write used to mint a Dolt history commit + whatever `dolt.auto-commit` said, and `bd dolt commit` was refused with + `proxy.dolt_commit.unsupported`. Under `batch`/`off` a proxied write now + lands in the server's working set without advancing Dolt history, and + `bd dolt commit` is the flush point, as on the direct route. Commands that + are explicit commit points (`bd batch`, `bd mol bond`/`pour`/`squash`, + `bd mol wisp create`, and the wisp half of `bd mol burn`) still commit, and + `bd serve` writes still commit per write. `on`, the default, is unchanged. + **Behaviour change:** a proxied workspace that set `batch` or `off` and was + getting per-write commits anyway now accumulates uncommitted changes until + `bd dolt commit`, and `dolt log`, `dolt push` and backups do not see them + before that. `bd vc commit`'s proxied refusal now points at `bd dolt commit`. - **`bd ready` and `bd list --ready` now return issues whose custom status is in the `active` category** ([#5918](https://github.com/gastownhall/beads/pull/5918), @@ -409,25 +281,158 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- **`bd dolt start` no longer puts a second sql-server over a proxied - workspace's data directory.** On a proxied-server workspace the default root - IS `.beads/dolt`, and `bd dolt start` knew nothing about proxied mode: with - the proxy quiesced it launched an unsupervised `dolt sql-server` directly - over that root, after which ordinary bd commands failed (`invalid - connection`) because the relaunched proxy found a foreign server on its - port; with the proxy live it instead adopted the proxy's own dolt child into - bd's classic PID/port records, leaving two managers for one process. It is - now a typed refusal with code `proxy.dolt_start.conflict`. The proxy owns - its backend's lifecycle — it starts on demand and `bd dolt stop` shuts it - down. Direct-server and embedded workspaces are unaffected. +- **An explicit `BEADS_DIR` is authoritative during workspace discovery.** + When `BEADS_DIR` named a directory without project files yet (missing, + empty, or not initialized), discovery ignored it and walked up from the + current directory instead, then rebound `BEADS_DIR` to whatever ancestor + workspace it found. `bd init` refused with "already initialized" because of + the parent workspace, and data commands such as `bd create` and `bd list` + read and wrote the parent's store. Discovery now reports no workspace for + such a `BEADS_DIR`, so `bd init` initializes the named directory and other + commands fail with "no beads database found". `bd import`/`bd setup` and + `bd bootstrap` target the named directory when they create a workspace. + Behaviour with `BEADS_DIR` unset is unchanged. **Behaviour change:** a + `BEADS_DIR` that points at a project root rather than its `.beads` + directory (for example `BEADS_DIR=$repo` instead of `BEADS_DIR=$repo/.beads`), + or at a `.beads` that does not exist yet, used to work by accident because + discovery walked up to the nearest workspace; it now fails with "no beads + database found". Point `BEADS_DIR` at the `.beads` directory itself. + ([#6938](https://github.com/gastownhall/beads/pull/6938)) -- **`bd dolt status` tells the truth on a proxied workspace.** It read the - classic PID file, which proxied mode never writes, and so reported `Dolt - server: not running` while the proxy was serving CRUD — the wrong answer - that sent operators to `bd dolt start` in the first place. It now reads the - proxy's own records and reports the proxy and its dolt backend separately, - without starting either. The `--json` payload changes shape to carry that — - see **Changed** above before parsing it. +- **A proxied workspace's proxy retires when its Dolt backend exits + cleanly.** The proxy noticed its `dolt sql-server` child exiting only when + the exit status was non-zero. A backend that shut down gracefully (for + example on SIGTERM) left the proxy up and adoptable in front of a dead store: + with `--proxied-server-idle-timeout 0` it never went away, `bd dolt status` + called it running, and every command failed until `bd dolt stop`. The proxy + now retires on any backend exit, as it already did on a crash, so the next bd + command (including `bd ping`) starts a fresh proxy and backend. `bd dolt + status` no longer reports a proxy whose managed backend is gone as running: + the text says "not serving" and the JSON reports `running: false` (with + `proxy_pid` still set), including for a stranded proxy started by an older + bd. ([#6937](https://github.com/gastownhall/beads/pull/6937)) + +- **`bd show --watch` works under `--proxied-server`.** It was refused with + `proxy.watch.unsupported` although the provider can answer the poll the way + `bd list --watch` already does, so every proxied workspace — the default + transport for managed-local setups — had no way to watch a bead. The proxied + route now shares the direct route's loop: render once, re-read every 2s, + redraw only when the issue's status or `updated_at` changes, stop cleanly on + Ctrl+C/SIGTERM, and require exactly one id. Each poll opens its own short + unit of work, so a long watch never pins a transaction, and a poll that + fails (the issue was deleted, the backend blipped) keeps the last render + without printing, as on the direct route. On both routes a watch whose id + cannot be found now exits 1 instead of 0 ([#6933](https://github.com/gastownhall/beads/pull/6933)). + +- **A server-mode workspace with an empty `.beads/dolt` is no longer refused + as legacy** + ([#5682](https://github.com/gastownhall/beads/issues/5682), + [#6935](https://github.com/gastownhall/beads/pull/6935)). bd creates `.beads/dolt` on use even when the data lives on an + external Dolt server, and leaves it empty. When the gitignored + `.local_version` witness was missing — after a fresh checkout, or when a + provisioner created the empty root before `bd init --server --external` — + the legacy-upgrade guard refused every command, `bd init` included, as a + "legacy Dolt server workspace", and no bd command could repair it. An empty + root holds nothing a legacy release could have left, so the guard now admits + it: `bd init` writes the witness, and an existing workspace in that state + re-seeds it on the next command. A pre-1.0 witness, or a `.beads/dolt` that + holds anything, is still refused. + +- **`bd types` now lists exactly the types `bd create --type` accepts** + ([#6934](https://github.com/gastownhall/beads/pull/6934)). The two resolved + the type set through different code and could disagree: a server-mode + workspace's `bd types` dropped `.beads/config.yaml` `types.custom` entries + once the database had any custom type, although create accepts them; in + proxied mode `bd config set storage-class.` checked `config.yaml` alone + and refused custom types registered in the database; and the built-in system + types `message`, `molecule`, `gate` and `event` were never listed. Every + mode now composes custom types by one rule (the `custom_types` table, else + the `types.custom` config row, always unioned with `config.yaml`). `bd types` + gains a "System types" section, and `bd types --json` an additive + `system_types` field; `core_types` and `custom_types` are unchanged. A + failure to read the custom types now fails `bd types`, `bd create --graph` + and `bd config set storage-class.*` instead of being silently treated as "no + custom types". + +- **`bd sql` no longer drops the rows of CTE queries or CALL result sets** + ([#6932](https://github.com/gastownhall/beads/pull/6932)). In proxied-server + mode, `WITH name(cols) AS (...) SELECT ...` and `WITH RECURSIVE ...` queries + were misread as writes and printed `OK, 0 rows affected` with no error; they + now return their rows. Statements are now classified with the Dolt SQL + parser (shared by the proxied and direct paths): reads return rows, plain + writes (including CTE-prefixed `UPDATE`/`DELETE`) commit and report + `rows_affected`, and anything that may both write and return rows (`CALL`, + `EXPLAIN ANALYZE` of a write, `RETURNING`) or that the parser cannot + classify is committed and prints whatever rows it returns, so + `CALL DOLT_BRANCH(...)`-style result sets are rendered instead of discarded. + Visible changes for scripts: in direct server mode a multi-statement write, + or a single write statement the parser cannot classify, now prints `OK` / + `{"status":"ok"}` (as proxied mode already did) instead of + `OK, N rows affected` / `{"rows_affected":N}`; and `--readonly` now refuses + statements it cannot parse (for example `PRAGMA`) instead of treating them + as reads. + +- **Unblocking two blockers of one dependent at the same time no longer leaves + the dependent stuck as blocked** ([#6716](https://github.com/gastownhall/beads/issues/6716); [#6719](https://github.com/gastownhall/beads/pull/6719), [#6942](https://github.com/gastownhall/beads/pull/6942)). + Closing both blockers in parallel (formula fan-in with parallel workers), or + a close racing a `bd dep remove` or a delete of the other, could leave the + dependent `is_blocked=1` and missing from `bd ready` until + `bd recompute-blocked`. A write that takes a blocker away (a close, an update + to an inactive status, a dependency removal, a delete) now rechecks the + dependents it recomputed once its transaction has committed. This covers the + Dolt store write transactions, every write the proxied-server (uow/domain-db) + route serves — under `--proxied-server`, the default topology, that is + `bd close` (single and batch), `bd update --status`, `bd dep remove`, + `bd delete`, `bd batch` and `bd serve` — `RunInTransaction` (`bd batch` + direct, `bd cook`, `bd mol squash`/`burn`, and SDK callers that close or + update inside a transaction), the wisp close, update and delete writers, and + the legacy dependency removal behind `bd duplicates --merge`. Demote-to-wisp + moves a row between planes without taking a blocker away and is unchanged. + +- **`bd close` now exits non-zero when any issue in a batch fails to close** + ([#6648](https://github.com/gastownhall/beads/issues/6648); [#6740](https://github.com/gastownhall/beads/pull/6740), [#6771](https://github.com/gastownhall/beads/pull/6771)). A batch with one + refused id used to exit 0 as long as another id closed, so scripts could not + tell that part of the batch was left open. The closable ids still close, the + refusals are still printed, and a final `Error: N of M issues failed to close` + line is added; ids that were already closed still count as success. Both the + direct and proxied-server routes behave this way for ids the close policy or + the engine refuses. (An id that cannot be *resolved* at all is unchanged and + still differs by route: the direct route aborts the whole command before any + close runs, while the proxied route refuses that argument and closes the + rest.) In `--json` mode the summary is instead a compact JSON line on stderr + naming the failed ids, matching `bd update`'s partial-failure report, while + stdout keeps the usual closed-issues array. Each `failed[]` entry carries the + refusal as the engine worded it, identically on both routes; the `--force` + hint and the route's own framing stay on the human-readable stderr line, which + is unchanged. `--claim-next` still claims when + part of the batch closed — the claim commits inside the batch's own + transaction and a sibling's refusal does not roll it back — so the summary + names the claimed id rather than leaving it silently assigned. + +- **`bd query` no longer silently stops at 50 rows when its output is piped** + ([#6229](https://github.com/gastownhall/beads/issues/6229), [#6744](https://github.com/gastownhall/beads/pull/6744)). `bd list` has + treated piped stdout as unlimited since GH#4094, but `bd query` read its + `--limit` default of 50 straight through, so `bd query '...' | consumer` + dropped every row past 50 with no hint. An unflagged `bd query` now resolves + through the same policy as `bd list`: an explicit `--limit` wins, piped + stdout is unlimited, agent mode on a terminal gets 20, and a terminal gets 50. + +- **Deleting or purging a wisp no longer leaves orphan rows behind on stores + without the wisp foreign keys** ([#6563](https://github.com/gastownhall/beads/pull/6563), [#6627](https://github.com/gastownhall/beads/pull/6627), [#6487](https://github.com/gastownhall/beads/issues/6487)). + On a store whose `wisp_dependencies` and wisp auxiliary tables lack the + migration-0047 FK constraints, every wisp delete (including `bd purge`, + `bd mol burn` and wisp GC) left its `wisp_dependencies` edges on both sides + and its `wisp_labels`, `wisp_events`, `wisp_comments` and + `wisp_child_counters` rows in place, which showed up as dangling parent + references and store bloat. The delete paths now remove them. This is a + forward fix: orphans already in a store are not cleaned up. + +- **A clone missing its local `events`, `bd_events_journal` or `bd_events_seq` + table recreates it on open** ([#6547](https://github.com/gastownhall/beads/pull/6547)). Those tables are dolt-ignored, so + a fresh clone can arrive at the latest schema version without them, and bd + then treated the schema as complete and never created them. Their absence + now makes the migration runner replay the migrations that create them, from + each table's own floor rather than from zero. - **The smart migrate gate no longer auto-migrates a clone whose data is behind the remote** ([#6575](https://github.com/gastownhall/beads/issues/6575)). The @@ -529,23 +534,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 --force` / `BD_ALLOW_REMOTE_MIGRATE=1` are still consulted before the gate and still override it. -- **A dotted config key now round-trips: what `bd config set` writes, - `bd config get` and bd's own readers find** - ([#6578](https://github.com/gastownhall/beads/pull/6578), bd-zj95). Setting a - key like `sync.remote` or `dolt.host` into a `config.yaml` with no matching - section appended a key whose *name* contained the dot — a top-level - `sync.remote: "..."` — instead of nesting it under `sync:`. Viper finds a key - spelled that way; the direct reader `GetStringFromDir` splits on the dot and - walks nested mappings, so it does not. The value was set and invisible at the - same time depending on which reader asked, and `bd init --remote` in a fresh - workspace was the ordinary way in: the remote was recorded and then not found. - Dotted keys are written nested now, and `bd config unset` removes the nested - form — it only ever matched the flat spelling, so unsetting - `sync.remote` silently left the remote live. Keys the write does not own, - including dotted ones somebody else put there, are left exactly as they are. - Two consequences of the nested write — the flat-key migration and two new - failure exits — are under **Changed** above. - - **`bd list --status --ready` honors `--status`** ([#5924](https://github.com/gastownhall/beads/pull/5924), [#5832](https://github.com/gastownhall/beads/issues/5832)). Combining the two @@ -643,6 +631,95 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 context. Durable rejections and an unresolvable hostname still fail without retrying. See **Changed** for the latency this adds against a dead endpoint. +## [1.3.1-rc.1] - 2026-09-16 + +### Changed + +- **`bd dolt status --json` changes shape on a proxied workspace** + ([#6580](https://github.com/gastownhall/beads/pull/6580)). It now emits + `{"mode": "proxied-server", "root": ..., "running": , "proxy_pid": + ..., "proxy_port": ..., "backend_managed": ..., "backend_running": ..., + "backend_pid": ..., "backend_port": ..., "backend_endpoint": ..., + "idle_timeout": ...}` instead of the always-false `{"running": false, "pid": + 0, "port": 0}` — `pid` and `port` are gone, replaced by the `proxy_*` and + `backend_*` pairs. `running` no longer reports a bd-managed dolt PID file; it + describes the proxy, the endpoint every bd command connects through, so it is + now usually true where it used to be always false. `backend_managed` is false + on external proxied topologies, where the dolt server is not bd's process to + report on and `backend_endpoint` names it instead. Anything parsing the old + payload changes behaviour on upgrade. Direct-server, shared, embedded and + externally-managed workspaces keep the output they had. + +- **`bd config set ` deletes a flat top-level key of the same + literal name** ([#6578](https://github.com/gastownhall/beads/pull/6578)). + Dotted keys are now written as a nested mapping (see **Fixed** below for why), + and where the file already carried the flat spelling — a top-level key whose + *name* contains the dot, such as `dolt.host: "10.0.0.1"` — that entry is + removed and the value re-emitted under `dolt:`. Only the key being written is + touched; other dotted keys, including ones bd did not write, are left exactly + as they are. + + `.beads/config.yaml` is not exclusively bd's, so this can be visible outside + bd: an external tool that both writes *and* reads flat dotted keys there will + stop seeing any value bd migrates this way. Any dotted key bd stores in + `config.yaml` rather than the database can collide; in practice that means + `dolt.*`, `export.auto`, and `backup.enabled`. If the other writer then + re-adds its flat key, the file carries both spellings and bd's own readers + disagree — Viper-backed reads (`bd config get`, and the `dolt.auto-start` + lookup on the store-open path) return the **flat** value, while + `GetStringFromDir` returns the **nested** one, so a `bd config set` can be + shadowed by a stale flat entry. Tracked as + [#6594](https://github.com/gastownhall/beads/issues/6594). Until that lands, + in a workspace whose config another tool maintains, let that tool own its + keys, or check `.beads/config.yaml` for a duplicated key after running + `bd config set`. + +- **Two `bd config set` writes that used to exit 0 now exit 1** + ([#6578](https://github.com/gastownhall/beads/pull/6578)), because both wrote + a value no reader could see: setting a dotted key under a parent that already + holds a scalar (`sync: enabled`, then `bd config set sync.remote`), and + setting one in a file whose top level is not a mapping. A script that + tolerated the old exit 0 now has to handle the failure. + +### Fixed + +- **`bd dolt start` no longer puts a second sql-server over a proxied + workspace's data directory.** On a proxied-server workspace the default root + IS `.beads/dolt`, and `bd dolt start` knew nothing about proxied mode: with + the proxy quiesced it launched an unsupervised `dolt sql-server` directly + over that root, after which ordinary bd commands failed (`invalid + connection`) because the relaunched proxy found a foreign server on its + port; with the proxy live it instead adopted the proxy's own dolt child into + bd's classic PID/port records, leaving two managers for one process. It is + now a typed refusal with code `proxy.dolt_start.conflict`. The proxy owns + its backend's lifecycle — it starts on demand and `bd dolt stop` shuts it + down. Direct-server and embedded workspaces are unaffected. + +- **`bd dolt status` tells the truth on a proxied workspace.** It read the + classic PID file, which proxied mode never writes, and so reported `Dolt + server: not running` while the proxy was serving CRUD — the wrong answer + that sent operators to `bd dolt start` in the first place. It now reads the + proxy's own records and reports the proxy and its dolt backend separately, + without starting either. The `--json` payload changes shape to carry that — + see **Changed** above before parsing it. + +- **A dotted config key now round-trips: what `bd config set` writes, + `bd config get` and bd's own readers find** + ([#6578](https://github.com/gastownhall/beads/pull/6578), bd-zj95). Setting a + key like `sync.remote` or `dolt.host` into a `config.yaml` with no matching + section appended a key whose *name* contained the dot — a top-level + `sync.remote: "..."` — instead of nesting it under `sync:`. Viper finds a key + spelled that way; the direct reader `GetStringFromDir` splits on the dot and + walks nested mappings, so it does not. The value was set and invisible at the + same time depending on which reader asked, and `bd init --remote` in a fresh + workspace was the ordinary way in: the remote was recorded and then not found. + Dotted keys are written nested now, and `bd config unset` removes the nested + form — it only ever matched the flat spelling, so unsetting + `sync.remote` silently left the remote live. Keys the write does not own, + including dotted ones somebody else put there, are left exactly as they are. + Two consequences of the nested write — the flat-key migration and two new + failure exits — are under **Changed** above. + ## [1.3.0] - 2026-09-15 The first tested release off `main` since the 1.1 line. [1.2.2] was a recovery diff --git a/cmd/bd/info.go b/cmd/bd/info.go index d33378517..94fadebb7 100644 --- a/cmd/bd/info.go +++ b/cmd/bd/info.go @@ -220,6 +220,31 @@ type VersionChange struct { // versionChanges contains agent-actionable changes for recent versions var versionChanges = []VersionChange{ + { + Version: "1.3.1-rc.2", + Date: "2026-09-29", + Changes: []string{ + "RC: second candidate for the 1.3.1 patch line. Still no schema migration, so upgrading from 1.3.0 or 1.3.1-rc.1 is a binary swap. The 1.3.1-rc.1 entry below still applies.", + "NEW: on a proxied-server workspace whose Dolt server bd runs (managed-local, the default), 'bd backup init/sync/status/restore/remove' now work, and 'backup.enabled=true' turns on auto-backup (default stays OFF). On an external or team-server proxied topology the family stays refused with 'proxy.backup.unsupported' and \"reason\": \"design\". Proxied refusals now carry a 'reason' field ('design' or 'unimplemented'). A per-workspace '.beads/backup.lock' serializes backups: 'bd backup sync' fails after 5s if another backup is running.", + "NEW: 'bd purge --wisps-plane --older-than --force [--limit N] --json' can replace a raw DELETE FROM wisps retention step. With --limit it purges oldest-closed first and reports 'remaining' and 'has_more'; loop while has_more is true.", + "CHANGE: 'bd purge' ALWAYS keeps a closed bead that a not-done bead depends on through parent-child, tracks or blocks, and reports the count as 'live_dependent_skipped'. There is no flag to disable it.", + "CHANGE: '--older-than' on 'bd purge' and 'bd prune' takes hour values exactly: '12h' no longer rounds up to a day and '36h' no longer floors to one, so 'bd prune --older-than 12h' now deletes rows closed 12-24 hours ago. Day values (7, 7d, 2w) are unchanged; a value too large to represent is refused.", + "CHANGE: 'bd show --watch ' works under --proxied-server, and on both routes a watch whose id cannot be found exits 1 instead of 0.", + "CHANGE: 'bd sql' classifies statements with the SQL parser. Proxied CTE queries ('WITH ... AS (...) SELECT') return their rows instead of 'OK, 0 rows affected', and CALL result sets are printed. In direct server mode a multi-statement write, or a write the parser cannot classify, prints 'OK' / {\"status\":\"ok\"} instead of 'OK, N rows affected' / {\"rows_affected\":N}. 'bd sql --readonly' refuses statements it cannot parse (e.g. PRAGMA).", + "CHANGE: 'bd types' lists exactly what 'bd create --type' accepts, with a new 'System types' section and an additive 'system_types' JSON field. A failure to read custom types now fails 'bd types', 'bd create --graph' and 'bd config set storage-class.*' instead of meaning 'no custom types'.", + "CHANGE: an explicit BEADS_DIR is authoritative. Point it at the .beads directory itself: a BEADS_DIR naming a project root, or a .beads that does not exist yet, now fails with 'no beads database found' instead of resolving to the nearest workspace above the current directory. 'bd init' initializes the named directory.", + "CHANGE: 'bd close' with several ids exits non-zero when any id fails to close (the rest still close). With --json, a summary naming the failed ids goes to stderr.", + "CHANGE: an unflagged piped 'bd query' is no longer capped at 50 rows; it follows the 'bd list' limit policy.", + "CHANGE: on a proxied workspace, dolt.auto-commit=batch/off now defers Dolt history commits and 'bd dolt commit' is supported as the flush point. Explicit commit points ('bd batch', 'bd mol bond/pour/squash', 'bd mol wisp create') still commit. The default 'on' is unchanged.", + "CHANGE: 'bd ready' and 'bd list --ready' include issues whose custom status is in the 'active' category; 'bd list --status --ready' honors --status; bare 'bd list --wisp-type' is refused (use --include-ephemeral); 'bd doctor' warns about a missing .beads/dolt-server-config.yaml gitignore pattern until 'bd doctor --fix'.", + "CHANGE: on a proxied-server workspace or under 'bd serve', opening a store against an unreachable Dolt server retries for up to ~40s before failing, instead of failing at once.", + "FIX: two blockers of one dependent removed at the same time (parallel closes, a close racing 'bd dep remove' or a delete) no longer leave the dependent stuck blocked and hidden from 'bd ready' (#6716), on every store route including --proxied-server.", + "FIX: a proxy whose Dolt backend exits cleanly now retires, so the next command starts a fresh one; 'bd dolt status' reports it as not serving.", + "FIX: a server workspace with an empty .beads/dolt is no longer refused as a legacy workspace; 'bd init' writes the version marker.", + "FIX: the smart migrate gate no longer auto-migrates a clone whose data is behind its remote. It stops with 'run bd dolt pull, then retry', and 'bd dolt pull' works from that state.", + "FIX: deleting or purging a wisp no longer orphans wisp_dependencies, wisp_labels, wisp_events, wisp_comments or wisp_child_counters rows on stores without the wisp foreign keys (existing orphans are not cleaned). A clone missing its dolt-ignored events tables recreates them on open.", + }, + }, { Version: "1.3.1-rc.1", Date: "2026-09-16", diff --git a/cmd/bd/version.go b/cmd/bd/version.go index be648b593..3b15822b1 100644 --- a/cmd/bd/version.go +++ b/cmd/bd/version.go @@ -16,7 +16,7 @@ import ( var ( // Version is the current version of bd (overridden by ldflags at build time) - Version = "1.3.1-rc.1" + Version = "1.3.1-rc.2" // Build can be set via ldflags at compile time Build = "dev" // Commit and branch the git revision the binary was built from (optional ldflag) diff --git a/cmd/bd/winres/winres.json b/cmd/bd/winres/winres.json index c497b8a90..0033c3540 100644 --- a/cmd/bd/winres/winres.json +++ b/cmd/bd/winres/winres.json @@ -18,12 +18,12 @@ "Comments": "AI-supervised issue tracker for coding workflows", "CompanyName": "Steve Yegge", "FileDescription": "beads - AI-supervised issue tracker", - "FileVersion": "1.3.1-rc.1", + "FileVersion": "1.3.1-rc.2", "InternalName": "bd", "LegalCopyright": "Copyright (c) 2024-2026 Steve Yegge. MIT License.", "OriginalFilename": "bd.exe", "ProductName": "beads", - "ProductVersion": "1.3.1-rc.1" + "ProductVersion": "1.3.1-rc.2" } } } diff --git a/default.nix b/default.nix index 5ec1275c6..f3e3c72f8 100644 --- a/default.nix +++ b/default.nix @@ -7,7 +7,7 @@ }: buildGoModule { pname = "beads"; - version = "1.3.1-rc.1"; + version = "1.3.1-rc.2"; src = self; diff --git a/integrations/beads-mcp/pyproject.toml b/integrations/beads-mcp/pyproject.toml index 5832ba943..5ef179c88 100644 --- a/integrations/beads-mcp/pyproject.toml +++ b/integrations/beads-mcp/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "beads-mcp" -version = "1.3.1-rc.1" +version = "1.3.1-rc.2" description = "MCP server for beads issue tracker." readme = "README.md" requires-python = ">=3.10" diff --git a/integrations/beads-mcp/src/beads_mcp/__init__.py b/integrations/beads-mcp/src/beads_mcp/__init__.py index 8118d4656..ef0328f6c 100644 --- a/integrations/beads-mcp/src/beads_mcp/__init__.py +++ b/integrations/beads-mcp/src/beads_mcp/__init__.py @@ -4,4 +4,4 @@ This package provides an MCP (Model Context Protocol) server that exposes beads (bd) issue tracker functionality to MCP Clients. """ -__version__ = "1.3.1-rc.1" +__version__ = "1.3.1-rc.2" diff --git a/integrations/beads-mcp/uv.lock b/integrations/beads-mcp/uv.lock index 1ccf25835..06aab85ee 100644 --- a/integrations/beads-mcp/uv.lock +++ b/integrations/beads-mcp/uv.lock @@ -175,7 +175,7 @@ wheels = [ [[package]] name = "beads-mcp" -version = "1.3.1rc1" +version = "1.3.1rc2" source = { editable = "." } dependencies = [ { name = "fastmcp" }, diff --git a/npm-package/package.json b/npm-package/package.json index c9382b627..fadd26e7b 100644 --- a/npm-package/package.json +++ b/npm-package/package.json @@ -1,6 +1,6 @@ { "name": "@beads/bd", - "version": "1.3.1-rc.1", + "version": "1.3.1-rc.2", "description": "Beads issue tracker - lightweight memory system for coding agents with native binary support", "main": "bin/bd.js", "bin": { diff --git a/plugins/beads/.claude-plugin/plugin.json b/plugins/beads/.claude-plugin/plugin.json index 0ee54c8c7..846f0aadd 100644 --- a/plugins/beads/.claude-plugin/plugin.json +++ b/plugins/beads/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.1-rc.1", + "version": "1.3.1-rc.2", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" diff --git a/plugins/beads/.codex-plugin/plugin.json b/plugins/beads/.codex-plugin/plugin.json index 96958575f..d6ba3db26 100644 --- a/plugins/beads/.codex-plugin/plugin.json +++ b/plugins/beads/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "beads", - "version": "1.3.1-rc.1", + "version": "1.3.1-rc.2", "description": "AI-supervised issue tracking for coding workflows with agent skills.", "author": { "name": "Steve Yegge", diff --git a/plugins/beads/.copilot-plugin/plugin.json b/plugins/beads/.copilot-plugin/plugin.json index dd5cd6eff..5b2331b1b 100644 --- a/plugins/beads/.copilot-plugin/plugin.json +++ b/plugins/beads/.copilot-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.1-rc.1", + "version": "1.3.1-rc.2", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" From 498742a275ea356bd076528623ba01b24b0f5121 Mon Sep 17 00:00:00 2001 From: Gas City Adopt-PR Date: Wed, 30 Sep 2026 19:11:53 +0000 Subject: [PATCH 35/45] release: prep v1.3.1 Promote v1.3.1-rc.2 to stable per RELEASING.md "Validate and promote", on the same branch the RC validated (hotfix/1.3.1, rc.2 = 696e3967be). ./scripts/update-versions.sh 1.3.1 drops the -rc.2 suffix from version.go, the plugin manifests, marketplace.json, npm package.json, the MCP pyproject/__init__/uv.lock, default.nix, the .githooks markers and the winres strings. ./scripts/check-versions.sh is green at 1.3.1. CHANGELOG: a dated [1.3.1] section, following the [1.1.0] precedent: it points at [1.3.1-rc.2] and [1.3.1-rc.1] for the full notes and says the code is unchanged since rc.2. The rc sections are untouched. cmd/bd/info.go: a 1.3.1 digest at index 0 of versionChanges. No code changes. Co-Authored-By: Claude Opus 5.5 --- .claude-plugin/marketplace.json | 2 +- .githooks/post-checkout | 4 ++-- .githooks/post-merge | 4 ++-- .githooks/pre-commit | 4 ++-- .githooks/pre-push | 4 ++-- .githooks/prepare-commit-msg | 4 ++-- CHANGELOG.md | 18 ++++++++++++++++++ cmd/bd/info.go | 7 +++++++ cmd/bd/version.go | 2 +- cmd/bd/winres/winres.json | 4 ++-- default.nix | 2 +- integrations/beads-mcp/pyproject.toml | 2 +- .../beads-mcp/src/beads_mcp/__init__.py | 2 +- integrations/beads-mcp/uv.lock | 2 +- npm-package/package.json | 2 +- plugins/beads/.claude-plugin/plugin.json | 2 +- plugins/beads/.codex-plugin/plugin.json | 2 +- plugins/beads/.copilot-plugin/plugin.json | 2 +- 18 files changed, 47 insertions(+), 22 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index f8f85baf2..0a50b06ad 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ "name": "beads", "source": "./plugins/beads", "description": "AI-supervised issue tracker for coding workflows", - "version": "1.3.1-rc.2" + "version": "1.3.1" } ] } diff --git a/.githooks/post-checkout b/.githooks/post-checkout index af46d18a8..797c1199c 100755 --- a/.githooks/post-checkout +++ b/.githooks/post-checkout @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- +# --- BEGIN BEADS INTEGRATION v1.3.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.2 --- +# --- END BEADS INTEGRATION v1.3.1 --- diff --git a/.githooks/post-merge b/.githooks/post-merge index 2c1d4a142..07a8f632f 100755 --- a/.githooks/post-merge +++ b/.githooks/post-merge @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- +# --- BEGIN BEADS INTEGRATION v1.3.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.2 --- +# --- END BEADS INTEGRATION v1.3.1 --- diff --git a/.githooks/pre-commit b/.githooks/pre-commit index 4bb400076..850121d71 100755 --- a/.githooks/pre-commit +++ b/.githooks/pre-commit @@ -40,7 +40,7 @@ if [ -n "$staged_go_files" ]; then echo "$staged_go_files" | xargs git add fi -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- +# --- BEGIN BEADS INTEGRATION v1.3.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -97,4 +97,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.2 --- +# --- END BEADS INTEGRATION v1.3.1 --- diff --git a/.githooks/pre-push b/.githooks/pre-push index e732dc462..675d4edce 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -39,7 +39,7 @@ fi exec < <(printf '%s\n' "$push_refs") -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- +# --- BEGIN BEADS INTEGRATION v1.3.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -96,4 +96,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.2 --- +# --- END BEADS INTEGRATION v1.3.1 --- diff --git a/.githooks/prepare-commit-msg b/.githooks/prepare-commit-msg index 3bed11aa6..333f71b51 100755 --- a/.githooks/prepare-commit-msg +++ b/.githooks/prepare-commit-msg @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.1-rc.2 --- +# --- BEGIN BEADS INTEGRATION v1.3.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1-rc.2 --- +# --- END BEADS INTEGRATION v1.3.1 --- diff --git a/CHANGELOG.md b/CHANGELOG.md index 314de4705..a8362f509 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.3.1] - 2026-09-30 + +First stable release of the 1.3.1 patch line. It ships the same code as +[1.3.1-rc.2]; the only change since that candidate is the version stamp. The +full notes are in the [1.3.1-rc.2] and [1.3.1-rc.1] sections below, which +together cover everything since [1.3.0]. + +There is **no schema migration**: upgrading from 1.3.0, 1.3.1-rc.1 or +1.3.1-rc.2 is a binary swap. + +### Upgrade notes + +- Coming from 1.3.0, read the upgrade notes in [1.3.1-rc.2] and the + **Changed** entries in [1.3.1-rc.1]. The rc.1 change most likely to affect + scripts is the new `bd dolt status --json` shape on a proxied workspace + ([#6580](https://github.com/gastownhall/beads/pull/6580)). +- Coming from 1.3.1-rc.2, nothing changes in behaviour. + ## [1.3.1-rc.2] - 2026-09-29 Second release candidate for 1.3.1, still a patch line on top of 1.3.0 with diff --git a/cmd/bd/info.go b/cmd/bd/info.go index 94fadebb7..30e1a13f9 100644 --- a/cmd/bd/info.go +++ b/cmd/bd/info.go @@ -220,6 +220,13 @@ type VersionChange struct { // versionChanges contains agent-actionable changes for recent versions var versionChanges = []VersionChange{ + { + Version: "1.3.1", + Date: "2026-09-30", + Changes: []string{ + "STABLE: first stable 1.3.1 release. Same code as 1.3.1-rc.2; only the version stamp changed. No schema migration, so upgrading from 1.3.0 or any 1.3.1 RC is a binary swap. Coming from 1.3.0, the 1.3.1-rc.2 and 1.3.1-rc.1 entries below both apply; read the [1.3.1-rc.2] upgrade notes in CHANGELOG.md.", + }, + }, { Version: "1.3.1-rc.2", Date: "2026-09-29", diff --git a/cmd/bd/version.go b/cmd/bd/version.go index 3b15822b1..34efb4af8 100644 --- a/cmd/bd/version.go +++ b/cmd/bd/version.go @@ -16,7 +16,7 @@ import ( var ( // Version is the current version of bd (overridden by ldflags at build time) - Version = "1.3.1-rc.2" + Version = "1.3.1" // Build can be set via ldflags at compile time Build = "dev" // Commit and branch the git revision the binary was built from (optional ldflag) diff --git a/cmd/bd/winres/winres.json b/cmd/bd/winres/winres.json index 0033c3540..226a357a8 100644 --- a/cmd/bd/winres/winres.json +++ b/cmd/bd/winres/winres.json @@ -18,12 +18,12 @@ "Comments": "AI-supervised issue tracker for coding workflows", "CompanyName": "Steve Yegge", "FileDescription": "beads - AI-supervised issue tracker", - "FileVersion": "1.3.1-rc.2", + "FileVersion": "1.3.1", "InternalName": "bd", "LegalCopyright": "Copyright (c) 2024-2026 Steve Yegge. MIT License.", "OriginalFilename": "bd.exe", "ProductName": "beads", - "ProductVersion": "1.3.1-rc.2" + "ProductVersion": "1.3.1" } } } diff --git a/default.nix b/default.nix index f3e3c72f8..64dadbfa7 100644 --- a/default.nix +++ b/default.nix @@ -7,7 +7,7 @@ }: buildGoModule { pname = "beads"; - version = "1.3.1-rc.2"; + version = "1.3.1"; src = self; diff --git a/integrations/beads-mcp/pyproject.toml b/integrations/beads-mcp/pyproject.toml index 5ef179c88..e908d2f57 100644 --- a/integrations/beads-mcp/pyproject.toml +++ b/integrations/beads-mcp/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "beads-mcp" -version = "1.3.1-rc.2" +version = "1.3.1" description = "MCP server for beads issue tracker." readme = "README.md" requires-python = ">=3.10" diff --git a/integrations/beads-mcp/src/beads_mcp/__init__.py b/integrations/beads-mcp/src/beads_mcp/__init__.py index ef0328f6c..d3e4dd84e 100644 --- a/integrations/beads-mcp/src/beads_mcp/__init__.py +++ b/integrations/beads-mcp/src/beads_mcp/__init__.py @@ -4,4 +4,4 @@ This package provides an MCP (Model Context Protocol) server that exposes beads (bd) issue tracker functionality to MCP Clients. """ -__version__ = "1.3.1-rc.2" +__version__ = "1.3.1" diff --git a/integrations/beads-mcp/uv.lock b/integrations/beads-mcp/uv.lock index 06aab85ee..eee08ed62 100644 --- a/integrations/beads-mcp/uv.lock +++ b/integrations/beads-mcp/uv.lock @@ -175,7 +175,7 @@ wheels = [ [[package]] name = "beads-mcp" -version = "1.3.1rc2" +version = "1.3.1" source = { editable = "." } dependencies = [ { name = "fastmcp" }, diff --git a/npm-package/package.json b/npm-package/package.json index fadd26e7b..65257d62f 100644 --- a/npm-package/package.json +++ b/npm-package/package.json @@ -1,6 +1,6 @@ { "name": "@beads/bd", - "version": "1.3.1-rc.2", + "version": "1.3.1", "description": "Beads issue tracker - lightweight memory system for coding agents with native binary support", "main": "bin/bd.js", "bin": { diff --git a/plugins/beads/.claude-plugin/plugin.json b/plugins/beads/.claude-plugin/plugin.json index 846f0aadd..6ca3bc1a1 100644 --- a/plugins/beads/.claude-plugin/plugin.json +++ b/plugins/beads/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.1-rc.2", + "version": "1.3.1", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" diff --git a/plugins/beads/.codex-plugin/plugin.json b/plugins/beads/.codex-plugin/plugin.json index d6ba3db26..2428a46e1 100644 --- a/plugins/beads/.codex-plugin/plugin.json +++ b/plugins/beads/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "beads", - "version": "1.3.1-rc.2", + "version": "1.3.1", "description": "AI-supervised issue tracking for coding workflows with agent skills.", "author": { "name": "Steve Yegge", diff --git a/plugins/beads/.copilot-plugin/plugin.json b/plugins/beads/.copilot-plugin/plugin.json index 5b2331b1b..3ebeccd59 100644 --- a/plugins/beads/.copilot-plugin/plugin.json +++ b/plugins/beads/.copilot-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.1-rc.2", + "version": "1.3.1", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" From d1840c043f82620f998df0ec658d768e85e2dcbd Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 28 Sep 2026 21:20:51 -0700 Subject: [PATCH 36/45] fix(dolt): publish the blocked-state recheck after its transaction commits (#6876) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(dolt): publish the blocked-state recheck after its transaction commits Post-merge review of #6719 (landed 511849496..1ed5b3330) raised two majors against the post-commit blocked-state recheck. Both are fixed here. 1. Publication ordering. recheckBlockedAfterCommit minted its Dolt commit with doltAddAndCommitInTx while its own recompute transaction was still open. That helper's doc block declares the ordering a live production-observed lost-update hazard (LatentLabsSpace/NEXUS#92): DOLT_ADD stages the whole table from the session's BEGIN-time root, so the minted commit writes every concurrently committed row in the staged tables back to its BEGIN-time value. The recheck is the worst possible adopter, because it runs precisely and only when concurrent unblocking writes are racing on issues. It now mirrors runIssueOperationTxWithMessage: the recompute runs alone inside withRetryTx, its IssueRowsChanged verdict is captured by plain assignment, and doltAddAndCommitPostTx publishes after the SQL transaction has committed, against the post-merge root. A failed trailing commit is accounted like that path's — post_tx_commit_dropped plus a log line — and not as a recheck failure, because the flag repair itself is committed and durable, so telling an operator the rows may be stale and prescribing `bd recompute-blocked` would be false. 2. Test evidence. The wiring from a committed public write to the post-commit recheck had no regression pin on either tier: deleting both Dolt runner tails left the whole TestCloseRecheckBlocked suite green, because those tests either call recheckBlockedAfterCommit themselves or take the pending set out of the transaction before the runner can. TestCloseRecheckBlocked_RunnerTailRechecksACommittedWrite now drives the race through withRetryTx and withWriteTx, one arm each (they carry separate tail calls), and asserts the dependent settles with no help from the test; each arm reads the dependent's flag inside the write's own transaction first, so a collapsed race window fails loudly instead of passing vacuously. TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded does the same for withConn's tail on the serialized embedded tier, leaving the pending set in the transaction for the store to take. That embedded pin needed a second pass: its first form asserted on types.Issue.IsBlocked, which GetIssue never hydrates — is_blocked is not in sqlbuild.IssueBaseColumns — so the assertion could not fail and the arm survived deleting the tail. It now reads the stored flag through embeddedIsBlocked. The pre-existing "graph ends settled" assertion in the sibling test had the same defect and is fixed the same way, with its comment corrected to say what it actually pins (the in-transaction recompute, since each write there takes its own pending set). The ordering itself cannot be pinned behaviorally — the hazard needs a writer that commits inside the recheck's own transaction window, and no seam exposes it — so it gets a structural guard in the shape the journal- and version-scope completeness guards already use, with liveness anchors so a rename cannot turn it into a tautology. Restoring the in-tx ordering leaves every behavioral test in the package green and reds only that guard, which is why it is here. Mutation evidence for both pins, re-measured for this push. Deleting all three runner tails (both Dolt, plus embedded withConn) reds exactly the two new tests — both arms of the Dolt one at close_recheck_blocked_test.go:240, and the embedded one at blocked_recheck_internal_test.go:180 — and nothing else in either package. Restoring the in-tx publication ordering reds only TestPostCommitRecheckPublishesOutsideItsTransaction. The non-gating disclosure corrections the same review raised ride the next commit. The rest — a recheck-scope completeness test, noteBlockedRecheck on the sibling in-transaction recomputes, and routing RemoveDependencyWithOptions through withRetryTx — stay out of this PR and are tracked as beads. Co-Authored-By: Claude Opus 5 * docs(blocked-recheck): correct the residue claim and the out-of-scope list Three non-gating items from the same post-merge review of #6719, all disclosure rather than behavior. blocked_recheck.go's package doc claimed the writes it does not record can at worst leave a dependent "flagged ready while blocked". A sibling comment two files away refutes that universal: addDependencyInTx documents that parent-child adds are not monotonic, because adding an already-closed child can satisfy an any-children waits-for gate. So an add of a still-open child racing the close of that child strands the waiter in the other direction — flagged blocked while ready and hidden from `bd ready`, which is the #6716 shape the recheck exists to fix, reached through `bd dep add` or `bd create --parent`. The doc now states both directions. Recording the add paths is the code fix, and it stays tracked separately. The CHANGELOG's list of writes that still need the `bd doctor` / `bd recompute-blocked` repair named `bd batch`, `bd cook`, `bd mol squash`, `bd mol burn`, `bd duplicates --merge` and the wisp writes, but not the proxied-server route: under `--proxied-server` the ordinary single verbs are served by the uow provider (`bd close` reaches runCloseProxiedServer and its BatchCloser), and no uow write is scoped. A proxied-mode operator reading that list would have concluded their plain `bd close` races were fixed. The code doc already disclosed this correctly; only the user-facing text did not. closeIssueInTx hand-concatenated its recheck label instead of calling statusChangeRecheckLabel, which exists so that a close through update and a close through CloseIssue mint the same message. The helper returns the identical string for the closed status, so no recheck commit message moves — close_recheck_blocked_test.go:129 and :242 still assert the same text and still pass. Co-Authored-By: Claude Opus 5 * build(bazel): add the recheck publication guard to the dolt_test srcs The post-merge review of #6719 found that this PR's own new test file, internal/storage/dolt/blocked_recheck_publication_test.go, never reached the go_test(name = "dolt_test") target. That target uses a gazelle-managed explicit srcs list -- it names close_recheck_blocked_test.go and journal_scope_completeness_test.go -- and the new file was missing from it, so `bazel test` silently skipped the only pin on the ordering half of the finding-1 fix. Nothing caught it. The package's go_srcs filegroup is a glob(["**/*.go"]), and `make bazel-sync-check` (tools/bazel/go_srcs.py --check) plus scripts/bazel_policy_test.go only ever compare those glob filegroups, never a go_test srcs list, so the sync check stays green while the target drops the file. The glob filegroup sitting next to the explicit list is a decoy: `bazel query 'kind("source file", deps(...))'` also reports the file as present through it, and only `labels(srcs, //internal/storage/dolt:dolt_test)` distinguishes the two -- that query returns the file with this line and returns nothing without it. Generated with `bazel run //:gazelle` and reduced to this PR's own file. The same run also wants to touch twelve other BUILD.bazel files and to add cli_mirror_ref_test.go to this same list; all of that drift reproduces byte-identically from this branch's base commit with no part of this PR applied, so it belongs to whoever landed those files, not here. //internal/storage/dolt:dolt_test builds with the entry added, so the file needs no new dep -- journalscan was already in the target's deps. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 Co-authored-by: Gas City Adopt-PR (cherry picked from commit 73018a69ebff9f7d347bddff68f0759785f89cfa) --- .../dolt/blocked_recheck_publication_test.go | 63 ++++++++++ .../dolt/close_recheck_blocked_test.go | 116 ++++++++++++++++-- internal/storage/dolt/store.go | 41 ++++++- .../blocked_recheck_internal_test.go | 92 ++++++++++++-- internal/storage/issueops/blocked_recheck.go | 10 +- 5 files changed, 304 insertions(+), 18 deletions(-) create mode 100644 internal/storage/dolt/blocked_recheck_publication_test.go diff --git a/internal/storage/dolt/blocked_recheck_publication_test.go b/internal/storage/dolt/blocked_recheck_publication_test.go new file mode 100644 index 000000000..86676e212 --- /dev/null +++ b/internal/storage/dolt/blocked_recheck_publication_test.go @@ -0,0 +1,63 @@ +package dolt + +import ( + "testing" + + "github.com/steveyegge/beads/internal/storage/journalscan" +) + +// TestPostCommitRecheckPublishesOutsideItsTransaction is the structural guard +// for the recheck's publication ordering (the lost-update hazard +// doltAddAndCommitInTx documents, LatentLabsSpace/NEXUS#92). +// +// The hazard needs a writer that commits inside the recheck's OWN transaction +// window, and nothing in the store exposes that window to a test: the recheck +// body is not test-supplied, so the in-transaction ordering cannot be turned +// into a red assertion the way the runner tails can +// (close_recheck_blocked_test.go). Restoring it would therefore be invisible — +// every test in this package stays green either way — which is exactly the +// situation the journal- and version-scope completeness guards exist for, so +// the ordering gets the same kind of defense. +// +// Known limit, shared with those guards: the check is name-based over this +// package's syntax. It covers a publication call anywhere in the recheck's own +// body, including inside the transaction closure it passes to withRetryTx +// (journalscan walks nested function literals), but not one moved out into a +// named helper of its own. The liveness anchors below are what keep a rename +// from turning the whole guard into a tautology. +func TestPostCommitRecheckPublishesOutsideItsTransaction(t *testing.T) { + fns, err := journalscan.ParsePackage(".") + if err != nil { + t.Fatalf("parse dolt package: %v", err) + } + + const ( + inTx = "doltAddAndCommitInTx" + postTx = "doltAddAndCommitPostTx" + recheck = "DoltStore.recheckBlockedAfterCommit" + // Anchors: one function known to publish with each ordering. + knownInTxPublisher = "DoltStore.demoteToWispInTx" + knownPostTxPublisher = "DoltStore.runIssueOperationTxWithMessage" + ) + publishes := func(key, helper string) bool { + f := fns[key] + if f == nil { + t.Fatalf("%s is not in the parsed package — this guard is not measuring what it names", key) + } + return f.CallsAnyOf(map[string]bool{helper: true}) + } + + if !publishes(knownInTxPublisher, inTx) { + t.Fatalf("%s no longer reads as an in-transaction publisher: the %s analysis is broken, so its verdict on %s means nothing", knownInTxPublisher, inTx, recheck) + } + if !publishes(knownPostTxPublisher, postTx) { + t.Fatalf("%s no longer reads as a post-transaction publisher: the %s analysis is broken, so its verdict on %s means nothing", knownPostTxPublisher, postTx, recheck) + } + + if publishes(recheck, inTx) { + t.Fatalf("%s publishes with %s: DOLT_ADD stages the whole table from the transaction's BEGIN-time root, so a Dolt commit minted inside the recheck's own transaction writes every concurrently committed row back to its BEGIN-time value — and a recheck runs only when concurrent unblocking writes are racing. Publish with %s after the transaction commits instead.", recheck, inTx, postTx) + } + if !publishes(recheck, postTx) { + t.Fatalf("%s no longer publishes with %s: a corrected is_blocked flag that reaches no Dolt commit sits dirty in the working set until an unrelated write sweeps it in", recheck, postTx) + } +} diff --git a/internal/storage/dolt/close_recheck_blocked_test.go b/internal/storage/dolt/close_recheck_blocked_test.go index f7fcca35d..bbbb90ed1 100644 --- a/internal/storage/dolt/close_recheck_blocked_test.go +++ b/internal/storage/dolt/close_recheck_blocked_test.go @@ -5,6 +5,8 @@ import ( "database/sql" "errors" "slices" + "strings" + "sync" "testing" "github.com/steveyegge/beads/internal/storage/issueops" @@ -14,12 +16,19 @@ import ( // seedSiblingBlockers creates rb-a and rb-b, both blocking rb-c. func seedSiblingBlockers(t *testing.T, ctx context.Context, store *DoltStore) { t.Helper() - for _, id := range []string{"rb-a", "rb-b", "rb-c"} { + seedSiblingBlockersAs(t, ctx, store, "rb-a", "rb-b", "rb-c") +} + +// seedSiblingBlockersAs is seedSiblingBlockers for a named triple, so one store +// can carry more than one independent fixture. +func seedSiblingBlockersAs(t *testing.T, ctx context.Context, store *DoltStore, blockerA, blockerB, dependent string) { + t.Helper() + for _, id := range []string{blockerA, blockerB, dependent} { createPerm(t, ctx, store, id) } - addDependency(t, ctx, store, "rb-c", "rb-a", types.DepBlocks) - addDependency(t, ctx, store, "rb-c", "rb-b", types.DepBlocks) - assertIsBlocked(t, ctx, store, "issues", "rb-c", true) + addDependency(t, ctx, store, dependent, blockerA, types.DepBlocks) + addDependency(t, ctx, store, dependent, blockerB, types.DepBlocks) + assertIsBlocked(t, ctx, store, "issues", dependent, true) } // beginRacingWrite opens a transaction on its own pooled connection, pins its @@ -28,6 +37,13 @@ func seedSiblingBlockers(t *testing.T, ctx context.Context, store *DoltStore) { // paths do. It returns the dependents the write recorded for the post-commit // recheck and a commit step that lands the write and frees the connection. func beginRacingWrite(t *testing.T, ctx context.Context, store *DoltStore, name string, write func(tx *sql.Tx) error) (pending issueops.BlockedRecheck, commit func()) { + t.Helper() + return beginRacingWriteOn(t, ctx, store, name, []string{"rb-a", "rb-b"}, write) +} + +// beginRacingWriteOn is beginRacingWrite for a named fixture: blockers are the +// ids whose open state pins the snapshot. +func beginRacingWriteOn(t *testing.T, ctx context.Context, store *DoltStore, name string, blockers []string, write func(tx *sql.Tx) error) (pending issueops.BlockedRecheck, commit func()) { t.Helper() conn, err := store.db.Conn(ctx) if err != nil { @@ -39,13 +55,20 @@ func beginRacingWrite(t *testing.T, ctx context.Context, store *DoltStore, name } clearScope := issueops.ScopeBlockedRecheckTransaction(tx) defer clearScope() + placeholders := make([]string, 0, len(blockers)) + args := make([]any, 0, len(blockers)) + for _, id := range blockers { + placeholders = append(placeholders, "?") + args = append(args, id) + } var open int if err := tx.QueryRowContext(ctx, - "SELECT COUNT(*) FROM issues WHERE id IN ('rb-a', 'rb-b') AND status <> 'closed'").Scan(&open); err != nil { + "SELECT COUNT(*) FROM issues WHERE id IN ("+strings.Join(placeholders, ", ")+") AND status <> 'closed'", + args...).Scan(&open); err != nil { t.Fatalf("pin snapshot for %s: %v", name, err) } - if open != 2 { - t.Fatalf("snapshot for %s sees %d open blockers, want 2", name, open) + if open != len(blockers) { + t.Fatalf("snapshot for %s sees %d open blockers, want %d", name, open, len(blockers)) } if err := write(tx); err != nil { t.Fatalf("%s in tx: %v", name, err) @@ -144,6 +167,85 @@ func TestCloseRecheckBlocked_PublicClosePaths(t *testing.T) { } } +// TestCloseRecheckBlocked_RunnerTailRechecksACommittedWrite pins the wiring +// this feature IS: a write that commits on a store write runner gets its +// recorded dependents rechecked by that runner, with no help from the test. +// +// Every other test in this file reaches the recheck another way — it calls +// recheckBlockedAfterCommit itself, or takes the pending set out of the +// transaction before the runner can — so all of them stay green when both +// runner tails are deleted, and the tier #6716 was reported on had no +// regression pin on its own deliverable. +// +// Each arm makes the in-transaction recompute provably blind rather than +// hoping for an interleaving: a racing close is held open so the runner's +// transaction cannot see it, the arm reads the dependent's flag inside that +// transaction to record what its own recompute concluded, and the racing close +// is landed from inside the runner's body — before the runner commits, after +// its snapshot was taken. Only a recheck on a fresh snapshot can settle the +// dependent from there. withRetryTx and withWriteTx carry separate tail calls, +// so each gets its own arm and its own fixture. +func TestCloseRecheckBlocked_RunnerTailRechecksACommittedWrite(t *testing.T) { + store, cleanup := setupConcurrentTestStore(t) + defer cleanup() + ctx, cancel := testContext(t) + defer cancel() + + for _, arm := range []struct { + runner string + run func(context.Context, func(*sql.Tx) error) error + racing string // closed by the transaction the runner cannot see + committed string // closed through the runner + dependent string + }{ + {"withRetryTx", store.withRetryTx, "rt-a", "rt-b", "rt-c"}, + {"withWriteTx", store.withWriteTx, "wt-a", "wt-b", "wt-c"}, + } { + t.Run(arm.runner, func(t *testing.T) { + seedSiblingBlockersAs(t, ctx, store, arm.racing, arm.committed, arm.dependent) + _, landRacingClose := beginRacingWriteOn(t, ctx, store, "close of "+arm.racing, + []string{arm.racing, arm.committed}, func(tx *sql.Tx) error { + _, err := issueops.CloseIssueInTx(ctx, tx, arm.racing, "done", "tester", "") + return err + }) + + var landed sync.Once + var blockedInTx int + if err := arm.run(ctx, func(tx *sql.Tx) error { + if _, err := issueops.CloseIssueInTx(ctx, tx, arm.committed, "done", "tester", ""); err != nil { + return err + } + // What this transaction's own recompute concluded, read before + // the racing close lands so it cannot be anything else. + if err := tx.QueryRowContext(ctx, + "SELECT is_blocked FROM issues WHERE id = ?", arm.dependent).Scan(&blockedInTx); err != nil { + return err + } + // Land the racing close inside the window: this transaction + // began before it, so its snapshot cannot include it, and the + // dependent is stale the moment this transaction commits. + // sync.Once because withRetryTx may re-run this body. + landed.Do(landRacingClose) + return nil + }); err != nil { + t.Fatalf("%s: a committed close returned %v, want nil", arm.runner, err) + } + if blockedInTx == 0 { + t.Fatalf("%s: %s was already unblocked inside the write's own transaction, so this arm measures nothing about the tail — the runner retried and re-ran the body on a snapshot that already included the racing close", arm.runner, arm.dependent) + } + + // Nothing here called recheckBlockedAfterCommit or touched the + // pending set: the runner's tail is the only thing that can have + // settled the dependent and published the correction. + assertIsBlocked(t, ctx, store, "issues", arm.dependent, false) + requireCleanTables(ctx, t, store, "issues") + if !doltHasCommitMessage(ctx, t, store, "bd: recheck blocked after close of "+arm.committed) { + t.Fatalf("%s: the recheck corrected %s but minted no Dolt commit for it", arm.runner, arm.dependent) + } + }) + } +} + // TestCloseRecheckBlocked_ConcurrentCloseAndDependencyRemoval is #6716 with a // dependency removal as the second writer: a close of rb-a racing a removal of // the rb-c -> rb-b edge. Each transaction still sees the other blocker in diff --git a/internal/storage/dolt/store.go b/internal/storage/dolt/store.go index 58c182966..6850340f7 100644 --- a/internal/storage/dolt/store.go +++ b/internal/storage/dolt/store.go @@ -1256,6 +1256,20 @@ func (s *DoltStore) commitWriteTx(ctx context.Context, fn func(tx *sql.Tx) error // the corrected flag reaches history the way the close's own rows did rather // than sitting dirty in the working set. // +// Publication ordering (lost-update fix, LatentLabsSpace/NEXUS#92): the Dolt +// commit runs AFTER the recompute transaction has committed, through +// doltAddAndCommitPostTx, exactly as runIssueOperationTxWithMessage does and +// for a sharper reason than that path had. DOLT_ADD stages the whole table +// from the session's transaction root, so minting the commit inside the still +// open transaction would write every row a concurrent writer committed inside +// that window back to its BEGIN-time value — the hazard doltAddAndCommitInTx +// documents. A recheck runs precisely and only when concurrent unblocking +// writes are racing on issues, so that window is never idle here. Committing +// the SQL transaction first lets Dolt's commit-time merge reconcile this +// repair with those writers, and the post-commit staging then stages the +// merged state. This also keeps the append-only versioned-history tables that +// withVersionedHistoryTables adds to the staged set out of that hazard. +// // It runs on issueops.BlockedRecheckContext: the write it follows is durable, // so the repair must outlive that write's cancellation, and a recheck must // never start another recheck. The returned failure is for the caller to log, @@ -1266,16 +1280,39 @@ func (s *DoltStore) recheckBlockedAfterCommit(ctx context.Context, pending issue } ctx, cancel := issueops.BlockedRecheckContext(ctx) defer cancel() + var rowsChanged bool err := s.withRetryTx(ctx, func(tx *sql.Tx) error { + // Plain assignment, never |=: withRetryTx re-runs this body on a retry + // and the rolled-back attempt's verdict must not survive into the + // publication decision below (the closure contract on withRetryTx). + rowsChanged = false result, err := issueops.RecomputeIsBlockedInTxWithResult(ctx, tx, pending.IssueIDs, pending.WispIDs) - if err != nil || !result.IssueRowsChanged { + if err != nil { return err } - return s.doltAddAndCommitInTx(ctx, tx, []string{"issues"}, pending.CommitMessage()) + rowsChanged = result.IssueRowsChanged + return nil }) if err != nil { return issueops.BlockedRecheckFailed(err) } + if !rowsChanged { + return nil + } + commitMsg := pending.CommitMessage() + if err := s.doltAddAndCommitPostTx(ctx, []string{"issues"}, commitMsg); err != nil { + // The repair itself is committed and durable: only its audit commit is + // missing, and the corrected flag rides the next Dolt commit on the + // branch. This is the post_tx_commit_dropped situation, not the recheck + // failure logBlockedRecheckFailure reports — that line tells an operator + // rows are possibly stale and prescribes `bd recompute-blocked`, which + // would be false here and would fire the counter a fleet alerts on for + // rows that are already correct. Accounted exactly like + // runIssueOperationTxWithMessage's trailing commit instead. + doltMetrics.postTxCommitDropped.Add(ctx, 1) + log.Printf("dolt: post-tx dolt commit failed for %q (blocked-state repair already committed; change rides the next dolt commit): %v", + commitMsg, err) + } return nil } diff --git a/internal/storage/embeddeddolt/blocked_recheck_internal_test.go b/internal/storage/embeddeddolt/blocked_recheck_internal_test.go index d96b235d4..66cc7b806 100644 --- a/internal/storage/embeddeddolt/blocked_recheck_internal_test.go +++ b/internal/storage/embeddeddolt/blocked_recheck_internal_test.go @@ -14,6 +14,24 @@ import ( "github.com/steveyegge/beads/internal/types" ) +// embeddedIsBlocked reads the STORED is_blocked flag for id. +// +// It exists because GetIssue cannot answer this question: is_blocked is not in +// the hydrated column list (sqlbuild.IssueBaseColumns), so types.Issue.IsBlocked +// is always false on a hydrated issue and an assertion on it holds no matter +// what the flag says. The read runs with commit=false, so it takes nothing out +// of the recheck scope and triggers no recheck of its own. +func embeddedIsBlocked(t *testing.T, ctx context.Context, store *EmbeddedDoltStore, id string) bool { + t.Helper() + var blocked int + if err := store.withConn(ctx, false, func(tx *sql.Tx) error { + return tx.QueryRowContext(ctx, "SELECT is_blocked FROM issues WHERE id = ?", id).Scan(&blocked) + }); err != nil { + t.Fatalf("read is_blocked for %s: %v", id, err) + } + return blocked != 0 +} + // TestEmbeddedBlockedRecheckRecordsUnblockingWrites pins, on a real engine, // that an unblocking write running on the embedded store's own scoped // transaction records the dependents it recomputed for the post-commit @@ -92,17 +110,77 @@ func TestEmbeddedBlockedRecheckRecordsUnblockingWrites(t *testing.T) { return issueops.DeleteIssueInTx(ctx, tx, "rb-b", "") }), "delete of rb-b", "rb-b") - // Through the public surface the recheck runs after commit and the graph - // ends settled: both blockers gone, rb-c unblocked. - got, err := store.GetIssue(ctx, "rb-c") - if err != nil { - t.Fatalf("get rb-c: %v", err) - } - if got.IsBlocked { + // The graph ends settled: both blockers gone, rb-c unblocked. Each write + // above took its pending set out of the transaction, so what settled the + // flag here is the in-transaction recompute, not the post-commit recheck — + // that one is pinned by TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded. + if embeddedIsBlocked(t, ctx, store, "rb-c") { t.Fatal("rb-c is still blocked after its last blocker was deleted") } } +// TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded pins withConn's tail on +// this tier: the recheck must run for whatever the committed transaction +// recorded, with the test taking nothing out of that transaction first. The +// test above takes the pending set inside the body — which is what it is for, +// but it also means the store's tail always rechecks an empty set there, so +// nothing was measuring the tail itself. +// +// Embedded transactions serialize (see recheckBlockedAfterCommit), so the +// snapshot skew of gastownhall/beads#6716 cannot be staged here. The +// transaction instead leaves the residue that skew leaves behind — a dependent +// still flagged blocked once every blocker is closed — by closing the second +// blocker with a raw status write, which no recompute observes. That stands in +// for the concurrent writer this tier cannot host, and from there only a +// post-commit recheck of the recorded dependent can settle the flag. +func TestEmbeddedBlockedRecheckSettlesWhatTheWriteRecorded(t *testing.T) { + ctx := context.Background() + store, err := Open(ctx, filepath.Join(t.TempDir(), ".beads"), "recheckafter", "main") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = store.Close() }) + if err := store.SetConfig(ctx, "issue_prefix", "rb"); err != nil { + t.Fatal(err) + } + for _, id := range []string{"rb-a", "rb-b", "rb-c"} { + iss := &types.Issue{ID: id, Title: id, Status: types.StatusOpen, Priority: 2, IssueType: types.TypeTask} + if err := store.CreateIssue(ctx, iss, "tester"); err != nil { + t.Fatalf("create %s: %v", id, err) + } + } + for _, blocker := range []string{"rb-a", "rb-b"} { + if err := store.AddDependency(ctx, &types.Dependency{IssueID: "rb-c", DependsOnID: blocker, Type: types.DepBlocks}, "tester"); err != nil { + t.Fatalf("add dependency rb-c -> %s: %v", blocker, err) + } + } + + var blockedInTx int + if err := store.withConn(ctx, true, func(tx *sql.Tx) error { + // Records rb-c and recomputes it: rb-b is open here, so rb-c stays + // blocked. + if _, err := issueops.CloseIssueInTx(ctx, tx, "rb-a", "done", "tester", ""); err != nil { + return err + } + // The blocker whose close no recompute sees. + if _, err := tx.ExecContext(ctx, "UPDATE issues SET status = 'closed' WHERE id = ?", "rb-b"); err != nil { + return err + } + // The pending set is deliberately left in the transaction for the + // store's own tail to take. + return tx.QueryRowContext(ctx, "SELECT is_blocked FROM issues WHERE id = ?", "rb-c").Scan(&blockedInTx) + }); err != nil { + t.Fatalf("close rb-a: %v", err) + } + if blockedInTx == 0 { + t.Fatal("rb-c was already unblocked inside the transaction, so this test measures nothing about the post-commit recheck") + } + + if embeddedIsBlocked(t, ctx, store, "rb-c") { + t.Fatal("rb-c is still blocked after a committed write recorded it: withConn's tail did not recheck what the transaction recorded") + } +} + // TestEmbeddedBlockedRecheckFailureKeepsTheWrite pins what a *failing* recheck // does at the store seam, which is the half of the contract the wrapper's unit // test cannot reach: the write it follows has already committed, so the store diff --git a/internal/storage/issueops/blocked_recheck.go b/internal/storage/issueops/blocked_recheck.go index d31b67c66..98dcd1ba6 100644 --- a/internal/storage/issueops/blocked_recheck.go +++ b/internal/storage/issueops/blocked_recheck.go @@ -28,8 +28,14 @@ import ( // flag has to move — removing the one closed child of an any-children gate // can block its waiter, and the recheck settles that too. Writes that add a // blocker (a reopen, a dependency add, a parent-child add) recompute the same -// way and are not recorded here; a racing pair where the last committer adds -// a blocker can leave a dependent flagged ready while blocked. +// way and are not recorded here, and the residue they leave runs in both +// directions. A racing pair whose last committer adds a plain blocks edge can +// leave a dependent flagged ready while blocked. A parent-child add is not +// monotonic, though — an already-closed child can satisfy an any-children +// waits-for gate (see addDependencyInTx) — so an add of a still-open child +// racing the close of that child leaves the waiter flagged blocked while +// ready and hidden from `bd ready`: the #6716 shape itself, reached through +// `bd dep add` or `bd create --parent`. // // # What is scoped, and what is not // From 35fea9b049d7a8e940387ac66eda2427bc179332 Mon Sep 17 00:00:00 2001 From: Rongjun GENG Date: Sat, 3 Oct 2026 14:54:09 -0700 Subject: [PATCH 37/45] docs(changelog): record the #6876 backport to hotfix/1.3.1 (#7030) Fold-in note for the preceding cherry-pick. Documents the gap this closes for a release-branch reader: the hotfix batches that ported #6716's fan-in fix carried the post-commit blocked-state recheck in the form #6876 later corrected on main, missing it between batches 2 and 4. Co-Authored-By: Claude Sonnet 5 --- CHANGELOG.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a8362f509..7a7c48743 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- **Backported #6876 to this release line: the post-commit blocked-state + recheck now publishes outside its own transaction.** The hotfix batches + that ported #6716's fan-in fix ([#6719](https://github.com/gastownhall/beads/pull/6719)/[#6936](https://github.com/gastownhall/beads/pull/6936), + [#6942](https://github.com/gastownhall/beads/pull/6942)/[#6947](https://github.com/gastownhall/beads/pull/6947)) carried the recheck in the form #6876 + later corrected on `main`: the Dolt commit was minted while the recompute's + own SQL transaction was still open, so `DOLT_ADD` staged the table from the + session's BEGIN-time root and could write concurrently committed `issues` + rows back to their BEGIN-time values — reachable only when the recheck + runs while unblocking writes are racing on `issues`. The recompute now runs + alone in its own retried transaction and the trailing Dolt commit publishes + after it commits; a failed trailing commit is counted as + `post_tx_commit_dropped` instead of surfacing as a recheck failure. + ## [1.3.1] - 2026-09-30 First stable release of the 1.3.1 patch line. It ships the same code as From 97bfa4c149ad0708d64f7013ef6f130cbf298321 Mon Sep 17 00:00:00 2001 From: Rongjun GENG Date: Sun, 4 Oct 2026 07:10:10 -0700 Subject: [PATCH 38/45] ci: trigger pull_request CI on hotfix/** branches, not just release/** (#7148) Hand port of #7198 (main 3c65c99c29) onto hotfix/1.3.1. NOT a clean cherry-pick; adapted for this branch: - conformance.yml and cross-version-smoke.yml: main added `paths:` filters that this branch does not have. Only the `branches:` line is changed here; the path filters are not brought over. - CHANGELOG.md: the [Unreleased] entry is dropped. The next release-prep on this branch records it in its own section. - scripts/ci_workflow_test.go: this branch has no readPolicyFile helper and no prRiskWorkflowName constant. The test reads the files with os.ReadFile and names pr-risk.yml literally. The assertions are unchanged. The pull_request trigger is read from the PR's merge ref, so this has to be on hotfix/1.3.1 itself before PRs into it (the v1.3.2 backports) run the full CI and produce `CI Gate / Required`. (ported from commit 3c65c99c29686535bdfd0ae54cbf6f44a11361c5) Co-Authored-By: Claude Opus 5.5 --- .github/workflows/conformance.yml | 2 +- .github/workflows/cross-version-smoke.yml | 2 +- .github/workflows/pr-risk.yml | 2 +- .github/workflows/pr.yml | 2 +- .github/workflows/regression.yml | 2 +- scripts/ci_workflow_test.go | 41 +++++++++++++++++++++++ 6 files changed, 46 insertions(+), 5 deletions(-) diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml index e981539db..b79c71bbb 100644 --- a/.github/workflows/conformance.yml +++ b/.github/workflows/conformance.yml @@ -5,7 +5,7 @@ name: Conformance on: pull_request: - branches: [ main, 'release/**' ] + branches: [ main, 'release/**', 'hotfix/**' ] merge_group: concurrency: diff --git a/.github/workflows/cross-version-smoke.yml b/.github/workflows/cross-version-smoke.yml index d9f5c7a71..5f165fca7 100644 --- a/.github/workflows/cross-version-smoke.yml +++ b/.github/workflows/cross-version-smoke.yml @@ -10,7 +10,7 @@ on: # feedback. Release-targeted PRs are exactly the ones where cross-version # upgrade breakage is most expensive to discover at tag time. pull_request: - branches: [ main, 'release/**' ] + branches: [ main, 'release/**', 'hotfix/**' ] # Allow manual trigger with custom version count workflow_dispatch: diff --git a/.github/workflows/pr-risk.yml b/.github/workflows/pr-risk.yml index 60ba8ca75..e667d691e 100644 --- a/.github/workflows/pr-risk.yml +++ b/.github/workflows/pr-risk.yml @@ -2,7 +2,7 @@ name: PR Risk on: pull_request: - branches: [ main, 'release/**' ] + branches: [ main, 'release/**', 'hotfix/**' ] merge_group: concurrency: diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 2215173da..7e62dde1b 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -5,7 +5,7 @@ on: # release/** is here because release-prep and release-fix PRs are based on # the release branch, not on main. Without it they run almost no CI and the # first full signal is the tag build, where a failure burns the tag. - branches: [ main, 'release/**' ] + branches: [ main, 'release/**', 'hotfix/**' ] merge_group: concurrency: diff --git a/.github/workflows/regression.yml b/.github/workflows/regression.yml index 46237752b..c3eec84bb 100644 --- a/.github/workflows/regression.yml +++ b/.github/workflows/regression.yml @@ -5,7 +5,7 @@ on: push: branches: [main] pull_request: - branches: [main, 'release/**'] + branches: [main, 'release/**', 'hotfix/**'] # Allow manual trigger workflow_dispatch: diff --git a/scripts/ci_workflow_test.go b/scripts/ci_workflow_test.go index b92665797..c7839da49 100644 --- a/scripts/ci_workflow_test.go +++ b/scripts/ci_workflow_test.go @@ -6,6 +6,7 @@ import ( "path/filepath" "reflect" "regexp" + "slices" "strings" "testing" @@ -35,6 +36,46 @@ func TestCIWorkflowArtifactOwnership(t *testing.T) { } } +// TestPullRequestWorkflowsTriggerOnHotfixBranches is the #7148 regression. +// pr.yml's own comment states why release/** is in the pull_request +// trigger: "release-prep and release-fix PRs are based on the release +// branch, not on main. Without it they run almost no CI and the first +// full signal is the tag build, where a failure burns the tag." A hotfix +// backport PR (hotfix/1.3.1 and its siblings) is in exactly that position +// — based on a branch other than main — so it needs the same trigger, +// alongside release/**, not instead of it. +func TestPullRequestWorkflowsTriggerOnHotfixBranches(t *testing.T) { + type triggers struct { + On struct { + PullRequest struct { + Branches []string `yaml:"branches"` + } `yaml:"pull_request"` + } `yaml:"on"` + } + root := sourceRepoRoot(t) + for _, name := range []string{ + "pr.yml", "pr-risk.yml", "conformance.yml", "cross-version-smoke.yml", "regression.yml", + } { + t.Run(name, func(t *testing.T) { + data, err := os.ReadFile(filepath.Join(root, ".github", "workflows", name)) + if err != nil { + t.Fatalf("read %s: %v", name, err) + } + var doc triggers + if err := yaml.Unmarshal(data, &doc); err != nil { + t.Fatal(err) + } + branches := doc.On.PullRequest.Branches + if !slices.Contains(branches, "release/**") { + t.Fatalf("%s pull_request.branches = %v, want release/** present (this test asserts the baseline it extends)", name, branches) + } + if !slices.Contains(branches, "hotfix/**") { + t.Errorf("%s pull_request.branches = %v, want hotfix/** alongside release/** — a hotfix backport PR is based on a branch other than main, same as a release-prep PR", name, branches) + } + }) + } +} + func TestPRCIGateRequiresPolicyAndLintWrappers(t *testing.T) { gate := readCIWorkflow(t, "pr.yml").job(t, "ci-gate") gateEnv := gate.step(t, "Evaluate CI gate").Env From 09e6397f1e89744879eda7f0f07fa5b1cb097c04 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 5 Oct 2026 03:36:31 +0000 Subject: [PATCH 39/45] ci(hotfix/1.3.1): install pinned golangci-lint instead of the disallowed action Release-only adaptation; there is no main commit to cherry-pick. The repo's allowed-actions policy does not list golangci/golangci-lint-action, so with pull_request now triggering pr.yml on hotfix/** the whole workflow fails at startup (startup_failure on PR #7217). main dropped the action in #7127 (bc19f6e47) and installs the pinned v2.10.1 release binary with scripts/ci/install-golangci-lint.sh. This takes that script verbatim from main (4b2852ce2) and keeps this branch's single Lint job, with --new-from-rev set to the PR base in place of only-new-issues. Locally, golangci-lint v2.10.1 --new-from-rev=origin/hotfix/1.3.1 reports 0 issues. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/pr.yml | 16 +++-- scripts/ci/install-golangci-lint.sh | 97 +++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+), 5 deletions(-) create mode 100755 scripts/ci/install-golangci-lint.sh diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 7e62dde1b..65cd53f80 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -1014,14 +1014,20 @@ jobs: go-version-file: 'go.mod' cache: false + # golangci/golangci-lint-action is not on this repository's allowed-actions + # list, so a workflow that names it fails at startup. Install the pinned + # release binary (the same script main uses) and report only issues new + # relative to the PR base, as only-new-issues did. + - name: Install golangci-lint + run: ./scripts/ci/install-golangci-lint.sh + - name: golangci-lint - uses: golangci/golangci-lint-action@ba0d7d2ec06a0ea1cb5fa41b2e4a3ab91d21278a # v9 env: GOWORK: "off" - with: - version: v2.10.1 - only-new-issues: true - args: --config=.golangci.yml --modules-download-mode=readonly --timeout=5m --build-tags=gms_pure_go + NEW_FROM_REV: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha }} + run: >- + golangci-lint run --config=.golangci.yml --modules-download-mode=readonly + --timeout=5m --build-tags=gms_pure_go --new-from-rev="$NEW_FROM_REV" windows-make-shell: name: Windows Make shell (${{ matrix.host }}) diff --git a/scripts/ci/install-golangci-lint.sh b/scripts/ci/install-golangci-lint.sh new file mode 100755 index 000000000..f9c0182ce --- /dev/null +++ b/scripts/ci/install-golangci-lint.sh @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Install the pinned golangci-lint release binary instead of `go install` +# (F5.2): the release tarball drops the install step from about 58s to about +# 3s, because it is a straight download-and-verify rather than a compile. +# CI-only: this script requires $RUNNER_TEMP and $GITHUB_PATH, so the local +# `.githooks/pre-commit` hook keeps using +# `go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.10.1`, +# which needs no change here. `make ci-pr-lint` (scripts/pr-lint) looks up +# whatever `golangci-lint` binary is on PATH; it does not use `go run`. +# +# Keep this version and the sha256 values in sync with CONTRIBUTING.md, +# engdocs/LINTING.md, .github/workflows/pr.yml, .github/workflows/main.yml, +# .github/workflows/ci-measurements.yml and .githooks/pre-commit. Verified +# locally against the release's own golangci-lint-2.10.1-checksums.txt. +readonly version="2.10.1" +readonly max_attempts=3 +readonly retry_delay_seconds=5 + +# A case function rather than `declare -A`: associative arrays need bash 4, +# and macOS's /bin/bash 3.2 silently parses `declare -A` as an indexed array, +# so `[amd64]=` becomes an arithmetic lookup that dies under `set -u` before +# the Linux-only guard below can print its message. +sha256_for_arch() { + case "$1" in + amd64) printf '%s' "dfa775874cf0561b404a02a8f4481fc69b28091da95aa697259820d429b09c99" ;; + arm64) printf '%s' "6652b42ae02915eb2f9cb2a2e0cac99514c8eded8388d88ae3e06e1a52c00de8" ;; + esac +} + +: "${RUNNER_TEMP:?RUNNER_TEMP is required; this script is CI-only}" +: "${GITHUB_PATH:?GITHUB_PATH is required; this script is CI-only}" + +os="$(uname -s | tr '[:upper:]' '[:lower:]')" +if [[ "$os" != "linux" ]]; then + printf 'install-golangci-lint.sh only supports Linux runners, got: %s\n' "$os" >&2 + exit 1 +fi + +arch="$(uname -m)" +case "$arch" in + x86_64 | amd64) arch="amd64" ;; + aarch64 | arm64) arch="arm64" ;; + *) + printf 'Unsupported architecture for pinned golangci-lint install: %s\n' "$arch" >&2 + exit 1 + ;; +esac + +expected_sha256="$(sha256_for_arch "$arch")" +: "${expected_sha256:?no pinned sha256 for arch: $arch}" +readonly asset="golangci-lint-${version}-${os}-${arch}.tar.gz" +readonly url="https://github.com/golangci/golangci-lint/releases/download/v${version}/${asset}" + +workdir="$(mktemp -d)" +trap 'rm -rf "$workdir"' EXIT + +for ((attempt = 1; attempt <= max_attempts; attempt++)); do + # See install-dolt.sh for why the status capture can't live in the `if`. + status=0 + curl -fsSL -o "$workdir/$asset" "$url" || status=$? + if ((status == 0)); then + break + fi + if ((attempt == max_attempts)); then + printf 'Failed to download %s after %d attempts (curl exit %d).\n' "$url" "$max_attempts" "$status" >&2 + exit "$status" + fi + printf 'Failed to download %s (attempt %d/%d); retrying in %d seconds.\n' \ + "$url" "$attempt" "$max_attempts" "$retry_delay_seconds" >&2 + sleep "$retry_delay_seconds" +done + +actual_sha256="$(sha256sum "$workdir/$asset" | awk '{print $1}')" +if [[ "$actual_sha256" != "$expected_sha256" ]]; then + printf '%s sha256 mismatch: got %s, want %s\n' "$asset" "$actual_sha256" "$expected_sha256" >&2 + exit 1 +fi + +tar -xzf "$workdir/$asset" -C "$workdir" + +install_dir="$RUNNER_TEMP/golangci-lint" +mkdir -p "$install_dir" +install -m 0755 "$workdir/golangci-lint-${version}-${os}-${arch}/golangci-lint" "$install_dir/golangci-lint" + +echo "$install_dir" >> "$GITHUB_PATH" + +# Fail loudly if the extracted binary does not report the pinned version: a +# bad tarball layout would otherwise silently put a stale or wrong binary on +# PATH. Compare the version token exactly, as install-dolt.sh does. +installed="$("$install_dir/golangci-lint" version)" +if [[ "$installed" != *"version $version"* ]]; then + printf 'golangci-lint reports "%s", want version %s\n' "$installed" "$version" >&2 + exit 1 +fi +printf 'Installed pinned golangci-lint %s to %s (sha256 %s)\n' "$version" "$install_dir" "$actual_sha256" From d9a09105bc4d4fca683e78f22d1172cca7b896c9 Mon Sep 17 00:00:00 2001 From: Rongjun GENG Date: Sun, 4 Oct 2026 20:01:35 -0700 Subject: [PATCH 40/45] fix(purge): preserve transitive live dependents (#7041) Propagate sweep protection through closed candidates so an open descendant keeps every candidate ancestor in its protecting dependency chain. Cover the shared helper, dry-run, real sweep, and all storage contract wirings. Agent-Signature: codex-gpt-5-unknown-reasoning on behalf of Rongjun GENG (cherry picked from commit 5db60f42ac852d6aec7d70b6f9abf0c71f1525b1) --- backend/conformance/role_bundle_cases.go | 1 + backend/conformance/sweeper_contract.go | 43 +++++++++++++++++++ .../storage/dolt/sweeper_contract_test.go | 3 ++ .../embeddeddolt/sweeper_contract_test.go | 3 ++ internal/storage/uow/sweeper_contract_test.go | 3 ++ internal/workapi/sweep.go | 31 ++++++++++--- internal/workapi/sweep_test.go | 24 +++++++++++ 7 files changed, 102 insertions(+), 6 deletions(-) diff --git a/backend/conformance/role_bundle_cases.go b/backend/conformance/role_bundle_cases.go index e2dee40d8..769eb1c14 100644 --- a/backend/conformance/role_bundle_cases.go +++ b/backend/conformance/role_bundle_cases.go @@ -640,6 +640,7 @@ var roleContractCases = []roleContract{ RunSweeperWispsPlaneClearsTheWholeWispsTable, RunSweeperWispsPlaneRequiresAFilter, RunSweeperProtectsLiveDependents, + RunSweeperProtectsTransitiveLiveDependents, RunSweeperProtectsLiveDependentsAcrossPlanes, RunSweeperLimitTakesTheOldestClosedFirst, ), diff --git a/backend/conformance/sweeper_contract.go b/backend/conformance/sweeper_contract.go index bae89c58d..8cbf6225a 100644 --- a/backend/conformance/sweeper_contract.go +++ b/backend/conformance/sweeper_contract.go @@ -747,6 +747,49 @@ func RunSweeperProtectsLiveDependents(t *testing.T, ctx context.Context, fixture sweeperAssertWispRows(t, ctx, fixture, 4, openChild, convoy, hooked, relater) } +// RunSweeperProtectsTransitiveLiveDependents pins that protection propagates +// through closed candidates: an open grandchild keeps both its closed parent +// and closed grandparent. Otherwise sweeping the grandparent orphans a live +// hierarchy even though the direct parent was correctly retained. +func RunSweeperProtectsTransitiveLiveDependents(t *testing.T, ctx context.Context, fixture SweeperFixture) { + t.Helper() + if fixture.AddDependencies == nil { + t.Skip("fixture has no AddDependencies hook; cannot wire the hierarchy this case needs") + } + root := sweeperSeed(t, ctx, fixture, sweeperIssue(fixture, "transitive", "root", true), nil) + middle := sweeperSeed(t, ctx, fixture, sweeperIssue(fixture, "transitive", "middle", true), nil) + leafIssue := sweeperIssue(fixture, "transitive", "leaf", true) + leafIssue.Status = types.StatusOpen + leafIssue.ClosedAt = nil + leaf := sweeperSeed(t, ctx, fixture, leafIssue, nil) + + if err := fixture.AddDependencies(ctx, publicops.AddDependenciesRequest{ + Actor: "sweeper-seed", + Edges: []publicops.DependencyEdge{ + {IssueID: middle, DependsOnID: root, Type: publicops.DepParentChild}, + {IssueID: leaf, DependsOnID: middle, Type: publicops.DepParentChild}, + }, + }); err != nil { + t.Fatalf("wiring transitive hierarchy: %v", err) + } + + request := publicops.SweepRequest{ + Tier: publicops.SweepWispsPlane, + IDPattern: sweeperPattern(fixture, "transitive"), + ClosedBefore: &sweeperCutoff, + ProtectLiveDependents: true, + } + preview := sweeperSweep(t, ctx, fixture, sweeperAsDryRun(request)) + if preview.Swept != 0 || preview.Skipped.LiveDependent != 2 { + t.Fatalf("dry run: Swept = %d, LiveDependent = %d; want 0 and 2", preview.Swept, preview.Skipped.LiveDependent) + } + result := sweeperSweep(t, ctx, fixture, request) + if result.Swept != 0 || result.Skipped.LiveDependent != 2 { + t.Fatalf("sweep: Swept = %d, LiveDependent = %d; want 0 and 2", result.Swept, result.Skipped.LiveDependent) + } + sweeperAssertWispRows(t, ctx, fixture, 3, root, middle, leaf) +} + // RunSweeperProtectsLiveDependentsAcrossPlanes pins that the live-dependent // protection reads BOTH dependency tables: a closed wisp that a live DURABLE // (issues-table) bead depends on through parent-child, tracks or blocks is diff --git a/internal/storage/dolt/sweeper_contract_test.go b/internal/storage/dolt/sweeper_contract_test.go index d01d2e96b..23afaebbb 100644 --- a/internal/storage/dolt/sweeper_contract_test.go +++ b/internal/storage/dolt/sweeper_contract_test.go @@ -44,6 +44,9 @@ func TestSweeperContract(t *testing.T) { t.Run("ProtectsLiveDependents", func(t *testing.T) { conformance.RunSweeperProtectsLiveDependents(t, ctx, fixture) }) + t.Run("ProtectsTransitiveLiveDependents", func(t *testing.T) { + conformance.RunSweeperProtectsTransitiveLiveDependents(t, ctx, fixture) + }) t.Run("ProtectsLiveDependentsAcrossPlanes", func(t *testing.T) { conformance.RunSweeperProtectsLiveDependentsAcrossPlanes(t, ctx, fixture) }) diff --git a/internal/storage/embeddeddolt/sweeper_contract_test.go b/internal/storage/embeddeddolt/sweeper_contract_test.go index 7a2e550e0..7d1694e64 100644 --- a/internal/storage/embeddeddolt/sweeper_contract_test.go +++ b/internal/storage/embeddeddolt/sweeper_contract_test.go @@ -49,6 +49,9 @@ func TestSweeperContract(t *testing.T) { t.Run("ProtectsLiveDependents", func(t *testing.T) { conformance.RunSweeperProtectsLiveDependents(t, ctx, fixture) }) + t.Run("ProtectsTransitiveLiveDependents", func(t *testing.T) { + conformance.RunSweeperProtectsTransitiveLiveDependents(t, ctx, fixture) + }) t.Run("ProtectsLiveDependentsAcrossPlanes", func(t *testing.T) { conformance.RunSweeperProtectsLiveDependentsAcrossPlanes(t, ctx, fixture) }) diff --git a/internal/storage/uow/sweeper_contract_test.go b/internal/storage/uow/sweeper_contract_test.go index e054b5226..1f43b50eb 100644 --- a/internal/storage/uow/sweeper_contract_test.go +++ b/internal/storage/uow/sweeper_contract_test.go @@ -44,6 +44,9 @@ func TestSweeperContract(t *testing.T) { t.Run("ProtectsLiveDependents", func(t *testing.T) { conformance.RunSweeperProtectsLiveDependents(t, ctx, fixture) }) + t.Run("ProtectsTransitiveLiveDependents", func(t *testing.T) { + conformance.RunSweeperProtectsTransitiveLiveDependents(t, ctx, fixture) + }) t.Run("ProtectsLiveDependentsAcrossPlanes", func(t *testing.T) { conformance.RunSweeperProtectsLiveDependentsAcrossPlanes(t, ctx, fixture) }) diff --git a/internal/workapi/sweep.go b/internal/workapi/sweep.go index 24af4ec50..b8a028c57 100644 --- a/internal/workapi/sweep.go +++ b/internal/workapi/sweep.go @@ -332,18 +332,37 @@ func SweepLiveDependentSources(incoming map[string][]*types.Dependency) []string return ids } -// SweepLiveDependentTargets returns the candidates a live row depends on: the -// targets of protecting edges whose source is in live. +// SweepLiveDependentTargets returns the candidates a live row depends on, +// directly or through other candidates. A protected closed candidate must act +// as live for the next edge up the chain; otherwise an open grandchild keeps +// its closed parent while the same sweep deletes the closed grandparent. func SweepLiveDependentTargets(incoming map[string][]*types.Dependency, live map[string]bool) map[string]bool { - protected := make(map[string]bool) + targetsBySource := make(map[string][]string) for target, deps := range incoming { for _, dep := range deps { - if dep != nil && isSweepProtectingEdge(dep.Type) && live[dep.IssueID] { - protected[target] = true - break + if dep != nil && isSweepProtectingEdge(dep.Type) { + targetsBySource[dep.IssueID] = append(targetsBySource[dep.IssueID], target) } } } + + protected := make(map[string]bool) + queue := make([]string, 0, len(live)) + for id, isLive := range live { + if isLive { + queue = append(queue, id) + } + } + for next := 0; next < len(queue); next++ { + source := queue[next] + for _, target := range targetsBySource[source] { + if protected[target] { + continue + } + protected[target] = true + queue = append(queue, target) + } + } return protected } diff --git a/internal/workapi/sweep_test.go b/internal/workapi/sweep_test.go index af26fd9fb..046da4595 100644 --- a/internal/workapi/sweep_test.go +++ b/internal/workapi/sweep_test.go @@ -430,6 +430,30 @@ func TestSweepLiveDependentsCountOnlyProtectingEdgesFromLiveSources(t *testing.T } } +func TestSweepLiveDependentTargetsPropagatesThroughProtectedCandidates(t *testing.T) { + dep := func(source, target string) *types.Dependency { + return &types.Dependency{IssueID: source, DependsOnID: target, Type: types.DepParentChild} + } + incoming := map[string][]*types.Dependency{ + "root": {dep("middle", "root")}, + "middle": {dep("leaf", "middle")}, + "cycle-a": {dep("cycle-b", "cycle-a")}, + "cycle-b": {dep("cycle-a", "cycle-b")}, + } + + protected := SweepLiveDependentTargets(incoming, map[string]bool{"leaf": true}) + for _, id := range []string{"middle", "root"} { + if !protected[id] { + t.Errorf("%s not protected; it is transitively depended on by a live source", id) + } + } + for _, id := range []string{"cycle-a", "cycle-b"} { + if protected[id] { + t.Errorf("%s protected without a live path into its closed cycle", id) + } + } +} + func TestValidateSweepRequestRefusesANegativeLimit(t *testing.T) { err := ValidateSweepRequest(issueops.SweepRequest{Tier: issueops.SweepEphemeral, Limit: -1}) if !errors.Is(err, issueops.ErrValidation) { From 711d238b96621120a79f7de590b4b5c475cebbc9 Mon Sep 17 00:00:00 2001 From: Rongjun GENG Date: Sun, 4 Oct 2026 20:01:43 -0700 Subject: [PATCH 41/45] fix(backup): anchor watermark before snapshot (#7044) Record the commit read before BackupToDir as the newest revision guaranteed to be in the snapshot. A commit landing during sync then remains pending for the next backup instead of being hidden by an advanced watermark. Agent-Signature: codex-gpt-5-unknown-reasoning on behalf of Rongjun GENG (cherry picked from commit 874e2866ff20f01f937b3a882403b176a11c52f9) --- cmd/bd/backup_export.go | 19 ++++----- cmd/bd/backup_export_watermark_test.go | 55 ++++++++++++++++++++++++++ 2 files changed, 65 insertions(+), 9 deletions(-) create mode 100644 cmd/bd/backup_export_watermark_test.go diff --git a/cmd/bd/backup_export.go b/cmd/bd/backup_export.go index 654ba7c47..4b3d7ed38 100644 --- a/cmd/bd/backup_export.go +++ b/cmd/bd/backup_export.go @@ -161,12 +161,16 @@ func runBackupExport(ctx context.Context, backend localBackupBackend, force bool return nil, err } + // Read the watermark before the sync. A concurrent commit may land while + // BackupToDir is running; recording a later HEAD would claim that commit is + // present in a snapshot that started before it existed. + currentCommit, err := backend.CurrentCommit(ctx) + if err != nil { + return nil, fmt.Errorf("failed to get current commit: %w", err) + } + // Change detection: skip if nothing changed (unless forced) if !force { - currentCommit, err := backend.CurrentCommit(ctx) - if err != nil { - return nil, fmt.Errorf("failed to get current commit: %w", err) - } if currentCommit == state.LastDoltCommit && state.LastDoltCommit != "" { debug.Logf("backup: no changes since last backup (commit %s)\n", truncateHash(currentCommit)) return state, nil @@ -190,11 +194,8 @@ func runBackupExport(ctx context.Context, backend localBackupBackend, force bool return nil, err } - // Update watermarks - currentCommit, err := backend.CurrentCommit(ctx) - if err != nil { - return nil, fmt.Errorf("failed to get current commit for state: %w", err) - } + // The pre-sync commit is the newest commit this backup is guaranteed to + // contain. If HEAD moved during the sync, the next backup must see it. state.LastDoltCommit = currentCommit state.Timestamp = time.Now().UTC() diff --git a/cmd/bd/backup_export_watermark_test.go b/cmd/bd/backup_export_watermark_test.go new file mode 100644 index 000000000..06d0fd2a5 --- /dev/null +++ b/cmd/bd/backup_export_watermark_test.go @@ -0,0 +1,55 @@ +package main + +import ( + "context" + "os" + "path/filepath" + "testing" + + "github.com/steveyegge/beads/internal/config" +) + +type advancingBackupBackend struct { + commit string + backupCommit string + commitAfterSync string +} + +func (b *advancingBackupBackend) CurrentCommit(context.Context) (string, error) { + return b.commit, nil +} + +func (b *advancingBackupBackend) BackupToDir(context.Context, string) error { + b.backupCommit = b.commit + b.commit = b.commitAfterSync + return nil +} + +func TestRunBackupExportWatermarkDoesNotAdvancePastSnapshot(t *testing.T) { + for _, force := range []bool{false, true} { + t.Run(map[bool]string{false: "changed", true: "forced"}[force], func(t *testing.T) { + repo := t.TempDir() + if err := os.MkdirAll(filepath.Join(repo, ".git"), 0o755); err != nil { + t.Fatal(err) + } + t.Setenv("BD_BACKUP_GIT_REPO", repo) + config.ResetForTesting() + t.Cleanup(config.ResetForTesting) + if err := config.Initialize(); err != nil { + t.Fatalf("config.Initialize: %v", err) + } + + backend := &advancingBackupBackend{commit: "head-before-sync", commitAfterSync: "head-after-sync"} + state, err := runBackupExport(context.Background(), backend, force) + if err != nil { + t.Fatalf("runBackupExport: %v", err) + } + if backend.backupCommit != "head-before-sync" { + t.Fatalf("backup captured %q, want head-before-sync", backend.backupCommit) + } + if state.LastDoltCommit != backend.backupCommit { + t.Fatalf("watermark = %q, backup captured %q; a commit that landed during sync must remain pending", state.LastDoltCommit, backend.backupCommit) + } + }) + } +} From af9d553565ec2966617a40be043ae8a48474fa34 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Mon, 5 Oct 2026 04:25:26 +0000 Subject: [PATCH 42/45] release: prep v1.3.2-rc.1 RC bump per RELEASING.md "Cut an RC", cut on hotfix/1.3.1. That branch is the v1.3.1 tag plus the hotfix CI-trigger port (#7217) and the v1.3.2 backports (#7041 and #7044, via #7220). ./scripts/update-versions.sh 1.3.2-rc.1 stamps version.go, the plugin manifests, marketplace.json, npm package.json, the MCP pyproject/__init__/uv.lock (1.3.2rc1), default.nix, the .githooks markers and the winres strings. The PE numeric fields stay at the 1.3.2 base. ./scripts/check-versions.sh is green at 1.3.2-rc.1. CHANGELOG: a dated [1.3.2-rc.1] section with: - an upgrade note for #7037: after upgrading from bd <=1.3.0, run `bd recompute-blocked` once per workspace, with the reason (migration 0059 vs dolthub/dolt#11886 on Dolt < 2.4.0 and on the embedded engine) - Fixed entries for #7031/#7041 and #7032/#7044 - a CI entry for #7148/#7198/#7217 [Unreleased] stays empty. cmd/bd/info.go: a 1.3.2-rc.1 digest at index 0 of versionChanges. Co-Authored-By: Claude Opus 5.5 --- .claude-plugin/marketplace.json | 2 +- .githooks/post-checkout | 4 +- .githooks/post-merge | 4 +- .githooks/pre-commit | 4 +- .githooks/pre-push | 4 +- .githooks/prepare-commit-msg | 4 +- CHANGELOG.md | 62 +++++++++++++++++++ cmd/bd/info.go | 10 +++ cmd/bd/version.go | 2 +- cmd/bd/winres/manifest.xml | 2 +- cmd/bd/winres/winres.json | 8 +-- default.nix | 2 +- integrations/beads-mcp/pyproject.toml | 2 +- .../beads-mcp/src/beads_mcp/__init__.py | 2 +- integrations/beads-mcp/uv.lock | 2 +- npm-package/package.json | 2 +- plugins/beads/.claude-plugin/plugin.json | 2 +- plugins/beads/.codex-plugin/plugin.json | 2 +- plugins/beads/.copilot-plugin/plugin.json | 2 +- 19 files changed, 97 insertions(+), 25 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 0a50b06ad..64af598d5 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ "name": "beads", "source": "./plugins/beads", "description": "AI-supervised issue tracker for coding workflows", - "version": "1.3.1" + "version": "1.3.2-rc.1" } ] } diff --git a/.githooks/post-checkout b/.githooks/post-checkout index 797c1199c..864cb33b3 100755 --- a/.githooks/post-checkout +++ b/.githooks/post-checkout @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.1 --- diff --git a/.githooks/post-merge b/.githooks/post-merge index 07a8f632f..147a8d0a3 100755 --- a/.githooks/post-merge +++ b/.githooks/post-merge @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.1 --- diff --git a/.githooks/pre-commit b/.githooks/pre-commit index 850121d71..921f27383 100755 --- a/.githooks/pre-commit +++ b/.githooks/pre-commit @@ -40,7 +40,7 @@ if [ -n "$staged_go_files" ]; then echo "$staged_go_files" | xargs git add fi -# --- BEGIN BEADS INTEGRATION v1.3.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -97,4 +97,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.1 --- diff --git a/.githooks/pre-push b/.githooks/pre-push index 675d4edce..27b5b3d71 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -39,7 +39,7 @@ fi exec < <(printf '%s\n' "$push_refs") -# --- BEGIN BEADS INTEGRATION v1.3.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -96,4 +96,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.1 --- diff --git a/.githooks/prepare-commit-msg b/.githooks/prepare-commit-msg index 333f71b51..c5a1f0869 100755 --- a/.githooks/prepare-commit-msg +++ b/.githooks/prepare-commit-msg @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.1 --- diff --git a/CHANGELOG.md b/CHANGELOG.md index a8362f509..5a84454f8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,68 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.3.2-rc.1] - 2026-10-05 + +First release candidate for 1.3.2, a patch on top of 1.3.1 with **no schema +migration**: upgrading from 1.3.1 is a binary swap. It fixes `bd purge` +deleting the closed root of a chain that still has live work under it, and an +auto-backup watermark that could claim a commit the snapshot does not contain. +It also carries an upgrade note for stores migrated by 1.3.0 or 1.3.1 +([#7037](https://github.com/gastownhall/beads/issues/7037)). + +### Upgrade notes + +- **After upgrading from bd ≤1.3.0, run `bd recompute-blocked` once per + workspace** ([#7037](https://github.com/gastownhall/beads/issues/7037)). + Migration 0059, first shipped in 1.3.0, recomputes the denormalized + `is_blocked` flag with a recursive query. On Dolt servers older than 2.4.0, + and on the embedded engine, a query-planner bug + ([dolthub/dolt#11886](https://github.com/dolthub/dolt/issues/11886), fixed + in Dolt 2.4.0) drops the query's `parent-child` edge filter. Blockedness then + spreads across every dependency type, including `relates-to`, + `discovered-from` and `tracks`. Beads that are not actually blocked are + marked blocked and silently disappear from `bd ready` and `bd list --ready`. + The flag is stored, so the damage persists until it is recomputed: nothing in + 1.3.2 repairs it automatically. + + `bd recompute-blocked` rebuilds the flag from the dependency graph. It is + idempotent and reports `rows_corrected`. It applies to every workspace whose + store crossed migration 0059 under bd 1.3.0 or later, which includes a + workspace now on 1.3.1 that was originally created by 1.2.x or earlier. + Running it on a workspace that was not affected changes nothing. + +### Fixed + +- **`bd purge` keeps the whole closed chain above a live bead, not just its + direct parent** ([#7041](https://github.com/gastownhall/beads/pull/7041), + fixes [#7031](https://github.com/gastownhall/beads/issues/7031)). The + live-dependent protection added in 1.3.1 only protected a closed candidate + that a not-done bead pointed at directly. Take a closed root R, a closed child + S1 and an open grandchild S2. S1 was protected because S2 is live, but R was + not, because its only dependent S1 is closed. Purge deleted R, which orphaned + S1 and S2, and `--dry-run` reported the same deletion. Protection now + propagates through closed candidates until nothing changes, on every backend + (dolt server, embedded, proxied/uow). An orchestrator retention sweep + (`bd purge --wisps-plane --older-than ...`) is the common way to hit this. +- **The backup watermark no longer runs ahead of the snapshot** + ([#7044](https://github.com/gastownhall/beads/pull/7044), fixes + [#7032](https://github.com/gastownhall/beads/issues/7032)). Auto-backup read + Dolt HEAD *after* the sync and recorded that as the newest commit backed up. + A commit that landed during the sync was therefore recorded as backed up even + though the snapshot did not contain it, and auto-backup reported "no changes + since last backup" until HEAD moved again. HEAD is now read before the sync, + so such a commit stays pending for the next backup. Only opted-in auto-backup + (`backup.enabled: true`) was affected; `bd backup sync` does not record a + commit watermark. + +### CI + +- PRs into `hotfix/**` branches now run full CI, including the required + `CI Gate / Required` check + ([#7198](https://github.com/gastownhall/beads/pull/7198), fixes + [#7148](https://github.com/gastownhall/beads/issues/7148); ported to this + branch in [#7217](https://github.com/gastownhall/beads/pull/7217)). + ## [1.3.1] - 2026-09-30 First stable release of the 1.3.1 patch line. It ships the same code as diff --git a/cmd/bd/info.go b/cmd/bd/info.go index 30e1a13f9..fe7d1753d 100644 --- a/cmd/bd/info.go +++ b/cmd/bd/info.go @@ -220,6 +220,16 @@ type VersionChange struct { // versionChanges contains agent-actionable changes for recent versions var versionChanges = []VersionChange{ + { + Version: "1.3.2-rc.1", + Date: "2026-10-05", + Changes: []string{ + "RC: first candidate for 1.3.2, a patch on top of 1.3.1 with no schema migration, so upgrading from 1.3.1 is a binary swap.", + "UPGRADE: after upgrading from bd <=1.3.0, run 'bd recompute-blocked' once per workspace (#7037). Migration 0059 (first shipped in 1.3.0) recomputes is_blocked with a recursive query whose parent-child filter is dropped by a Dolt query-planner bug on Dolt servers older than 2.4.0 and on the embedded engine (dolthub/dolt#11886). Blockedness then spreads across relates-to, discovered-from, tracks and other edges, and unblocked beads silently vanish from 'bd ready'. The flag is stored, so the damage persists until recomputed. This includes a workspace now on 1.3.1 that was first created by 1.2.x or earlier. The command is idempotent and reports rows_corrected.", + "FIX: 'bd purge' keeps every closed bead in a chain above a live bead, not just the one a live bead points at directly (#7031). A closed root over a closed child over an open grandchild used to lose its root, orphaning the rest; --wisps-plane retention sweeps hit this.", + "FIX: auto-backup records the Dolt commit read before the sync as its watermark (#7032). A commit landing during the sync used to be recorded as backed up although the snapshot lacked it, and auto-backup then reported 'no changes' until HEAD moved again.", + }, + }, { Version: "1.3.1", Date: "2026-09-30", diff --git a/cmd/bd/version.go b/cmd/bd/version.go index 34efb4af8..3120e048f 100644 --- a/cmd/bd/version.go +++ b/cmd/bd/version.go @@ -16,7 +16,7 @@ import ( var ( // Version is the current version of bd (overridden by ldflags at build time) - Version = "1.3.1" + Version = "1.3.2-rc.1" // Build can be set via ldflags at compile time Build = "dev" // Commit and branch the git revision the binary was built from (optional ldflag) diff --git a/cmd/bd/winres/manifest.xml b/cmd/bd/winres/manifest.xml index aae5b5d6b..a710519f0 100644 --- a/cmd/bd/winres/manifest.xml +++ b/cmd/bd/winres/manifest.xml @@ -3,7 +3,7 @@ beads - AI-supervised issue tracker diff --git a/cmd/bd/winres/winres.json b/cmd/bd/winres/winres.json index 226a357a8..7da38b692 100644 --- a/cmd/bd/winres/winres.json +++ b/cmd/bd/winres/winres.json @@ -9,8 +9,8 @@ "#1": { "0000": { "fixed": { - "file_version": "1.3.1", - "product_version": "1.3.1", + "file_version": "1.3.2", + "product_version": "1.3.2", "type": "App" }, "info": { @@ -18,12 +18,12 @@ "Comments": "AI-supervised issue tracker for coding workflows", "CompanyName": "Steve Yegge", "FileDescription": "beads - AI-supervised issue tracker", - "FileVersion": "1.3.1", + "FileVersion": "1.3.2-rc.1", "InternalName": "bd", "LegalCopyright": "Copyright (c) 2024-2026 Steve Yegge. MIT License.", "OriginalFilename": "bd.exe", "ProductName": "beads", - "ProductVersion": "1.3.1" + "ProductVersion": "1.3.2-rc.1" } } } diff --git a/default.nix b/default.nix index 64dadbfa7..9e58d034b 100644 --- a/default.nix +++ b/default.nix @@ -7,7 +7,7 @@ }: buildGoModule { pname = "beads"; - version = "1.3.1"; + version = "1.3.2-rc.1"; src = self; diff --git a/integrations/beads-mcp/pyproject.toml b/integrations/beads-mcp/pyproject.toml index e908d2f57..2090923fb 100644 --- a/integrations/beads-mcp/pyproject.toml +++ b/integrations/beads-mcp/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "beads-mcp" -version = "1.3.1" +version = "1.3.2-rc.1" description = "MCP server for beads issue tracker." readme = "README.md" requires-python = ">=3.10" diff --git a/integrations/beads-mcp/src/beads_mcp/__init__.py b/integrations/beads-mcp/src/beads_mcp/__init__.py index d3e4dd84e..4032264c9 100644 --- a/integrations/beads-mcp/src/beads_mcp/__init__.py +++ b/integrations/beads-mcp/src/beads_mcp/__init__.py @@ -4,4 +4,4 @@ This package provides an MCP (Model Context Protocol) server that exposes beads (bd) issue tracker functionality to MCP Clients. """ -__version__ = "1.3.1" +__version__ = "1.3.2-rc.1" diff --git a/integrations/beads-mcp/uv.lock b/integrations/beads-mcp/uv.lock index eee08ed62..761afc223 100644 --- a/integrations/beads-mcp/uv.lock +++ b/integrations/beads-mcp/uv.lock @@ -175,7 +175,7 @@ wheels = [ [[package]] name = "beads-mcp" -version = "1.3.1" +version = "1.3.2rc1" source = { editable = "." } dependencies = [ { name = "fastmcp" }, diff --git a/npm-package/package.json b/npm-package/package.json index 65257d62f..f0b862a66 100644 --- a/npm-package/package.json +++ b/npm-package/package.json @@ -1,6 +1,6 @@ { "name": "@beads/bd", - "version": "1.3.1", + "version": "1.3.2-rc.1", "description": "Beads issue tracker - lightweight memory system for coding agents with native binary support", "main": "bin/bd.js", "bin": { diff --git a/plugins/beads/.claude-plugin/plugin.json b/plugins/beads/.claude-plugin/plugin.json index 6ca3bc1a1..ec1cca3af 100644 --- a/plugins/beads/.claude-plugin/plugin.json +++ b/plugins/beads/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.1", + "version": "1.3.2-rc.1", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" diff --git a/plugins/beads/.codex-plugin/plugin.json b/plugins/beads/.codex-plugin/plugin.json index 2428a46e1..eba4a392e 100644 --- a/plugins/beads/.codex-plugin/plugin.json +++ b/plugins/beads/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "beads", - "version": "1.3.1", + "version": "1.3.2-rc.1", "description": "AI-supervised issue tracking for coding workflows with agent skills.", "author": { "name": "Steve Yegge", diff --git a/plugins/beads/.copilot-plugin/plugin.json b/plugins/beads/.copilot-plugin/plugin.json index 3ebeccd59..9215cd82a 100644 --- a/plugins/beads/.copilot-plugin/plugin.json +++ b/plugins/beads/.copilot-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.1", + "version": "1.3.2-rc.1", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" From e29f0509c15f5024271af92c9a2452743c0a1fe7 Mon Sep 17 00:00:00 2001 From: Julian Knutsen Date: Sat, 10 Oct 2026 07:28:36 +0000 Subject: [PATCH 43/45] release: prep v1.3.2-rc.2 Second RC bump per RELEASING.md "Cut an RC", cut on hotfix/1.3.1. Owner decision (2026-10-10): v1.3.2 = v1.3.1 + #7041 + #7044 (rc.1) + #7146, the backport of #6876 (blocked-state recheck published after its transaction commits, fixes #7030). This branch sits on top of #7146's head (bb18223fbd), so it must merge after #7146. Once #7146 merges, this PR's diff is only this commit. ./scripts/update-versions.sh 1.3.2-rc.2 stamps version.go, the plugin manifests, marketplace.json, npm package.json, the MCP pyproject/__init__/uv.lock (1.3.2rc2), default.nix, the .githooks markers and the winres strings. The PE numeric fields stay at the 1.3.2 base. ./scripts/check-versions.sh is green at 1.3.2-rc.2. CHANGELOG: a dated [1.3.2-rc.2] section that takes over #7146's [Unreleased] entry as a Fixed entry and restates that the rc.1 #7037 upgrade note still applies. [1.3.2-rc.1] is unchanged and [Unreleased] is empty again. cmd/bd/info.go: a 1.3.2-rc.2 digest at index 0 of versionChanges. Co-Authored-By: Claude Opus 5.5 (1M context) --- .claude-plugin/marketplace.json | 2 +- .githooks/post-checkout | 4 +- .githooks/post-merge | 4 +- .githooks/pre-commit | 4 +- .githooks/pre-push | 4 +- .githooks/prepare-commit-msg | 4 +- CHANGELOG.md | 40 +++++++++++++------ cmd/bd/info.go | 8 ++++ cmd/bd/version.go | 2 +- cmd/bd/winres/winres.json | 4 +- default.nix | 2 +- integrations/beads-mcp/pyproject.toml | 2 +- .../beads-mcp/src/beads_mcp/__init__.py | 2 +- integrations/beads-mcp/uv.lock | 2 +- npm-package/package.json | 2 +- plugins/beads/.claude-plugin/plugin.json | 2 +- plugins/beads/.codex-plugin/plugin.json | 2 +- plugins/beads/.copilot-plugin/plugin.json | 2 +- 18 files changed, 58 insertions(+), 34 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 64af598d5..b31296c69 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ "name": "beads", "source": "./plugins/beads", "description": "AI-supervised issue tracker for coding workflows", - "version": "1.3.2-rc.1" + "version": "1.3.2-rc.2" } ] } diff --git a/.githooks/post-checkout b/.githooks/post-checkout index 864cb33b3..496003ad6 100755 --- a/.githooks/post-checkout +++ b/.githooks/post-checkout @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.2-rc.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.2 --- diff --git a/.githooks/post-merge b/.githooks/post-merge index 147a8d0a3..e99d18348 100755 --- a/.githooks/post-merge +++ b/.githooks/post-merge @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.2-rc.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.2 --- diff --git a/.githooks/pre-commit b/.githooks/pre-commit index 921f27383..f708ef665 100755 --- a/.githooks/pre-commit +++ b/.githooks/pre-commit @@ -40,7 +40,7 @@ if [ -n "$staged_go_files" ]; then echo "$staged_go_files" | xargs git add fi -# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -97,4 +97,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.2-rc.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.2 --- diff --git a/.githooks/pre-push b/.githooks/pre-push index 27b5b3d71..2307fa2a0 100755 --- a/.githooks/pre-push +++ b/.githooks/pre-push @@ -39,7 +39,7 @@ fi exec < <(printf '%s\n' "$push_refs") -# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -96,4 +96,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.2-rc.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.2 --- diff --git a/.githooks/prepare-commit-msg b/.githooks/prepare-commit-msg index c5a1f0869..c4ea7884e 100755 --- a/.githooks/prepare-commit-msg +++ b/.githooks/prepare-commit-msg @@ -1,5 +1,5 @@ #!/usr/bin/env sh -# --- BEGIN BEADS INTEGRATION v1.3.2-rc.1 --- +# --- BEGIN BEADS INTEGRATION v1.3.2-rc.2 --- # This section is managed by beads. Do not remove these markers. if command -v bd >/dev/null 2>&1; then export BD_GIT_HOOK=1 @@ -56,4 +56,4 @@ if command -v bd >/dev/null 2>&1; then fi if [ "$_bd_exit" -ne 0 ]; then exit "$_bd_exit"; fi fi -# --- END BEADS INTEGRATION v1.3.2-rc.1 --- +# --- END BEADS INTEGRATION v1.3.2-rc.2 --- diff --git a/CHANGELOG.md b/CHANGELOG.md index 75cc8cbbe..61c3914f1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,20 +7,36 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [1.3.2-rc.2] - 2026-10-10 + +Second release candidate for 1.3.2. Like rc.1 it has **no schema migration**: +upgrading from 1.3.1 or 1.3.2-rc.1 is a binary swap. It adds one fix on top of +rc.1, the backport of [#6876](https://github.com/gastownhall/beads/pull/6876), +which keeps the post-commit blocked-state recheck from reverting concurrent +writes. The rc.1 upgrade note (run `bd recompute-blocked` once per workspace +after upgrading from bd ≤1.3.0, [#7037](https://github.com/gastownhall/beads/issues/7037)) +still applies. + ### Fixed -- **Backported #6876 to this release line: the post-commit blocked-state - recheck now publishes outside its own transaction.** The hotfix batches - that ported #6716's fan-in fix ([#6719](https://github.com/gastownhall/beads/pull/6719)/[#6936](https://github.com/gastownhall/beads/pull/6936), - [#6942](https://github.com/gastownhall/beads/pull/6942)/[#6947](https://github.com/gastownhall/beads/pull/6947)) carried the recheck in the form #6876 - later corrected on `main`: the Dolt commit was minted while the recompute's - own SQL transaction was still open, so `DOLT_ADD` staged the table from the - session's BEGIN-time root and could write concurrently committed `issues` - rows back to their BEGIN-time values — reachable only when the recheck - runs while unblocking writes are racing on `issues`. The recompute now runs - alone in its own retried transaction and the trailing Dolt commit publishes - after it commits; a failed trailing commit is counted as - `post_tx_commit_dropped` instead of surfacing as a recheck failure. +- **The post-commit blocked-state recheck no longer reverts concurrent + writes** ([#7146](https://github.com/gastownhall/beads/pull/7146), backport + of [#6876](https://github.com/gastownhall/beads/pull/6876), fixes + [#7030](https://github.com/gastownhall/beads/issues/7030)). The 1.3.1 line + carried the #6716 fan-in fix + ([#6719](https://github.com/gastownhall/beads/pull/6719)/[#6936](https://github.com/gastownhall/beads/pull/6936), + [#6942](https://github.com/gastownhall/beads/pull/6942)/[#6947](https://github.com/gastownhall/beads/pull/6947)) + in the form #6876 later corrected on `main`: on the Dolt server backend the + recheck minted its Dolt commit while its own SQL transaction was still open. + `DOLT_ADD` stages the table from the session's BEGIN-time root, so that + commit could write `issues` rows that other sessions had just committed back + to their BEGIN-time values. The recheck runs only when unblocking writes + (closes, dependency removals, deletes) race on `issues`, which is exactly + when this matters. The recompute now runs alone in its own retried + transaction and the Dolt commit is published after it commits. A failed + trailing commit is counted as `post_tx_commit_dropped` and logged, instead + of being reported as a recheck failure, because the flag repair itself is + already committed. ## [1.3.2-rc.1] - 2026-10-05 diff --git a/cmd/bd/info.go b/cmd/bd/info.go index fe7d1753d..fe7dd92cf 100644 --- a/cmd/bd/info.go +++ b/cmd/bd/info.go @@ -220,6 +220,14 @@ type VersionChange struct { // versionChanges contains agent-actionable changes for recent versions var versionChanges = []VersionChange{ + { + Version: "1.3.2-rc.2", + Date: "2026-10-10", + Changes: []string{ + "RC: second candidate for 1.3.2, rc.1 plus one fix, with no schema migration, so upgrading from 1.3.1 or 1.3.2-rc.1 is a binary swap. The rc.1 upgrade note still applies: after upgrading from bd <=1.3.0, run 'bd recompute-blocked' once per workspace (#7037).", + "FIX: the post-commit blocked-state recheck publishes its Dolt commit after its own transaction commits (#7030, backport of #6876). Minted inside the transaction, DOLT_ADD staged issues from the BEGIN-time root and could write concurrently committed rows back to their old values while unblocking writes raced. A failed trailing commit now counts as post_tx_commit_dropped instead of a recheck failure.", + }, + }, { Version: "1.3.2-rc.1", Date: "2026-10-05", diff --git a/cmd/bd/version.go b/cmd/bd/version.go index 3120e048f..bd756ec32 100644 --- a/cmd/bd/version.go +++ b/cmd/bd/version.go @@ -16,7 +16,7 @@ import ( var ( // Version is the current version of bd (overridden by ldflags at build time) - Version = "1.3.2-rc.1" + Version = "1.3.2-rc.2" // Build can be set via ldflags at compile time Build = "dev" // Commit and branch the git revision the binary was built from (optional ldflag) diff --git a/cmd/bd/winres/winres.json b/cmd/bd/winres/winres.json index 7da38b692..c86a24254 100644 --- a/cmd/bd/winres/winres.json +++ b/cmd/bd/winres/winres.json @@ -18,12 +18,12 @@ "Comments": "AI-supervised issue tracker for coding workflows", "CompanyName": "Steve Yegge", "FileDescription": "beads - AI-supervised issue tracker", - "FileVersion": "1.3.2-rc.1", + "FileVersion": "1.3.2-rc.2", "InternalName": "bd", "LegalCopyright": "Copyright (c) 2024-2026 Steve Yegge. MIT License.", "OriginalFilename": "bd.exe", "ProductName": "beads", - "ProductVersion": "1.3.2-rc.1" + "ProductVersion": "1.3.2-rc.2" } } } diff --git a/default.nix b/default.nix index 9e58d034b..f4f9c7dc1 100644 --- a/default.nix +++ b/default.nix @@ -7,7 +7,7 @@ }: buildGoModule { pname = "beads"; - version = "1.3.2-rc.1"; + version = "1.3.2-rc.2"; src = self; diff --git a/integrations/beads-mcp/pyproject.toml b/integrations/beads-mcp/pyproject.toml index 2090923fb..cb445d89e 100644 --- a/integrations/beads-mcp/pyproject.toml +++ b/integrations/beads-mcp/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "beads-mcp" -version = "1.3.2-rc.1" +version = "1.3.2-rc.2" description = "MCP server for beads issue tracker." readme = "README.md" requires-python = ">=3.10" diff --git a/integrations/beads-mcp/src/beads_mcp/__init__.py b/integrations/beads-mcp/src/beads_mcp/__init__.py index 4032264c9..df3dedcd4 100644 --- a/integrations/beads-mcp/src/beads_mcp/__init__.py +++ b/integrations/beads-mcp/src/beads_mcp/__init__.py @@ -4,4 +4,4 @@ This package provides an MCP (Model Context Protocol) server that exposes beads (bd) issue tracker functionality to MCP Clients. """ -__version__ = "1.3.2-rc.1" +__version__ = "1.3.2-rc.2" diff --git a/integrations/beads-mcp/uv.lock b/integrations/beads-mcp/uv.lock index 761afc223..6042a67b4 100644 --- a/integrations/beads-mcp/uv.lock +++ b/integrations/beads-mcp/uv.lock @@ -175,7 +175,7 @@ wheels = [ [[package]] name = "beads-mcp" -version = "1.3.2rc1" +version = "1.3.2rc2" source = { editable = "." } dependencies = [ { name = "fastmcp" }, diff --git a/npm-package/package.json b/npm-package/package.json index f0b862a66..8f57f14ef 100644 --- a/npm-package/package.json +++ b/npm-package/package.json @@ -1,6 +1,6 @@ { "name": "@beads/bd", - "version": "1.3.2-rc.1", + "version": "1.3.2-rc.2", "description": "Beads issue tracker - lightweight memory system for coding agents with native binary support", "main": "bin/bd.js", "bin": { diff --git a/plugins/beads/.claude-plugin/plugin.json b/plugins/beads/.claude-plugin/plugin.json index ec1cca3af..dd9763920 100644 --- a/plugins/beads/.claude-plugin/plugin.json +++ b/plugins/beads/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.2-rc.1", + "version": "1.3.2-rc.2", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" diff --git a/plugins/beads/.codex-plugin/plugin.json b/plugins/beads/.codex-plugin/plugin.json index eba4a392e..d3417ceb5 100644 --- a/plugins/beads/.codex-plugin/plugin.json +++ b/plugins/beads/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "beads", - "version": "1.3.2-rc.1", + "version": "1.3.2-rc.2", "description": "AI-supervised issue tracking for coding workflows with agent skills.", "author": { "name": "Steve Yegge", diff --git a/plugins/beads/.copilot-plugin/plugin.json b/plugins/beads/.copilot-plugin/plugin.json index 9215cd82a..15af0a1ae 100644 --- a/plugins/beads/.copilot-plugin/plugin.json +++ b/plugins/beads/.copilot-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "beads", "description": "AI-supervised issue tracker for coding workflows. Manage tasks, discover work, and maintain context with simple CLI commands.", - "version": "1.3.2-rc.1", + "version": "1.3.2-rc.2", "author": { "name": "Steve Yegge", "url": "https://github.com/steveyegge" From 65b406fe566792b2ce23abccb5aaafe09e14fd96 Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 12:46:26 -0700 Subject: [PATCH 44/45] docs(engdocs): re-review ERROR_HANDLING.md against cmd/bd and refresh its marker (#7267) The Last reviewed marker (2026-07-07) passed the 90-day limit in scripts/check-doc-freshness.sh at 2026-10-06 00:00 UTC, so the required "Check doc flags freshness" job fails on every PR and merge-group run (#7266). Re-reviewed against cmd/bd at e72cd8b31 and corrected what had drifted: - createConfigYaml takes three arguments. - cmd/bd/sync.go has no warning or ignored-error sites any more; its citations are gone. Line-number citations into init.go and create.go no longer matched and are replaced by search strings. - internal/syncbranch was removed with the sync-branch pipeline; the syncbranch.Set example and the sync.branch bullet are gone. - No code writes last_import_hash; the tracking-metadata examples now show the bd_version write (SetLocalMetadata) and verifyMetadata as init.go has them. - CheckReadonly also exits ExitMigrationFrozen under a migration freeze. - HandleProxyCapabilityError is added to the helper list. Fixes #7266 Co-authored-by: Claude Fable 5.1 (cherry picked from commit 800d43eb8d6eaf372347fbbdda9c4c366c2b6c94) --- engdocs/ERROR_HANDLING.md | 56 +++++++++++++++++---------------------- 1 file changed, 25 insertions(+), 31 deletions(-) diff --git a/engdocs/ERROR_HANDLING.md b/engdocs/ERROR_HANDLING.md index e394c8c02..37ebbaccd 100644 --- a/engdocs/ERROR_HANDLING.md +++ b/engdocs/ERROR_HANDLING.md @@ -1,6 +1,6 @@ # Error Handling Guidelines -Last reviewed: 2026-07-07 +Last reviewed: 2026-10-05 Freshness source: `cmd/bd/*.go`, especially command error exits and JSON error helpers in `cmd/bd/errors.go`. @@ -47,7 +47,8 @@ every `defer`), and `main()` maps the sentinel to exit code 1. **Narrow exceptions still using `os.Exit`:** process-level gates that run before or outside the `RunE` error path — e.g. `CheckReadonly`, which aborts a blocked -command after flushing metrics first. A handful of pre-existing direct +command after flushing metrics first (exit 1 in read-only mode, +`ExitMigrationFrozen` under a migration freeze). A handful of pre-existing direct `os.Exit(1)` calls also remain inside handler bodies; new command code should return a `HandleError*` value instead of adding more. @@ -67,7 +68,7 @@ return a `HandleError*` value instead of adding more. **Example:** ```go -if err := createConfigYaml(beadsDir, false); err != nil { +if err := createConfigYaml(beadsDir, false, ""); err != nil { fmt.Fprintf(os.Stderr, "Warning: failed to create config.yaml: %v\n", err) // Non-fatal - continue anyway } @@ -80,10 +81,9 @@ if err := createConfigYaml(beadsDir, false); err != nil { - Core functionality still works **Files using this pattern:** -- `cmd/bd/init.go` (lines 155-157, 161-163, 167-169, 188-190, 236-238, 272-274, etc.) -- `cmd/bd/sync.go` (lines 156, 257, 281, 329, 335, 720-722, 740, 743, 752, 762) -- `cmd/bd/create.go` (lines 333-334, 340-341) -- `cmd/bd/sync.go` *(handles Dolt sync operations)* +- `cmd/bd/init.go` — search for `"Warning: failed` (config.yaml, `.gitignore`, + git exclude, and tracking-metadata writes) +- `cmd/bd/create.go`, `cmd/bd/gc.go` — search for `WarnError(` --- @@ -107,9 +107,7 @@ _ = os.Remove(tempPath) - Primary error already reported **Files using this pattern:** -- `cmd/bd/init.go` (line 209, 326-327) -- `cmd/bd/sync.go` (lines 696-698) -- `cmd/bd/sync.go` *(sync cleanup)* +- `cmd/bd/init.go` — search for `_ = store.Close()` - Dozens of other locations throughout the codebase --- @@ -166,7 +164,7 @@ if err != nil { ### Creating Auxiliary Config Files → Pattern B (Warn) ```go -if err := createConfigYaml(localBeadsDir, false); err != nil { +if err := createConfigYaml(localBeadsDir, false, ""); err != nil { fmt.Fprintf(os.Stderr, "Warning: failed to create config.yaml: %v\n", err) // Non-fatal - continue anyway } @@ -186,8 +184,8 @@ defer func() { ### Optional Metadata Updates → Pattern B (Warn) ```go -if err := store.SetMetadata(ctx, "last_import_hash", currentHash); err != nil { - fmt.Fprintf(os.Stderr, "Warning: failed to update last_import_hash: %v\n", err) +if err := store.SetLocalMetadata(ctx, "bd_version", Version); err != nil { + fmt.Fprintf(os.Stderr, "Warning: failed to write bd_version local metadata: %v\n", err) } ``` @@ -205,7 +203,7 @@ if err := store.CreateIssue(ctx, issue, actor); err != nil { ```go // BAD: Same type of operation handled differently -if err := createConfigYaml(dir, false); err != nil { +if err := createConfigYaml(dir, false, ""); err != nil { fmt.Fprintf(os.Stderr, "Warning: %v\n", err) // Warns } if err := createReadme(dir); err != nil { @@ -216,7 +214,7 @@ if err := createReadme(dir); err != nil { ```go // GOOD: Consistent pattern for similar operations -if err := createConfigYaml(dir, false); err != nil { +if err := createConfigYaml(dir, false, ""); err != nil { fmt.Fprintf(os.Stderr, "Warning: failed to create config.yaml: %v\n", err) } if err := createReadme(dir); err != nil { @@ -285,15 +283,10 @@ defer store.Close() if err := store.SetConfig(ctx, "issue_prefix", prefix); err != nil { return HandleError("failed to set issue prefix: %v", err) } - -if err := syncbranch.Set(ctx, store, branch); err != nil { - return HandleError("failed to set sync branch: %v", err) -} ``` **Examples:** - `issue_prefix` - Defines how all issue IDs are generated -- `sync.branch` - Critical for git synchronization workflow **Rationale:** These settings are prerequisites for basic operation. Without them, the system cannot function correctly. A failure here indicates a serious problem (e.g., filesystem issues, database corruption). @@ -303,28 +296,25 @@ Tracking metadata **enhances functionality** but the system works without it: ```go // Pattern B: Warn and continue -if err := store.SetMetadata(ctx, "bd_version", Version); err != nil { - fmt.Fprintf(os.Stderr, "Warning: failed to store version metadata: %v\n", err) +// bd_version is clone-local state, so it goes through SetLocalMetadata. +if err := store.SetLocalMetadata(ctx, "bd_version", Version); err != nil { + fmt.Fprintf(os.Stderr, "Warning: failed to write bd_version local metadata: %v\n", err) // Non-fatal - continue anyway } -if err := store.SetMetadata(ctx, "repo_id", repoID); err != nil { - fmt.Fprintf(os.Stderr, "Warning: failed to set repo_id: %v\n", err) -} - -if err := store.SetMetadata(ctx, "last_import_hash", hash); err != nil { - fmt.Fprintf(os.Stderr, "Warning: failed to update last_import_hash: %v\n", err) +// verifyMetadata writes the field, reads it back, and warns on failure. +if verifyMetadata(ctx, store, "repo_id", repoID) && !quiet { + fmt.Printf(" Repository ID: %s\n", repoID[:8]) } ``` **Examples:** - `bd_version` - Enables version mismatch warnings on upgrades - `repo_id` / `clone_id` - Helps with collision detection across clones -- `last_import_hash` - Optimizes staleness detection (falls back to mtime if unavailable) **Rationale:** System degrades gracefully if tracking metadata is unavailable. Core functionality (creating issues, importing data) still works. Failures here might indicate temporary issues (e.g., read-only filesystem) that shouldn't block the entire operation. -**See also:** `cmd/bd/init.go` lines 206-272 for detailed inline documentation of this distinction. +**See also:** the `CONFIGURATION METADATA` and `TRACKING METADATA` comment blocks in `cmd/bd/init.go` for inline documentation of this distinction. ### File Permission Errors @@ -378,6 +368,11 @@ func HandleErrorWithHintRespectJSON(message, hint string) error // Exit 1 with no message, when the error was already reported func SilentExit() error +// Proxied-server capability refusals: a nil error passes through as nil, a +// *ProxyCapabilityError is rendered (JSON-aware) with its own exit code, and +// any other error falls back to HandleErrorRespectJSON +func HandleProxyCapabilityError(err error) error + // Pattern B — prints "Warning: ..." to stderr and returns nothing func WarnError(format string, args ...interface{}) ``` @@ -393,4 +388,3 @@ func WarnError(format string, args ...interface{}) - `cmd/bd/errors.go` - The `HandleError*` / `WarnError` / `SilentExit` helpers and the `exitError` sentinel that `main()` maps to an exit code - `cmd/bd/defer.go` - Clean example of Pattern A: `return HandleError(...)` from a `RunE` with `SilenceUsage`/`SilenceErrors` set - `cmd/bd/init.go` - Examples of all three patterns -- `cmd/bd/sync.go` - Examples of Pattern B for metadata operations and Pattern C for cleanup operations From 1d71e70fe4a60a7a56313ec0d693ee2f86ddd070 Mon Sep 17 00:00:00 2001 From: Bee Date: Tue, 6 Oct 2026 23:03:01 +0000 Subject: [PATCH 45/45] docs(ide-setup): re-review against cmd/bd/setup and refresh its marker (#7268) The Last reviewed marker (2026-07-10) passes the 90-day limit in scripts/check-doc-freshness.sh at 2026-10-09 00:00 UTC, when the required "Check doc flags freshness" job would fail on every PR and merge-group run, as it did for ERROR_HANDLING.md (#7266). Re-reviewed against cmd/bd/setup.go, cmd/bd/setup/ and internal/recipes/ at e72cd8b31 and corrected what had drifted: - bd setup cursor installs the agent skill and .cursor/hooks.json (sessionStart, preCompact, postToolUse) besides the rules file, and takes --global (hooks and skill only). - bd setup claude writes its managed section to AGENTS.md when CLAUDE.md is an @AGENTS.md import stub. - bd setup codex takes --global. - bd init runs the Claude Code, Codex and Cursor project setup, not Codex alone. The marker is 2026-10-05, not the UTC date, because the script rejects a future date and a Pacific shell still reads 10-05. Co-authored-by: Claude Fable 5.1 (cherry picked from commit 8ae1a2beb35bafa99a20fb400b158107b3fbd7b7) --- docs/getting-started/ide-setup.md | 34 +++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/docs/getting-started/ide-setup.md b/docs/getting-started/ide-setup.md index 47a4537da..4eea8a09e 100644 --- a/docs/getting-started/ide-setup.md +++ b/docs/getting-started/ide-setup.md @@ -5,7 +5,7 @@ description: Configure bd setup recipes, hooks, and instruction files for Claude Configure your IDE or coding agent for optimal beads integration. -Last reviewed: 2026-07-10 +Last reviewed: 2026-10-05 Freshness source: `cmd/bd/setup*.go` and `internal/recipes/`. @@ -23,10 +23,10 @@ bd setup claude --remove # Uninstall | Recipe | Files written | Details | |--------|---------------|---------| | `claude` | `.claude/settings.json` (or `~/.claude/settings.json` with `--global`) + `CLAUDE.md` section | [Claude Code](/integrations/claude-code) | -| `cursor` | `.cursor/rules/beads.mdc` | [Cursor](/integrations/cursor) | +| `cursor` | `.cursor/rules/beads.mdc` + `.agents/skills/beads/` + `.cursor/hooks.json` (with `--global`: `~/.cursor/hooks.json` + `~/.agents/skills/beads/`, no rules file) | [Cursor](/integrations/cursor) | | `gemini` | `~/.gemini/settings.json` (or `.gemini/settings.json` with `--project`) + `GEMINI.md` section | [Gemini CLI](/integrations/gemini) | | `copilot` | `.copilot-plugin/plugin.json` + `.github/copilot-instructions.md` | [Copilot CLI](/integrations/copilot-cli) | -| `codex` | `.agents/skills/beads/` + `AGENTS.md` section + `.codex/` hooks | [Codex](/integrations/codex) | +| `codex` | `.agents/skills/beads/` + `AGENTS.md` section + `.codex/` hooks (with `--global`: `~/.agents/skills/beads/` + `$CODEX_HOME` or `~/.codex`) | [Codex](/integrations/codex) | | `factory` | `AGENTS.md` section | [Factory.ai Droid](/integrations/factory) | | `mux` | `AGENTS.md` section (+ `.mux/` layers with `--project`/`--global`) | [Mux](/integrations/mux) | | `opencode` | `AGENTS.md` section | [OpenCode](/integrations/opencode) | @@ -95,7 +95,7 @@ bd setup claude --global # Global install: ~/.claude/settings.json This installs: - **SessionStart hook** - Runs `bd prime --hook-json`, which wraps the workflow context in the JSON envelope Claude Code expects. SessionStart fires when a session starts, resumes, or clears, and again after context compaction — no separate compaction hook is needed. -- **Minimal beads section in `CLAUDE.md`** - A pointer to `bd prime`, managed with hash/version markers for safe updates and `--check` freshness detection. +- **Minimal beads section in `CLAUDE.md`** - A pointer to `bd prime`, managed with hash/version markers for safe updates and `--check` freshness detection. If `CLAUDE.md` is a thin stub that imports `AGENTS.md` (an `@AGENTS.md` line of its own), the section is written to `AGENTS.md` instead, and a stale beads block left in the stub by an older bd is removed. If the [beads Claude Code plugin](/integrations/claude-code-plugin) is installed, hooks are plugin-managed and `bd setup claude` skips writing them, so `bd prime` doesn't fire twice per session. @@ -143,11 +143,26 @@ If you prefer manual configuration, add the hook to your Claude Code settings: ## Cursor IDE ```bash -bd setup cursor # Always-applied rules file +bd setup cursor # Project install: rules file + agent skill + hooks +bd setup cursor --global # Global install: hooks + agent skill (no rules file) ``` -This creates `.cursor/rules/beads.mdc` with beads-aware rules that Cursor -re-includes every turn. +A project install writes three things: + +- **`.cursor/rules/beads.mdc`** - an always-applied rules file that Cursor + re-includes every turn. +- **`.agents/skills/beads/`** - the beads agent skill, the same one + `bd setup codex` installs. `--remove` keeps it while the Codex integration + still uses it. +- **`.cursor/hooks.json`** - `sessionStart`, `preCompact`, and `postToolUse` + hooks that run `bd cursor-hook `, so `bd` must be on `PATH` for + Cursor. Your own entries in `hooks.json` are left alone. + +`--global` writes the hooks to `~/.cursor/hooks.json` and the skill to +`~/.agents/skills/beads/`. Cursor has no file-based global rules location, so +add global rules in Cursor Settings or run `bd setup cursor` per project. + +Restart Cursor (or start a new `cursor-agent` session) after installing. **Verify:** ```bash @@ -201,9 +216,10 @@ bd setup factory # Factory.ai Droid — AGENTS.md section bd setup mux # Mux — AGENTS.md section (+ --project/--global layers) bd setup opencode # OpenCode — AGENTS.md section bd setup codex # Codex — beads skill + AGENTS.md guidance + native hooks +bd setup codex --global # Same, under ~/.agents/skills/beads and $CODEX_HOME (or ~/.codex) ``` -These create or update a managed section in `AGENTS.md` (see Managed Sections above). `bd init` runs the project Codex setup automatically unless `--skip-agents` or `--stealth` is used. In worktree, shared, or `BEADS_DIR` setups, use `bd where` to confirm the resolved workspace — these integrations do not require a local `./.beads`. Restart the tool after setup if it is already running. +These create or update a managed section in `AGENTS.md` (see Managed Sections above). `bd init` runs the project Claude Code, Codex, and Cursor setup automatically unless `--skip-agents` or `--stealth` is used (or the repository is bare). In worktree, shared, or `BEADS_DIR` setups, use `bd where` to confirm the resolved workspace — these integrations do not require a local `./.beads`. Restart the tool after setup if it is already running. Details: [Factory.ai Droid](/integrations/factory), [Mux](/integrations/mux), [OpenCode](/integrations/opencode), [Codex](/integrations/codex). @@ -349,7 +365,7 @@ bd setup --print | Type | Description | Used by | |------|-------------|---------| -| `file` | Write the template to a single file | windsurf, cody, kilocode, kiro | +| `file` | Write the template to a single file | windsurf, cody, kilocode, kiro, cursor (which also installs a skill and hooks) | | `hooks` | Modify JSON settings to add hooks | claude, gemini | | `section` | Inject a marked section into an existing file | factory, codex, mux, opencode | | `multifile` | Write multiple files | aider, copilot, junie |